Effectiveness of Auckland Council's post-implementation review process

1 Post-implementation reviews (PIRs) evaluate how well a project has been managed and whether the benefits of that project have been achieved. They are an important part of an organisation’s performance management framework, providing information about performance and lessons for future projects. PIRs also provide assurance that a project is achieving its intended objectives.

2 We reviewed the process Auckland Council (the Council) used for carrying out the PIRs of two of its projects – “Contact Centre Consolidation” and “Libraries Fit for the Future”. For each project, we looked at how effective, fair, and balanced the PIR process was. We commend the Council for supporting a strong performance management culture by carrying out PIRs.

3 We provide details of our findings in Appendix 1, and a suggested post-implementation review checklist in Appendix 2.

4 Our work is carried out under section 104 of the Local Government (Auckland Council) Act 2009, which requires the Auditor-General to review the service performance of the Council and each of its council-controlled organisations from time to time.

Our main findings

5 The two PIRs we reviewed were largely run and managed well. They resulted in balanced, high-quality, and timely reports to the Mayor’s office.

6 The process used for carrying out the PIRs was robust and followed good practice guidance. The planning and fieldwork was effective, which meant that relevant evidence was collected.

7 Staff carrying out the PIRs (PIR teams) understood their roles and responsibilities. Management oversight and quality assurance of the PIR processes and findings were appropriate.

8 There were some matters that, in our view, could be improved. We address these in our recommendations below.

Recommendations

9 Based on our review of the two PIRs, we have made the following recommendations for the Council:

• Further develop internal PIR capacity and capability, particularly for report writing.
• Where appropriate, use staff with knowledge of the council service being reviewed to help plan the PIR fieldwork, more easily source information, and provide context.
• Review the PIR process to ensure that resource planning and timelines are appropriate to the size and nature of the review.
• Implement a more consistent approach to taking notes for interviews that clearly groups evidence against the relevant line of inquiry.
• Improve communication between the PIR team and the council service being reviewed. This is to ensure that there is a transparent approach that keeps everyone informed of the PIR’s progress.
• Put a process in place to identify lessons learned from the PIR, which can be applied to future PIRs.

10 We encourage the Council to implement these recommendations, to develop a more effective and efficient PIR process that will be useful, deliver strong performance information, and be sustainable for future PIRs.

Introduction

11 The Council proposed to carry out four PIRs in 2018 in response to requests from the Mayor’s office. The Mayor’s office wanted an overview of the significant projects that had made changes to some of the Council’s services and been completed in 2017. These were:

• Contact Centre Consolidation;
• Libraries Fit for the Future;
• Community Facilities; and
• Regulatory Future Shape.

12 The Council postponed the PIRs of Community Facilities and Regulatory Future Shape projects until 2019. This was because changes were taking place in these services and the Council considered that it was not the right time to carry out a PIR for these projects. The PIRs of Contact Centre Consolidation and Libraries Fit for the Future have been completed.

13 To ensure that our findings are reported in a timely way and lessons are able to be shared with the Council, we are reporting our assessment of the two completed PIRs (Contact Centre Consolidation and Libraries Fit for the Future).

14 We are not intending to assess the postponed PIRs because, in our view, we have enough evidence from the two completed PIRs to support our findings and recommendations.

What we reviewed

15 We carried out an independent review of the Council’s PIR process to establish whether it is a fair and balanced approach to assessing project implementation.

16 Our approach consisted of:

• regular contact and discussion with the two PIR teams throughout all stages of the process;
• observing the planning and set-up meetings for each PIR;
• observing the desk-based assessment of project documents carried out for each PIR, including the use and understanding of performance information;
• observing a selection of interviews and focus groups;
• observing advisory group challenge meetings, where PIR teams analysed and assessed evidence to make conclusions;
• assessing planning documentation, interview notes, and the draft and final reports; and
• interviewing the PIR teams and those with sponsorship, governance, and management roles after the reviews were completed.

What we did not review

17 We have not commented on the implementation of the projects because this is the purpose of the PIRs. Although we refer to the PIRs’ main findings, our focus is on the PIR process.

18 For our review of the PIR process, we were not able to look at all parts of the process. For example, we did not observe all the interviews that took place. We focused on aspects of the PIR process that we considered would provide sufficient assurance for assessing the effectiveness of the PIR process.

Our expectations

19 We expected the PIRs to evaluate whether the projects have achieved their intended objectives.

20 To do this, we expected:

• the PIRs to be appropriately planned and clearly scoped, and that there would be a clear and effective process to carry them out;
• the PIRs to be done at an appropriate time, and timelines and review milestones to be met;
• enough resources to be allocated to carry out the PIRs within agreed timelines;
• the PIRs to be carried out by staff who have the requisite independence,
• experience, and expertise;
• roles and responsibilities for PIR teams, sponsors, governance, and
• management to be clear and effective;
• effective and appropriate quality assurance processes;
• effective collection and analysis of all relevant PIR evidence;
• PIR reports that reflect the evidence collected; and
• lessons learned from the projects and the PIR process to be used to make improvements.

Process used for post-implementation reviews

21 In our view, the Council had an effective process in place to carry out the PIRs. The Council prepared a process that was based on previous internal reviews and had been refined using lessons from other reviews. The process also incorporated good practice guidance from consultants and published research (see Figure 1).

Figure 1: Auckland Council’s six-phase process for carrying out post-implementation reviews of projects

Auckland Council’s six phases for carrying out a post-implementation review of a project are: mobilisation and discovery, desk-top assessment, conduct interviews, draft report, review and feedback, and finalise report. Each phase has a specified timeline.

Source: Adapted from Auckland Council.

22 We spoke with the Council about adding a seventh phase to the six-phase process that would identify lessons from the PIR process. These lessons could then be used to improve future PIRs. The Council informed us that, even though there was not a formal seventh phase, lessons from the first PIR were applied to the second PIR.

23 Each phase in the PIR process has a specified timeline (for example, Phase 1 has a timeline of one week). However, there was not enough understanding of how much time was needed to complete each phase. The PIR teams told us that timelines for each phase of the process were tight, although overall deadlines for reporting to the Mayor’s office were met. Project milestones were delayed in some instances, and some timelines had to be reset.

24 We investigated the reasons for these delays and were informed that some PIR team members had only recently joined the Council and took longer to become familiar with the process and the council service that they were reviewing. The timelines for the process did not take into account the complexity of each council service, the individual circumstances of PIR teams, or the time needed to complete each review. As a result, the tight timelines put PIR team members under pressure but they were able to complete the work. Timelines were also adjusted to compensate for underestimating the time needed to complete the PIRs.

Planning meetings

25 Planning meetings between the PIR teams, sponsors, governors, advisors, and managers were held before the reviews were started. In these meetings, the PIR team leader clearly communicated the approach to be taken.

26 The two PIR teams agreed on the scope of their respective reviews during these planning meetings. The teams agreed to focus on whether the objectives described in the business case for each project had been achieved.

27 These planning meetings were important to ensure that everyone involved understood the scope of the PIRs. This limited the risk of the PIR going beyond scope and also ensured that all members of the PIR teams had clear expectations of what the PIR aimed to achieve.

28 The planning meetings outlined five questions for each PIR:

• Are the identified financial and non-financial metrics and customer and community benefits being realised?
• Have any risks or issues emerged?
• Are follow-up projects required?
• Is the outcome having a positive effect on our customers and communities overall?
• Are the outcomes improving trust and confidence in the Council?

Timing of the post-implementation reviews

29 The Contact Centre Consolidation and Libraries Fit for the Future projects were both completed in 2017. The Mayor’s office requested the PIRs to be completed by 30 June 2018.

30 We questioned whether enough time had passed to fairly assess whether the projects had met their objectives. However, the PIR teams and others involved did not consider it too early to start the review the process. Having observed the PIR process, we agree with this view.

31 The only objective for the PIRs that was considered too soon to judge was whether there had been improvements in the public’s trust and confidence in the Council.

32 In the projects’ business cases, some of the objectives were intended to be achieved in the first year. For the Contact Centre Consolidation project, assessment of progress against first-year objectives was considered useful in assessing whether later stages of the project would be carried out.

33 A matter for consideration, which was highlighted in both reports, was the unrealistic timelines for achieving some of the objectives described in the projects’ business cases.

34 It was also difficult for the PIR teams to obtain performance data for some of the objectives.

35 These issues were not part of our review, but the Council should consider them when preparing future business cases and performance monitoring. These issues could seriously undermine not only the PIR process but also the Council’s ability to monitor future project implementation.

Resourcing the post-implementation reviews

36 Each of the PIR teams had two main team members who did most of the fieldwork, with additional people involved with the interviews. Other roles in the PIR teams related to sponsorship, management oversight, and quality assurance. The PIR team structure is shown in Figure 2.

Figure 2: Structure for the PIR teams

The PIR team members had different roles, with oversight from the PIR leader.

Source: Adapted from Auckland Council.

37 The Council made a deliberate decision to use its own staff for the PIR teams instead of recruiting externally. This enabled the Council to develop staff capability for future PIRs, support continuous improvement in the PIR process, and have PIR teams that understand the Council’s culture, structures, and services.

38 During our audit, we discussed whether resourcing PIR teams internally would prevent a fully independent assessment. The Council mitigated this issue by putting in place strong quality assurance processes and using staff from the Strategy & Capability and Finance & Commercial units of the Council.

Capability of the PIR teams

39 Each PIR team had a staff member who was new to the Council but had previous experience of carrying out reviews. One staff member was an internal auditor who had experience carrying out review and assessment work within the Council.

40 Both PIR teams initially lacked detailed knowledge of the council service they were reviewing. This resulted in more time being needed to understand the council service, know where to obtain the required service performance information, and identify the appropriate people to interview. In our view, a staff member from the council service should work with the PIR team to help them source the information they need and more quickly understand the service. The PIR teams could then be more focused and efficient in their evidence collection.

41 In our view, despite some members of the PIR teams being new to the Council, both teams had the capability to carry out the planning, fieldwork, and analysis to the standard the Council required.

42 We were informed that the Council wanted its staff to improve their report drafting capability. The final PIR reports were of a good standard, but this followed considerable investment from management in the drafting process as part of their quality assurance role.

Resource planning

43 There was a lack of resource planning in the PIR planning process. Although PIR teams were identified, there was no assessment of the actual resources required to achieve the desired outcomes within the allocated timelines in the PIR process. As a result, the teams were under considerable pressure to complete the work on time. This resulted in some delays meeting timelines. We were informed that lessons from the first PIR were used to improve resource planning in the second PIR.

44 In our view, it be would be better to have a clear understanding at the planning stage of the type and amount of evidence needed. This would ensure that the appropriate people are assigned to the PIR teams.

45 The Council’s other priorities also affected the PIR teams because staff were assigned to other tasks. In our view, the PIR teams worked effectively despite these obstacles to deliver high-quality review assessments.

46 We observed planning meetings for both PIR teams. The meetings clearly set out roles and responsibilities for the PIR teams to ensure that they would be effective. Subsequent discussions we had with the teams, sponsors, governors, and managers supported this view.

Evidence collection

47 Both PIR teams initially collected performance information and other information, such as investment summaries, to assess progress against the projects’ objectives. Performance information was also measured against “service area metrics” to benchmark and assess performance. For example, the global satisfaction target level for contact centre customer satisfaction is 90%. The Council’s contact centre performance was measured against this target as well as the target described in the project’s business case.

48 The PIR teams struggled to collect all the information required to fully assess performance. In our view, the PIR teams did the best they could and effectively analysed and understood the reasons behind the levels of performance the council services achieved. It was clear that not all performance was being monitored effectively, which hampered assessment.

49 The final PIR reports noted that a range of documents were reviewed as part of the PIR process. We were informed that the PIR team kept a record of all documents reviewed and also noted their own insights as part of the process.

50 PIR teams carried out a range of interviews and focus groups. Other activities, such as observing how staff handled calls at the contact centre, were also carried out. We observed a small number of interviews and focus groups.

51 Each interview and focus group opened with a thorough introduction that outlined the purpose of the interview or focus group. For interviews, one PIR team member interviewed the person while another took notes. Interviews were done in a friendly and professional manner and interviewees were put at ease. Information relevant to the PIR was recorded. The team members demonstrated a good level of knowledge about the council services and the PIR process.

52 We noted two matters where improvements could be made to interviewing and managing focus groups. Some of the focus groups we observed could have been better directed to the objectives of the PIR. This was particularly evident in the focus groups for the Libraries Fit for the Future. Most of the hour-long focus group meetings were taken up by staff expressing views on the change management process. Although this provided good context, only some of the information could be used for the PIR.

53 Another matter for improvement was the note-taking of interviews. There was guidance for PIR teams on conducting interviews and the teams took comprehensive notes. However, these notes were not recorded in a standard format. It would be useful for the Council to provide a template to ensure that all interviewers take relevant notes in a standardised format. The notes were also not linked to the objectives in the project’s business case. Grouping notes against the relevant objective would make analysis and use of this evidence easier and ensure that the interviews covered all the objectives.

54 The PIR team for Contact Centre Consolidation carried out 30 interviews, which it considered to be too many. In their view, they overcompensated for a lack of understanding about council services by interviewing a wide range of staff members and stakeholders. It became clear during the later stages of the interview process that not much new information was being collected. Having a better initial understanding of the council services and individual roles and better planning could have made this process more efficient. We observed that this lesson was implemented into the second PIR for the libraries service, where there was a more focused approach to who was interviewed.

Evidence analysis, quality assurance, and reporting

55 While carrying out our review, we had access to a sample of the evidence that had been collected and observed how this evidence was analysed and assessed to prepare the initial findings and conclusions. We attended an advisory group challenge meeting. In this meeting, members of the PIR team from the strategy and capability and financial and commercial units were presented with the initial findings.

56 We observed a healthy and constructive challenge process during this meeting. In our view, this strengthened the findings and brought some wider context and understanding to the process. As a result of the challenge process, the presentation of some of the findings changed.

57 We did not observe other management oversight on the findings but discussed how this was carried out with the sponsors and manager of the PIR. Their view was that they were kept informed at appropriate times during the progress of the PIRs. Quality assurance was carried out by the PIR team leader (Corporate Strategy Manager) to substantiate the reports. Staff involved in the PIR process who we talked to felt this was an effective process that strengthened the final reports, ensuring a fair and balanced approach.

58 Overall, communication between the PIR teams and the council service being reviewed was effective. Requests for information were dealt with quickly.

59 Both PIR teams put their draft reports through a consultation process to ensure transparency and natural justice, and to correct any factual inaccuracies. The PIR teams described this as a broadly well-managed process that ensured that the reports were fair and balanced.

60 The consultation process for the PIR for Libraries Fit for the Future was slightly longer than anticipated. Feedback from the PIR team and the Libraries Service indicated that this was a result of poor communication during the later stages of the PIR process. However, matters of disagreement were resolved.

61 The report consultation process for the PIR of Contact Centre Consolidation was smoother. The final report contained a management response outlining progress since the fieldwork, as well as some contextual information.

62 The PIR team told us that their relationship with the Mayor’s office improved as the process progressed. They noted that some initial concern was reported from the Mayor’s office about a perceived lack of independence in the PIR process. However, these concerns were alleviated after discussions about initial findings.

63 The PIR team’s view was that there were no concerns from the Mayor’s office about independence when the final reports were presented. There was positive feedback from the Mayor’s office on the quality and fairness of the reports. Future PIRs will be able to build on the level of trust and confidence in the process.

64 Early drafts of the reports were not made available to us, but we were informed that a level of redrafting was needed before they could proceed to the consultation process. We have already commented on the need to build capability in report drafting to develop a more efficient process.

65 The later drafts that we reviewed were concise, balanced, outlined findings clearly, and stated whether objectives had been achieved, partially achieved, or not achieved, or whether there was not enough data to make an assessment. Benefit gaps were outlined and recommendations for improvement were included. All five objectives outlined in planning meetings were covered by the reports. It was too early to determine, at this stage, any improvements in public trust and confidence in the Council.

Lessons learned

66 The Council is open to learning from the first two PIRs completed under this process. The Council is building capability and capacity, considering how to be more flexible with the process so as to not have a one-size-fits-all approach, and incorporating more knowledge about the council services to help planning and fieldwork. The Manager for Strategy & Capability will oversee and implement PIR improvements.

67 We discussed with the Council how the lessons learned from the PIRs would be implemented and how recommendations would be monitored. We were told that council-wide lessons, such as better monitoring of business objectives, would be considered centrally in the Council through the Strategy & Capability and Finance & Commercial functions.

68 Service-related improvements will be monitored by the Chief Operating Officer and the operational heads of service.