B.29[16s]

Using Gateway reviews to support public sector projects

Presented to the House of Representatives under section 20 of the Public Audit Act 2001.

September 2016

ISBN 978-0-478-44248-9

Contents

Auditor-General’s overview

Our recommendations

Part 1 – Introduction

The purpose of our audit

What we audited

What we did not audit

How we carried out our audit

Structure of this report

Part 2 – About Gateway reviews

What Gateway reviews involve

Part 3 – Introducing Gateway reviews in New Zealand

How Gateway reviews could help New Zealand projects

The case for introducing Gateway reviews

Part 4 – Operating Gateway reviews in New Zealand

Operating Gateway reviews

Most projects have Gateway reviews as required

How Gateway reviews fit with other monitoring and assurance activities

Part 5 – Capability and capacity for Gateway reviews

Agencies

The Treasury

Gateway review teams

Part 6 – Benefits delivered by Gateway reviews

Benefits for individual projects

Broader benefits

Part 7 – The future for Gateway reviews

Limitations of Gateway reviews

The Treasury’s intentions

Figures

1 – Delivery confidence ratings

2 – Description of gates for projects

3 – Number of projects that have included Gateway reviews, by agency

4 – Investment-intensive agencies

5 – Differences between Gateway reviews and Independent Quality Assurance reviews

6 – New Zealand Gateway reviewers by country and sector

7 – Percentage of recommendations by category in the 2011, 2013, and 2015 reports on lessons learned

Auditor-General’s overview

Government agencies manage significant assets that are critical for providing public services. Every year, those agencies invest millions of dollars in projects and programmes to acquire or develop new assets, so that they can continue to provide the public services New Zealanders need. The public needs to have confidence that these projects and programmes are using money in the right way and on the right things, and that the benefits that the agencies have undertaken to deliver will be realised.

Gateway reviews are a peer review process designed to provide independent and timely advice to the people responsible for major projects and programmes. The reviews take place at key stages in the project or programme’s life cycle. Gateway reviews make recommendations to help projects and programmes succeed. They do not pass or fail a project and, despite their name, are not designed to stop a project or programme that is not going well. There are 25-30 reviews carried out each year with a total annual cost of about $2 million.

Since their introduction in 2008, there have been over 180 reviews of more than 80 projects and programmes involving about 40 agencies and with a total value of about $45 billion. Projects and programmes that have had Gateway reviews include the Transmission Gully project, a patient information system for South Island district health boards, the refurbishment of Government House, and the Inland Revenue Department’s Business Transformation programme. These are some examples of important asset developments for New Zealand’s future.

My staff looked at whether Gateway reviews have been designed and implemented to deliver benefits to major projects and programmes in the public sector.

In my view, Gateway reviews have delivered benefits to individual projects and programmes. Project sponsors have reported that they find Gateway reviews beneficial. My staff saw specific examples where a Gateway review had provided advice that helped a project reach an important milestone. For example, a Gateway review was able to help a project reach agreement between stakeholders who were in conflict. Without this agreement, the project would not have been able to progress.

For many projects and programmes, specific benefits, including cost savings, are harder to identify. There are many other influences on the success of projects and programmes, and the impact of Gateway reviews can be difficult to isolate. A cost-benefit analysis of Gateway reviews in the United Kingdom found that projects that had reviews experienced cost savings of 2-4% of the total project costs. This type of analysis is very complex and has not been replicated in New Zealand.

Gateway reviews can be effective only if they are applied to projects and programmes where they can make a difference. All projects and programmes considered to be high-risk, based on common causes of project failure, are required to have Gateway reviews. The Treasury has effective processes to identify high-risk projects and programmes and make sure that they do actually have Gateway reviews at the right times.

Having the right review team is also critical to the success of Gateway reviews. There is a large pool of reviewers to choose from, and the Treasury puts a lot of effort into getting a review team with the best mix of skills and experience for each project or programme. This is usually done well.

When review team members come from government agencies themselves, they can take what they have learned from a review and apply it to their own agency. Opportunities for this can be limited, however, as it is hard for senior public servants to dedicate a week to a Gateway review, and currently only about 20% of reviewers come from the New Zealand public service.

Gateway reviews can have a wider impact than just the projects and programmes that are reviewed. Some lessons from Gateway reviews are already shared, but I consider that this could be done more effectively. I would like to see the Treasury share more examples of good practice and find ways to help make sure that lessons are put into practice in other projects and programmes. In this way, such reviews could have a broader beneficial effect for the public sector as a whole.

When Gateway reviews were introduced to New Zealand, they had already been running successfully overseas, and it was reasonable to assume that they would benefit projects and programmes here. However, there was not a strong case to show how Gateway reviews would specifically meet the needs of New Zealand projects and programmes.

Gateway reviews do have limitations. They are a “one size fits all” approach and are unlikely to be the best way of helping all projects and programmes at all times.

The Treasury is aware of the limitations of Gateway reviews and is planning to introduce new types of investment reviews to run as well as, or instead of, Gateway reviews. I would expect that, in considering the types of investment reviews it offers, the Treasury would ensure that it takes into account the limitations of Gateway reviews, the specific needs of projects and programmes in New Zealand, and the information and assurance needs of major stakeholders.

I thank the Treasury for co-operating with our audit. I also thank other people we spoke to, including agency staff, Gateway reviewers, and people from other parts of the assurance system, who all kindly shared their experiences of Gateway reviews.

Lyn Provost
Controller and Auditor-General

15 September 2016

Our recommendations

We recommend that the Treasury:

1. determine how to more effectively share lessons learned through Gateway reviews, including examples of good practice; and

2. ensure that the work it has under way to consider new types of investment review takes into account:

the limitations of Gateway reviews;

the specific needs of projects and programmes in New Zealand; and

the information and assurance needs of stakeholders such as Ministers, chief executives, boards, and the Corporate Centre.

Part 1 – Introduction

1.1
In this Part, we discuss:

the purpose of our audit;

what we audited;

what we did not audit;

how we carried out our audit; and

the structure of this report.

The purpose of our audit

1.2
Every year, government agencies invest hundreds of millions of dollars in new products and services. Ministers need to know that there is a system for encouraging and providing adequate incentives for the effective and efficient use of capital resources.

1.3
Gateway™ reviews are independent peer reviews of a project or programme and are repeated at key points in the project or programme’s life cycle. These reviews provide advice and support to the project sponsor.¹ The reviews focus on how to help the project or programme progress successfully rather than finding out what might have gone wrong.

1.4
Gateway reviews were introduced in New Zealand in 2008, among a range of initiatives to strengthen capital asset management in the public sector. Up to
31 May 2016, there have been over 180 reviews covering more than 80 projects and programmes, involving about 40 departments and agencies.

1.5
Examples of projects and programmes that have had Gateway reviews are the Transmission Gully project, a patient information system for South Island district health boards, the refurbishment of Government House, the New Zealand Defence Force’s new Pilot Training Capability programme, and the Inland Revenue Department’s Business Transformation programme.

1.6
Each Gateway review costs the agency $75,000. This fee covers the direct costs of the reviews as well as the cost of administering the reviews. With 25-30 reviews carried out each year, Gateway reviews have an annual cost of about $2 million.

1.7
Initially run by the State Services Commission, Gateway reviews moved to the Treasury at the end of 2013. The reviews are currently managed by the Treasury’s Investment Management and Asset Performance team.

1.8
We carried out a performance audit to assess whether Gateway reviews have been designed and implemented to deliver benefits to major projects and programmes in New Zealand.

What we audited

1.9
Specifically, we looked at:

whether Gateway reviews are an appropriate tool for major capital projects and programmes in New Zealand;

whether Gateway reviews are planned, designed, and implemented effectively to deliver benefits to major capital projects and programmes;

whether agencies, review teams, and the Treasury have the capacity and capability to ensure that Gateway reviews are run effectively and efficiently; and

whether there is evidence that benefits are being realised as a result of using Gateway reviews.

1.10
We looked at Gateway reviews that were completed before and after 2013, when the Gateway review function moved to the Treasury. However, where we looked at how Gateway reviews are managed, our focus was on how the Treasury currently runs the reviews.

What we did not audit

1.11
We did not audit other monitoring and assurance activities for public sector projects and programmes. However, we did look at how Gateway reviews fit in with those other activities.

How we carried out our audit

1.12
To carry out our audit:

We analysed data about all the Gateway reviews that had been completed up to 31 May 2016.

We looked at 67 Gateway review reports, and followed the progress of eight projects through all their Gateway reviews.

We spoke to reviewers who, between them, have been involved in about 60 Gateway reviews.

We spoke to agencies with a combined experience of 63 Gateway reviews.

We reviewed documents about the implementation and management of Gateway reviews.

Structure of this report

1.13
In Part 2, we describe what a Gateway review involves.

1.14
In Part 3, we consider whether Gateway reviews are an appropriate tool for major capital projects in the New Zealand public sector.

1.15
In Part 4, we discuss whether Gateway reviews are planned, designed, and implemented effectively.

1.16
In Part 5, we consider whether there is adequate capability and capacity for Gateway reviews.

1.17
In Part 6, we discuss whether there is evidence that Gateway reviews are bringing benefits to projects and programmes.

1.18
In Part 7, we consider the future for Gateway reviews.

1.19
In this report, we use the term “Gateway unit” to mean the team of people responsible for running Gateway reviews, previously in the State Services Commission and now in the Treasury. We also use the term “projects” to mean both projects and programmes, unless we specifically state otherwise.

1 The project sponsor is often called the Senior Responsible Owner, or SRO. For Gateway reviews, this means the most senior person with direct responsibility for the funding and performance of the project or programme. They are usually part of an executive team.

Part 2 – About Gateway reviews

2.1
In this Part, we describe what a Gateway review involves. We also clarify some common misunderstandings about Gateway reviews.

2.2
During our audit, we came across many people who did not have a good understanding of Gateway reviews. Quite a few people told us that they had not understood Gateway reviews properly until they had taken part in one. By explaining the main features of Gateway reviews, we hope to provide an accurate context for our audit findings in Parts 3 to 6.

What Gateway reviews involve

Gateway reviews provide advice, not judgements

2.3
A Gateway review is a short, focused, independent peer review of a project.

2.4
Gateway reviews are forward-looking. They do not focus on what might have gone wrong. Instead they look for problems that might impede progress or prevent the project from achieving its intended outcomes, and make recommendations to address those problems.

2.5
Gateway reviews do not pass or fail a project. The reviews look ahead to help a project succeed, rather than focus on what might have gone wrong in the past. Even when a review finds serious problems, this does not mean that the project has to stop. It simply means that major risks and issues need to be addressed so the project is less likely to fail.

2.6
Gateway reviews provide advice to project sponsors and make recommendations. It is up to each project sponsor to decide whether, and how, to implement any recommendations. The project sponsor is still accountable for the project and its outcomes.

Gateway review week

2.7
A review team, selected by the Gateway unit, spends a week interviewing people involved in the project and other stakeholders, such as Ministers and chief executives. The review team also looks at documents, such as business cases, risk registers, and project reports.

2.8
At the end of each day, the review team meets with the project sponsor. During this conversation, the review team can alert the project sponsor to any serious concerns. The project sponsor also helps to clarify any issues that have come up during the day’s interviews. This helps the team to focus on the most important matters in the limited time available.

2.9
At the end of the week, the review team produces a report for the project sponsor. This report gives the project a delivery confidence rating, sets out the main findings, and presents the team’s recommendations.

Projects are given delivery confidence ratings

2.10
A delivery confidence rating is based on the review team’s assessment of the project’s ability, at the time of the review, to deliver its aims and objectives successfully. To decide on the overall rating, the review team considers factors such as the project’s time and cost expectations, whether it is likely to deliver its expected outcomes to the quality required, and compliance with any formal approval requirements.

2.11
There are five levels of delivery confidence rating, ranging from green, through amber, to red (see Figure 1).

Figure 1
Delivery confidence ratings

Source: The Treasury.

2.12
The Treasury introduced delivery confidence ratings at the start of 2014 to replace a system of red/amber/green ratings that indicated the urgency and criticality of any remedial action required for the project to succeed.

There is an escalation process for projects with a low rating

2.13
Along with delivery confidence ratings, the Treasury also introduced an escalation process for projects assessed as red or amber/red. This is consistent with international Gateway practice.

2.14
The escalation process means that appropriate people, such as the agency’s chief executive and people with a monitoring role, are informed when a project is found to be at risk of failing. They can then provide additional monitoring and support for the project sponsor. This provides an opportunity to make sure that the project gets the help it needs to resolve any problems it is facing and continue towards a successful outcome.

2.15
Since the escalation process was introduced, eight out of 77 reviews have been escalated.

Outputs from Gateway reviews

2.16
The main outputs of a Gateway review are daily discussions with the project sponsor and the review team’s report. The report is written for the project sponsor and is confidential. This means that the project sponsor does not have to share the report with anyone else. In practice, most project sponsors choose to share the reports within their agencies, and sometimes outside them.

2.17
The Gateway unit keeps a copy of each report, but it is available only to staff working directly on Gateway reviews, for archiving and analysis of recommendations as an input to lessons learned reporting. Other parts of the Treasury get to see a report only if the project sponsor chooses to share it with them.

Most projects have several Gateway reviews

2.18
Projects have a series of Gateway reviews before key decision points, such as business case approval. The review process is similar each time, but will focus on different matters at different stages of the project. Figure 2 shows our description of the six “gates” that apply to projects.

Figure 2
Description of gates for projects

Gate | Gate name | Project stage/review focus
Gate 0 | Strategic assessment | Start up
Gate 1 | Business justification and options | Indicative business case
Gate 2 | Delivery strategy | Detailed business case
Gate 3 | Investment decision | Contract signing
Gate 4 | Readiness for service | Implementation
Gate 5 | Operational review and benefits realisation | 6-12 months after implementation, then repeated at intervals

2.19
The sequencing of Gateway reviews is different for programmes. Programmes, unlike projects, tend not to have a linear sequence of key decision points. Programmes undergo a series of reviews, repeated during the life of the programme and tailored according to the programme’s stage. Sometimes a programme review will also consider key decision points for a project that is part of the programme.

Projects do not always go through gates in a linear fashion

2.20
Although most qualifying projects have Gateway reviews, many do not have reviews for each gate. In a few cases, projects miss the earlier gates – for example, because the agency had not submitted a risk profile assessment and the project did not get picked up by the Gateway unit until a later stage. In other cases, the risk profile might increase as a project progresses and the project becomes eligible for Gateway only at a later stage. If a project has not had a Gateway review before the contract is signed, then it will not have any Gateway reviews because the potential benefits would be too few.

2.21
Gate 5 is not used in New Zealand. A Gate 5 review focuses on whether the project is achieving the results set out in the business case and whether the operational assets and/or services are running smoothly. It was found to be difficult to continue with Gateway reviews once a project had been implemented and the project team had disbanded. Agencies often opt for a different type of post-implementation review that covers some of the same matters as a Gate 5 review. This is consistent with the approach in other jurisdictions.

2.22
Gateway reviews often combine two gates in certain circumstances – for example, when there is a short time between two stages. In practice, there is flexibility as to which gate is applied when. Part of the planning for each review is to confirm which is the most appropriate gate, or combination of gates, to apply.

Part 3 – Introducing Gateway reviews in New Zealand

3.1
In this Part, we consider whether Gateway reviews are an appropriate tool for major capital projects in the New Zealand public sector.

Our expectations

3.2
We expected to see:

evidence that Gateway reviews could benefit projects in New Zealand; and

that there was a need for an intervention like Gateway reviews.

Summary of our findings

3.3
In our view it was reasonable to assume that Gateway reviews could provide some benefit for projects in New Zealand. Gateway reviews are based on a sound methodology and, before being introduced here, had been operating successfully and adding value to projects in the United Kingdom and Australia. Gateway reviews were also tested successfully with five pilot projects before being rolled out in New Zealand.

3.4
However, we did not see a strong case for introducing Gateway reviews in New Zealand. There was little analysis to show where intervention was most needed or why Gateway reviews in particular were the solution. This means that there may have been a better way to benefit projects in New Zealand.

How Gateway reviews could help New Zealand projects

Gateway reviews had been shown to benefit projects overseas

3.5
Before Gateway reviews were introduced in New Zealand, they had been operating successfully in the United Kingdom and parts of Australia, delivering both financial and non-financial benefits to projects. Given the similarities between these jurisdictions and New Zealand, in our view it was reasonable to assume that Gateway reviews could also benefit projects here.

Gateway reviews were tested before being fully introduced

3.6
Gateway reviews were tested with five pilot projects before being fully rolled out in New Zealand. The pilot reviews used experienced reviewers from Australia alongside New Zealand public servants. Feedback from the pilot reviews was very positive and indicated that Gateway reviews were likely to deliver benefits to projects. This meant there was confidence that Gateway reviews could be used effectively for all qualifying projects.

Gateway reviews target projects that are more likely to need help

3.7
Projects are selected for Gateway reviews based on a risk profile assessment, which uses a standard set of risk factors based on research into common causes of project failure. The assessment uses a series of questions set by the Treasury and has been customised from the version used in the United Kingdom.

3.8
The risk profile assessment considers factors such as:

cost;

the number of people involved in the project;

the degree of innovation involved (such as new technology, methods, or services);

the expected impact of the project;

the degree of change the project will lead to; and

the agency’s experience in the particular type of project.

3.9
The agency running the project makes a provisional assessment. If the risk profile rating comes out as high or medium, the agency has to submit the assessment to the Treasury. The Treasury and other relevant agencies review each assessment and confirm or amend the risk profile rating. All projects with a final rating of high are required to have Gateway reviews.

3.10
Ministers can also request that a project has a Gateway review.

3.11
By focusing on projects that are considered to have a higher risk of failing, Gateway reviews are more likely have a positive impact on those projects.

The case for introducing Gateway reviews

3.12
We did not see a strong case for introducing Gateway reviews to New Zealand. We had expected to see some analysis of what had caused problems with major projects in the past, and what types of intervention might address the problems and reduce the chance of their recurring. This would determine the intended purpose and objectives for introducing Gateway reviews. For example, objectives might include reducing costs and improving skills.

3.13
The case for Gateway reviews was based on them working well in the United Kingdom and Australia, jurisdictions with similar systems to New Zealand, where the reviews had been shown to add value to projects. In our view, it was reasonable to assume that Gateway reviews could deliver benefits in New Zealand, but a stronger argument should have been made to support their introduction.

3.14
We also expected to see an analysis of options, matching different types of intervention with the problems identified, so that the most effective and efficient measures could be put in place.

3.15
Gateway reviews were introduced in New Zealand in 2008 after a major review in 2006, led by the Treasury, of capital asset management policy settings and practices in departments and Crown entities. As a result of that review, the Government introduced new processes and standards to strengthen capital asset management in the public sector. Gateway reviews were one of these. The others were:

a common framework for all assets controlled or monitored by departments and Crown entities;

a whole-of-life approach to capital asset management;

a requirement for capital-intensive agencies to demonstrate an advanced standard of capital asset management, and other agencies to demonstrate a core standard;

a two-stage Cabinet approval process for all new capital investment proposals above a certain threshold; and

an expectation that capital-intensive agencies would manage their asset portfolios with a 20-year planning horizon.

3.16
The Treasury’s review found that capital asset management was poorly understood, and there was room for improvement at all stages of an asset’s life cycle. However, we did not find any detailed analysis of how Gateway reviews could benefit projects in New Zealand or why Gateway reviews were the best option.

3.17
Without clear objectives, it is not possible to assess whether Gateway reviews have been successful. Although we found that Gateway reviews have delivered some benefits to projects in New Zealand, it is not possible to say whether an alternative approach might have delivered more.

Part 4 – Operating Gateway reviews in New Zealand

4.1
In this Part, we:

consider how well Gateway reviews have been operating in New Zealand;

examine whether the right projects have Gateway reviews at the right time; and

consider how well Gateway reviews fit with other types of project monitoring and assurance.

Our expectations

4.2
We expected to see:

Gateway reviews following a set methodology;

effective processes to make sure that qualifying projects have Gateway reviews at the right time; and

a clear understanding of how Gateway reviews fit with other monitoring and assurance functions.

Summary of our findings

4.3
Gateway reviews have been operating well in New Zealand. They have been carried out in accordance with the official Gateway methodology. This has been achieved through training and support and helps make sure that projects get the benefits associated with Gateway reviews.

4.4
Most projects are having Gateway reviews as required. The Treasury is aware that a few projects may be missed but, in general, it has effective processes to make sure that qualifying projects are identified and kept track of as they progress.

4.5
There is not a clear understanding of how Gateway reviews fit with other monitoring and assurance activities, in particular those activities run by different parts of “the Corporate Centre”.² As a result, these activities may not be as effective and efficient as they could be and may not deliver the best outcomes for projects.

Operating Gateway reviews

Gateway reviews follow a set methodology

4.6
Gateway reviews are planned and operated in a way that helps to ensure that they follow the Gateway methodology. There is some flexibility in the way that each review is carried out to best meet the needs of each individual project, but the general approach has to be followed and the main features included. These features include the confidentiality of the reports and the advisory nature of the reviews.

4.7
The following processes have been put in place to help review teams follow the methodology:

all Gateway reviewers have to attend training before they can take part in a review;

review teams always include some experienced reviewers who have been through the process several times;

there is a planning day held for each review, which is attended by the project sponsor, the review team, and someone from the Gateway unit, when the review scope and approach are confirmed;

review teams are provided with guidance and templates; and

someone from the Gateway unit also attends at times during the review week, to provide advice on the Gateway review process and its application.

4.8
The reviewers we spoke to had a good understanding of the Gateway approach. Some of them had taken part in reviews where new reviewers had started to explore areas that are out of scope of Gateway reviews, such as policy decisions. This was picked up and resolved by the rest of the team.

4.9
The methodology has to be followed in order to use the Gateway brand, which is owned by the United Kingdom Government. This means that any other jurisdiction that uses Gateway reviews has limited scope to change how the reviews are applied. A few small changes have been made to fit the New Zealand context (for example, changes in terminology in supporting documents). Generally, however, the reviews here are carried out in the same way as they are in the United Kingdom and Australia.

4.10
By following the Gateway methodology, reviews are more likely to deliver the benefits associated with Gateway reviews.

Most projects have Gateway reviews as required

4.11
The Treasury has processes to make sure that most qualifying projects have the right Gateway review at the right time. We found consistency between the agencies most likely to have high-risk projects and the agencies that have Gateway reviews. This shows that the Treasury’s processes are largely effective.

4.12
Agencies with qualifying projects are supposed to submit a risk profile assessment to the Treasury, but this does not always happen. This might be intentional or it might reflect a lack of awareness. The Gateway unit looks at information from various sources to identify potentially qualifying projects for which an assessment has not been submitted. These information sources include:

other parts of the Treasury, such as the people who monitor capital expenditure and review business cases;

other central agencies with a monitoring role; and

the online register of government tenders (GETS), which the Gateway unit monitors regularly.

4.13
Sometimes people connected with an agency will alert the Gateway unit about projects that need to complete a risk profile assessment. The Gateway unit can then ask an agency to comply. The Treasury is confident that more than 90% of all qualifying projects have Gateway reviews.

4.14
The Treasury is aware of one or two agencies that actively avoid engagement with the Treasury’s monitoring and review processes, including Gateway reviews. This is being dealt with at an organisational level. Some other agencies engage with Gateway reviews but only when they are prompted. The Treasury has found that, when an agency has been through a few reviews, it is much more likely to engage proactively with Gateway reviews.

4.15
We looked at which agencies have had Gateway reviews and compared this with the agencies that the Treasury has identified as investment-intensive. We expected to see a reasonable level of consistency between them. Figure 3 shows how many projects from each agency have had Gateway reviews. Figure 4 shows the current list of the most investment-intensive agencies (categorised by the Treasury as “Tier 1” and “Tier 2”).

4.16
Figures 3 and 4 show that most of the investment-intensive agencies have had Gateway reviews, as expected. Those that have not had reviews might own or control a high value of assets but might not have carried out any large or high-risk projects since Gateway reviews were introduced. Figures 3 and 4 support the Treasury’s assertion that most eligible projects have Gateway reviews.

Figure 3
Number of projects that have included Gateway reviews, by agency

Source: Our analysis of Treasury data.

Figure 4
Investment-intensive agencies

Tier 1 | Tier 2
Ministry of Education | Ōtākaro Limited*
New Zealand Defence Force | Department of Conservation
Department of Corrections | Department of Internal Affairs
Inland Revenue | Ministry of Business, Innovation and Employment
Accident Compensation Corporation | Ministry of Foreign Affairs and Trade
Housing New Zealand | Ministry of Justice
New Zealand Transport Agency | Ministry of Social Development
Ministry of Health | New Zealand Customs Service
Auckland District Health Board | New Zealand Police
Canterbury District Health Board | Tertiary Education Commission
| Counties Manukau District Health Board
| Waitemata District Health Board
| Waikato District Health Board
| Capital and Coast District Health Board
| Southern District Health Board
| Northland District Health Board

* Ōtākaro Limited is responsible for projects that previously came under the Canterbury Earthquake Recovery
Authority.

Source: The Treasury; http://www.treasury.govt.nz/statesector/investment-intensive-agencies.

4.17
Once a project has had its first Gateway review, the Treasury makes sure that subsequent reviews happen at the right time. Some project sponsors contact the Gateway unit when their project is ready for the next review. For others, the Gateway unit keeps track of each project and contacts the project sponsor when the next Gateway review is due. If a project has been delayed then the next review might be postponed. A few projects have been cancelled or reduced in scope and therefore no longer required Gateway reviews.

4.18
For most projects, Gateway reviews, like many other types of intervention, can be most useful early on, before the project has committed to a particular solution. This is when ideas can be tested and there is more scope to modify the project to help it succeed.

Improvements have been made to Gateway reviews over time

4.19
The Gateway unit uses up-to-date information to make changes to Gateway reviews. In particular, the Gateway unit keeps in close contact with the United Kingdom Major Projects Authority – the agency with responsibility for the Gateway brand. This relationship led to the introduction of delivery confidence ratings at the start of 2014, a change that had already been made in some other jurisdictions. New Zealand also takes part in an Australasian Gateway forum, where Gateway units from different jurisdictions can share ideas for improving Gateway reviews.

4.20
The Gateway unit also uses feedback from project sponsors and reviewers to make minor adjustments to the way it operates Gateway reviews.

How Gateway reviews fit with other monitoring and assurance activities

It is not clear how Gateway reviews fit with other monitoring and assurance activities

4.21
We found that there was not a clear understanding of how Gateway reviews fit with other types of monitoring and assurance activities that apply to most large projects. In particular, we found that roles and responsibilities in the Corporate Centre for monitoring and assurance of major projects were not clear. The Treasury confirmed this and told us that recent attempts to reach agreement on these roles and responsibilities had not been successful.

4.22
Our audit did not look in depth at these other types of monitoring and assurance activities or how well all these activities work together to get the best outcomes for projects. However, we did expect to find a clear understanding of how Gateway reviews fit with those other activities.

4.23
Without this understanding, monitoring and assurance activities can be less effective and efficient. There is the potential for overlap if different reviews look at the same things. In some cases, this could lead to conflicting advice, especially if the purpose of each review is not clear. There is also the potential for reviews to completely miss some aspects of a project.

Gateway reviews and Independent Quality Assurance

4.24
One particular type of review that also applies to most projects eligible for Gateway reviews is Independent Quality Assurance (IQA). An IQA review is carried out by an external provider; this is usually a consulting firm. Some people we spoke to did not understand the difference between an IQA review and a Gateway review. The main differences between them are shown in Figure 5.

Figure 5
Differences between Gateway reviews and Independent Quality Assurance reviews

Gateway | Independent Quality Assurance
Focus is at a high level. | Focus is at a detailed level.
Forward-looking. | Backward-looking.
Provides advice to the project sponsor, at a peer level, to help the project succeed. | Reviews project’s progress and the processes, standards, guidance, and practice used in its management and governance.
Not suitable for use as an input for performance and approval purposes. | Can be used as an input for performance and approval processes.

4.25
There are some similarities between Gateway reviews and IQA reviews. For example, both tend to use interviews and document reviews. Some duplication in the matters they look at is also possible. However, agencies have some discretion to direct the scope or focus of an IQA review and so minimise any duplication.

4.26
We note that, in the United Kingdom, where Gateway reviews were designed, IQA reviews are not in common use. It is not clear how much effort went into ensuring that Gateway reviews did not duplicate IQA reviews when Gateway reviews were introduced in New Zealand. However, the State Services Commission was responsible for both introducing Gateway reviews and helping agencies to plan IQA reviews, and was well placed to ensure a clear distinction. Documentation that we looked at included a table showing the differences between Gateway reviews and IQA reviews.

2 “The Corporate Centre” is a term used to describe public entities that have some oversight role in large government projects and programmes. The Corporate Centre includes the Treasury (investment assurance), the State Services Commission (capability assurance), the Government Chief Information Officer (information and communication technology assurance), and the Department of the Prime Minister and Cabinet (co-ordinating and supporting government priorities). The Corporate Centre also includes the public entities that have lead responsibilities throughout government for procurement and property (Ministry of Business, Innovation and Employment) and shared ICT capabilities (Department of Internal Affairs).

Part 5 – Capability and capacity for Gateway reviews

5.1
In this Part, we consider whether agencies, the Treasury, and review teams have the capability and capacity to carry out their respective roles in Gateway reviews effectively.

Our expectations

5.2
We expected to see:

agencies able to take part in reviews and implement the resulting recommendations;

appropriate resourcing to support Gateway reviews;

a sufficient pool of suitably qualified and experienced Gateway reviewers; and

processes to make sure that the best team is selected for each Gateway review.

Summary of our findings

5.3
In our view, agencies have the capability and capacity to pay for and participate in Gateway reviews. The $75,000 cost of each Gateway review is reasonable, especially when considered as a percentage of the overall cost of each project. For the review itself, there is some burden on the agency, but this is limited. Project sponsors told us they did not find Gateway reviews too onerous.

5.4
The Treasury has adequate resources to run Gateway reviews and does this well. However, it has limited capacity for additional work, such as making improvements.

5.5
Having the right review team is critical to the success of Gateway reviews. There is a large pool of reviewers available, which helps make sure that the best team is put together for each review. This can be challenging, but the Treasury puts a lot of effort into doing this well.

5.6
Review team independence is also important and has been shown to bring benefits to projects.

Agencies

The cost of Gateway reviews is reasonable

5.7
In our view, the cost of Gateway reviews is reasonable, especially when compared to the overall budget of the types of project that have Gateway reviews. Each review costs the agency $75,000. For a project of $50 million, five Gateway reviews will cost 0.75% of the budget. Most projects going through Gateway reviews have much larger budgets, and few of them have all five reviews.

5.8
Agencies with more experience of Gateway reviews are more likely to build the costs into their budgets. Not planning for Gateway review costs will inevitably make the costs harder to accommodate.

Gateway reviews are not too onerous for agencies

5.9
We consider that, on balance, Gateway reviews are not too onerous for agencies. Any type of project review will place demands on the agency and project team involved. Project teams are often under a lot of pressure already, and reviews require people to give up their time to provide documents and attend interviews.

5.10
Gateway reviews are designed to reduce the burden on agencies and project teams. For example, the Gateway unit selects and contracts the review team and makes all arrangements for the team’s travel and accommodation. Most people involved from within the agency will be required to attend only one interview of an hour’s duration. Although review teams need to see documentation, this would comprise key project documents that we would expect to be complete and easily available as a matter of course for any project.

5.11
The burden of a Gateway review is greater on the project sponsor, who will have to engage more frequently with the review team, including attending the daily debriefs. Project sponsors told us that they did not find this burden too much and accepted the need to allow for Gateway reviews as part of doing a major public sector project. Feedback received by the Treasury shows that most project sponsors agree that one week is a suitable duration for a review. Project sponsors have commented that one week is a good balance between giving the review team enough time to understand the project while also allowing the team to report back its findings quickly.

5.12
The burden of a Gateway review does not stop at the end of the review week. After receiving the Gateway report, the project sponsor will have recommendations to consider and implement. We discuss how well agencies implement Gateway recommendations in Part 6. However, we have not seen any evidence that agencies lack the capacity and capability to implement recommendations made during Gateway reviews.

The Treasury

The Treasury has enough resourcing to run Gateway reviews but little capacity for improvement

5.13
The Gateway unit has adequate capability and capacity to run Gateway reviews, but no spare capacity. The unit currently has two full-time staff members and one part-time staff member and supports 25-30 reviews each year. Over the last few years, the number of reviews has increased, but the number of staff in the Gateway unit has decreased. This means that there is no spare capacity for adding value through activities such as researching and implementing improvements.

5.14
The Gateway unit has been run effectively and efficiently by the same person for several years. However, this has meant that the unit is heavily dependent on that one person’s knowledge and experience.

Gateway review teams

There are enough reviewers available to carry out Gateway reviews to a high standard

5.15
There is a pool of nearly 700 reviewers from New Zealand, Australia, and the United Kingdom available for carrying out Gateway reviews in New Zealand. Having a large pool of reviewers makes it easier to find the best team of available reviewers for each review.

5.16
Review team members come from various backgrounds. Each team is put together to get the best mix of skills and experience for each project. For example, the review team for a construction project that is about to sign a contract might include people with experience in construction, contract management, and project management.

5.17
Currently, about a third of the review pool is Australian, with a few from the United Kingdom. Most review teams include at least one Australian. Overseas reviewers may have expertise not available in New Zealand. They are often used as review team leaders because they typically have more experience, particularly for large or complex projects. Overseas reviewers also have fewer conflicts of interest with New Zealand agencies.

5.18
The review team is critical to the effectiveness of a Gateway review. When a team does not have the right skills and experience, or does not work well together, the advice for the project sponsor is likely to be of less value.

New Zealand is still dependent on reviewers from overseas

5.19
New Zealand still uses a lot of reviewers from overseas. The United Kingdom and Australia have been running Gateway reviews for longer than New Zealand and have more projects that have been through Gateway reviews. This means that reviewers from those countries often have a lot more experience than New Zealand reviewers. When Gateway reviews were first introduced, most reviewers came from Australia and some from the United Kingdom, because there was little experience in New Zealand. Over time, more New Zealanders have become reviewers, and there is less reliance on reviewers from overseas.

5.20
Reviewers can come from the private or the public sector. For some projects, reviewers from the private sector have the most suitable skills and experience. Review teams usually include at least one reviewer from the public sector, because they understand the system that public sector projects operate in.

5.21
Figure 6 shows how the mix of reviewers taking part in reviews has changed since Gateway reviews were introduced in New Zealand.

Figure 6
New Zealand Gateway reviewers by country and sector

Source: The Treasury.

The Treasury trains and assesses reviewers

5.22
The Gateway unit makes sure that reviewers are prepared and suitable to work in and lead a Gateway review team. An application process makes sure that all new reviewers have the right skills and experience. New reviewers have to attend a workshop about Gateway reviews before they can take part in a review. The workshop explains how the reviews work and what reviewers should be looking for.

5.23
After each review, the review team leader gives feedback to the Gateway unit on the team members. This feedback can identify people with the potential to become team leaders, or those who are not suitable for Gateway reviews. Review team members also provide feedback on the team leader. Feedback from project sponsors is also used.

5.24
Gateway reviews are carried out by a team of four reviewers, one of whom is the review team leader. To become a review team leader, a reviewer usually has to complete 8-10 reviews with consistently good feedback. They are then invited to a workshop for review team leaders.

The Treasury understands how to put a high-performing review team together

5.25
The Gateway unit puts a lot of effort into getting the best team for each review. Our interviews, together with feedback provided to the Gateway unit, confirmed that this is usually done well.

5.26
Gateway reviews are high-level and intensive, and are based on interviews. Review team members therefore need the right mix of technical and professional skills, combined with suitable interpersonal skills. Technical and professional skills are needed to give high-quality advice quickly.

5.27
People with the right skills are often ruled out of a review team by a conflict of interest. This can be a particular problem in specialist areas in New Zealand, where only a few people have expertise in areas such as education funding systems, heritage construction, or urban renewal.

5.28
The right interpersonal skills are important, because reviewers have to work as a team and communicate effectively with the agency.

5.29
The Gateway unit also considers the mentoring and development needs of review team members. A new team leader will always have an experienced team leader in their team. Each review will include only one new reviewer. Experienced reviewers can be placed with an experienced leader, who can assess whether they are ready to take on the role of team leader.

5.30
Although there is a large pool of reviewers, taking all these matters into consideration means that it can sometimes be difficult to find the right mix of available people for each review. We heard about some instances where a reviewer was not suitable, and this affected how well the team was able to work with the project sponsor, and the quality of advice they provided.

Review team independence is important

5.31
Review team members are independent of the project being reviewed and the agency managing the project. They are contracted directly by the Treasury, not the agency, and are restricted from providing any services to the project after the review. This independence means that review teams can offer impartial advice.

5.32
One project we looked at had been fast-tracked, and the project sponsor had been concerned that he might have missed something important. The review team’s independence brought a more objective view, and this was able to reassure the project sponsor that he had not missed anything critical.

5.33
For another project that was making slow progress, the independent review team was able to see that the agency was working in silos when people within the agency could not see this themselves. Having this view expressed by an independent team meant that it was more easily accepted by the agency, which then committed to resolve the problem.

Part 6 – Benefits delivered by Gateway reviews

6.1
In this Part, we explore whether there is evidence that Gateway reviews have delivered value to projects in New Zealand.

Our expectations

6.2
We expected to see:

evidence that Gateway reviews have delivered value for individual projects;

an assessment of the performance of Gateway reviews; and

evidence that Gateway reviews have delivered broader benefits to capital asset management systems.

Summary of our findings

6.3
Gateway reviews have delivered benefits to individual projects. Sometimes Gateway reviews help projects by providing a sounding board for the project sponsor. In other instances, Gateway reviews have suggested specific actions, which have directly affected a project’s success.

6.4
The effectiveness of Gateway reviews in New Zealand has not been measured. Analysis in the United Kingdom found that Gateway reviews delivered cost savings to projects. This analysis was complex and has not been reproduced here. However, in our view, Gateway reviews are likely to have resulted in cost savings for New Zealand projects.

6.5
One way that Gateway delivers value is through the recommendations made by review teams. These recommendations are of mixed quality. Most, but not all, are implemented. Poor-quality recommendations substantially reduce the value that Gateway reviews can add to a project.

6.6
Gateway reviews can benefit more than just the individual projects that are reviewed. The skill level of people taking part in reviews are improved by giving them exposure to different projects in different agencies and from working with different reviewers.

6.7
There is scope to improve the way lessons from Gateway reviews are shared. Currently, the Treasury categorises Gateway review recommendations by themes and publishes the results. Not everyone we spoke to knew about this. The Treasury has started to collect information about examples of good practice observed during reviews, but is not yet collating and analysing this information. Sharing lessons more effectively could benefit all projects.

6.8
Confidentiality is an important part of a Gateway review, and it does bring benefits. However, it also conflicts with transparency and accountability, which are important values in New Zealand’s public sector.

Benefits for individual projects

Gateway reviews deliver value to individual projects

6.9
We found that Gateway reviews can deliver value to individual projects. Project sponsors gave examples where these reviews had made a difference to their projects. Feedback collected by the Gateway unit is also positive, with 100% of project sponsors³ agreeing, or agreeing strongly, that the Gateway review “was beneficial and will impact positively on the outcome of the project”.

6.10
In many instances, the main benefit to the project sponsor came from using the review team as a sounding board to work through issues. Projects can also benefit more directly from Gateway reviews.

6.11
For one project, the agency had been struggling to get approval from its different stakeholders and was unable to progress. The Gateway review helped the project sponsor to decide what actions were needed to get stakeholders to agree. As a result, a way for the project to move forward was agreed.

6.12
In another instance, the agency was not confident that its business case for the project would be approved. The review team provided advice and guidance to help the agency to rewrite the business case so that it presented a more compelling proposal. Cabinet approved the business case and the project avoided delays and costly rewrites.

The Treasury has not measured the performance of Gateway reviews in New Zealand

6.13
The Treasury has not made a formal assessment of how Gateway reviews have performed in New Zealand. Without clear objectives (as we discussed in Part 3), it is impossible to say whether Gateway reviews have achieved what they were intended to achieve. However, it would still be possible to assess the impact that Gateway reviews have had.

6.14
A cost benefit analysis of Gateway in the United Kingdom estimated that 2-4% of project costs were avoided as a result of Gateway reviews. It is reasonable to assume there would be savings in New Zealand, but there has been no analysis to show what level of savings has been achieved here. Gateway reviews follow the same process here, but projects are much smaller. The analysis in the United Kingdom was skewed towards projects with a budget in excess of £1 billion.

6.15
The analysis in the United Kingdom was very complex and would be difficult to replicate. It would be particularly difficult to isolate the effect of Gateway reviews from other interventions in the same period. For example, Better Business Cases were introduced in 2010 and have been found to have significantly improved the quality of business cases. We note that one of the reasons for introducing Better Business Cases was the high number of recommendations about problems with business cases in early Gateway reviews.

6.16
Another way to consider whether Gateway reviews have been successful would be to find out whether the reviews had stopped any projects from failing. This would also be difficult, because we would need evidence of whether, and how many, projects were failing before Gateway reviews were introduced. Secondly, we would need to attribute a project’s success to the Gateway reviews. Even if we found that a Gateway review had picked up a critical problem, we cannot assume that another process would not have done the same. However, what we do know is that no projects that have included Gateway reviews have failed. Some projects have been cancelled or re-scoped.

For most reviews, recommendations are an effective means of adding value to the project

6.17
Some of the recommendations in Gateway review reports are better than others. We expected to see variation in the number and type of recommendations, depending on the nature of projects. We looked at the recommendations in 44 review reports and found that those reports had between 5 and 21 recommendations, which varied in quality.

6.18
In our view, a high-quality recommendation is:

specific about the action required, with sufficient detail;

practical and achievable by the project sponsor;

focused on the key issues; and

ranked by priority.

6.19
Features of lower-quality recommendations that we found included:

recommendations requiring action by people outside the project sponsor and project team;

impractical recommendations;

unclearly worded recommendations;

recommendations that did not get to the bottom of issues; and

obvious recommendations or recommendations about matters that were already being addressed.

6.20
Recommendations are one of the main ways that a Gateway review can influence a project. Poor-quality recommendations therefore reduce the value that a Gateway review can bring.

6.21
We also looked at whether the recommendations were implemented. Most Gateway reports include a table showing progress on recommendations from the previous review, where there was one.

6.22
We found that most recommendations were implemented but several were not. Sometimes this was because the recommendation was no longer applicable, but it does call into question the relevance of some recommendations. Although recommendations are only advisory, we would expect project sponsors to implement most of them.

Broader benefits

Taking part in Gateway reviews can improve skill levels for everyone

6.23
When the Government introduced Gateway reviews, one of the intended benefits was that by taking part in reviews, reviewers would improve their own skill levels through the experience of looking at projects outside their own agency. They would also benefit from the input of the other reviewers on the team. Those reviewers would then be able to bring back what they had learned and apply it to projects in their own agency.

6.24
We saw some evidence of this. For example, one chief executive told us that even after being part of only one review team, he was able to bring back lessons and apply them to projects in his own agency.

6.25
The impact of this is limited however. Although over 40 agencies have had employees take part in review teams, only about 20% of reviewers come from the New Zealand public service. It is often difficult for senior public servants to be part of a review team. Review team members have to dedicate an entire week for each review and, for most people, there is a limit to how often this is feasible.

There is scope to improve the communication and implementation of lessons learned from Gateway reviews

6.26
The Treasury already shares some of the lessons learned from Gateway reviews. There is room for improvement by making sure that the right people are informed and that lessons are applied to other projects.

6.27
Reports on the lessons learned categorise Gateway review recommendations by themes and summarise the main findings under each theme. These reports also include examples of recommendations from Gateway reports. The examples do not include any contextual information that could identify the projects they come from.

6.28
So far there have been three reports on lessons learned – one for every 50 Gateway reviews. The reports show how the distribution of recommendation themes has changed over time. Figure 7 shows the distribution of recommendations for each report.

Figure 7
Percentage of recommendations by category in the 2011, 2013, and 2015 reports on lessons learned

Source: Adapted from the Treasury (2015), New Zealand Gateway Reviews: Lessons Learned Report 2015, Wellington.

6.29
The Treasury could do more to make sure that these lessons are shared proactively. The reports are distributed to government agencies and are also available on the Treasury’s website. Some people told us they find the reports useful, while others had not seen them.

6.30
There is also scope to share the examples of good practice observed by Gateway reviewers. The reports on lessons learned look only at recommendations on aspects of the projects that are not being done as well as they could be. We expect that there are also many examples of good practice, which could benefit other projects. Since 2015, Gateway reports have included a small section where reviewers can give these examples. The Gateway unit has not started to use this information yet.

6.31
Gateway reviews provide a unique opportunity to delve into a wide range of projects in the public sector. Sharing lessons from these projects effectively should help people to apply them to other projects.

Recommendation 1

We recommend that the Treasury determine how to more effectively share lessons learned through Gateway reviews, including examples of good practice.

The confidentiality of Gateway reviews has benefits but limits the reviews’ potential

6.32
Confidentiality is a fundamental principle of Gateway reviews. Most project sponsors share the report within their agencies and sometimes more widely, but they can control what they share and when. This is important, because Gateway reviews are intended to help the project sponsor, not to assess their performance and hold them accountable.

6.33
Each interview during a review is also confidential. This helps people feel free to speak openly about the project. Without confidentiality, many Gateway reviews would not be able to find out what the real issues are.

6.34
However, transparency and accountability are important values in our public sector. Confidential Gateway reviews are not in keeping with these values.

6.35
Central agencies with a monitoring role may not be able to carry out this role effectively if they are not informed about important findings from a Gateway review. This risk has been mitigated by the escalation process described in Part 2, which makes sure that the relevant people are informed when a project is at risk of failing.

3 This is with an 89% total response rate since 2008.

Part 7 – The future for Gateway reviews

7.1
In this Part we:

identify limitations of Gateway reviews; and

explain the Treasury’s intentions for addressing those limitations.

Limitations of Gateway reviews

7.2
We found that, although Gateway reviews do help projects, they also have some limitations.

7.3
Gateway reviews have a set approach. While they allow for some flexibility to meet the needs of individual projects, they are still very much a “one size fits all” approach. For some projects, a different approach might be more beneficial – for example, a shorter and more-focused review when a project is facing a specific challenge.

7.4
There can be a long time between successive reviews; in many cases, it has been more than two years. This might happen for a large and complex project where it does take a long time to complete each stage. It can also happen when a project has problems and is changed or delayed. In these situations, projects might benefit from an interim review.

7.5
As we discussed in Part 6, the confidential nature of Gateway reviews is not consistent with values of transparency and accountability. Some stakeholders might want more information about how a project is performing overall.

7.6
Gateway reviews provide only a limited type of assurance. They can provide a level of assurance that the project has been thoroughly and independently reviewed using an approach recognised by central government agencies. The delivery confidence rating also gives an indication of the project’s likelihood of success. This gives the project sponsor some assurance that the project is on track and ready to proceed to the next stage. However, Gateway reviews are not designed to provide assurance to the Government, or to the public, that a project is being managed well and is on track to achieve its intended results.

The Treasury’s intentions

7.7
The Treasury is aware of the limitations of Gateway reviews. In response, the Treasury is considering introducing new types of investment review to run instead of, or as well as, Gateway reviews. The aim is to offer a selection of review types that provide more flexibility to meet the various needs of different projects. As part of this, the Treasury is designing and will be implementing a new benefits realisation review consistent with its new focus on benefits delivery from projects. In our view, the Treasury also needs to make sure that there is a clear business need for any new reviews that it introduces.

Recommendation 2

We recommend that the Treasury ensure that the work it has under way to consider new types of investment review takes into account:

the limitations of Gateway reviews;

the specific needs of projects and programmes in New Zealand; and

the information and assurance needs of stakeholders such as Ministers, chief executives, boards, and the Corporate Centre.

7.8
This work is happening as part of a wider initiative for the Treasury to take the lead in the investment management system throughout the State sector. As part of this initiative, the Treasury set up an Investment Management and Asset Performance (IMAP) team. The IMAP team was formed from existing functions in the Treasury and the State Services Commission concerned with the performance of major projects, investments, and assets in the State sector. The Investment Reviews unit, which currently runs Gateway reviews, is part of the IMAP team.

7.9
The IMAP team has introduced new requirements for management of investments and assets in the State sector.⁴ These include the new requirements for investment planning and investor confidence ratings. These ratings of an agency’s investment management environment provide an indicator of the confidence Ministers and other stakeholders have in an agency’s capacity and capability to realise a promised investment result if funding were committed.

7.10
Investor confidence ratings can affect the degree of central control over each agency’s investment management. One effect could be that agencies with a high rating will have less scrutiny from the Corporate Centre. For example, they might not be required to have Gateway reviews.

4 The new requirements were released as Cabinet Office Circular CO (15) 5 Investment Management and Asset Performance in the State Services.