Part 8: Evaluation of grant programmes

• Introduction
• Roles and responsibilities for grant programme evaluation
• Evaluation activity occurring within NZTE
• Assessment of the evaluation of NZTE’s grant programmes

Introduction

8.1
Programme evaluation plays a key role in indicating how effective government programmes are. As NZTE identified in its first Strategic Plan, dated July 2003 –

We need to understand how effectively our activities contribute to the Government’s desired policy outcomes. In developing and delivering our services and programmes we will clearly identify the desired outcomes and collect the data and information that will enable us to evaluate the extent to which our programmes and services achieve those outcomes.

8.2
The quality of the evaluation activity undertaken by NZTE should be assessed in the context of both the level of understanding of evaluation and the capability to undertake it within the New Zealand public service as a whole.

8.3
The programmes examined as part of this audit present particular evaluation difficulties, which include:

• the diversity of organisations receiving the grants;
• the proportionately small amounts of funding that can go to organisations – this is particularly relevant for the Enterprise Development Fund;
• the likelihood that multiple other factors affect the organisations being funded at the same time as they receive a grant; and
• the complexity of the links between the activities of organisations and the final outcomes for the wider sector.

8.4
Further, information about appropriate evaluation models is scarce. A May 2004 report, which looked at lessons for New Zealand from 4 benchmark countries on export development and promotion,¹⁷ found that internationally there was little evidence of systematically applied performance evaluation.

8.5
MED, MFAT, and NZTE all play a role in evaluating NZTE’s grant programmes. For the purposes of this report, we have looked only at programme evaluation for the grant programmes that were the subject of our audit.

Roles and responsibilities for grant programme evaluation

8.6
MED and MFAT are jointly responsible for monitoring and evaluating NZTE’s services. This responsibility primarily sits within MED’s Industry and Regional Development Branch. One of the Branch’s outputs is to–

... undertake systematic evaluation of the range of business support and facilitation programmes administered by NZTE, to identify those that are most effective in increasing aggregate business performance; and progressively refocusing effort on these.

8.7
Within the Industry and Regional Development Branch is the Research, Evaluation and Monitoring team (REM). REM is responsible for designing and implementing the Branch’s research programme, evaluating the impacts of government programmes, and monitoring the performance of NZTE.

8.8
REM’s evaluation activities include:

• in-depth programme reviews to assess the efficiency and effectiveness of NZTE’s programmes; and
• an annual review (referred to by REM as a “stocktake”) of industry and regional development policy to broadly evaluate programmes.

8.9
In undertaking its in-depth reviews, REM works closely with MED’s sector development policy team, who are responsible for the design of sector and industry programmes delivered through NZTE.

8.10
A joint MED/MFAT monitoring and evaluation unit (MEU) performs the monitoring functions of NZTE. The MEU is located within MED, and includes one permanent MFAT staff member in the unit. The MEU is primarily responsible for providing advice to relevant Ministers on NZTE’s accountability framework, including the development of supporting documents such as the Output Plan, Statement of Intent, and the relationship management protocol.

NZTE’s Strategy and Evaluation Group

8.11
NZTE has a Strategy and Evaluation Group that is responsible for:

• consultation on the scope of evaluation products;
• co-ordination of information inputs into evaluation products and internal peer review of evaluation findings and conclusions; and
• undertaking components of research activity to support assessment of programme effectiveness and efficiency.

8.12
NZTE’s Strategy and Evaluation Group works closely with MED to ensure that there is no duplication of work by the 2 organisations. Figure 17 below sets out the roles and responsibilities for conducting in-depth or programme evaluations.

Figure 17
In-depth evaluation roles and responsibilities

Evaluation activity occurring within NZTE

Background

8.13
Before the merger of Trade New Zealand and Industry New Zealand on 1 July 2003, different evaluation approaches were applied. MFAT evaluated Trade New Zealand¹⁸ (responsible for the previous version of the Enterprise Networks programme), while MED evaluated Industry New Zealand¹⁹ (responsible for the programmes that became the Growth Services Fund, Enterprise Development Grants, the Major Events Fund, and the Strategic Investment Fund).

8.14
Both Trade New Zealand and Industry New Zealand had mechanisms for measuring their performance. Industry New Zealand focused on activity measures, while Trade New Zealand focused on outcome measures.

8.15
After the establishment of NZTE, MED and MFAT have undertaken some evaluative activity. This has included an in-depth review of the implementation of Investment New Zealand, MED’s annual review (or stocktake), and the development of 2 draft evaluation and research plans.

In-depth reviews

8.16
From the Cabinet minutes establishing NZTE, we noted that programme evaluations, or in-depth reviews, were to include an assessment of:

• the adequacy of the programme’s delivery and administration;
• the effectiveness of the programme’s various services and whether they met their objectives;
• whether the services provide value for money; and
• whether the services were still necessary.

8.17
MED has planned a series of in-depth evaluations of the NZTE grant programmes that were subject to our audit. As part of the establishment of NZTE, an in-depth evaluation timetable has been agreed. Figure 18 on the next page sets out the current evaluation timetable.

Figure 18
Evaluation timetable for the grant programmes subject to our audit

8.18
As at 31 October 2004, MED had completed one in-depth review or programme evaluation for the grant programmes we examined. The Evaluation of the Implementation of Investment New Zealand was a review conducted by MFAT and MED in consultation with NZTE and the Treasury. Included within this review was a consideration of Investment New Zealand’s implementation of the Strategic Investment Fund.

8.19
The review undertaken by MFAT and MED found that it was not evident that the evaluation of grants undertaken by Investment New Zealand met the level of detailed analysis originally anticipated by Cabinet. Further, the review found that the departments would expect to see a more robust analysis being undertaken by Investment New Zealand in support of proposals. For cash grants and guarantees of funding, the review identified a need for Investment New Zealand to consider seeking independent expert advice on the costs and benefits of proposed investment, particularly “spill-over” benefits.

8.20
After completion of the review, the Minister for Economic Development and the Minister for Trade Negotiations recommended to Cabinet:

• development of a strategic policy framework for Investment New Zealand; and
• changes to the criteria and administration of the Strategic Investment Fund.

The Ministry of Economic Development’s annual review

8.21
MED undertakes an annual review (or stocktake) that provides an assessment of the industry and regional development programme performance to date. It summarises available information on service implementation, delivery and performance, which can then be used to improve delivery or to direct further research or policy development work.

Evaluation and research plans

8.22
MED has prepared draft evaluation and research plans for the Enterprise Development Fund and the Growth Services Fund. These draft plans were prepared in May 2004. MED has yet to develop evaluation and research plans for the Strategic Investment Fund or the Major Events Fund, primarily because the policy foundations for these programmes were under review.

8.23
Included within the plans are sections on:

• programme objectives and descriptions of the service being delivered;
• intervention logic models for the programmes; and
• evaluation objectives and design.

Assessment of the evaluation of NZTE’s grant programmes

8.24
The different evaluation approaches applied by MED and MFAT to NZTE’s predecessor organisations, and the effects of the merger that took place to form NZTE, have affected the timeliness of evaluation activity. A number of evaluation framework elements were developed progressively over the course of programme implementation, such as:

• timelines;
• relationship agreements;
• organisational evaluation role descriptions; and
• evaluation priority setting.

8.25
The following should have occurred, but did not occur, at the start of the planning process:

• ensuring that monitoring data collected in the grant programmes was consistent with later evaluation efforts; and
• ensuring that high-level risks to the evaluation – particularly institutional change – were appropriately managed.

Evaluation planning for each grant programme

8.26
Various evaluation activities have taken place, including some formative evaluation²⁰ work, such as:

• before the merger, customer surveys by Industry New Zealand and various Trade New Zealand client research and performance measurement reports;
• considerable developmental evaluation work by NZTE to ensure that programmes were well formed;
• some collection of monitoring data;
• the MED annual review (or stocktake);
• in-depth reviews; and
• evaluation plans for the EDF and GSF programmes.

8.27
However, there was a lack of early evaluation and planning within MED. For example, at the time of our audit, evaluation and research plans had been developed for only 2 of the grant programmes.

Timing of evaluation planning

8.28
The timing of evaluation planning is important to ensure that data that can be collected only at a certain point in time is actually collected, so that it can be available for evaluation purposes.

8.29
Evaluation planning did not take place early enough. Although it may be possible to retrospectively collect some information using one-off surveys as part of later evaluation design, this can mean that the most effective and efficient opportunity for data capture is missed.

17: Export Development and Promotion: Lessons from Four Benchmark Countries, The Boston Consultancy Group. The countries were Chile, Denmark, Malaysia, and the United Kingdom.

18: We note that, for a period, the Treasury was responsible for evaluating Trade New Zealand.

19: MED undertook various pieces of evaluation work while responsible for Industry New Zealand, including a review of the Regional Partnership Programme and a review of the BIZ Investment Ready scheme.

20: Formative evaluation uses a disciplined approach to ensure that a programme is optimized for success. It includes checking that needs have been assessed, the literature reviewed, programme objectives and intervention logic fully developed, pre-testing and piloting carried out, the development of monitoring data collection systems, and appropriate plans for later evaluation put in place.