Part 5: Evaluating, monitoring, and reporting on settlement outcomes
5.1
In this Part, we look at the responsibilities that MBIE has for evaluating and reporting on settlement outcomes based on expectations laid out in:
- Cabinet papers for the Settlement Strategy;
- approvals for Migrant Levy funding for the Migration Research, Evaluation and Analysis Team; and
- Immigration New Zealand's business plans and other corporate documents.
Summary of our findings
5.2
MBIE has made progress towards preparing and putting into effect various parts of a settlement evaluation and monitoring framework. The information it gathers about settlement indicators and outcomes has improved and will continue to improve as work progresses on the Integrated Data Infrastructure project with Statistics New Zealand.
5.3
We found that Immigration New Zealand has not yet put in place the integrated settlement evaluation and monitoring framework for the whole-of-government Settlement Strategy, including links to the national and regional action plans as set out in the approved 2006 Cabinet proposal for the Settlement Strategy. Because of this, reporting on the Settlement Strategy and action plans has not been able to show what "measurable and tangible settlement outcomes" the Settlement Strategy and associated action plans have achieved. The Migration Research, Evaluation and Analysis Team has not carried out the evaluation of the Migrant Levy-funded initiatives throughout government that was needed.
5.4
Since 2008, Immigration New Zealand has progressed work on an evaluation and monitoring framework for the settlement services that it purchases and provides to new migrants, which was ready to be put into effect in 2012/13. Although it took a long time to prepare and finalise the framework, this work should improve the information that the Settlement Unit can report on for settlement services and programmes and possible links to settlement outcomes for new migrants.
5.5
The Settlement Unit has contracted several reviews of the settlement services and programmes that it is responsible for as it tries to improve effectiveness. These reviews have resulted in identifying where improvements are needed, and changes have been put into effect as a result of the Settlement Unit's approach to continual improvement.
5.6
The Settlement Unit is responsible for leading and co-ordinating the Settlement Strategy, the national and regional action plans, and the National Settlement Structure. It does not report on the results of its work against annual business plans. This makes it difficult to identify what has been achieved towards the Settlement Strategy objectives.
5.7
Improvements are needed for some of the settlement indicators that MBIE reports.
Recommendation 4
We recommend that Immigration New Zealand implement evaluation and monitoring frameworks and outcome reporting for whole-of-government settlement activities, including the services purchased by Immigration New Zealand, to ensure effective and efficient use of resources that meet the needs of new migrants. Target time frame: by June 2015.
Recommendation 5
We recommend that the Ministry of Business, Innovation and Employment improve the quality of information it reports to the Government and more widely by including appropriate context information and data about principal and secondary skilled migrants and improving methods for reporting on retention. Target time frame: by June 2014.
What we expected from integrated settlement evaluation, monitoring, and reporting
5.8
We expected that Immigration New Zealand would have met intended objectives set out for the Settlement Strategy related to evaluation and reporting on settlement outcomes. This included putting in place an integrated settlement evaluation and monitoring framework and reporting on work towards the Settlement Strategy objectives and goals. Specifically, we expected that Immigration New Zealand's evaluation and reporting would incorporate measures and results from four parts of the Settlement Strategy:
- national and regional action plans;
- government programmes funded by the Migrant Levy;
- settlement support services and programmes Immigration New Zealand purchases and provides to new migrants; and
- the Settlement Unit's progress towards its objective to "support migrants to settle and work" and the objectives of the Settlement Strategy.
5.9
As a part of the Settlement Strategy, Immigration New Zealand was responsible for setting in place evaluation frameworks and reporting processes for the national and regional settlement working groups and the action plans.
5.10
Other agencies that receive funding for providing settlement support services have been responsible for evaluating the services that they deliver. The Settlement Unit's governance role included collating and reporting on those programme evaluations. The Department indicated it would:
…coordinate the development of an over-arching evaluation framework focussed on settlement outcomes for migrant and refugees….and include the identification of medium and high-level indicators that may be used to understand settlement outcomes and the contribution of the Settlement Strategy in supporting good settlement outcomes. The development of these will incorporate the identification of data needs and an assessment of the adequacy of current monitoring information.
5.11
In 2007/08, the Settlement Unit's Business Plan indicated that:
…as the agency with the lead in coordinating the implementation and development of the New Zealand Settlement Strategy, DoL will ensure that proposed interventions to support good settlement outcomes are clear about their objectives and about how they will be measured and evaluated. The Strategy's evaluation framework needs to ensure that any evaluation of specific interventions is linked to an overall assessment framework of their impact on the Strategy's goals.5
5.12
The Settlement Unit's Business Plan also specified that:
… this comprehensive evaluation framework still needs to be implemented. We intend this to be addressed as part of the Settlement National Action Plan with funding from the Migrant Levy.6
5.13
We expected that these evaluation frameworks would have been completed and implemented to ensure that the intended objectives of better use of resources throughout government for settlement support and a clear and measureable assessment of settlement outcomes showing the contribution of interventions under the Settlement Strategy were achieved.
5.14
As noted in Part 2, the Settlement Unit is responsible for:
- leading the Settlement Strategy;
- co-ordinating how the Settlement Strategy is put into effect;
- the Settlement Strategy's governance structure; and
- purchasing the delivery of some settlement support services for new migrants.
5.15
The Migration Research, Evaluation and Analysis Team is responsible for research and evaluation to help understand how the immigration system works in terms of settlement outcomes for migrants, their families, their employers, and society and the economy in general.7
Slow but now improving evaluation and reporting of settlement outcomes need to be integrated
5.16
Immigration New Zealand has made progress setting up various parts of a settlement evaluation framework to show indications of successful settlement. Some settlement outcome measures have been put in place. Improvements in data gathering to indicate progress towards improved settlement for new migrants has been made as a result of work led by the Migration Research, Evaluation and Analysis Team.
5.17
Several reports, programmes, and surveys are used to monitor migration trends and report against the main measures and outcomes in the corporate statement of intent. The Migration Research, Evaluation and Analysis Team has produced migration trend monitoring reports since 2000/01. These reports summarise indicators of annual migration trends together and provide some in-depth research.8 These reports enable Immigration New Zealand to monitor trends and patterns of migration and to highlight trends, including global migration trends, and what implications there might be for New Zealand.
5.18
The former Department of Labour developed the Longitudinal Information Survey New Zealand of migrants who were approved for residence between 2004 and 2005. This was self-reported information based on interviews that took place six months, 18 months, and 36 months after migrants had gained permanent residence in New Zealand. This provided Immigration New Zealand with a rich source of information on the employment outcomes and labour market integration of permanent residents in New Zealand. This data is becoming dated.
5.19
The Immigration Survey Monitoring Programme, which was piloted in 2009 and has been running since, is designed to gather information every year from service users (migrants, employers, and community stakeholders) providing information about new migrants' settlement experiences, views from employers and the community about matters related to new migrants' settlement and work in New Zealand, and satisfaction with the services that Immigration New Zealand provides.9 Results from the programme are used to track the main measures that are part of MBIE's reporting, including in the statement of intent and annual reports. These measures are designed to provide service improvement information and be able to benchmark improvements over time.
5.20
Client satisfaction with settlement services has been measured every year since 2011 through a survey carried out by an external provider. These client satisfaction surveys, based on the Common Measurement Tool, are administered to newcomers who access settlement services.
5.21
The Migration Research, Evaluation and Analysis Team also regularly collates benefit data as a way of identifying employment patterns of Immigration New Zealand's clients.
5.22
MBIE has plans to improve migrant performance outcome measures through the Integrated Data Infrastructure project that Statistics New Zealand is leading. This project will integrate data gathered from Immigration New Zealand through various visa streams, the Inland Revenue Department (on employment), and the Ministry of Social Development (on the use of benefits). The project is a notable achievement because it should provide better information about settlement outcomes data and better reporting in the future. Although it is too early to tell, this data is expected to improve MBIE's performance monitoring and reporting in the future by providing evidence-based outcome data and more opportunity for trend analysis because it includes data about various migrant visa categories, use of benefits, and income since 1998.
5.23
An integrated settlement evaluation and monitoring framework is needed to show what has been achieved through Immigration New Zealand's co-ordination of whole-of-government settlement work and the settlement services it delivers. As a result of insufficient reporting for the Settlement Strategy, the national and regional action plans, and the Settlement Unit, it is difficult to identify links between these and achieving settlement outcomes.
5.24
The former Department of Labour was to co-ordinate the overall development of evaluation work on the national strategy, focused on settlement outcomes for migrants and refugees.
5.25
Until 2009, Immigration New Zealand reported progress on the Settlement Strategy and the Action Plan. Reporting was of milestones achieved against the Action Plan, which is different than the type of reporting that an evaluation and monitoring framework would have provided by linking work to progress achieving measurable settlement outcomes.
5.26
Immigration New Zealand staff identified the lack of an evaluation framework for the Auckland Strategy in 2008 as a priority needing to be addressed to make clear what was being done towards achieving settlement outcomes. This work has not been progressed since then because the focus has been on revising the Auckland Strategy and its structure.
5.27
The 2010 review of the Wellington Strategy showed that focusing on reporting on milestones was at the expense of considering settlement outcomes. We are aware that Immigration New Zealand was having another review of the Wellington Regional Settlement Strategy carried out during our audit. That new review might help identify ways to improve evaluation and reporting practices.
5.28
The approved Cabinet paper for the Settlement Strategy in 2006 specified that evaluations of the individual programmes and initiatives within the settlement strategy would be prepared by the lead organisations. The Settlement Unit was tasked with collating and reporting information from evaluations of Budget 2004 initiatives.10 The Settlement Unit did not complete this and indicated to us that it did not think it was responsible for reporting on other settlement programme evaluations. Since the Settlement Strategy and the Settlement Unit have been in place, there has been no whole-of-government analysis of how effective settlement services are or any rationalisation of settlement support funding, which was a part of the purpose of having a Settlement Secretariat and, in our view, was needed.
5.29
Our audit also looked for reporting that might link the work of the Settlement Unit and the activities outlined in its business plans to progress towards achieving settlement outcomes. In our view, the Settlement Unit needs to develop and implement better reporting to be able to show the results of work towards the Settlement Strategy objectives and goals.
Better reporting on Migrant Levy funded cross-government settlement support is needed
Immigration New Zealand needs to improve some reporting methods to evaluate results
5.30
In 2007/08, the Migration Research, Evaluation and Analysis Team received a further $500,000 to:
… evaluate Migrant Levy funded initiatives … The results will be used to inform future funding of settlement services through the Migrant Levy, and the investment in settlement services more generally.11
5.31
The Migration Research, Evaluation and Analysis Team used this funding to prepare an evaluation and monitoring framework for the settlement services that it purchases and oversees because this was a priority in their view. The Migration Research, Evaluation and Analysis Team did not evaluate programmes funded by the Migrant Levy throughout government as it was intended it do to inform future decisions about how best to use the Migrant Levy funding.
5.32
The reporting requirements for Migrant Levy funding do not assess what outcomes have been achieved. The application process for Migrant Levy funding is detailed and requires the applicant to estimate what will be achieved with the requested funding. However, the reporting requires information only on whether funding was spent as intended. It does not include any outcome reporting requirements, which is needed to assess effectiveness.
5.33
Immigration New Zealand needs better information on settlement support funding throughout government. In March 2012, Immigration New Zealand submitted a proposal to the Cabinet Economic Growth and Infrastructure Committee for allocations of Migrant Levy funding. Immigration New Zealand's proposal showed overlaps between agencies:
… agencies could take a more collaborative approach to produce savings and efficiencies … better use could be made of each others' avenues for providing information to migrants [and] a whole-of-Government overview of funding for migrant settlement and research was needed to inform future migrant levy allocations.
5.34
As discussed in Part 3, when this audit was under way, the Settlement Unit started to meet with the staff from other public entities working on new migrant settlement to discuss funding.
Evaluation and monitoring of Immigration New Zealand's settlement services is being improved
5.35
Various parts of an evaluation and monitoring framework for the Settlement Unit's settlement services were put into effect between 2008 and 2011. The last part of the framework, an outcomes framework, was completed in 2013. The evaluation and monitoring framework should improve the Settlement Unit's ability to report on the results of settlement services and programmes and identify possible links to settlement outcomes in the future.
5.36
Between 2008 and 2011, the Migration Research, Evaluation and Analysis Team developed an evaluation and monitoring framework for the settlement services that MBIE purchases. The framework included:
- an intervention logic for the settlement services purchased by MBIE;
- preparing a good practice matrix to help service providers to assess how well they work compared with international good practice;
- processes for providers to report feedback;
- carrying out client satisfaction surveys; and
- local network surveys for Settlement Support New Zealand.
5.37
A drawback of the good practice matrix is that it has no option for less-than-acceptable practice, leading to a positive bias in service providers' self-evaluation. Results from these surveys and service providers' self-assessments are reported every year.
5.38
In 2011, the Migration Research, Evaluation and Analysis Team handed the evaluation and monitoring framework over to the Settlement Unit. At that time, the outcomes framework, which was a main part of the evaluation and monitoring framework, had not been completed. However, the outcomes framework was completed in 2013 by an external contractor and should enable service providers to monitor and report outcomes as programmes are delivered.
5.39
The Settlement Unit has focused on continuous improvement of its settlement services by ordering several programme reviews of the settlement services that it purchases and provides to new migrants in an effort to provide those services more effectively.
5.40
Immigration New Zealand has reviewed its settlement services and programmes over the years in an effort to improve their effectiveness. For example, reviews have been carried out on Settlement Support New Zealand in 2007, a review of the services the Settlement Unit purchases in 2010, and in 2011 a review of the delivery model for Settlement Support New Zealand was carried out. The Settlement Unit has used these reviews to make its settlement services more effective and efficient.
Reporting of migrant settlement outcomes needs to improve
5.41
High-level performance outcome reporting needs to be improved. Immigration New Zealand needs to take care when reporting performance data to ensure that it is accurate.
5.42
Our audit looked into the performance outcomes that Immigration New Zealand reports to the public. High-level performance outcomes that Immigration New Zealand reports on include:
- economic contributions that migrants provide compared to New Zealand-born citizens;
- retention rates of migrants; and
- the percentage of new migrants working at a level they are qualified to work.
5.43
In our view, Immigration New Zealand needs to improve how it reports data about performance so that the public receives a more accurate interpretation of how the data relates to Immigration New Zealand's work on new migrant settlement.
5.44
Immigration New Zealand needs to provide more context and explanation when reporting on the economic contributions of migrants compared to New Zealand-born citizens. Further information about the differences between migrant and New Zealand populations is needed to avoid over-emphasising the economic benefits of migrants compared to New Zealand-born citizens.
5.45
Immigration New Zealand needs to address methodological issues with the way retention data is gathered and reported on. Using cumulative retention rates over extended periods of time (1998-2011) portrays an over-optimistic picture of retention. Tracking retention rates within the initial settlement period of between two to five years would provide a more valid data set to assess the effect of recent efforts to retain migrants. MBIE has found that there is a lack of international data and there are no precise comparable figures to benchmark retention rates against.
5.46
Immigration New Zealand reports the percentage of principal skilled migrants whose work matches their skills and qualifications but does not report on this target for secondary skilled migrants. Settlement work relates to both principal skilled migrants and their partners. Therefore, in our view, Immigration New Zealand needs to report results for both principal and secondary skilled migrants to provide a more balanced picture of the effect of new migrant settlement support.
5.47
Part 6 discusses some of the barriers that new migrants face in settling and finding work and how Immigration New Zealand's work has helped to overcome those barriers.
5: Settlement Unit 2007/08 Business Plan, page 17, paragraph 4.13.
6: Settlement Unit 2007/08 Business Plan, page 17, paragraph 4.13.
7: Ministry of Business, Innovation and Employment briefing (21 February 2013), Immigration policy, service support, research and evaluation work programme, page 3.
8: The reports are available at the former Department of Labour's website, www.dol.govt.nz.
9: The report is available at the former Department of Labour's website, www.dol.govt.nz.
10: Settlement Division Strategic Plan 2006/07 – 2008/09, pages 10-11.
11: Cabinet paper.
page top