Part 3: Service performance reporting
3.1
In this Part, we discuss DHBs' service performance reporting, including our audit opinions in 2012/13 and the sector-wide issue about the reporting of third-party performance information, which affected the audit opinions of all DHBs and the Ministry.
About service performance reporting
3.2
Service performance reporting is an important part of Parliament's accountability system and helps demonstrate effectiveness, efficiency, and value for money in the public sector. It also enables organisations to manage performance more effectively, and supports good decision-making.
3.3
There are long-standing legislative requirements for DHBs and many other entities in the public sector to report on their service performance, and for the Auditor-General to audit that reported information.
3.4
Since 2006, we have stressed the importance of high-quality performance information to aid people's understanding of the public sector's effectiveness. We have reported extensively to public entities and Parliament on the need to improve service performance reporting, and the quality of reporting has improved significantly in recent years.13
3.5
We have also been improving our approach to auditing service performance information, which includes progressively implementing a revised auditing standard on auditing performance information. The standard requires auditors to report on whether an entity's reported performance complies with generally accepted accounting practice and fairly reflects actual performance for the year. To do this, auditors verify material aspects of the reported service performance and make sure that the reporting appropriately covers the entity's activities, and the effect of those activities.
3.6
We have been working with DHBs to help improve performance reporting since 2008/09. Our early focus was to help improve the quality of statements of intent, given that performance is reported against information in those statements. More recently, our focus has been on DHBs continuing to improve the appropriateness, clarity, and reliability of their reported information.
3.7
In the 2012/13 audits, we applied the revised auditing standard to DHBs' service performance reports for the first time. DHBs were among the last Crown entities to have this auditing standard applied to the audit of their service performance reports. There are a few remaining public entities, such as subsidiaries of DHBs, that are still to have this standard applied to their audits.
The 2012/13 audits
3.8
Audit work is planned and performed to obtain evidence on a sample basis, because public entities carry out too many transactions for an auditor to look at them all. Auditors use their professional judgement to assess the evidence and ensure that they have reasonable assurance that there are not material misstatements in the financial statements and service performance information.
3.9
Auditors assess whether an entity has systems and controls that the auditor can rely on. This assessment includes testing these systems and controls to confirm their reliability. If there are not reliable systems and controls, then auditors must perform substantive testing of data to confirm its reliability.
3.10
The 2012/13 audits of DHBs' performance information included evaluating the national health targets.14 These targets report performance from DHBs' hospital services and services that DHBs purchase from third parties, such as PHOs and other non-governmental organisations.
3.11
DHBs generally have reliable systems and controls over the performance information that they report for their hospital services, with two exceptions, which we discuss in paragraphs 3.25-3.29. However, DHBs generally have limited controls over much of the performance information that they report from third-party health providers.
Third-party performance information
3.12
DHBs could not demonstrate how they know that they can rely on third-party performance information. DHBs did monitor how performance was tracking against targets, but could not provide us with evidence that they were checking that the information reported to them by third parties was reliable.
3.13
It appeared that DHBs and other entities, such as the Ministry, either assumed someone else was checking the data or they largely operated on trust.
3.14
There were no practical audit procedures we could use to determine the effect of this limited control. For example, the primary care measure that includes providing smokers with advice and support to quit relies on information from general practitioners that we are unable to independently test. It is not our role to audit the third-party health providers. Our role is to check that the reported performance information fairly reflects the actual performance of each DHB.
3.15
As a result, we were not able to obtain the evidence needed to express an unmodified opinion on all of a DHB's performance information. For the third-party performance information, the scope of the audit was limited.
3.16
All DHBs' audit opinions for 2012/13 were qualified because of this problem with third-party performance information.
3.17
Our qualified opinion does not mean that the health target performance reporting by DHBs was wrong, or that there was a failure of DHBs' service delivery, or any wrongdoing or false reporting by DHBs, or that the information reported from general practitioners was wrong. It simply means that we were unable to verify some important performance information.
A sector-wide issue
3.18
The qualification on third-party performance information is a sector-wide issue that has affected the audits of all DHBs and the Ministry.
3.19
The Ministry reports national health target information as its impact measures. This information is reported to the Ministry by DHBs. We qualified the Ministry's audit opinion because we were not able to gain sufficient assurance about the health targets at the time that we completed our audit of the Ministry, which has a statutory reporting date a month earlier than the date for DHBs.
3.20
A significant amount of health funding is spent on third-party services, about $5.7 billion, and it is important that there is reliable performance information about these services to support decision-making and public accountability.
3.21
The audits have shed light on the limited control of DHBs over much of the performance information provided by third-party health providers. The limited control is an issue that we expect DHBs to address.
3.22
We recommended that the Ministry and DHBs work together, and with other relevant organisations, to identify a cost-effective approach for the sector to get sufficient assurance over the performance information from third parties. Because this is a sector-wide issue, we are encouraging the sector as a whole to consider whether the introduction of additional controls is appropriate and cost-effective and, if so, how best to introduce them.
3.23
We are aware that an Integrated Performance and Incentive Framework is being developed for primary care, and that there is already considerable audit activity in the health sector, both of which may provide opportunities to help address this issue.
3.24
By early 2014, limited progress had been made. Short-term approaches are being considered alongside longer-term ones. Our 2013/14 audits might also be affected by the limited control over third-party performance information. We will continue to work with the sector to help address this issue.
Other qualifications
3.25
As well as the qualification on third-party performance information, we further qualified our audit opinions on Lakes and Wairarapa DHBs. These matters were also related to performance information reporting.
3.26
In our audit of Lakes DHB, we found inconsistencies (between paper and electronic records) in the recorded times for the discharge of patients in the DHB's reported performance for the "shorter stays in emergency departments" measure. We were unable to quantify the extent of any misstatement in the reported performance so we could not rely on the reported performance.
3.27
As a result, we qualified our opinion and recommended that Lakes DHB improve its system for recording discharge times for patients in the emergency department.
3.28
In our audit of Wairarapa DHB, we found that the reported performance was materially overstated for the hospital performance measure for "smokers seen in hospital are offered advice and support to quit". Our testing identified an error in the data extracted to calculate the reported results. We were unable to quantify the extent of the overstatement so we could not rely on the reported performance. We therefore qualified our audit opinion.
3.29
We recommended that the DHB implement a robust system to ensure that appropriate evidence is retained to support the results of reported performance and that there is a system to check and monitor the accuracy and completeness of results.
Assessing service performance information and associated systems and controls
3.30
In Part 2, we discussed our assessments and grades of management control environments and financial information systems and controls. In this section, we discuss our grading of service performance information and associated systems and controls for DHBs. Appendix 2 sets out the grades for each DHB for 2012/13 and the prior year.
3.31
DHBs have made steady progress (see Figure 7) with improving their performance reporting in recent years, including:
- preparing appropriate performance frameworks to show what they do (services or outputs) and the difference they make (impacts and health outcomes);
- reporting performance against performance measures and targets to show what was achieved and how well they performed; and
- reporting on the health status of their population to show progress in achieving improved health outcomes for their districts, including for population groups such as Māori, who have disparities in health outcomes (see Part 5).
3.32
Canterbury DHB provides a good example of a full and informative performance story in its 2012/13 annual report.
3.33
Our assessment and grading of service performance information in 2012/13 excluded the sector-wide issue with third-party performance information.
3.34
The grades reflect the auditor's recommendations for improvement based on what is considered best practice for the entity and how far short an entity is from an appropriate standard of best practice. Because this was a sector-wide issue affecting all DHBs, there is no standard of best practice for DHBs to be assessed against yet.
3.35
Figure 7 shows the grades that our auditors gave DHBs for each of the past four years.
Figure 7
Grades for district health boards' service performance information and associated systems and controls, 2009/10 to 2012/13
3.36
Three DHBs (Auckland, Bay of Plenty, and Whanganui) improved their grades from "needs improvement" in 2011/12 to "good" in 2012/13.
3.37
Lakes DHB's grade changed from "good" to "needs improvement" because it needs to improve its system for recording the discharge times for emergency department patients.
3.38
We will continue to exclude the sector-wide issue with third-party performance information from our assessment and grading of service performance information in our 2013/14 audits, to allow time for the sector to address this issue. We will review our approach for the 2014/15 audits.
3.39
We will continue to work with DHBs and the Ministry to help DHBs to continue to improve their performance reporting, including their reporting of third-party performance information.
3.40
We will also consider the effect of changes that may result from the amended Crown Entities Act (see Part 1) on performance information reporting.
13: Our publications about performance reporting are available at oag.govt.nz/reports/performance-reporting.
14: See "How is my DHB performing?" at www.health.govt.nz.