Evaluation Culture and Practice in the Public Service

Dr Jacqueline Cumming, Dr Janet McDonald, Dr Ausaga Fa‘asalele Tanuvasa, Dr Lynne Russell, Dr Clive Aspin (including contributions from the late Dr Jenny Neale, and from Dr Sue Buckley) report on a major new research project on evaluation in the public sector.

The Role of Evaluation in Policy-Making Cycles

Evaluation is recognised as a key component in policy-making cycles. It is particularly relevant in assessing the implementation and outcomes of chosen solutions to key policy problems and using these learnings to revise and improve policies and programmes. High-quality evaluations tell us whether services are being delivered effectively. Without them, we risk wasting millions of dollars on services that do not improve people’s lives.

We know that the use of evaluation varies within the Aotearoa New Zealand public service, but we know little about why this is so. In this article, we look at what we know about the recent history of evaluation in the public sector and set out the details of a research project we are leading that explores the topic of evaluation in more depth.

Evaluation in the Aotearoa New Zealand Public Sector

Over the past 20 years, we have seen the establishment and disestablishment of evaluation units in various departments and ministries (for example, the Social Policy Evaluation and Research Unit – Superu), as well as the pepper potting of evaluators throughout organisations. Various agencies have provided guidance about good evaluation practice, such as the Public Service Commission, Superu, the Department of the Prime Minister and Cabinet, Treasury, and the Prime Minister’s Chief Science Advisor. Recurrent issues include variable evaluation practice across government agencies, a focus on evaluating new services with limited evaluation of the performance of existing programmes, and questions of evaluation capability and capacity.

Māori evaluators have highlighted the importance of kaupapa Māori evaluation that seeks aspirational and transformative outcomes for Māori. Data sovereignty and the ownership of Māori evaluation data and the privileging of mātauranga Māori (Māori knowledge) and Māori worldviews are significant Te Tiriti o Waitangi considerations. Other voices also need to be heard in evaluation practice, including Pacific voices. Health Research Council guidelines on Pacific research are highly relevant to evaluation with Pacific peoples, emphasising that major social policies designed to improve Pacific peoples’ wellbeing need to come from Pacific perspectives.

The Aotearoa New Zealand Evaluation Association (ANZEA) and the Australian Evaluation Society (AES) both support the professional development of evaluators through, for example, competency standards, regular conferences to share experiences, and publication of the Evaluation Journal of Australasia.

There are, however, key gaps in our knowledge about evaluation practice in Aotearoa New Zealand, including how decisions are made about what is – and is not – evaluated (both in terms of the types of policies or programmes that are evaluated and the populations they focus on); the reasoning behind choices of evaluation approach; how evaluations are considering the increasing diversity of our population and how policies affect different groups in practice; and, most crucially, the ways evaluations influence public policy decision making.

Aotearoa New Zealand’s unique population make-up, in particular, the Māori population and the many Pacific migrants living here, alongside its unique institutional and policy settings, mean that local evaluations of key initiatives are essential. Many more scaled-up initiatives should develop from local experiences, but that is difficult where local evaluations have not been done.

Learning from Other Jurisdictions

Evaluation policy and practice has been a key focus in a number of other countries in the last few years. Recent Australian work noted strong support for the importance of evaluation throughout the policy process, alongside poor use of evaluation in practice. The authors concluded that the skills and capacity of public servants were not in doubt, but there was a lack of an institutional framework that embeds and values learning from evaluation. It was therefore recommended that there be stronger centralised prioritisation and oversight of evaluation. Similarly, a recent review of the Australian Public Service included a recommendation to embed evaluation within evidence-based policy and programmes.

A 2013 report by the UK National Audit Office focused on evaluation of impact and cost effectiveness across the 17 government departments, concluding there was variable coverage, quality, and use of evaluation while recommending greater transparency about what is evaluated, including publication of all reports and the actions planned in response. In parallel, in 2013, the UK government announced the establishment of a network of independent What Works centres (currently numbering 13 and covering a broad range of social policy areas) to “embed a culture of rigorous testing and evaluation in the design of policy and the delivery of services”. The UK Treasury produces extensive guidance and resources for evaluation, including its Green and Magenta Books.

In 2016, the Canadian government published its Policy on Results and associated Directive on Results, setting out requirements for performance information and evaluation within federal departments. A variety of resources for evaluation is available.

In the United States, the Foundations for Evidence-Based Policymaking Act of 2018 mandates executive government agencies to have a designated Evaluation Officer and produce an annual evaluation plan. The 2021 memorandum for agencies producing evaluation plans noted the importance of learning from evaluation (including “negative” results) and using evaluation to drive improvement. It also highlighted the importance of understanding how context influences whether and how well something works and noted, therefore, that effectiveness may vary in different communities.

All these other countries highlight the usefulness of clear, central direction for evaluation with the provision of appropriate supports.

Understanding the 'Black Box' of Evaluation Culture and Practice in Aotearoa New Zealand

Given the lack of research into Aotearoa New Zealand’s evaluation culture and practice, researchers from Te Hikuwai Rangahau Hauora Health Services Research Centre at Te Herenga Waka Victoria University of Wellington have set out to explore what is happening currently with respect to evaluation of key social policies.

The project is supported by the Marsden Fund and seeks to systematically describe what is happening with respect to evaluation culture and practice in the public service and to identify how evaluation culture and practice can be improved to better support policy making.

Ultimately, we are seeking to find what makes for a strong evaluation culture, with high quality evaluation practice and the systematic use of evaluation findings in policy work. Our focus is on health, education, social development, and housing, particularly where there have been many new initiatives and where the Aotearoa New Zealand environment is likely to be so different from that of other countries (for example, in population mix, beliefs and, behaviours), such that Aotearoa New Zealand-based evaluations are essential to understanding whether and how the policies and programmes work. A key focus of our research is on policies and programmes aimed at improving Māori wellbeing and Pacific wellbeing.

The team has started with interviews in central government agencies to get a broad understanding of current evaluation activity in social policy. Next, we will undertake some in-depth case studies with several social service agencies, focusing on the use of evaluation in policy making. We also want to talk with key evaluation practitioners about their experiences with evaluation.

Please get in touch if you would like to contribute or know more.