Summary of the impact

King's was vital to the innovative development of a research based model, termed `dialogic evaluation', which reshaped the practice of Performance Audits on Public-Private Partnerships (PPPs) undertaken by national audit offices in the UK, and particularly, in Australia. `Dialogic evaluation' introduced a theoretical model for evaluating the value-for-money of the post-decision operational stage of Public Private Partnerships (PPPs) in Performance Audits. Building on research and impact with the UK National Audit Office — the British agency — mandated with evaluating public spending — research on Australian PPPs in collaboration with partners at the University of Sydney and the Victorian Auditor-General's Office (VAGO) in Australia itself reshaped VAGO's practice. Whilst PPPs are used in many countries, the UK and Australia are world leaders in these arrangements. In the UK alone, by 2007, more than 870 projects worth £65.5bn had been procured as PPPs. Yet, the national audit offices in both these countries faced considerable challenges in their attempts to judge value-for-money of PPPs over their lengthy (25+ years) contract period. The theoretical evaluation model provided a way to resolve these problems and was adopted by VAGO, the national agency for the evaluation of public (government) spending in Australia.

Underpinning research

The `dialogic evaluation' model for judging value-for-money was developed in research by Professors Richard Laughlin (King's College London) and Jane Broadbent (Royal Holloway). The key characteristic of this evaluation model involves a process called `dialogic evaluation' whereby value-for-money is not seen as an objective, scientific pre-given but comes out of a critical, structured dialogue over time between key experts and stakeholders. (Refs 1 and 2). This model was extended in a research project from 1999 to 2002 with a specific emphasis on the evaluation of health PPPs, funded by the Chartered Institute of Management Accountants. (Refs. 3, 4, 5, 7 and 8). This model was further developed when Professors Laughlin and Broadbent in collaboration with Professor James Guthrie and Associate Professor Linda English (both of the University of Sydney) secured a major Australian Research Council Linkage Grant (2006-2008) to undertake a research project entitled `Developing a Model for the Evaluation of Australian Public Private Partnerships'. (Refs. 6 and 9). A major partner in this linkage grant was VAGO.

Performance Audits involve national audit offices making judgements on the value-for-money of public expenditure and have been particularly complex in the case of PPPs, which are also referred to as Private Finance Initiative (PFI) projects in the UK. One of the unique characteristics of PPPs involves extensive contract negotiations at the pre-decision stage between procurers and suppliers about service provision and risk identification. There is considerable government guidance on how to construct these long term contracts providing national audit offices with a framework to be able to undertake Performance Audits of the pre-decision and decision stages of any PPPs. Yet there has been virtually no guidance on how to undertake Performance Audits of the much longer post-decision operational stage of PPPs. This has caused considerable problems for national audit offices, particularly those dealing with many PPPs. The theoretical model provided a way of thinking as well as a design for Performance Audits to fill this gap conceptually and practically.

References to the research

Where a DOI or URL is not supplied, hard copies are available upon request.

Grants Received:

7. Professor Jane Broadbent (Royal Holloway (Grant Holder)) and Professor Richard Laughlin (King's College London), `An Exploration and Evaluation of the Private Finance Initiative with Particular Reference to the National Health Service (Part 1)',Chartered Institute of Management Accountants (CIMA), 1/4/1999 to 1/9/2000, £12,185 (direct costs)

8. Professor Jane Broadbent (Royal Holloway and then the University of Roehampton, London (Grant Holder)) and Professor Richard Laughlin (King's College London), `An Exploration and Evaluation of the Private Finance Initiative with Particular Reference to the National Health Service (Part 2)', Chartered Institute of Management Accountants (CIMA), 1/4/2001 to 31/12/2002, £40,339 (direct costs)

9. Professor James Guthrie (University of Sydney (Grant Holder)) Associate Professor Linda English (University of Sydney), Professor Jane Broadbent (Royal Holloway and then the University of Roehampton, London) and Professor Richard Laughlin (King's College London), `Developing a Model for the Evaluation of Australian Public Private Partnerships', Australian Research Council (ARC) Linkage Grant, 1/1/2006 to 31/12/2008, Aus$1,125,188 (full economic cost) (£774,877 at £1= A$1.45)

Details of the impact

The `dialogic evaluation' theoretical model had a direct impact on VAGO's performance audits. As noted in Box/Section 2, the direct research that underpinned this impact was developed and conducted in collaboration with VAGO and the University of Sydney. Laughlin and Broadbent's earlier research and impact on the National Audit Office (NAO) in the UK was influential, enabling the Auditor-General of Victoria (Des Pearson) and VAGO to work with the Australian Research Council (ARC) funded research team between 2006 and 2008. That earlier research, which introduced the theory of `dialogic evaluation', reshaped the thinking and practice of the UK's Auditor-General and the UK National Audit Office.

Laughlin and Broadbent's research indicates that the adoption of `dialogic evaluation' provides the framework for necessary changes involved in moving away from a more reactive `watchdog' emphasis in value-for-money audits and audit thinking. Evidence that this change in thinking was linked to the work of Laughlin and Broadbent is confirmed in Public Sector Auditing by Sir John Bourn, written as he was preparing to retire as Comptroller and Auditor General of the UK (Source B: 73):

`I use the terms effectiveness audit and evaluation interchangeably as in essence they are broadly the same, particularly with what has come to be known as `dialogic evaluation'. Indeed, Richard Laughlin, Jane Broadbent and Jas Gill in their research on evaluating Private Finance Initiative deals highlight the close similarities in approaches adopted by evaluators and value for money auditors.'

Performance Audits to judge value-for-money of PFI projects in the UK had been a constant problem for the NAO (Source C). It was recognised that judging the effectiveness of the operational stage of PFI projects was particularly problematic and that a new audit framework to fill this gap was needed. Laughlin and Broadbent were invited to present their CIMA-funded research, along with others to the NAO. The NAO embraced `dialogic evaluation' as a methodology and replaced the 1999 value-for-money audit practice guide with the NAO's highly influential 2006 Performance Audit guidelines (Source D). These guidelines, informed by `dialogic evaluation', provided the foundation for the research funded by the Australian Research Council and the work with VAGO, which led, as with the UK precedent, to the development of new thinking and practice founded on `dialogic evaluation.' The three-year long Australian Research Council project was presented and extensively discussed in December 2008 at a two day meeting with the Auditor-General and VAGO senior staff. The `dialogic evaluation' theoretical model and the research in the Australian context for the Performance Audit of the operational stage of PPPs were adopted by VAGO.

A letter dated 25 March 2009 from the Auditor-General to each of the researchers, confirmed that the research `will assist our future endeavours in this important area.' (Source F). On 28^th September 2010, the Auditor-General of VAGO commended the 2010 publication and confirmed impact and operationalisation of the model by linking the research and this publication to a recently completed Performance Audit of the operational stage of a PPP for the provision of prison accommodation services:

`On 15 September 2010, I tabled before Parliament a performance audit report on "Management of Prison Accommodation Services Using PPPs". This audit focussed on the operational stage of four prisons across Victoria and found that an ongoing challenge for the public sector is the delivery of anticipated value-for-money proposition over the life of these long-term contracts.

In contrast to the more positive findings in regard to the early stages of PPPs in Victoria, we observed a number of weaknesses in the monitoring and management of contractor performance during the operational stage of the reviewed PPPs. The Department of Justice acknowledged that as a result of this audit, it has initiated changes to its contract management and administration of prison PPPs. The Department of Treasury and Finance is re-examining the extent to which it provides guidance and support to agencies managing PPPs; to date its focus had been on the procurement phase of these contracts.' (Sources F and E).

This impact was clarified further and developed with indications of future influence in May 2013 by Jacquie Stepanoff, Manager — Policy and Co-ordination, VAGO:

`....your work went beyond the scope of that Prisons audit, and has stimulated us to think differently about what is and what is not feasible in these evaluations.

Paul O'Connor, Director, Policy-Co-ordination comments that whilst your research encouraged him to consider more carefully the concept of whole of life VFM during the planning for the Prisons audit, in many cases, the actual phase of the project defines just how far an audit can realistically go in forming that judgement, as it's impossible to conclude on VFM after only 5 years of a 30 year contract.

So, although its application was limited in the prisons audit, your research stimulated Paul to nominate for our forward program a review of a "mature operational" infrastructure PPP instead of the typical post-transaction review we had done in the other cases. This topic is still under consideration but looks likely to be added in later years. This is a direct reflection of the way your work stimulated VAGO's interest in whole of life cycle VFM.' (Source G).

The impact of `dialogic evaluation' on the thinking of all those involved in the operational stage of PPPs, not just in Victoria, is captured in a paper written by the Former Auditor-General of Australia (Source A). Commenting on Reference 6 he pointed out:

`The evaluation and audit of PPPs have received little attention in the literature in recent years, particularly in any analytical way...... The article is a very worthwhile contribution in drawing attention to the issues associated with assessing the longer-term efficiency and effectiveness of PPPs particularly in a changing environment. It should encourage more thoughtful analysis and development by all those involved, not least from audit offices. In particular, there is a need to focus on assessments of Value for Money (VFM) over long periods of time as results are actually achieved.' (Source A: 99).

In this regard Des Pearson, Auditor-General of VAGO, had indicated in his letter of 25 March 2009 that he was going to refer the research to the Auditors-General in other States in Australia: `Due to the recent endorsement of the National Public Private Partnership Policy and Guidelines by the Council of Australian Governments, I intend to refer the research outputs to the Australasian Council of Auditors General (ACAG) for further comment and action.' (Source F). Jacquie Stepanoff, Manager — Policy and Co-ordination, VAGO, in her email of 12 May 2013, made clear this did occur and implies that the research and VAGO's performance audit experience will be influential when evaluating PPPs over the operational stage of these long term contracts becomes more of a pressing problem in other States in Australia. (Source G).

Taken together the `dialogic evaluation' theoretical model has had comprehensive impact on and reshaped VAGO's audit/evaluation of the value-for-money of the operational stages of PPPs.

Sources to corroborate the impact

All documents are available on reqest.

A. Barrett, P. (2011) `Commentary: Performance Audit of PPPs — Getting the Basics Right', Australian Accounting Review, 21(1): 99-106

B. Bourn, J. (2007) Public Sector Auditing, London: Wiley

C. Examining the Value for Money of Deals under the Private Finance Initiative, HC739 Session 1998-99

D. National Audit Office (NAO) (2006 and 2006a) A Framework for Evaluating the Implementation of Private Finance Initiative Projects: Volumes 1 and 2, London: HMSO

E. VAGO (2010) Management of Prison Accommodation Services Using PPPs http://www.audit.vic.gov.au/reports_and_publications/reports_by_year/2010- 11/20101509_prisons.aspx

F. Letters from the Auditor-General, VAGO, Level 24, 35 Collins Street, Melbourne, Victoria, Australia 3000 dated 25 March 2009 and 28 September 2010 [confirms impact on VAGO's prisons PPP audit]

G. Email from the Manager — Policy and Co-ordination, VAGO, dated 12 May 2013 [confirms work on project `Developing a model for the Evaluation of Australian PPPs']