Summary of the impact

Research undertaken at Royal Holloway developed a theoretical Performance Audit (PA) model for evaluating the value for money (v-f-m) of the post-decision operational stage of Private Finance Initiatives (PFIs) and Public Private Partnerships (PPPs). Performance Audits (PAs) undertaken by national audit offices encountered difficulties in evaluating v-f-m, given their lifespan of 25 years or more. The model has impact in the UK and Australia, influencing national auditors in their development of PA of PPP/PFI, initially in the UK National Audit Office and directly during the REF 2014 period on the PAs in the Victorian Auditor General's Office (VAGO), Australia.

Underpinning research

The underpinning research for this impact is a body of research developed over 15 years by Professor Jane Broadbent (Royal Holloway 1997 to 2006, and 2012-current, with an intervening position as Deputy V-C at Roehampton) and her colleague Professor Richard Laughlin (Kings College, London). The theoretical model for Performance Audits (PAs) that provided the initial foundation for this work has its roots in two papers published in the mid-1990s, authored by Broadbent and Laughlin (publications 1 and 2), that explore `dialogic evaluation'. This sees evaluation of value for money (v-f-m) as emerging from a critical, structured dialogue between experts and key stakeholders. This theoretical model, for which impact is claimed, was then advanced and developed during Broadbent's time at Royal Holloway through two funded research projects for the Chartered Institute of Management Accountants (CIMA), which ran from 1999 to 2002 and centred upon health PPPs. These documented the implementation of PFI schemes in the first phase of their use in the NHS, and were based on extensive interviews to ascertain how contracts were being implemented as well as an analysis of the contractual documentation to see how they should have been implemented. They pointed to the need to evaluate such initiatives during their lifespan and not simply during their development phase. The requirement to consider the range of stakeholder needs and balance the demands of different stakeholders through a dialogic evaluation was also highlighted. The final stage of this research was conceived during 2004/5 when Broadbent was still at Royal Holloway. An application for funding to the Australian Research Council Linkage Programme was developed with a team from the University of Sydney, with the linkage partner from industry being VAGO. This programme of research used the advanced theoretical model developed in the CIMA research (publication 5), in a practical initiative operationalising the model in a `real-life' setting in the period 2006 to 2008.

PAs involve national audit offices evaluating the v-f-m of programs, functions and operations in order to report to parliament. V-f-m is fraught with conceptual difficulties and awkward dilemmas because it assesses how well the PPP is fulfilling not just financial audit considerations but societal responsibilities too. National audit offices are well placed to undertake PAs relating to the pre-decision and decision stages for PPPs because they have a wealth of documentary guidance from Government for those stages. But, surprisingly, very little support and guidance was provided by government on how to undertake PAs of the much longer post-decision operational stage of PPPs. Consequently, this has provided difficulties for national audit offices.

The research carried out by Broadbent at RHUL (papers 3 and 4) provided the basis for the funding application to ARC and the partnership with VAGO allowed her to develop these insights in a practical situation, considering the use of the approach in a variety of PPPs in Victoria, Australia. This project demonstrated the centrality of using national audit offices to undertake such evaluations on a systematic basis.

References to the research

Research Grants:

• Jane Broadbent and Professor Richard Laughlin, `An Exploration and Evaluation of the Private Finance Initiative with Particular Reference to the National Health Service (Part 1)' Chartered Institute of Management Accountants (CIMA) 1/4/1999 to 1/9/2000; £12,185.

• Jane Broadbent and Professor Richard Laughlin `An Exploration and Evaluation of the Private Finance Initiative with Particular Reference to the National Health Service (Part 2)', Chartered Institute of Management Accountants (CIMA) 1/4/2001 to 31/12/2002, £40,339.

• James Guthrie (University of Sydney (Grant Holder)), Linda English (University of Sydney), Jane Broadbent and Professor Richard Laughlin `Developing a Model for the Evaluation of Australian Public Private Partnerships' Australian Research Council (ARC) Linkage Grant 1/1/2006 to 31/12/2008 Aus $1,125,188.

Details of the impact

Broadbent and Laughlin's theoretical model for PAs has influenced auditing practice in the UK, and more recently has been incorporated into models of practice in Australia.

The model derived from the CIMA-funded research project and first incorporated into the UK National Audit Office's Framework for Evaluation in 2006. The influence was evidenced by a quotation from the former Comptroller and Auditor General Sir John Bourne in his book Public Sector Auditing (source 2, p. 73):

`I use the terms effectiveness audit and evaluation interchangeably as in essence they are broadly the same. Particularly with what has come to be known as `dialogic evaluation'. Indeed Richard Laughlin, Jane Broadbent and Jas Gill in their research on evaluating PFI deals highlight the close similarities in approaches adopted by evaluators and V-F-M auditors.'

Two key publications by the NAO (source 3) were informed by `dialogic evaluation'. NAO's development of these ideas began in 2003 when they called together a group of experts, including Broadbent and Laughlin to advise them in their development of a model to deal with the complexity of evaluating PFI/PPPs.

This same model formed the foundations of the PA evaluation model adopted by the Victorian Auditor General's Office (VAGO) in Australia in 2009.

The findings of the ARC-funded project were fed into VAGO following extensive dialogue and meetings, especially with the Auditor-General and VAGO senior staff. A key part of this interaction with these end users concerned the theoretical model for the PA of the operational stage of PPPs. In a letter dated 25 March 2009 (source 4) the Auditor-General wrote to the individual members of the research team:

"Your research efforts, the many stimulating conversations that they generated by the team's work, and the final paper that you provided in December 2008, are appreciated and will assist our future endeavours in this important area."

The Auditor-General was sufficiently impressed with the insights that he was keen to implement these in VAGO and that the best practice should be disseminated and adopted by other Auditor- Generals in other Australian States. This perspective is also apparent in the supporting letter, dated 25 March 2009 (source 4):

"Due to the recent endorsement of the National Public Private Partnership Policy and Guidelines by the Council of Australian Governments, I intend to refer the research outputs to the Australasian Council of Auditors General for further comment and action."

The wider resonation of the ideas and their importance with the Auditor-General of VAGO is illustrated further in a later letter to Associate Professor Linda English on 28 September 2010 (source 5) where he explains how a performance audit report (source 6) tabled before Parliament on 15 September 2010, which focussed on the operational stage of four prisons across Victoria, found that an on-going challenge for the public sector is the delivery of anticipated v-f-m proposition over the life of these long-term contracts. The PA helped to identify weaknesses in the operational stages of prison PPPs and subsequently led to:

"The Department of Justice acknowledged that as a result of this audit, it has initiated changes to its contract management and administration of prison PPPs. The Department of Treasury and Finance (DTF) is re-examining the extent to which it provides guidance and support to agencies managing PPPs..."

The further dissemination of the ideas, and greater impact of the ideas upon a wider set of end users beyond Victoria, specifically with regard to the operational stage of PPPs, is clearly stated in a paper written by the Former Auditor-General of Australia (source 7, commenting on source 8), where he pointed out:

`The evaluation and audit of PPPs have received little attention in the literature in recent years, particularly in any analytical way, in my opinion. The article is a very worthwhile contribution in drawing attention to the issues associated with assessing the longer term efficiency and effectiveness of PPPs particularly in a changing environment. It should encourage more thoughtful analysis and development by all those involved, not least from audit offices. In particular, there is a need to focus on assessments of Value for Money over long periods of time as results are actually achieved.'

Sources to corroborate the impact

1. Broadbent, J., Gill, J. and Laughlin, R. (2004) The Private Finance Initiative in the National Health Service: Nature, Emergence and the Role of Management Accounting in Decision Making and Post-Project Evaluation, London: Chartered Institute of Management Accountants (CIMA). This CIMA-funded and published work developed the theoretical model for Performance Audits subsequently adopted by the NAO. Online Access:
   http://www.cimaglobal.com/Documents/ImportedDocuments/tech_resrep_the_private_finance_initiative_in_the_national_health_service_2004.pdf.
2. Bourn, J. (2007) Public Sector Auditing, London: Wiley. Includes a quotation (p. 73) from the former Comptroller and Auditor General demonstrating the influence of the CIMA research and resulting theoretical model on the NAO's `Framework for Evaluation'.
3. National Audit Office (NAO) (2006) A Framework for Evaluating the Implementation of Private Finance Initiative Projects: Volumes 1 & 2, London: HMSO. These publications incorporate the `dialogic evaluation' methodology developed by Broadbent & Laughlin who also acted as advisors to the NAO on the development of a suitable model to evaluate PFI/PPPs.
4. Letter from Auditor-General, Victorian Auditor-General's Office, The letter expresses appreciation for the research team's efforts, cites their assistance to future VAGO endeavours in the field of PPP evaluation and referral of the research outputs to Australasian Council of Auditors General.
5. Letter from Auditor-General, Victorian Auditor-General's Office, dated 28 September 2010. The letter confirms that a performance audit report highlighted weaknesses in the operational stages of prison PPPs. As a consequence, the Department of Justice initiated changes to its contract management and administration of prison PPPs.
6. VAGO (2010) Management of Prison Accommodation Services Using PPPs — An example of a performance audit undertaken by VAGO informed by the Broadbent/Laughlin PPP evaluation model. http://www.audit.vic.gov.au/reports_and_publications/reports_by_year/2010-11/20101509_prisons.aspx
7. Barrett, P. (2011) `Commentary: Performance Audit of PPPs — Getting the Basics Right', Australian Accounting Review, 21(1): 99-106. Corroborates the further dissemination and impact of the ideas highlighted in source 8 upon a wider set of end users beyond Victoria, specifically with regard to the operational stage of PPPs. The author is the Former Auditor-General of Australia.
8. Broadbent, J., English, L., Guthrie, J. & Laughlin, R. (2010) Performance Audit of the Operational Stage of Long Term Partnerships for the Private Sector Provision of Public Services, Australian Accounting Review, Vol 1, p. 64-75. Reflects on the authors' familiarity with the audit of PPPS in Australia and the UK to identify the building blocks required in the design of future systems by audit offices to assess the VFM of the mature operational stage of PPPSs.