Summary of the impact

The European Social Survey (ESS) is a major biennial cross-national social survey in which 36 countries in and beyond the European Union have participated. It monitors change and stability in the social, moral and political fabric of Europe and uses and sets high standards of cross-national measurement. Its impact is extensive, both nationally and internationally, and occurs in two main domains:

Improving survey methodology: The ESS sets new and improved methodological and coordination standards in cross-national survey research, providing higher quality data outcomes for ESS and, through the subsequent adoption of these standards and practices by other national and international survey programmes, higher standards of measurement in policy-oriented surveys and commercial survey practice. As a result the ESS methodology has pioneered better quality statistics which contribute to improved quality and reliability for outcomes that rely on the data across Europe.

Influencing the policy making process and society: ESS's high-quality biennial social survey datasets and associated dissemination activities have facilitated immediate and easy use of ESS data and findings by a wide variety of stakeholders. Academics worldwide have analysed ESS data and research results. Alongside the direct use of ESS data by policy-makers, this has influenced policy-making.

Underpinning research

The ESS has been based at City University London since 2003 and was directed by Sir Roger Jowell until his death in 2011. Other key researchers in the City team include Rory Fitzgerald (at City since 2004 and ESS Director since 2012) and Eric Harrison (at City since 2006). The ESS Director provides academic leadership across the ESS network, which includes a Core Scientific Team (CST) (six leading European social science institutions), a Scientific Advisory Board, a Methods Board, Question Module Design Teams (leading European social scientists who win a competition to design a module in collaboration with the CST) and National Coordinators (who realise the ESS in their countries according to the Director's specification).

The underpinning research comprises the rounds of the ESS and the substantive analysis and methodological innovation arising from them. While ESS outputs are necessarily collaborative, the ESS Director and team are the driving force of the enterprise, ensuring extensive pan-European collaboration, ensuring high methodological standards and overseeing the research agenda. The Director and team have particular responsibility for co-designing and pre-testing the question modules. The underpinning research is structured as follows.

Survey methodology: This includes the cumulative effects of the ESS methodology, which set new standards in cross-national research¹, in areas such as sampling, questionnaire design, response measurement, fieldwork design, cross-national harmonisation, data archiving and access². Previous cross-national survey enterprises were often blighted by a lack of comparability due to poor or non-existent harmonisation and coordination. The ESS therefore adopted a more directive approach, with the CST working closely with national teams in developing the survey. The methodological aspirations of the ESS are summarised in a specification³ outlining how the survey is to be conducted by each National Coordinator. The ESS Director and CST lead the coordination and development of the survey's methodology.

Substantive datasets: These include data from five rounds of the ESS along with cumulative datasets⁴. The datasets are based on the ESS core questionnaire (e.g., institutional trust, political engagement and well-being) and rotating modules designed with substantive experts following a Europe-wide competition (e.g., immigration, democracy and health inequalities). The questionnaire takes an average of one hour to complete. The ESS Director works closely with the teams, developing questions that are effective cross-nationally and meet the underlying measurement aims. A new methodology for cross-national pre-testing was developed specifically for this⁵. The ESS Director and team at City are also responsible for the content of the core questionnaire. Over 60,000 users worldwide have registered on the ESS website, around 90% of whom are academic users, many then publishing using the data, which in turn generates non-academic impact. Around 10% of users are `non-academic' users, for example Government, private enterprise, non-governmental organisations (NGOs) and private citizens. A major component of the ESS's recent outreach strategy since 2010 has been to embed `pathways to impact' in the project's arrangements, in particular in producing and disseminating results from its rotating modules and core questionnaire.

References to the research

The ESS was awarded the Descartes Prize for `Excellence in Scientific Collaborative Research' in 2005 in recognition of its world-leading quality. An independent high-level scientific review<sup>6</sup>, conducted by the former head of the US Census Bureau in 2008, emphasised the high quality of the ESS and its impact. The impact and relevance of the ESS was recognised by its inclusion in the European Strategy Forum on Research Infrastructures in Europe `Roadmaps'⁷. Inclusion in the Roadmap (formulated by Research Ministries in Europe to identify priority research infrastructures) was a condition for Commission support to prepare for European Research Infrastructure Consortium (ERIC) status (2008-2010). This was a precursor for late 2013 when the ESS will become the first UK-hosted ERIC⁸. This represents the adoption by national governments of the ESS from the academic research councils, acknowledging the role that subjective social measurement has to play in policy. Selected papers are also published in highly regarded journals.

Details of the impact

Many state-of-the-art features of ESS procedures have found their way beyond the infrastructure, impacting the methodological standards employed by other survey organisations and programmes and feeding into their statistical outputs. For instance, a collaborative project led by Fitzgerald to develop a new methodology for cross-national pre-testing led to much of its approach being adopted from 2009 by a key collaborator in the USA, the National Center for Health Statistics. This is America's largest health protection agency and conducts national and international surveys. ESS methods have also been adopted by major international survey programmes. The European Quality of Life Survey, conducted by the European Foundation for the Improvement of Living and Working Conditions (Eurofound), used the ESS as a benchmark for improving its methodology in 2007⁹. The 2011 EQLS benefited from these methodological improvements. The work of Eurofound contributes to the planning and design of better living and working conditions in Europe via the European Commission. The Survey of Health, Ageing and Retirement in Europe (SHARE) was also influenced by the methodology of the ESS, for example adopting its translation approach¹⁰, benefiting SHARE fieldwork in 2008 and 2009. SHARE is Europe's first ERIC and its findings feed into policy discussions on the key issue of an ageing society. ESS question modules have had impact beyond the ESS. For example, many of the ESS Trust in Justice questions have been adopted as part of the EC measures for this domain (the Euro-Justis project to assess public confidence in justice), providing attitudinal indicators on this key societal issue at a European level.

The ESS methodology has been adopted in specific countries participating in the ESS by government departments, commercial fieldwork agencies and the private sector. A survey among ESS National Coordinators¹¹ in 2012 revealed that the Estonian Ministry of Social Affairs and Education had benefited from ESS methodology — seminars about the ESS were held in 2009 and learning from the ESS applied to Ministry-funded initiatives such as the Programme for the International Assessment of Adult Competencies (PIAAC) study in 2011; Polish ESS National Coordinators have provided ESS-based training for the National Bank of Poland and ARC Market and Opinion, a commercial research institute (in 2011 and 2012, respectively); and commercial fieldwork agency GfK Slovakia has since 2011 adopted the ESS contact form to monitor respondent contacts, a major fieldwork agency in Germany (INFAS, INFRATEST DIMAP). The ESS TRAPD (translation, review, adjudication, pre-testing and documentation) approach to translation was used in a survey of Muslims in North-Rhine Westphalia in 2011 and published by the Ministry for Work, Integration and Social Affairs.

ESS has also had a substantive impact. Developing better measures of well-being and progress is a shared goal internationally, supported by the Organisation for Economic Co-operation and Development (OECD) World Forum and European Commission. The UK Government is committed to developing wider measures of well-being so that Government policies can be more tailored. The All-Party Parliamentary Group on Wellbeing Economics was established in 2009 to support this agenda, with the New Economics Foundation (`nef') providing the secretariat. nef is a major player in UK and international debates in this area and regularly uses ESS data. In 2006-2007 nef worked with the ESS Director and team to develop an ESS module on well-being to inform its National Accounts of Wellbeing project¹², which calls for governments to directly and regularly measure individuals' subjective well-being. The project website was launched in 2010, including a tool enabling individuals to measure their own well-being based on ESS questions, which were used to create the National Accounts of Well-being Indicators. The Office of National Statistics (ONS) Measuring National Well-being Programme, which commenced in 2010 and publishes articles and reports on its findings, seeks to provide a fuller picture of society by supplementing existing economic, social and environmental measures. In support of the programme, the ONS has analysed ESS core questionnaire items measuring life satisfaction and happiness, and the Round 3 ESS module on Well-being. ONS measures of subjective well-being were also developed following discussions with nef about its Round 3 ESS findings. In addition, the Strategy and Analysis Team at the Cabinet Office has deployed evidence from the ESS to mainstream the concept of well-being across Government¹³.

The ESS module on Experiences and Expressions of Ageism (2008-2009), developed by Professor Dominic Abrams (University of Kent), the ESS Director and their teams, was used to `reframe the debate' on attitudes to ageing, another significant issue for governments worldwide. The module was complemented by an in-depth programme of work between the University of Kent, Age UK and the Department for Work and Pensions (DWP)¹⁴. In 2009 the team approached the DWP about an analysis of the attitudes towards ageing data. The DWP commissioned the team to produce reports using the ESS ageing data, working in collaboration with a senior researcher at DWP. The ESS Director and his team held a seminar at the DWP in 2013 on this topic to introduce the DWP and other Government analysts to the ESS dataset and present the ESS findings. The Coordinators of the EU and UK Year for Active Ageing and Intergenerational Solidarity also discussed ESS data at this event, underlining the importance of preparing for an ageing society.

The Round 5 module on Trust in Justice (2010-2011) was developed by Dr Jon Jackson (London School of Economics and Political Science), the ESS Director and their teams and was designed to provide evidence on procedural justice. The research has shown that fairness is not simply a desirable feature of justice systems but a precondition for effective justice. The project's impact has been to persuade key policy people in UK Government and the justice system that institutional legitimacy is central to the formation and application of justice policy¹⁵. Its impact is identifiable by changes in the attitudes of key stakeholders, reflected in commendations of the work (National Audit Office)¹⁶, commissioning of further related work (Ministry of Justice) and applications of principles (HM Inspectorate of Constabulary) where `legitimacy targets' have been added to senior officers' Key Performance Indicators. The work has featured in the broadcast media and in the national press. For example, the research has also been taken up in Radio 4 programmes including Andrew Marr's Start the Week and a three-part serial on policing by Mark Easton. The ESS Director and his team organised a seminar on the findings, which was hosted by the Centre for European Policy Studies, a leading Brussels think-tank, and attended by senior officials from the EC. The Director was invited to organise a seminar at the UK Cabinet Office on this topic further to discussions about the ESS dataset with members of the Well-being and Analysis Unit of the Cabinet Office. This took place in October 2012.

Also within the UK, the National Centre for Social Research, NatCen (where the UK ESS National Coordinator is housed) collaborated with Dr Rob Ford (University of Manchester) to use ESS immigration questions on the 2011 British Social Attitudes (BSA) Survey (to be repeated in 2013). When published in the 29^th BSA report in September 2012, this work attracted much comment and debate in the media and policy circles. For example, the Migration Observatory prepared a briefing on the issue using BSA and ESS data. The Migration Observatory provides authoritative, evidence-based analysis of data on migration and migrants in the UK to inform media, public and policy debates.

Tracking the non-academic impact of the ESS datasets beyond the UK is a formidable task, but examples show similar processes taking place at an international level. For instance, the ESS Director and his team were invited to convene a seminar at the OECD in Europe in January 2013 to discuss findings from the ESS module on work and the recession, feeding into their wider programme of work on this topic. They also convened a seminar in December 2012 at the European Policy Centre in Brussels on the results of the welfare module, attended by senior EC officials and representatives of many third-sector organisations. Examples from a survey of ESS National Coordinators (NCs) in 2012 show how ESS findings have been used in reports from local administrations in Germany, often as a reference distribution for local surveys, for example on migration. ESS findings have also appeared in reports by German Federal Offices and Ministries. The Bulgarian national team has produced publications to aid decisions in diverse policy fields. For example, the book authored by the NC, Wellbeing and trust, published in 2010, used ESS data and was used in the actualisation of immigration legislation.

ESS findings have also appeared in the press and are available to the general public. The launch in July 2013 of the ESS Findings booklet was covered in the UK by The Today Programme, during the CNBC News flagship morning news programme and in the Sunday Times and The Guardian. The ESS Director and his team gave media interviews. The Polish national ESS team held a series of dissemination conferences since the third round of the ESS from 2008 onwards for a mixed audience involving politicians, representatives of public administrations at all levels and NGOs. The Bulgarian national team has organised press conferences after each round of ESS that address key areas for policy. For example, the release of ESS data and analyses of satisfaction with the health service presented in 2011 at a press conference provoked wide public debate. The Slovakian NC team has held a similar event to that in Bulgaria and made a TV documentary about the ESS in 2012, aired on state television, including an interview with the ESS Director.

The ESS questionnaire, developed by the ESS Director and his City-based team in collaboration with other international experts and then fielded according to a methodology developed with his CST, has had a direct influence on the policy process in the UK¹⁷ and across Europe. The UK Universities Minister, David Willetts, writing to other Ministries represented on the ESS ERIC Steering Committee in June 2011, stated that the ESS enables `governments, policy analysts and scholars to keep up with societal trends that affect how democracy is working and how European citizens perceive their lives, their nation and the world'¹⁸

Sources to corroborate the impact

9. European Foundation for the Improvement of Living and Working Conditions (2007). First European Quality of Life Survey Methodological Review. Letter of confirmation from Eurofound.
10. Supan, A., & Jürges, H. (Eds.) (2005). Survey of Health, Ageing and Retirement in Europe Methodology Report.
11. Survey on the non-academic impact of the ESS among ESS National Coordinators (2012). Available from ESS team at City University London.
12. Michaelson, J., Abdallah, S., Steuer, N., Thompson, S., & Marks, N. (2009). National accounts of well-being. London: New Economics Foundation. www.nationalaccountsofwellbeing.org/learn/download-report.html.
13. Wellbeing and Civil Society Policy, Analysis and Insights Team. UK Cabinet Office.
14. Abrams, D., Russel, P., Vauclair, C., & Swift, H. (2011). Ageism in Europe: findings from the European Social Survey. Department for Work and Pensions. www.ageuk.org.uk/Documents/EN-GB/For-professionals/ageism_across_europe_report_interactive.pdf?dtrk=true.
15. Institute for Criminal Policy Research, Birkbeck University.
16. National Audit Office (2011). Comparing international criminal justice systems. www.nao.org.uk/publications/1012/criminal_justice_systems.aspx.
17. Drew, H., King, A., & Ritchie, F. (2013). Impact evaluation: Workplace Employment Relations Survey and European Social Survey. Final report to the ESRC, University of the West of England.
18. Letter from the Rt Hon. David Willetts to ESS ERIC Steering Committee Members, (15.06.2011).