Summary of the impact

This case study describes the international impact of research undertaken by Professor Kenna and co-workers into the concept of critical mass in research groups. The main impact arising from the research is upon public policy and services. The research has influenced policy debate in the UK Parliament, in France and more generally. Beneficiaries of the research include policy makers in higher education, governments, think tanks, and public sector organisations and societies.

Underpinning research

The concept of critical mass in research had been around for a long time without clear definition. It has been considered, discussed and debated by research managers and policy makers in academia and governments, especially in relation to which research areas to promote.

Kenna has been researching statistical physics at Coventry University since his appointment in 2002. Drawing on mean-field theory, he developed the first agent-based model, which takes account of interactions between individuals, to explain the relationship between research quality as produced by a group and the number of its members. The model manifests a linear relationship between research quality and group/departmental size, up to a discipline-dependent "upper critical mass" (similar to the Dunbar number in anthropology). Above this size, difficulties in communication akin to the Ringelmann effect in sociology result in a much reduced correlation between research quality and group size. Empirical support for the theory comes from rigorous statistical analysis of the RAE 2008 and its French counterpart. Kenna's research delivers the first and only quantitative definition of critical mass in the context of managing research groups.

His initial paper [1] was followed by a second [2] in which critical masses for various academic disciplines were determined. In an interview with Times Higher Education (THE) in 2009, Professor Dame Julia Higgins, Chair of the RAE2008 panel for Physical Sciences lamented the absence of an "intellectual basis" to compare RAE results across disciplines and called for "serious thinking" on the matter (http://tinyurl.com/msluy3r). In response to this, Kenna developed a method to overcome the problem of comparing peer-evaluation between disciplines [3]. Kenna's research also focussed on the optimum group/departmental size [4] and explained, in terms of critical mass, why some smaller, research-focused departments' quality is on a par to that of larger institutions [5]. He also explained why citation-based indicators are poorly correlated with group quality, although correlate well with peer-evaluated measures of group strength [6]. The initial paper [1] was ranked "Best of 2010" in European Physics Letters (EPL), placing it in the top 5% of the circa 800 papers published in EPL that year. It also featured in EPL's October 2010 "Highlights from Previous Volumes". Paper [3] was the 4^th, 4^th and 45^th most-read article in Research Evaluation in Jan, Feb & Mar 2012. Paper [4] was respectively the 13^th and 4^th and 18^th most-read article in the IMA Journal of Management Mathematics in Jan, Feb & Mar 2012.

References to the research

Details of the impact

Kenna and Berche's research on critical mass has been reported many times in print and online media and has had impact on research policy all over the world, leading directly to policy changes.

Impacts on public policy and services
There is documented evidence of Kenna's research having an impact on policy debate in Parliament, EU Directorate and policy-orientated think tanks. There is also evidence that the research has been used by lobbying organisations, professional associations and the media to inform or change policy on research quality.

Informing French research policy change: The Conseil National des Universités (CNU) is France's national body whose role includes the assessment of academics and evaluation of research groups (similar to the UK's REF but with a broader remit). Professor Georges Landa, President of CNU's section 28 (Physics), states that, prior to engaging with Kenna's research, the CNU assessed academics solely using metrics such as the numbers and quality of publications. However, now it takes the size of the research group into account. Landa believes that, as a result of Kenna's research, the CNU's assessment of individual researchers is now more accurate and fair [a]. He has also stated "I am totally convinced of the necessity to take into account the size of groups or labs" [a]. Through its impact on the CNU, Kenna's work has impacted on every university in France and, by extension, to every academic employed in French universities.

Influencing EU policy: Dr James Gavigan was Head of the EU's Unit for European Research Area policy in DG Research, from 2006 to 2012. His job was to develop initiatives to increase the amount, impact and overall efficiency of research carried out in the EU. He stated to Kenna "Your work on critical mass is certainly relevant and interesting for the research policy debate at EU level, as led by the European Commission in Brussels. In that regard, in my role as Head of the European Research Area Policy Unit which I occupied up to the end of August 2012, I read with interest your work and circulated it within the Research and Innovation Directorate General" [b].

Informing UK parliamentary debate: In 2010, Dr Jonathan Adams, Director of Research Evaluation at Evidence, a Thomson Reuters business (now called Research Analytics), wrote: "We were interested to see your article reported in the Times Higher Education... This is very helpful and we would very much like to get advice on when the papers go into publication... I am passing on your information to colleagues at BIS, and hope that is OK with you." As a result Kenna (with Berche, co-author) presented the research evidence to the House of Commons Science & Technology Committee on Peer Review. This is published in the Eighth Report of Session 2010-12 Volume 2 at http://tinyurl.com/kgxsubr.

Research used as the basis for critical reviews of policy in the media: Journalist Paul Jump is the senior science and research reporter at Times Higher Education (THE). One of his three articles on the implications of Kenna's work has been viewed by over 2,000 people online in addition to the 60,000 per week hard-copy readers. Jump believes Kenna's work is relevant to policy-making on "whether research funding should be concentrated on large research intensive universities, as the Russell Group argue", or whether size should be taken into account more explicitly in the allocation of research funding [c]. Jump's articles show there has been considered, wider, public engagement with the research, and a policy issue has been raised for research funding in the UK.

Stimulation of policy debate: The Foundation for Science and Technology provides a `neutral platform for debate of policy issues that have a science, engineering or technology element'. It is directed by a Council including heads of the Royal Society, British Academy, Research Councils and others. The Foundation provides support to around 140 learned & professional societies. Paper [1] influenced an Editorial by Sir John Enderby on "the dilemma of science and research funding" in the Journal of the Foundation for Science and Technology [Vol 20, No. 6 (2011) page 4]. The research also influenced the Russell Group policy report Jewels in the Crown: the Importance and Characteristics of the UK's World-Class Universities, Russell Group Papers, Issue 4, 2012, which stated `A study using research assessment exercises in the UK and France to look at the relationship between quality and number of researchers submitted, found evidence of maximum and minimum thresholds for group size in a range of subjects' [d].

Engagement with practitioners and professional services: Kenna and Berche have been commissioned to write articles for a number professional journals and newsletters. These included Research Intelligence, the membership publication of the British Educational Research Association; Reflets de la Physique, the magazine of the French Physics Society, of which 4,000 copies per issue are distributed; Mathematics Today, the membership publication of the Institute of Mathematics and its Applications which has a bimonthly readership of 4,500 professional mathematicians; Significance, which is the membership magazine for the Royal Statistical Society, the American Statistical Association and Reports of the National Academy of Sciences of Ukraine in 2013. In addition, Europhysics News published an article based on Kenna's research (The relationship between quality and quantity in research, Vol.41, No.5, 2010, page 15). This bimonthly publication is the "voice of the European Physical Society" which has 41 national Member Societies, representing over 120,000 members. Europhysics News has 25,000 copies per issue which go to relevant university departments and key players the world over. These and other articles were translated into many languages including Greek, Russian, Ukrainian and Vietnamese and reverberated around the world through websites, blogs, online media and discussion forums.

Influence on campaigns and debate by international lobbying groups and NGOs: The following give examples of use of the research to support campaigns to protect research funding:

• Sauvons l'Universite (formerly Sauvons la Recherche) is a French association working to defend values and research funding in the French university system, to promote collegial work and to campaign for reforms in the HE sector. In 2011 they ran a detailed report on papers [1- 3]. See http://www.sauvonsluniversite.com/spip.php?article4808 (in French).
• The Ukrainian Science Association is "organized to promote education and science reform in Ukraine", which "conducts independent analytical research and monitoring of education, science and technology developments in Ukraine and worldwide." In 2010, they ran a report on their website, inspired by Kenna's interview in Times Higher Education. See http://nauka.in.ua/news/archive/article_detail/5809 (in Ukrainian).
• The Association of Swedish Higher Education (SUHF in English) is "an organisation for institutional co-operation" comprising 41 universities in Sweden. The Association "aims at safeguarding the external interests of the institutions and at strengthening their internal co- operation." In November 2011, an SUHF Experts Committee on Quality issued a report on "Research Quality and the Role of University Leadership". Their recommendations regarding group size were influence by [2]. See http://tinyurl.com/mfywoy5.
• RAND (Research and Development) Corporation is "a non-profit institution that helps improve policy and decision making through research and analysis". In a 2011 review of the research and development system, prepared for the Greek Ministry of Education, Lifelong Learning and Religious Affairs, RAND drew on reference [1] and on Kenna's submission to Parliament to draw conclusions on the influence of research group size. See http://tinyurl.com/meg6mrp.
• The Norwegian Network for Private Higher Education Institutions advocates on issues concerning HE and research policy to the Norwegian parliament. Following publication of Kenna's OECD Higher Education Management and Policy paper [5], in March 2012 the Network placed a review on its website of the research and its implications. See http://www.nph.no/?idnr=167 (in Norwegian). Dr Arne J Eriksen, Secretary General of the Norwegian Network for Private Higher Education Institutes, stated that Kenna's research "is important for the development of national and institutional strategies/policies and priorities".
• Kenna's research was cited in a Policy Discussion on the use of metrics in evaluating research, documented in the Proceedings of the International Congress of Mathematicians, Hyderabad 2010, Volume 1 Plenary Lectures and Ceremonies, Ed. Rajendra Bhatia. See section called Round Table: The Use of Metrics in Evaluating Research - page 742 (by J.M. Ball, Oxford). Panel included Doug Arnold (President of SIAM), Malcolm MacCallum (Director of the Heilbronn Institute and Member, Research Policy Committee, London Mathematical Society), José Antonio de Pena, (Director of the Mathematical Institute at the National University of Mexico and former President of UMALCA (Mathematical Union of Latin America and the Caribbean)) and Frank Pacard (Scientific Advisor of Mathematics in the French Ministry of Higher Education and Research).

Articles and interviews in media: Articles in Times Higher Education (THE) [e], University Business [f] , Research Professional [g] and the New Zealand Herald appeared in 2010 following interviews with Kenna. THE publishes 28,000 copies per issue, has 60,000 readers per week and THE online registers over 650,000 users globally each month. University Business is read by senior management across 165 Universities and over 350 HE and FE colleges throughout the UK. It has a circulation of 6,000 and a readership of 8,500. In 2011, Research Trends reported on [1-4] [h]. In 2012 another interview with Kenna prompted further articles in THE [e] and University Business [f] following publication of [5]. In 2011 Kenna was also interviewed for Physics World [i]. Physics World has a circulation of 35,000 and a readership of 110,000. These articles and interviews were translated and reproduced many times worldwide. In 2013, following publication of [6] Kenna was again interviewed by Times Higher Education [e].

Influencing a commercial organisation's evaluation of excellence: Verisk Analytics is "a leading source of information about risk", offering "risk-assessment services and decision analytics to professionals in many fields" (6,000+ employees including 500 with advanced degrees, 200+ actuaries with $1.5billion in revenue in 2012). At their Investor Day in 2013, Scott Stephenson, president and Chief Executive Officer of Verisk Analytics, used Kenna's work in a presentation on "Operating Strategy and Innovation" showing that excellence is achievable at modest scale [j].

Conclusion
Kenna and co-workers' research has provided much-needed rigour to the concept of `critical mass' in research groups. Their work has directly led to a change in French policy on the assessment of its academics. Furthermore, it has stimulated significant discussion by policy makers and think tanks working in the field, in the UK, EU and beyond.

Sources to corroborate the impact

a. Georges Landa, Président de la section CNU 28 (Information collected by RAND Europe in two interviews, see report PR-514-CU and PR-843-CU)

b. Head of the EU's Unit for European Research Area policy in DG Research, from 2006 to 2012

c. Paul Jump, Senior Science and Research Reporter for Times Higher Education (Information collected by RAND Europe in an interview, see report PR-514-CU)

d. http://russellgroup.org/JewelsInTheCrown.pdf

e. Truth in numbers: study pinpoints 'critical mass' for research success, by Paul Jump, Times Higher Education, No. 1955, 8-14 July, 2010
Do too many researchers spoil the outcome? by Paul Jump, Times Higher Education No. 2036, 9-15 February, 2012
Skip the REF and count the notes?, by Paul Jump Times Higher Education No.2110, 18-24 July 2013

f. Research funding most effective when targeted at medium-sized research groups by Lucy Porter, University Business, 9 July 2010
Smaller universities often produce better research by Carley Drew, University Business, 13 February 2012

g. Concentrating research to get quality is a myth, says study by Elizabeth Gibney, Research Professional 8 and 14 July 2010

h. Two's company: how scale affects research groups, by Matthew Richardson, Research Trends (issue 25, November 2011)

i. Why 13 and 25 are magic numbers for physicists, by Michael Banks, Physics World, Volume 24, No. 6 (June 2011) page 7 and Volume 24, No. 12 (December 2011) page 52

j. See p106 of document available at http://tinyurl.com/odct5ls or from Coventry University