The REF is a system for assessing the quality of research in UK higher education institutions implemented for the first time in 2014.

The primary purpose of REF 2014 was to assess the quality of research and produce outcomes for each submission made by institutions:

• The four higher education funding bodies will use the assessment outcomes to inform the selective allocation of their grant for research to the institutions which they fund, with effect from 2015-16.
• The assessment provides accountability for public investment in research and produces evidence of the benefits of this investment.
• The assessment outcomes provide benchmarking information and establish reputational yardsticks, for use within the higher education (HE) sector and for public information.

For background information on the development of the REF, please visit: http://www.ref.ac.uk/about/background/

The introduction of the ‘Impact’ element is a key change to research assessment in the UK. As part of the REF, HEIs were required to showcase the impact of research beyond academia via impact case studies (REF3b) and statements on the HEIs approach to research impact (impact template. REF3a).

Other key features of REF 2014 that differ from RAE 2008 include the following:

• The RAE units of assessment (67 sub-panels under the guidance of 15 main panels) were amalgamated to form 36 REF units of assessment, under four main panel discipline areas. This has resulted in broader discipline areas being covered by one sub-panel.
• Research environment continues to be assessed. However the criteria and structure of this part of the assessment changed and it is therefore not directly comparable with the same element in the RAE.
• Additional measures to support equality and diversity were developed for the REF. This included the submission of Codes of Practice on the selection of staff, approved in advance by the REF Equality and Diversity Panel; and a systematic approach to considering individual staff circumstances that constrained the ability of staff to produce four research outputs.

For further information on REF requirements please see:

Assessment framework and guidance on submissions (available at: http://www.ref.ac.uk/pubs/2011-02/)

Panel criteria and working methods (available at: http://www.ref.ac.uk/pubs/2012-01/)

Higher education institution (HEI) is a term from the Further and Higher Education Act 1992. According to the Act, it means any provider which is one or more of the following: a UK university; a higher education corporation; a designated institution. HEFCE may choose to fund higher education institutions for teaching and research if they meet the conditions of grant. Higher education institutions are also required to subscribe to the Office of the Independent Adjudicator.

154 Higher Education Institutions submitted to the REF from across the UK. A full list of these is available on the REF website at: http://results.ref.ac.uk/Results/SelectHei

Institutions were invited to make REF submissions in 36 subject areas, called units of assessment  (UOAs). The REF submissions were assessed by an expert sub-panel for each UOA.

For further information see: http://www.ref.ac.uk/panels/unitsofassessment/

The expert sub-panels who assessed the REF submissions were grouped into broad subject areas and worked under the guidance of four main panels.

For further information sees: http://www.ref.ac.uk/panels/

The RAE units of assessment (67 sub-panels under the guidance of 15 main panels) were amalgamated to form 36 REF units of assessment, under four main panel discipline areas. This has resulted in broader discipline areas being covered by one sub-panel

In the REF, impact is defined as an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia.

REF impact was assessed in the form of impact case studies and impact templates, where HEIs provided further information about their approach to supporting and enabling impact. For access to the impact templates please see full REF submissions at: http://results.ref.ac.uk/Results

Each HEI submitted a selection of impact case studies for assessment in the REF. An impact case study is a four-page document, describing the impact of research undertaken within the submitting department. It also contains information about the research that underpins the impact that took place. Further information about the criteria for the submission of impact case studies can be found in the two key REF guidance documents:

Assessment framework and guidance on submissions (available at: http://www.ref.ac.uk/pubs/2011-02/)

Panel criteria and working methods (available at: http://www.ref.ac.uk/pubs/2012-01/)

A submitting institution is a Higher Education institution that submitted to the REF. A full list of the 154 Higher Education Institutions that submitted to the REF from across the UK is available on the REF website at: http://results.ref.ac.uk/Results/SelectHei

Each submitting HEI was required to submit a minimum of two impact case studies. The number of case studies required then depended on the number of staff submitted to each unit of assessment:

Number of Category A staff submitted (FTE)	Required number of case studies
Up to 14.99	2
15 – 24.99	3
25 – 34.99	4
35 – 44.99	5
45 or more	6, plus 1 further case study per additional 10 FTE

Joint submissions

Higher Education Institutions were able to make submissions with one or more other UK HEI where this is the most appropriate way of describing research they have developed or undertaken collaboratively. All impact case studies will have been submitted jointly, and could stem from research undertaken either collaboratively or at either of the submitting HEIs. The case studies are not designated to a particular HEI within the joint submission.

Multiple submissions

Institutions would normally make one submission in each unit of assessment (UOA) they submit in. They could, by exception, make multiple submissions (with prior approval) to one UOA. An HEI might have wanted to make more than one submission to a UOA if they had two bodies of research that fell within the scope of the assessment panel but were clearly academically distinct. Case studies in the same unit of assessment are not indexed by separate multiple submissions within this database.

For further information, see Assessment framework and guidance on submissions (available at: http://www.ref.ac.uk/pubs/2011-02/)

The REF Impact case studies database includes 6,637 documents (at 18 November 2015) submitted by UK Higher Education Institutions (HEIs) to the 2014 Research Excellence Framework (REF2014). The documents have been processed by Digital Science, normalised to a common format to improve searchability and tagged against select fields.

REF impact case studies generally follow a template as set by the REF criteria (see ‘Assessment framework and guidance on submissions’ Annex G for impact case study template and guidance ( http://www.ref.ac.uk/pubs/2011-02/) . This template has a Title and five main text sections, plus the name of the Submitting Institution and the Unit of Assessment.

Some submitting institutions omitted some of this information or modified the template. The name of the Submitting Institution and the Unit of Assessment has therefore been added as metadata tags.

In addition to the Title of the case study, the text sections of the template and the  indicative lengths, as recommended in the REF criteria are:

1	Summary of the impact	100 words
2	Underpinning research	500 words
3	References to the research	Six references
4	Details of the impact	750 words
5	Sources to corroborate the impact	10 references

In some case studies  the sections vary considerably from the indicative word length, however all case studies were restricted to four pages in total..

Some case studies include non-text items such as institutional shields, photographs, other images, tables and embedded links.

The format of all original documents has been modified in the database to bring them to a similar format on the website and to enable the content to be searched more easily.

Links have been added to references, which may be slightly modified for clarity, to enable users to link to commercial database for the full article record.

Metadata has been added to associate the impact case studies with research subject areas, impact locations and impact type.

The PDF of each original document, as submitted, can be downloaded for comparison from the page displaying the individual case study. All case studies are also available as part of the REF submissions data: http://results.ref.ac.uk/

Text has been removed (or ‘redacted’) from some case studies by the submitting institutions because it is commercially sensitive or otherwise needs to be restricted. Generally this is indicated by the formula phrase “[text removed for publication]”, which was recommended in the REF guidance, but variants of this phrase do occur. 

REF impact case study titles in this database are drawn from the HEFCE REF database. They are the titles inserted by the HEI into the REF submissions form when they submitted their case studies to the REF.

HEIs were able to notify the REF team that certain case studies were ‘not for publication’. These have not been included in the Database. For further information, please see: www.ref.ac.uk/about/guidance/datamanagement/confidentialimpactcasestudies/

There are also some impact case studies that are not in the Database in order to satisfy re-use and licensing arrangements. These were removed where all other case studies from a particular HEI have been made available under a CC BY 4.0 license.

The total number of Impact case studies submitted to the REF is 6,975.

The number of Impact case studies in the Database is 6,637 (at 18 November 2015).

Redaction is the censoring or obscuring of part of a text.

Some case studies have parts of the text  removed (redacted) for confidentiality reasons (for instance, commercial sensitivity). These case studies are mostly included in the database but with this text removed. Where HEIs notified us of the need to remove elements of the text after 1 October 2014, the entire document has been removed from the database.

Submitting institutions can be grouped by UK region (Northern Ireland, Scotland, Wales, and nine regions for England). This is shown as part of the ‘browse by index’ on the website.

Submitting institutions can be grouped according to their relative and absolute research income. The UK Higher Education Statistics Agency (HESA) has assigned them to economic peer groups on the basis of incomme data available in 2004-05 (see www.hesa.ac.uk). This is shown as part of the ‘browse by index’ on the website.

 

Case studies are assigned to a single ‘Summary Impact Type’ by text analysis of the ‘Summary of  the Impact’ (Section 1 of the Impact case study template) . This is an indicative guide to aid text searching and is not a definitive assignment of the impact described.

There are eight Summary Impact Types. These follow the PESTLE convention (Political, Economic, Societal, Technological, Legal, and Environmental) widely used in Government policy development. For the purposes of introductory guidance in REF impact searching, Health and Cultural impact types (otherwise subsumed within Societal) have been added to the six standard categories.

The category names have a particular meaning for the purposes of analysis. This may vary between users. For example, JISC suggests:

Political: worldwide, European and UK national and local Government directives, public body policies, national and local organizations’ requirements, institutional policy.

Economic: funding mechanisms and streams, business and enterprise directives, internal funding models, budgetary restrictions, income generation.

Societal: societal attitudes to and impacts of education, government directives and employment opportunities, lifestyle changes, changes in populations, distributions and demographics, the societal impact of different cultures.

Most REF impact case studies relate at some level to more than one type of impact. Some case studies arguably cover all eight. Tagging supports rapid initial searching that will reveal a deeper and more diverse range of impact; user perspectives on this will vary.

Some analysts would assign REF Impact case studies differently to the categorization applied here. For example, many REF impact case studies refer to spin-outs. What is the research impact of such research where it leads to a commercially valuable device of medical benefit? It has a proximate technological impact that, once developed, might have economic impact for a company and later leads to health impact for society. If the research is relatively recent and the spin-out is new then in this database the Summary Impact is tagged as technological since the economic and health impacts remain latent.

See also: http://www.jiscinfonet.ac.uk/tools/pestle-swot/

The REF Impact case studies are assigned to one or more Research Subject Areas (to a maximum of three) by text analysis of the ‘Underpinning research’ (Section 2 of the Impact case study template). This is an indicative guide to aid text searching via a more fine-grained disciplinary structure than is immediately available in the 36 REF Units of Assessment.  It is not a definitive assignment of research discipline.

The Research Subject Area is equivalent to the 4-digit Group level of granularity in the Fields of Research of the Australia-New Zealand Standard Research Classification (http://www.arc.gov.au/pdf/ANZSRC_FOR_codes.pdf). This is hierarchical with 22 Divisions at the 2-digit level and 157 Groups at the 4-digit level (there are also 1,238 Fields at the 6-digit level but these are not used here).

In search result lists, the impact case study details for Research Subject Area display the 2-digit Division name in bold and 4-digit Group name in normal type. Some case studies can be associated with multiple Research Subject Areas drawn from different 2-digit Divisions. For example, a case study might link Statistics (0104) in Mathematical Sciences (01) with Ecology (0602) in Biological Sciences (06). Such instances are tagged in the database as Interdisciplinary and can be filtered in search results.

REF impact case studies are tagged with one or more UK locations on the basis of places (UK cities and towns, as found in the GeoNames database http://www.geonames.org) referenced in the text of either Section 1 (Summary of impact) or Section 4 (Details of the impact) of the document. This is an indicative guide to aid text searching. It is not a definitive identification of where UK impact has occurred as some text makes passing references to associated locations; other text references impact beneficiaries without a specific location.

It should be noted that the automated indexing cannot distinguish between e.g. Dover as a town, as the name of a street and/or as a person’s surname.

REF impact case studies are tagged with one or more global locations on the basis of places (place names, as found in the GeoNames database http://www.geonames.org) referenced in the text of either Section 1 (Summary of impact) or Section 4 (Details of the impact) of the document. Global locations outside the UK are grouped by country. This is an indicative guide to aid text searching. It is not a definitive identification of where the impact has occurred as some text makes passing reference to associated locations, while other text references impact beneficiaries without a specific location.

It should be noted that the automated indexing cannot distinguish between e.g. Brazil as a country, as the name of a street and/or as a person’s surname.

Search results for REF impact case studies are ordered on case study title, which may include numbers and other tags inserted by the HEI, unless a text search is used. If a text search is used then the case studies will be ranked according to the number of occurrences of the search term in the document.

If a REF impact case study can be associated with multiple Research Subject Areas that are drawn from different broad Divisions then they are tagged in the database as Interdisciplinary.  This assignment is an indicative guide to aid text searching via a more fine-grained disciplinary structure than is immediately available in REF Units of Assessment . It is not a definitive assignment of a case study’s interdisciplinary nature.

REF impact case studies that are interdisciplinary in the terms of this database can be filtered in search results.

The similarity of REF impact case studies is estimated by text analysis of Section 2 (Underpinning research), using Latent Semantic Analysis (LSA). This gives a compact representation of key semantic concepts contained in documents as defined by the co-occurrence of words within documents. Similar documents are defined as those that refer to the same semantic concepts.

The “view similar case studies” button allows users to see associations between REF impact case studies that use similar source research or work in the same research area, though the impacts may differ. This is an indicative guide to aid text searching. It is not a definitive indicator of a specific aspect of similarity between case studies.

The References to the research (Section 3 of theimpact template) have been extracted and where they can be disambiguated and unequivocally identified, usually by Digital Object Identifier (DOI), then they are linked to external citation databases. Not all authors provided DOIs.

Thomson Reuters processed all article records, and some other documents, for the HEFCE REF impact case studies database so as to address the deficit in DOI links. Where possible, Thomson Reuters matched the author-provided information with article records in the Web of Science^TM. This has created a significant improvement in DOI-linked coverage which benefits the information recovery for users.

A DOI link is indicated by a small icon identifying the external source and placed below the reference. If the user is not a subscriber to the source then the link will take them to a preview page containing partial information rather than the fully annotated article record.

Where the case study refers to a letter of support for a claim made, this is held by the submitting HEI. These were required by the REF team where it was deemed necessary to verify the claim made and have not been systematically collected during the assessment process. Such letters are not available within this database.

Research references in REF impact case studies include journal articles.  Where possible, the number of times that each of these have been mentioned on mainstream and social media is indicated by a link to http://www.altmetric.com/, which collates and indexes these data and also displays the context for the mention. Altmetric mentions are complementary to and not necessarily correlated with conventional citations.

The research impact identified in the REF impact case studies has arisen from research supported/funded by many UK and overseas organisations. The project funders that can be searched automatically in this database are those funded through the UK Science Budget disbursed by the Department of Business, Innovation & Skills (BIS)
Bodies supported by the UK Science Budget are:

• Research Councils UK
• Arts and Humanities Research Council
• Biotechnology and Biological Sciences Research Council
• Economic and Social Research Council
• Engineering and Physical Sciences Research Council
• Medical Research Council
• Natural Environment Research Council
• Particle Physics and Astronomy Research Council
• Council for the Central Laboratory of the Research Councils
• Science and Technology Facilities Council
• Royal Society
• Royal Academy of Engineering
• British Academy
• UK Space Agency
• Innovate UK

The Wellcome Trust co-funded the development of this database. They are also listed as a searchable research funder.

If a set of search results produces no returns for any specific category then that category is not displayed in the results set.

Sets of case studies can be downloaded in the following formats:

• Excel spreadsheet
• HTML document
• Zipped PDF files

In the case of Excel and HTML downloads, there is no limit to the number of case studies that may be downloaded in a single session.

In the case of PDF files it is recommended that you restrict your download to the set of case studies that would be returned from a single index selection or specific text search.  There is an absolute maximum limit of 300 case studies in any one download and such a download may take an appreciable time.

The full Terms of Use for this database are available.

This summary is not designed to replace the Terms of Use.

Most HEIs represented in the Database have agreed to license their case studies under a CC BY 4.0 licence. The following use is permitted under these licence conditions: http://creativecommons.org/licenses/by/4.0/legalcode

A more user friendly version is available here but does not replace the full legal code: http://creativecommons.org/licenses/by/4.0/

31 HEIs who were not in a position to license their case studies under CC BY 4.0 and are listed under Terms of Use. For such case studies, users are able to search the Database and undertake their own analysis and must comply with all relevant laws including fair dealing provisions. In order to comply with fair dealing provisions you should only copy as much text as is needed to make the point and you may not use the material for commercial purposes (this includes income generation without profit); you must attribute the source; incidental copying for text or data mining is permitted for non-commercial research as is text and data analysis by researchers for the purpose of carrying out computational analysis of the work. You are able to do this without having to obtain additional permission from the rights holder(s).

For case studies from HEIs who have not agreed that they can be used under a CC BY 4.0 licence, any user who wishes to copy or re-use material from the Database for another purpose, will need to seek permission from the relevant rights holder (the HEI who submitted the case study in the first instance). For the avoidance of doubt you will need to seek their permission should you wish to make multiple copies, for example put the work on a shared drive, computer network, intranet or website; send the material by email to multiple recipients or put it on a discussion list etc.

It will be possible for impact case studies to be completely removed from the Database but we are unable to redact case studies and then replace the original with the redacted version.

Please note that all case studies are also available as part of the REF submissions data at: http://results.ref.ac.uk/. Requests for case studies to be further redacted or removed will be considered for the main REF website, although we cannot guarantee they will be accepted. To make such a request please email: ResearchPolicy@hefce.ac.uk