Summary of the impact

Outcomes of the research conducted at the University of Edinburgh (2001 to 2007) that have had the most far-reaching impact are a strong conceptualisation of the whole learning environment (including curricula, teaching, learning support, and assessment and feedback) and its influence on the quality of undergraduates' learning. What gave these outcomes added resonance was a concern for disciplinary distinctiveness as well as more generic features; an alertness to the pervasive implications for day-to-day teaching-learning practices of mass 21^st-century higher education; and a focus on enhancing as well as evaluating the student experience.

The reach of the impact extends to university teachers, middle and senior academic managers, local and national bodies with responsibilities for surveying quality and standards and, albeit less directly, students. Staff in at least 21 universities in 12 countries have used the Experiences of Teaching and Learning Questionnaire (ETLQ). The National Student Survey questionnaire was influenced by the ETLQ, and has continuing UK-wide impact on teaching through students' retrospective ratings of their experience. Project outputs were directed towards teaching staff through workshops, publications and invited presentations, followed by detailed advice on assessment and feedback of coursework.

Underpinning research

Two main research projects underpin the impact described here:

• ESRC/TLRP Award (L139251002 for £847,560, 2001-2005), evaluated as Very Good/Excellent. Enhancing Teaching-Learning Environments in Undergraduate Courses. Co-directors: Entwistle, Hounsell; Associate directors in Edinburgh: Anderson, McCune, with contributions from Durham and Coventry
• Higher Education Academy (£30,000, 2006-2007), Innovative Assessment Across the Disciplines (Director: Hounsell)

Key researchers, all at the University of Edinburgh, include Noel Entwistle, Bell Professor of Education (1978-2005), now Professorial Research Fellow (2013-2014); Dai Hounsell, (1985-2012), Professor of Higher Education, now Professor Emeritus; Dr. Velda McCune, (1999-2008, 2011-), Deputy Director, Institute for Academic Development and Dr. Charles Anderson (1987-), Senior Lecturer.

Enhancing Teaching-Learning Environments (ETL) was part of the highly selective ESRC Teaching and Learning Research Programme. It grew out of investigations into students' approaches to learning that raised questions about how teaching could better encourage understanding. The project was designed to involve university teachers in selected departments — electronic engineering, biological sciences, economics and history — with the specific aim of demonstrating how research can aid practice.

The ETL project developed an inventory (Experiences of Teaching and Learning Questionnaire — ETLQ) to collect students' descriptions of their approaches to learning and the teaching and learning environment they had experienced. Items included were based on previous research to develop a conceptual framework through which implications for teaching practice could be identified (Entwistle, McCune & Hounsell, 2003). During the project, the questionnaire was administered to over 6500 students in 11 British universities, involving 90 university teachers in each stage of the research.

The questionnaire dataset was complemented by a similarly substantial corpus of data derived from semi-structured interviews with nearly 670 students and 90 academic staff. Results from an initial round of data-gathering by questionnaire and interviews were discussed with the course teams involved in the project and, where feasible, led to changes in the way the course was taught the following year. The effects of the changes were investigated through a second round of data-gathering.

Drawing on both sources of data, the findings of the project threw new light on undergraduate courses as teaching-learning environments in 21^st-century mass higher education. It was suggested that the focus of courses should be on developing the distinctive ways of thinking and practising characteristic of the subject area concerned, rather than on lists of intended learning outcomes. It was shown how certain aspects of the teaching-learning environment — most notably, teaching for understanding, staff enthusiasm and support, and well-aligned assessment-for-learning and feedback on students' progress and performance — could have a powerful effect on the student experience in each of the course units investigated, including their ways of studying and their perceived academic achievement. The research also indicated how the key facets of an undergraduate teaching-learning environment needed to be coherent with the main aims of the course and congruent with each other, as well as with the previous knowledge and experience of the students.

Later follow-up research, including a systematic review of the literature on innovations in practices funded by the Higher Education Academy, focused particularly on the interwoven themes of assessment-for-learning and feedback to students, which had emerged as areas of greatest student dissatisfaction in successive National Student Surveys in the UK, as well as in surveys and empirical studies in Hong Kong and Australia (Hounsell et al. 2008). What particularly interconnected this follow-up work closely to the seminal ETL project was a focus on the interrelationships between assessment design and feedback provision and the quality of student learning, the crucial role of the subject/disciplinary dimension in shaping opportunities and affordances, and the role of evidence in understanding and enhancing practices on the ground. It has taken that work further forward by marrying research findings and conceptual insights to a large pool of documented initiatives grounded in specific subject and course settings. As reported in the sections below, the fruits of this synthesis are evident in the HEA Innovative Assessment project, Enhancement Themes initiatives and a website.

References to the research

The quality of the research is evidenced by the status of the refereed publications, the sources of funding, the evaluation of the ESRC project, the engagement with prominent national and international higher education bodies, and international book publishers.

Details of the impact

The impact of the research across and beyond the UK arises from its distinctive range and blend of outcomes and outputs. Given the quite substantial measure of autonomy which university teachers continue to be able to exercise in matters of curriculum, teaching-learning provision and approaches to assessment and feedback, they represent a primary target. They are, however, a difficult group with which to engage, not least because of the extent to which their professional identities, allegiances and channels of communication are allied to their chosen discipline or subject area. Our impact strategy has therefore been pursued along a variety of routes, combining generic and discipline-specific dissemination, and interaction through institutional, national and international forums.

One fertile pathway to impact stems from the evolution of the influential ETLQ survey instrument. A precondition of later impact at national level was Entwistle's involvement (2002-03) on a sub-committee designing the National Student Survey (NSS). His experience of developing the ETLQ and its predecessor contributed significantly to the selection of NSS items and the design of its rubric and format [5.1]. This has continued to have an impact during the REF period, as the NSS has grown in significance in terms of its capacity to highlight the importance of the student experience and consequently the need to direct university resources towards changing teaching practices.

The ETLQ instrument has also been adopted and used by higher education institutions nationally and internationally. The inventory was selected for inclusion in a peer-reviewed teaching resource bank developed for the Association of Medical Colleges in the USA [5.2]. The University of Helsinki has adapted versions of the questionnaire to use routinely and annually with all students as part of their portfolios [5.3]. Departments are provided with scale scores shown in relation with overall norms, and use the ETLQ results to introduce and monitor innovations based on the analyses. The scale scores are also used in counselling individual students, and findings from the growing corpus of ETLQ data at Helsinki are now being promulgated through articles in refereed international journals [5.4]. In 2013, Universities in Greece and Portugal were trialling the general use of ETLQ [5.5], while teachers have reported using the instrument individually, or with colleagues, at universities or colleges in Alberta, Antwerp, Cape Town, Galway, Lausanne, Granada, Prague, Punjab and The Hague. In addition, eighteen lecturers in nine British universities have reported using ETLQ to monitor students' perceptions of their teaching. The ETLQ's growing accessibility on websites and in publications increases the likelihood of other uses that have not been reported.

A complementary avenue to impact has focused on the findings as a whole, disseminated through a book written specifically for practising university teachers across the subject spectrum (Entwistle, 2009), in tandem with invited presentations to bodies concerned with improving the quality of teaching and learning. For example, Entwistle (2008) was invited to contribute a plenary paper for a conference organised for the Higher Education Quality Council of Ontario, with an audience of some 75 invited academics mainly involved in professional development [5.6]. Another nine invited presentations and workshops have been provided in UK, Sweden and Finland between 2008 and 2013.

Drawing on findings from the research, in the period 2008-2013 Hounsell has given more than 40 keynote and workshop presentations to a combined total of some 2,500 practitioners and academic managers in the UK, Sweden, Norway, Denmark, Canada, China and Australia, and to bodies including the Australian Learning and Teaching Council, the Australian Technology Universities Network, and Universitas21.

His follow-up research on assessment and feedback has also entailed pursuing more innovative modes of dissemination that combine reviews of empirical evidence and conceptual insights with a wealth of illustrative case-examples from across the disciplinary range that have been sourced internationally as well as from the UK. One such instance was the production in 2007 of a set of four guides for university teachers on integrative approaches to assessment and feedback [5.7]. Commissioned as part of the Enhancement Themes initiative by the Quality Assurance Agency Scotland in partnership with Universities Scotland, the Scottish Funding Council and the Higher Education Academy, the guides were disseminated to staff in all Scottish universities, and there has been continuing subsequent involvement in sector-wide enhancement [5.8] and communication with middle and senior university managers through the Leadership Foundation [5.9] throughout the REF period. A groundbreaking website, Enhancing Feedback, has been created to provide reviews of over thirty strategies for generating feedback to students and links these to more than 200 case-examples from a wide range of subject areas. Since its launch in 2010, there have been 27,000 visitors to the website from over one hundred countries [5.10].

Sources to corroborate the impact

Sources to corroborate the impact are indicated in square brackets in the text. Web pages have been archived at https://www.wiki.ed.ac.uk/display/REF2014REF3B/UoA+25

[5.1] For confirmation, contact member of the Open University team that conducted the pilot study for the National Student Survey and can corroborate the contribution of this research to its design.

[5.2] Directory and Repository of Educational Assessment Measures (DREAM) at
https://www.mededportal.org/about/initiatives/dream/

[5.3] See presentation at http://www.helsinki.fi/ktl/yty/tutkimus/LEARNresearch_proj160609.pdf that reports the use of the ETLQ.

[5.4] See for example article by Parpala, A., & Lindblom-Ylänne, S (2012) Using a research instrument for developing quality at the university. Quality in Higher Education 18, 313-328.
DOI:10.1080/13538322.2012.733493

[5.5] Details of a seminar on teachers and the quality of learning to which contacts at University of Evora, Portugal and University of Ioannina, Greece contributed:
www.ed.ac.uk/polopoly_fs/1.85836!/fileManager/ECS-QualityofLearningSeminar-25Apr2012.pdf

[5.6] This led to a further written contribution: Entwistle, N. J. (2010) Taking stock: An overview of research findings. In J. Christensen Hughes & J. Mighty (Eds.), Research on Teaching and Learning in Higher Education. Montreal: McGill-Queens University Press. Supplied on request.

[5.7] Reports: Hounsell, D., Xu, R. and Tai, C.M. (2007) Monitoring Students' Experiences of Assessment; Balancing Assessment of and Assessment for Learning; Blending Assignments and Assessments for High-Quality Learning; Managing Assessment Practices and Procedures. Gloucester/Glasgow: Quality Assurance Agency for Higher Education.

http://www.enhancementthemes.ac.uk/docs/publications/monitoring-students-experiences-of-assessment.pdf

http://www.enhancementthemes.ac.uk/docs/publications/guide-no-2---balancing-assessment-of-and-assessment-for-learning.pdf

http://www.enhancementthemes.ac.uk/docs/publications/guide-no-3---blending-assignments-and-assessments-for-high-quality-learning.pdf

http://www.enhancementthemes.ac.uk/docs/publications/guide-no-4---managing-assessment-practices-and-procedures.pdf?sfvrsn=18

[5.8] Report: Hounsell, D. (2011) Graduates for the 21st Century: Integrating the Enhancement Themes. Institutional Activities. Glasgow: QAA Scotland. (ISBN 978 1 84979 410 7).
www.enhancementthemes.ac.uk/enhancement-themes/completed-enhancement-themes/graduates-for-the-21st-century

[5.9] Paper: Hounsell, D. and Rigby, S. (2013) Leading Change in Assessment and Feedback. Case Examples and a Guide to Action. (Series Four Research, Commissioned Papers). London: Leadership Foundation for Higher Education.

www.lfhe.ac.uk/en/research-resources/published-research/2013-research.cfm

[5.10] Enhancing Feedback website. www.enhancingfeedback.ed.ac.uk or via
www.heacademy.ac.uk/assessment-projects-resources. Data on visitors was sourced from Google Analytics.