Summary of the impact

Academic English language skills are essential if overseas students are to perform effectively in university/college contexts and to engage in study with relative independence and adequate comprehension. The Centre for Research in English Language Learning and Assessment (CRELLA) has developed a theoretically sound and practically efficient methodology for the construction and validation of tertiary-level Academic English tests used in the selection of millions of students by universities worldwide. Greater precision in language tests enables academic institutions to target their resources in ways that bring maximum benefit to students, both pre- and post-entry. CRELLA's research into how English is employed in academic contexts has led to significant improvements in test design and practice, and added to the commercial impact of both the existing tests to which they have contributed and the new tests they have created. Over £1.6 million was generated by our development of the Password test alone — see Section 4 below.

Underpinning research

Supported by British Council funding, staff in CRELLA have conducted extensive research since 2005 into the validity of IELTS (International English Language Testing System), the internationally recognised and widely used Academic English proficiency test for evaluating students wishing to access higher education opportunities (over 2 million test takers in 2013) (Refs3.1, 3.2, 3.5 and 3.6). Our research has established criteria that enable testers to align their tests more precisely with the contexts and the cognitive processes associated with using English for academic purposes. It has facilitated the inclusion of these key parameters in practical test specifications and their incorporation in test provisions, with major implications for the validity of the test.

Contextual parameters in academic reading

Weir (Professor 2005-present), Green (Principal Lecturer 2006-2008, Reader 2008-2013, Professor 2013-present), Bax (Reader 2008-2012, Professor 2012-present) and Hawkey (Senior Research Fellow 2006-2009, Visiting Professor 2009-present) have been at the forefront of this research strand, winning funding for a series of projects. One study investigated the nature of Academic English reading in British universities and identified problems international students had in coping with these demands (Ref3.1: pp.37-119). IELTS was evaluated with respect to these data and improvements suggested. Ref3.2 employed automated text analysis tools to measure key textual features and explored the validity of the IELTS Academic English Reading test by comparing the features of texts used in the test with those of core textbooks used by undergraduates. The study identified disparities requiring intervention which were subsequently addressed by the test developers through changes to the specifications.

Cognitive parameters in academic reading

Ref3.1 (pp.212-269) also highlighted the cognitive processes engaged by the IELTS Academic Reading test compared with those demanded of students in higher education (see Ref3.3 for our theoretical framework for reading and Ref3.4 by Bax for later elaboration). Bax made innovative use of eye-tracking technology to investigate the cognitive processes of readers taking online Academic English reading tests. His research investigated whether test items required higher level cognitive processing from candidates (Ref3.5). Green and Hawkey (Ref3.6) investigated the extent to which appropriate contextual and cognitive parameters were considered and operationalised by item writers working on the IELTS Academic Reading test.

Cognitive and contextual parameters in academic listening and speaking

Three other staff broadened this investigation of Academic English literacy to cover other language skills as operationalised through IELTS. In the assessment of Academic Listening ability, Field (Senior Lecturer 2011-present) developed a research methodology for investigating the cognitive validity of an Academic English Listening test, i.e., the extent to which it taps into processes employed in real-world academic listening events. This methodology was employed (Ref3.1: pp.391-453) to examine the effects of test method on listener behaviour. Field edited a special issue of the Journal of English for Academic Purposes (2011 Vol.10/2) exploring research and thinking relating to listening in academic contexts. Taylor (Senior Lecturer 2011-present) contributed to the issue a major review article on the testing of academic listening. This work is complemented by innovative research by Nakatsuhara (Lecturer 2010-2012, Senior Lecturer 2012-present) into the relationship between test-takers' listening proficiency and their interactional performance on the IELTS Speaking test (Ref3.1: pp.519-573).

References to the research

*Language Testing is a fully peer-reviewed, international journal that publishes original research on language testing and assessment. Impact Factor: 1.080 Ranking: 33/160 in Linguistics. Source: 2012 Journal Citation Reports® (Thomson Reuters, 2013).

All of the above references are being submitted as Outputs in UoA 29 REF2014. Follow-on annual funding from major EAP test providers for this underpinning research evidences clear pathways to impact. Since 2008, our research into IELTS funded by the British Council has been awarded a total of £99,100 across 5 separate projects. Our research for the Eiken Foundation of Japan produced a total income of £102,884, in 2010-2013. We have received £216,621 from English Language Testing (Password) for annual research and development work since 2008.

Details of the impact

Tertiary students need appropriate English language skills to benefit fully from academic study. Furthermore, if under-qualified students are admitted to universities, the economic costs to the institutions and the negative social and personal impact on the students can be enormous (Ref5.1 provides empirical data on the severity of these costs). Soundly conceived Academic English tests provide accurate and valid information to universities for making access decisions and determining the length and nature of any remedial language instruction required.

CRELLA's research has contributed to a deeper understanding of the nature of Academic English and provided a theoretically sound and practically efficient framework for its measurement. Its research has been utilised by all five English language test providers officially credited as meeting the UK Border Agency's requirements for student visas (Cambridge, City and Guilds, IELTS, Pearson and Trinity College London), as well as by other international test providers.

Evidence of Commercial Impact

• The Password academic placement test

CRELLA's research into language-related academic proficiency underpinned the design and development of the Password test to assess students' English language levels before or after acceptance on tertiary-level language programmes (Ref5.2 and Ref5.3). Our original commercial proposal became the basis for a joint venture business (English Language Testing (ELT) Ltd) in 2008 with offices in London, Australia and the USA employing five full-time staff members, 15 item writers and seven outsourced suppliers. Now used by some 170 HE institutions around the world (see Ref5.3/Password clients), the test has had over 150,000 candidates since 2008 (generating over £1.6 million in business for ELT) and proved itself as an efficient means of assessing students' language levels and identifying those needing language support. One user, who has carried out an evaluation study, considers that: `There is no test that can compete with Password for its combination of state-of-the-art question design, security, reliability and simplicity of use, delivered at such a low cost' and describes it as `an extremely reliable indicator of academic language ability and predicted final [language course] grade' (see Ref5.3/testimonials). CRELLA recently developed Password Pupil, Password Intro, Password Arabia and the Password Reading Test. Listening and speaking tests are currently under development in CRELLA.

Evidence of Educational Impact

• The International English Language Testing System (IELTS)

IELTS is taken by over 2 million people in 125 countries by those needing to meet English language entry requirements for university, professional and immigration purposes. It facilitates access to education worldwide and is used increasingly as an entry criterion to professional bodies (Ref.5.4). More than 7,000 education institutions, faculties, government agencies and professional organisations around the world recognise the test.

The IELTS partners are committed to `continual research-based improvement'. According to Davidson and Pollock, Chief Executives of the British Council and IELTS Australia: `The steady evolution of IELTS in response to research outcomes ensures its contemporary relevance and continued usefulness for test stakeholders' (Ref5.5). CRELLA's research has significantly informed the revision of the test's design, content and production. 86 versions of the test are in use at any one time so receiving institutions need to know that these are equivalent. CRELLA's research has given IELTS a systematic methodology for ensuring the consistency of content from one version of the reading test to the next. Specific IELTS research studies by CRELLA staff have demonstrated the necessity of creating reading and listening tasks which are cognitively valid, i.e., involve cognitive processing representative of the processes occurring in real-life academic activities. This has been incorporated into the IELTS test validation process (Ref5.6).

CRELLA's research has also contributed to the test's strong international reputation and commercial success in recent years (Ref5.7 confirms). The annual number of IELTS test takers has increased from just over 1 million in 2008 to over 2 million in 2013 (at c£150 per candidate, generating over £300 million p.a.).

Educational institutions need to be able to predict students' performance on academic courses accurately from test results. IELTS makes a useful contribution to the prediction of overall performance on academic courses. Weir et al found that in general, the better the IELTS score the more likely the student is to perform well academically (Ref5.8: p28). Based on returns from 100 staff in 27 departments in 13 UK universities, Hyatt and Brooks report that 88% of respondents felt the IELTS Test to be a useful indicator of Academic English proficiency (Ref5.9: p.16) enabling them to reject students with inadequate English language proficiency and target appropriate remedial instruction for those who fall just below an acceptable threshold. Correct use of individual IELTS profile scores for acceptance purposes invariably contributes to a fall in the number of students underperforming in their degree courses because of language deficiency (Ref5.1).

• The Test of English for Academic Purposes (TEAP) in Japan

CRELLA's research and reputation in the Academic English field also led in 2011 to an invitation to develop the Test of English for Academic Purposes (TEAP), a new university entrance examination for Japan, in partnership with the Eiken Foundation of Japan (formerly STEP), the leading English language testing organisation in Japan, with over 2 million test takers annually and Sophia University, one of the leading private universities in Japan. TEAP has Japanese government support. The major aim behind the development of TEAP was to impact positively on English education in Japan at the high school level (3.4 million students). Ref5.10 states: "The long-term impact of the TEAP project will be felt not only at the immediate level of the universities which choose to recognize the results, but will play an important part in removing the impediment to the improvement of EFL teaching and learning practices in Japan that the university entrance exam system has posed".

The Eiken Foundation is conducting iterative washback studies working with Green of CRELLA to evaluate the success of this intention. TEAP is intended to supersede the disparate approaches to English testing currently used in university admissions in Japan in which speaking and writing skills go untested. It serves as a model of the full range of Academic English skills needed by Japanese students to cope with the requirements of tertiary-level English-medium study in Japan. Our research into Academic English provided the theoretical and empirical basis for the TEAP test specifications, and CRELLA developed both the speaking and writing tasks and the rating scales. (Ref5.10). The new test was announced in 2012 and the Eiken Foundation has made provision for up to 200,000 candidates in the first full scale administration in 2014.

• The Cambridge Academic Literacy Test

In 2013 CRELLA was commissioned to assist Cambridge English Language Assessment in the development of the new Cambridge Academic Literacy Test which will be aimed at all prospective UK undergraduate and postgraduate students. CRELLA provided the initial position paper on the nature of academic literacy and subsequently helped write the theoretical rationale for the test. It is currently helping develop the specification and test content.

Sources to corroborate the impact

Ref5.1 Banerjee, J. (2003) Interpreting and using proficiency test scores. Unpublished PhD dissertation, Lancaster University

Ref5.2 Testimonial from Managing Director, English Language Testing Ltd for Password

Ref5.3 http://www.englishlanguagetesting.co.uk/Academic-Information

Ref5.4 Merrifield, G, GBM & Associates (2012) An impact study into the use of IELTS by professional associations and registration entities: Canada, the UK and Ireland http://www.ielts.org/PDF/vol11_report_1_an_impact_study.pdf

Ref5.5 Davidson, M. and Pollock, A. (2011) Foreword in IELTS Research Reports, Volume 11 (see http://www.ielts.org/researchers/research/volume_11.aspx)

Ref5.6 Bridges, G. (2010) Demonstrating cognitive validity of IELTS Academic Writing Task 1, Research Notes 42. http://www.cambridgeesol.org/rs_notes/rs_nts42.pdf

Ref5.7 Testimonial Senior Research Adviser IELTS, on behalf of CELA/IDP Australia/British Council

Ref5.8 Weir, C.J., Chan, S.H.C. and Nakatsuhara, F. (2013) Examining the criterion related validity of the GEPT advanced reading and writing tests: Comparing GEPT with IELTS and real life academic performance, LTTC-GEPT Research Report, 1: 1-43. http://www.lttc.ntu.edu.tw/lttc-gept-grants/RReport/RG01.pdf

Ref5.9 Hyatt, D. and Brooks, G. (2009) Investigating stakeholders' perceptions of IELTS as an entry requirement for higher education in the UK, IELTS Research Reports, Volume 10. IDP, IELTS Australia and British Council

Ref5.10 Testimonial from Chief Researcher Eiken Foundation of Japan for TEAP test