Summary of the impact

The global workplace means that the staff at your local hospital, the pilot of your aeroplane or your teacher may be operating in a foreign language. Establishing their foreign language proficiency is crucial to ensuring effective communication. Not only this, establishing what one knows and does not know enables appropriately targeted teaching. We have enabled institutions and individuals throughout Europe to better understand the nature of foreign language proficiency, and, moreover, provided the means of measuring it. Our research led to the production of an on-line language assessment system, DIALANG, made publically available from 2001 in 14 European languages.

Underpinning research

The initial DIALANG research was funded by the European Commission, Directorate General of Education and Culture, through Socrates Lingua Action D [G1, G2]. The research team consisted of 20 prominent academic and non-academic institutions in Europe. It was coordinated by Professor Kari Sajavaara (University of Jyväskylä, Finland) and Professor Wolfgang Mackiewicz (Free University of Berlin, Germany). Charles Alderson, from the Department of Linguistics and English Language (LAEL), played a key role in this team, acting as academic consultant on the steering committee (1996-2004), scientific coordinator (1999-2004), and managing director of the software team (1996-1999). Dr. Caroline Clapham, also from LAEL, coordinated the analysis of data (1999-2004). Dr Dianne Wall and Dr Jayanti Banerjee (both from LAEL) participated, along with other members of the Lancaster Language Testing Research Group, in the development of a standard setting methodology to establish links with the Common European Framework of Reference. Software research was coordinated and undertaken by a team of 10 Lancaster University information technologists. (For a full list of project members and institutions, see Alderson 2005: vi-xii).

Our specific research contributed to all aspects of the design and development of DIALANG (http://www.lancs.ac.uk/researchenterprise/dialang/about.htm), including the content and methods used for self-assessment and diagnosis, the nature of its psychometric models, the development of the necessary technology, and the means of validating the system [R1, R3, R6, G1, G2]. It has been supported by a series of additional research grants (see section 3). In short it included:

• Research into the reading, listening, writing, grammar and vocabulary, and into their operationalisation for diagnostic purposes [R1, R3, R5, G1, G2, G3, G5].
• Research into standard setting procedures, i.e. the establishment of cut-off points between levels of language mastery for making diagnostic decisions (in particular for a multilingual target population). This resulted in manuals for standard setting (Kaftandjieva 2004; Kaftandjieva, Verhelst and Takala, 1999) [R6, G1, G2].
• Research into the self-assessment of foreign language ability: philosophy, construct and operationalization [R6, G1, G2, G8, G9].
• Research into modes of diagnostic feedback and advice and their effectiveness [R1, R3, R6, G1, G2, G8, G9].
• Psychometric research for diagnostic tests [R6, G1, G2, G8, G9].
• Research into the nature and performance of software for the development of online diagnostic language assessment [R1, R3, R6, G1, G2].

We are currently investigating the extension of DIALANG to Asian languages, and have recently entered into discussions with Beijing Foreign Studies University and the Foreign Language Teaching and Research Press to develop a version of DIALANG for the Chinese market. Lancaster University provided the following support to secure the research's impact:

Staff resources:

• Gillian McLaughlin was employed by Lancaster University on a 6-month contract to develop a business plan for the maintenance and commercialisation of DIALANG. She received funding to give commercial presentations and negotiate with organisations such as IBM, Microsoft and Pearson.
• Professor Alderson received funding to promote DIALANG at non-academic and teacher conferences (e.g. Danish language educationalist seminars, the opening conference for the European Year of Languages, British Council international seminars, conference of Hungarian National Language Centres). In addition, his teaching load was reduced and 50% of his DIALANG staff cost as managing director of the software team (in addition to 50% of his research coordinator and project consultancy role) was covered by Lancaster.

IT Resources:

• The DIALANG tests and website run on Lancaster University servers.

References to the research

Research findings have been reported in the following peer-reviewed articles:

and in the following book:

Grants held as part of the underpinning research described include:

		Principal Investigator	Title [Funder]	Award	Period
[G1]	European Commission LINGUA	Saajavaara, University of Jyväskylä	DIALANG: A European System for Diagnostic Language Testing. Grant number LINGUA D No: 39441-CP-1-96-1-FI-L. Phase 1	2,203,375 ECU, of which 1,650,000 ECU was the Lingua grant.	1996-1999
[G2]	European Commission LINGUA]	Mackiewicz, Free University of Berlin	DIALANG: A European System for Diagnostic Language Testing, Phase 2	€1,022,000	1999-2002
[G3]	Dutch Ministry of Education	Alderson, Lancaster	The theoretical construct underlying the construction of an item bank based on the Common European Framework.	€70,000	2003-2004
[G4]	European Commission	Alderson, Lancaster	ENLTA: European Network for Language Testing and Assessment.	€158,000	2003-2005
[G5]	Dutch Ministry of Education	Alderson, Lancaster	Continuation Project for the Dutch CEFR Construct	€32,000	2005
[G6]	EURO-CONTROL	Alderson, Lancaster	ELPACS - The validation of a test of English for air traffic controllers.	€32,000	2006-2007
[G7]	Lancaster University	Alderson, Lancaster	Research into Aviation English Testing	£6,997	2009
[G8]	Economic and Social Research Council .	Alderson, Lancaster	Diagnosing reading in a second or foreign language	£249,000	2010-2012
[G9]	Academy of Finland	Alderson, Lancaster	Diagnosing reading in a second or foreign language	€500,000	2010-2013
[G10]	Leverhulme Trust	Alderson, Lancaster	Diagnosing reading in a second or foreign language	£19,990	2012-2014

Details of the impact

Our research has increased knowledge of the nature of foreign language proficiency, its measurement and diagnosis, and our understanding of the standards embodied in the Common European Framework of Reference for Languages (CEFR). Understanding what to test and what the results mean was the crucial background to devising an easily accessible on-line assessment tool — DIALANG. By undertaking the assessment, learners can diagnose their abilities, providing an enhanced understanding of what they need to do to reach their own language use targets.

The impact of DIALANG at the level of the individual is evidenced by the number of test administrations recorded by the Research & Enterprises Services of Lancaster University (http://www.lancs.ac.uk/researchenterprise/dialang/about.htm). Between November 2006 and February 2012, 897,941 test sessions were recorded. On average, nearly 14,500 tests were taken per month and 475 per day during this period. In addition, the effects of DIALANG on individual test takers in terms of how they felt about taking the tests and in terms of the impact of test feedback is reported in Huhta (e.g. 2010) and Yang (2003).

Beyond the individual, institutions have deployed DIALANG in order to evaluate the language proficiency of their students and, related to that, their teaching effectiveness. A survey revealed that already in 2005 at least 49 institutions across Europe were using DIALANG, including the Netherlands, Germany, Belgium, Italy, Switzerland, Finland, France and the UK (see Huhta 2010, p.127). These included institutions of secondary, higher, and private education, and companies' in-house training sections. In addition, impact is visible from user testimonials from HU Utrecht, University of Twente, and University of Antwerp (collected in 2012), which reported that DIALANG is used for language proficiency screening of international students and for `home' students preparing for exchange programmes or on language degree programmes. In addition, DIALANG is used to diagnose the language proficiency of staff, both academic and support, teaching and operating in second languages. It has been called "of crucial importance" to those institutions' functioning (User testimonials, 2012).

DIALANG has had impact at a regional/international level, as can be seen in projects such as the European Survey on Language Competences — a survey carried out in 2011 on foreign language competence and knowledge in Europe and an initiative by the European Commission to support the development of language learning policies across Europe (http://www.surveylang.org). This project has based its development of systems on the concepts and design of the input, editing and piloting software developed for DIALANG, as well as the DIALANG software for automatic compilation of translations.

The impact of DIALANG-related research has also extended beyond the development and usage of an internet-based testing system into explorations of the possibilities of diagnosis in learning second and foreign languages in specific areas. For example, we have conducted studies of English proficiency tests for air traffic controllers and pilots, thereby improving the safety of a public service [R2, R4]. Professor Alderson conducted an 18-month validation study [G6] of a test of English for air traffic controllers (ELPAC) for Eurocontrol. That study led to important improvements to the ELPAC test and to its achieving widespread recognition as a high-quality, fit for purpose measuring instrument (see: www.elpac.info, and the ELPAC Final Validation Executive Summary July 2007; also available on request). A subsequent unfunded project surveyed available tests of aviation English (for air-traffic controllers and pilots) and found serious problems with the validity and reliability of many such tests, potentially leading to serious compromises in aviation safety (Alderson, 2010) [R2, G6, G7]. In 2009 Alderson headed a Task Force of the International Language Testing Association (ILTA) which proposed a voluntary self-sustaining system of endorsement/accreditation for tests of aviation English, in order to guarantee the quality of aviation English testing. As a result, the International Civil Aviation Organisation developed a mechanism, in conjunction with the ILTA, to endorse (quality check) tests of aviation English (http://www.icao.int/Newsroom/Pages/icao-promotes-aviation-safety-by-endorsing-english- language-testing.aspx; www.icao-aelte.org).

Work following on from the DIALANG Project has involved close collaboration with industry, notably, Pearson, as it relates to the Pearson Tests of English. The latest project is co-funded by the Australian Council for Educational Research, Pearson Language Assessment, the ESRC [G8], the Academy of Finland [G9], as well as the Leverhulme Trust [G10].

As a result of Lancaster's research into language proficiency, and particularly as contribution to development of the DIALANG assessment system, there have been improvements in the operation and safety of organisations across a range of sectors, particularly education and transport. The European success of DIALANG has led to an initiative to expand its use in China. In September 2013, Beijing Foreign Studies University, in collaboration with Alderson, led a nationwide pilot of DIALANG in China, involving 1000 students of English in 10 universities in 5 provinces, with the aim of adapting the system to Chinese users.

Sources to corroborate the impact

On the usefulness of the DIALANG Language Assessment System, see: Huhta, A. (2010) Innovations in Diagnostic Assessment and Feedback. Unpublished PhD dissertation, University of Jyvaskyla, Finland. Further information can be found in interim and final reports produced for the funding body: European Commission, Socrates Programme, LINGUA Action D, project 39441-CP-1-96-1-FI-LINGUA-LD (1996-1999), project 72072-CP-1-1999-1-DE-LINGUA-LD (1999-2002). All documents are available on request.

Contacts who can corroborate information on:

• The impact of DIALANG: Two named individuals from the Directorate Generate of Education and Culture, Centre for Research on Lifelong Learning (see REF submission system for details)
• The impact of The European Association for Language Testing and Assessment, which was set up by Alderson, for the impact of DIALANG, and also of the aviation English testing research: A former President of The European Association for Language Testing and Assessment; the current President of The European Association for Language Testing and Assessment (see REF submission system for details).
• DIALANG website statistics: The Intellectual Property Development Manager, Research and Enterprise Services, Lancaster University (see REF submission system for details)