Summary of the impact

This case study details the impact of a pioneering theoretical approach to English language testing. Recognised as the most influential test validation theory in modern assessment, the socio-cognitive framework, conceived by Weir and O'Sullivan, and operationalized and developed further by O'Sullivan at the University of Roehampton, focuses on three key elements: the test taker (social), the test system (cognitive), and the scoring system (evaluative). This framework is applied to give a meaningful measure of a candidate's performance, appropriate to the underlying traits or abilities being assessed. This research has had a significant impact in two distinct phases: 1) through a series of commissioned projects since 2008, the research has had a significant impact on testing bodies, organisations and test takers internationally, and 2) it has underpinned the development of innovative new business products by a leading international educational and cultural organisation since 2012.

Underpinning research

The research underpinning this case study was built on the premise that test validation should take into account not only the cognitive processing of language, but also the social conditions in which language is used. It is an innovative perspective that puts the test taker at the heart of assessment theory and practice. This led, in 2001, to the development by Professor Cyril Weir (Professor, 2000-5) and later the operationalization and refinement by Professor Barry O'Sullivan (Reader 2003-, Prof. 2007-12) of the first validation framework (published by Weir, 2005).

This model was further refined in the field during the EXAVER project (Universidad Veracruzana, Mexico) and QUALSPELL project (Quality Assurance in Language for Specific Purposes, Tallin Technical University, Estonia 2002-4), resulting in a framework that is not only theoretically robust but also practically useful. While other validation theories existed, they were difficult to operationalize and thus of limited use in the development of tests. Refinement and operationalization of the underlying model continued, with an updated version and rationale published (O'Sullivan, 2011a).

The socio-cognitive framework identifies and comprehensively defines three key elements: the candidate's cognitive and linguistic resources; the test system, which reflects social and performance variables of the tasks; and the scoring system, which is theoretically linked to the two other elements of the framework. The framework incorporates both the social and cognitive aspects of language and the interaction between the test taker and the test system, including both test tasks and test administration (O'Sullivan, 2008). This approach also means that scoring systems are theoretically robust, whilst the detailed definition of the framework elements means they are also operationalisable.

The framework focuses on all three aspects of the assessment process (test taker, test instrument, test scoring), and as a result it is highly sensitive to context. O'Sullivan's research argues that this is crucial (2011a) and also results in tests being localised, and specifically appropriate for the populations and skills being assessed (O'Sullivan, 2012). Moreover, this attention to context means the framework is highly adaptable. Not only can it be used to test different facets of English language competence, it has also been used to assess subject specific material.

References to the research

Indicators of quality:

The significance of this research has been recognised through the commissioning of projects and funding of research by organisations including the British Council (£280,000), City and Guilds (£42,500), and the International Council of Ophthalmology (£40,000).

Details of the impact

The socio-cognitive framework has been foundational in the benchmarking of existing tests and test design on an international scale. The framework, initially developed by Weir and O'Sullivan at the University of Roehampton, has continued to evolve as it is operationalized through commissioned research projects undertaken by CLARe (Centre for Language Assessment Research at the University of Roehampton) and CRELLA (Centre for Research in English Language Learning and Assessment at the University of Bedfordshire). Since 2008, O'Sullivan and CLARe have undertaken a total of ten research, validation and development projects in over a dozen countries. Projects underpinned by the framework have include a range of leading examination bodies, governments, institutions and organisations, all of which have had associated benefits for candidates taking the tests.

1) Impact on testing bodies, organisations and test takers since 2008:

In the globalised economy increased mobility and English language skills are desirable in a range of contexts. Through the application of the socio-cognitive framework alternatives to expensive and generic international tests are seen as crucial, in order to assess the subject-specific and transferrable skill sets.

One of the earliest applications of the research was the EXAVER project (2005-8). Working with a team from Universidad Veracruzana, and initially in partnership with Cambridge ESOL, the project led to the development of a full suite of language tests. In 2008, O'Sullivan took over management of the project and substantially refined the foundations of test development using, and in the process refining, the socio-cognitive framework. The Secretariat of Public Education in Mexico now accredits the tests for national use, and the impact of the research was two-fold. Firstly, the tests were made culturally appropriate for the people of Central and South America, and secondly, they cost less than one fifth of equivalent assessments making them much more accessible to local people.

The development of an advanced level test for the International Council of Ophthalmologists (2009-10) is one example that demonstrates the impact of the framework and the associated benefits for test takers. Through the research, the test was developed for specific-subject knowledge in English, and tested particularly for both precision and accuracy, which were seen as important characteristics for professional competency. The successful completion of the test now leads to recognition as a Fellow of the International Council of Ophthalmology, and is recognised as a "revolutionary examination" (Taylor and Quilter, 2011: 2).

Growing pressures for those operating English language tests to demonstrate validity have resulted in a series of commissioned projects in which the framework has been applied to ensure that the tests are appropriate and useful to the needs of examiners and testers. Examples which have been underpinned by the socio-cognitive framework since 2008 include Cambridge ESOL, and a range of examination boards that required their tests to be theoretically linked to the Common European Framework (CEFR), in the UK, Turkey, Mexico, Taiwan and Japan. The benchmarking of City and Guilds English language tests (2008-9), in line with the CEFR, is one example that has been completed by O'Sullivan. This project critically reviewed City and Guilds examination in the light of the socio-cognitive framework, reworking test specifications and empirically benchmarking against CEFR criteria. This was vital for the recognition of these exams. Moreover, the reports, recommendations and reviewing framework provided the expert advice required to navigate the changing policy environment and steer the continuing professional development of the organization. The project is now registered with the Council of Europe as one of the official linking projects.

A further example of the application of the socio-cognitive framework is the development of the International Language Assessment (ILA), in collaboration with CRELLA at the University of Bedfordshire, for the British Council (2007-9). This innovative assessment tool measures levels of English so that learners can continue their language learning at the appropriate stage. Not only does it ensure that students are able to register for the appropriate level of English language instruction, but it is also an affordable test. In the Ukraine, for example, it costs less than £4 (50UAH). O'Sullivan continues to work on the ILA, monitoring the test and conducting analyses of results by country as requested. The 300,000+ individuals who have taken the ILA test, to date, indicate the reach of this impact.

2) The development and delivery of APTIS with the British Council:

Since 2012 O'Sullivan has worked with the British Council as the Senior Advisor for English Language Assessment. O'Sullivan is driving the application of the socio-cognitive framework on a much wider scale and innovation in the development of a new testing system. The significance of the research is demonstrated by the British Council's recognition of the framework as `the most influential test validation theory in modern assessment' (http://www.britishcouncil.org/exams/aptis/research). O'Sullivan's version of the socio-cognitive framework has now been formally adopted by the British Council and is currently being developed for use in teaching as well as in relation to assessment. This research has had a significant impact on the British Council, as the organisation now has an agreed approach to language assessment, which is underpinned by a coherent philosophy and approach built on the socio-cognitive framework. This has acted as the foundation stone for a new direction in the organisational approach to the development of testing products.

This change is evident in the development of a new `testing system', APTIS, which is underpinned by the socio-cognitive framework. This is the first time the British Council has developed its own testing products, rather than acting as the distributor of other products. As a result there have been tangible economic benefits, as the product has created a significant new revenue stream. The APTIS service is a business-to-business product that is both flexible and accessible providing the British Council with an innovative and highly marketable English Language testing system. Because of its innovative modular structure, adaptable delivery system and customizable content, the APTIS service is appropriate for a wide range of contexts. Launched globally in August 2012, APTIS helps organisations benchmark and assess the English language skills of their employees, prospective employees, students and teachers. The test can also be used to conduct `language audits' to identify language training and development needs.

The benefits of the APTIS service which are underpinned by the framework are that 1) it can be completed quickly (a maximum of seventy-two, and normally twenty-four hours) and, 2) it is affordable (£30 for the four skill test), which means that organisations are able to bear the cost. It is a modular system, with 15 different configurations of four fundamental elements (reading, writing, speaking, listening). Clients can choose precisely what they need to assess and tailor components to local contexts. The system is also innovative in its delivery as it can be taken over the phone, on computer, or with pen and paper. This key development means that the test is suitable for a much wider range of applications than traditional instruments and that it can be used to access even remote and isolated communities.

The reach of this impact between the launch of the product in 2012 and 31 July 2013 is indicated by the 90,000 individuals who have taken the test, and its usage in over 40 countries, and a range of contexts including: a DFiD funded project in Rwanda, to assess English teachers' baseline competencies; the Malaysian Education Ministry, for the development of English language assessment and teacher training; Adecco in Poland, for recruitment purposes; the Universidad Católica de Oriente in Columbia to evaluate graduating students' English; Ciência Sem Fronteiras (Science without Frontiers) Brazil, as a gateway to this programme, which facilitates the international training and mobility of science students; Petrobras, Tanzania, to assess training needs; the Ministry of Foreign Affairs, Republic of Kosovo, for recruitment of foreign diplomats and consular staff; the Universidad de Alcalá de Henares, in Spain, to evaluate students' English. In Egypt, it is used by Pfizer and KPMG, in Hungary by MPS Group Henkel and in Brazil by Siemens.

The reach of the APTIS product continues to grow, and will also contribute to the refinement and development of the socio-cognitive framework.

Sources to corroborate the impact

i. Exaver Project:

Abad Florescano, A., O'Sullivan, B., Sanchez Chavez, C., Ryan, D E, Zamora Lara, E., Santana Martinez, L.A, Gonzalez Macias., M. I. Maxwell Hart, M., Grounds, P. E., Reidy Ryan, P., Dunne, R. A. and Romero Barradas, T de E. (2011) Developing Affordable `local' tests: the EXAVER project. In Barry O'Sullivan (ed.) Language Testing: Theory and Practice. Oxford: Palgrave, 228-43.

http://www.uv.mx/exaver/

http://www.cenni.sep.gob.mx/pdf/U.Veracruzana/EXAVER_1.PDF
http://www.cenni.sep.gob.mx/pdf/U.Veracruzana/EXAVER_2.PDF
http://www.cenni.sep.gob.mx/pdf/U.Veracruzana/EXAVER_3.PDF

ii. International Council of Opthalmology:

Taylor, D, O'Sullivan, B and Quilter, N (2009) Advanced International Ophthalmology Examination proposal. London: International Council of Ophthalmology

Taylor, D, and Quilter, N. (2011) "Guest Editorial: The International Council of Ophthalmology Examinations" Iranian Journal of Ophthalmology 3(3): 1-3.

http://www.icoexams.org/exams/advanced/

iii. City and Guilds:

O'Sullivan, B. (2009a). City & Guilds Communicator Level IESOL Examination (B2) CEFR Linking Project Case Study Report. City & Guilds Research Report.

O'Sullivan, B. (2009b). City & Guilds Achiever Level IESOL Examination (B1) CEFR Linking Project Case Study Report. City & Guilds Research Report.

O'Sullivan, B. (2009c). City & Guilds Expert Level IESOL Examination (C1) CEFR Linking Project Case Study Report. City & Guilds Research Report.

O'Sullivan, B (2010) City and Guilds Preliminary and Access Level IESOL Examinations (A1 and A2) CEFR Linking Project Case Study Report. City & Guilds Research Report.

http://www.cityandguildsenglish.com/CEFR

iv. The British Council (ILA and APTIS tests):

Independent Corroboration: Head of the Assessment Services, British Council.

http://www.britishcouncil.org/srilanka-learning-international-language-assessment.pdf

http://www.britishcouncil.org.ua/en/english/book-course

http://www.britishcouncil.org/exams/aptis/what-aptis

O'Sullivan, B. (2012) APTIS test development approach, APTIS technical report.
http://www.britishcouncil.org/sites/default/files/documents/aptis-test-dev-approach-report.pdf