Summary of the impact

Key areas of education policy and practice have been influenced by the work of Professor Underwood and her team:

• investigating the impact of digital technologies on user behaviours, particularly (but not exclusively) learners, providing robust evidence of effective use of technology for policymakers;
• acting as a change agent within a key stakeholder organisation: British Educational Communications and Technology Agency (Becta);
• developing tools to capture the complexity of organisations at various stages of technology innovation;
• contributing to the development of a self-assessment tool to improve professional practice;
• contributing to national guidelines on the use and abuse of technology.

Underpinning research

Professor Jean Underwood is an established expert on the impact of digital technologies on behaviour in general and learning in particular. She has added significantly to our understanding of technology acceptance, how technology enhances cognition and language, group communication and interaction and facilitates misdemeanours [References 1 & 2]. Underpinning research here highlights two aspects of Underwood's work.

Capturing the impact of technology: Becta, a non-departmental public body funded by the Department for Education, commissioned Underwood to develop tools to provide robust evidence on the impact of technology to inform the development of strategy and policy designed to improve outcomes for learners and the educational system as a whole. Historically, significant advances in technology have not always brought about measurable shifts in user behaviour even where potential users register positive attitudes.

Recognising that organisations are complex systems of interrelationships where context is not a neutral backdrop, work focused on the interplay between learners and schools through the lens of organisational psychology. Underwood, after Lesgold (2000), developed and tested maturity models to systematically capture the impact of structural and individual learner factors on performance as measured by national standard scores [Reference 3]. This work was initiated in the Test Bed project (2002-06) and was core to two further projects (IMPACT 7 and 8). Building maturity models involved an iterative cycle engaging a range of stakeholders - teachers, educational managers, assessors and policy-makers — thus ensuring the face validity of the tools.

A unique longitudinal four-year study of 24 institutions, 700 staff and 6,000 students provided a more holistic understanding of the impact of technology-rich environments on learner outcomes than previously possible. From this the concepts of e-maturity and institutional maturity were developed which, alongside learner investment in the learning process, produced a simple but powerful predictive model: Opportunity (e-maturity, institutional- maturity) + Learner Investment = Effective Learning.

The maturity model provided the first direct and quantifiable evidence of the "Technology Dip"; the initial decline in performance when technological innovation is introduced into a work setting. Structured self-assessment by schools mapped this dip and identified factors to ameliorate the problem. As a result, schools drawn from educational sink areas when the project started were performing above the national average four years later [Reference 4].

The basic model was adopted and developed by Becta into the Self-Review Framework used by schools to self-assess their progress and effectiveness. It was first distributed in 2006 and updated in 2010.

Academic Dishonesty: It has been argued that the use of digital technology makes academic dishonesty easier. Underwood's initial research in this area examined how groups work [Reference 1]. She established factors that facilitate or impede effective group work with technology and, importantly showed how the technology changed behaviours, supporting collaborative problem solving but also academic malpractice. She moved to systematically examine the prevalence, risk factors (individual and social) and characteristics of malpractice [References 5 & 6]. This early work led to the Qualifications and Curriculum Authority commissioning a report on the extent, causes and potential solutions to academic dishonesty.

References to the research

Grants awarded in open competition (Underwood PI unless stated otherwise)

1998-2000 Funder ESRC: Designated Research Centre for Development, Instruction and Training (c. £2 Million) Professor David Wood, University of Nottingham (PI), Underwood designated holder of sub-contracted grant includes 2.6 years funding after moving to NTU (£45K per annum).

1998-2001 Funder EU, MEDIAKIDS: an investigation of children's use of multi-media tools Professor Antonio Bartolome, University of Barcelona PI (428,820 Euros).

2003-06 Funder: DES /Becta Test Bed project: a longitudinal study of the impact of technology on school and student performance (£900K), Professor Somekh, Manchester Metropolitan University, and Underwood joint PI (£900K).

2006-07 Funder: Becta Impact 2007: Personalising Learning with Technology (£200K)

2007-08: Funder: Becta Impact 2008 (£185K).

2008-08 Funder Becta: Narrowing the Gap: an investigation into under achievement and school failure (£147,000).

Details of the impact

Public policy and services

1. Impact on public policy and services within the UK

In 2009/10 Underwood was seconded half-time to Becta [Evidence 1a] with the dual goals of (i) developing a research agenda to provide robust evidence of the impact of technology for learning, and (ii) acting as a catalyst for research skills development within this government agency. "Latterly, Underwood's major contribution to Becta's work was in the impact arena. Because of her ability to speak the languages of both research and policy, we requested a secondment for her into the evidence team" [Evidence 3b].

Reach: Her report on The Impact of Digital Technology [Evidence 1b] was circulated nationally by Becta and presented to more than 150 stakeholders and policymakers at the then Department for Children, Schools and Families (DCSF). Her reports on using technology to personalise learning (Impact 2007) and "Narrowing the Gap" informed discussions within the DCSF (2007-10) and were instrumental in convincing the Department that Becta could take on significant work in this area [Evidence 3a]. A pack highlighting Underwood's work on the "Narrowing the Gap" Project focusing on the national issue of low and underachieving adolescents as they prepare for life beyond the school gate, was distributed to English and Welsh schools by Becta.

Effect: Underwood was a key figure in bringing an understanding of the value of quantitative research methods to this government agency. Her secondment was used "to up-skill less experienced team members, provide a forum for professional debate amongst more experienced staff ... around the potential impact of technology. This needed a strong, independent authorial voice that had credibility with both the research and policy communities" [Evidence 3b].

2. Practitioners and services:

Maturity Modelling: Underwood's construction of a suite of maturity models to capture organisational behaviours (see section 2, Evidence 2a) underpinned the development of the Becta's Self Review Framework [Evidence 3a, 3b]. As part of the ICT Test Bed project, a flagship longitudinal study of factors in the use and impact of technology in schools, "Professor Underwood developed a series of maturity models which proved highly effective in analysing change. Becta developed the model as a self-assessment tool for school leaders and, later, for teachers and support staff" [Evidence 3a].

Reach: This tool had "4,500 users within the final 18 months of its availability." [Evidence 3b]. Warwick University's independent assessment [Evidence 2b] reported:

• 17,000 schools registered to use the Self Review Framework, an average uptake of 50% of schools in each region of England and Wales;
• Usage statistics showed at least 72,041 site visits [p. 6].

Effect:

• "The response to the Self Review Framework was overwhelmingly positive" [p. 4]; 83% of respondents (N=788) acknowledging the relevance of the tool [p. 15]
• "The process of going through the Self Review Framework was often deemed to be of greater value than achieving the ICT mark - the mark may have been a catalyst to doing the framework but the work itself was important." [p. 4]
• "Professional discussions arising from the process were highlighted as a positive result of using the Self Review Framework" [p. 4] with 73% of respondents stating that the tool had challenged their practice [p. 18]
• "I think it's a really useful tool - it throws up the actions you need to take." [p. 38]
• "It helped me with training. The framework helped me to see what we need to do." [p. 41]

The Self Review Framework sits on the National Archive and National Association of Advisors for Computers in Education sites [Evidence 2b].

Solutions to Academic Dishonesty: Underwood spearheaded the Qualifications and Curriculum Authority's mapping of the prevalence, characteristics and potential solutions to such malpractice.

Reach: Underwood's research on plagiarism is quoted in OfQual"s guidance to teachers in England and Wales. The key to preventing dishonest practice is the development of a culture of honesty that encourages and promotes original work through "Reducing the opportunities to cheat, developing environments where cheating is unacceptable and reducing the acceptability of cheating within any institution should be the foci of a programme to reduce malpractice" [Evidence 4a, p. 3].

Effect: As a consequence of the original QCA report and the OfQual Guidelines other stakeholders have accessed Underwood's expertise, including:

• The UKCCIS Better Education e-Safety Academic Group, which investigated young people's understanding of digital media risk. [Evidence 4b]
• TQA's Masters in Teaching and Learning Writing Group developed a new professional qualification for teachers in their first five years
• The BiERightOnline strategy for The United Church Schools Trust/United Learning Trust when developing practical ways of reducing online malpractice.

Sources to corroborate the impact

1. The impact of Underwood's research on policy

a. Her secondment to Becta (June 2009-March 2010) as "Manager/ Research Analyst, Research Reviews and Dissemination," is clear existence proof of the value of Underwood's work in this field:

b. "The Impact of Technology" (2009) [http://www.ictliteracy.info/rf.pdf/impact-digital-tech.pdf] http://publications.becta.org.uk/display.cfm?resID=41343 http://www.ictliteracy.info/rf.pdf/impact-digital-tech.pdf

2. Evidence of the impact of the maturity model on professional development is evidenced through:

a. Underwood, J. D.M. & Dillon, G. (2007). ICT Test Bed | Maturity Model Analyses: Year 4, 2006. Technical Report 20, 2006 (T20/06) 01 June 2007 http://dera.ioe.ac.uk/1588/1/becta_2006_icttestbed_maturitymodel_report.pdf

b. The Self-Review Framework is now housed in the National Archive http://webarchive.nationalarchives.gov.uk/20101102103713/ https://selfreview.becta.org.uk/ Further information about the tool and its associated kite mark, the `ICT Mark', can be found here: http://www.naace.co.uk/ictmark.

c. The University of Warwick's Self Review Framework Evaluation Report is not in the public domain but has been supplied by Head of the Technology Policy Unit, Department for Education in support of the claims made here.

3. Key stakeholders have corroborated the impact of Underwood's impact on policy and professional development within Becta and the wider educational sector:

a. Head of the Technology Policy Unit, Department for Education and formerly Director of e-Strategy, Becta

b. Former Head of Research and Analysis, Becta

4. The impact of Underwood's research on academic dishonesty is evidenced in the following reports:

a. The Office of the Qualifications and Examinations Regulator (2009). Authenticity: A Guide for Teachers. Ofqual/10/4533 (ISBN: 978-1-84962-253-0) directly quotes Underwood, J. (2006) Digital technologies and dishonesty in examinations and tests, Nottingham: Nottingham Trent University. Available at: www.ofqual.gov.uk/files/qca-06_digital-technologies-dishonesty-exams-testsreport.pdf

b. UKCCIS Better Education e-Safety Academic Group: Outcomes of the Scoping Workshop held 25^th May 2010. This report is not in the public domain but has been supplied by Head of the Technology Policy Unit, Department for Education