Summary of the impact

The research improved the design and distribution of educational tests and software, textbooks, teaching materials, qualifications, and associated guides and research briefings in mathematics education. The MaLT project test papers have achieved sales of 350,000, with 382 interactive software versions. Some 27,000 certifications have been awarded using the Free Standing Mathematics qualifications. Research has influenced courses designed to aid transition into STEM in higher education, especially 13 programmes in seven universities engaged in a HE STEM funded mathematical modelling project.

Underpinning research

The impact is based on research that took place in Manchester from 1997, with the first major research output in 2000. The key researchers (returned staff names in bold): Black (2004-date); Farnsworth (2006-2010); Hernandez-Martinez (2006-2010); Hutcheson (1998-date); Harris (2002- date); Pampaka (1999-date); Pepin (2008-2009); Petridou (2003-date); Prevett (nee Davis) (1997- date); Ryan (1996-1997); Wake (1994-2010); Williams (1984-date); Wo (2001-date). This case study is located in the Mathematics Education team within the Mathematics Education and Critical Pedagogies (MECP) Thematic Programme of Research (TPR) in the UoA25 submission.

The team have worked on a range of projects with a total funding in excess of £1.5m from the Leverhulme Trust, Nuffield Foundation, Astra-Zeneca, and the publisher Hodder Murray, and four ESRC projects known as the `Transmaths Projects': Mathematics learning, identity and educational practice: the transition to higher education (RES-062-23-1213), Mathematics teaching and learning in secondary schools: the impact of pedagogic practices on important learning outcomes (RES- 061-25-0538), Mathematics learning, identity and educational practice: the transition into post- compulsory education (RES-000-22-2890); a follow on project, Promoting participation and engagement in post-compulsory mathematics education for STEM (RES-189-25-0235); an ESRC seminar series Developing a `how things work' research agenda in education (RES-451-26-0576); plus two ESRC funded doctorates and two ESRC post-doctoral fellowships.

The aim of the research is to understand how mathematics can be learned and assessed for understanding and `use'. This has two connected strands. The initial strand of research demonstrated how diagnostic assessment and summative, age-standardised measurement could be integrated into mathematics assessment (for ages 5-14) [3.1, 3.3, 3.4]. This research benefited from ESRC doctoral and post doctoral research projects (Petridou) which validated the constructs [3.3]. The research involved developing new assessment tools usable by classroom teachers in order to improve assessment for learning. The research required to do this involved scaling and validating test instruments that are modified from items designed for research into the psychology of learning mathematics; this included trialling modifications in classroom application, and test standardisation, using new methods of validation through models of `person misfit' [3.3].

The second strand of research is focused on `Mathematics in Use' [3.2] and was funded by Leverhulme (2000-2002), with further research and development supported by the Nuffield Foundation (2003-7). This involved introducing `real world modelling' into `Use of Mathematics' qualifications and assessment. The validity of these developments was subsequently researched and evaluated as part of the four aforementioned ESRC (2006-11) projects which showed how students were more persistent on `Use of Mathematics' courses largely because of the opportunity to learn through modelling assessment. These findings have in turn led to a second wave of research (and impact) on post- 16 mathematics education, including university programmes, known as the Transmaths Projects (Transmaths.org). A key finding was that teacher centred mathematics is associated with declining student dispositions to study more mathematics: learner centred mathematics (of the sort promoted in `modelling' courses) is associated with slower declines in students' dispositions to study mathematics [3.5]. Findings show that students with lower grades (B and C) in GCSE Mathematics are significantly less likely to survive the traditional AS qualification than those following the `Use of Mathematics' programme, largely due to the assessment and curriculum design (i.e. not the pedagogy per se); on the other hand we also showed that students with very high A level grades now often struggle with the mathematics they are expected to understand in first year university courses in STEM, which we showed is associated with dysfunctional learning in traditional A-level mathematics courses [3.4, 3.5].

References to the research

(AOR- Available on request)

The high quality of the research is evidenced by peer review of competitive grant applications, the monitoring by project Advisory Groups and, if applicable, final report grades. The ESRC grants have been graded (good, very good, outstanding, the fourth review is pending). The research has been published in leading international journals for research in education.

Details of the impact

Context: Traditional programmes in mathematics in schools, colleges and universities involve teacher/lecturer-centred delivery, with timed and summative test-centred assessment, leading to a high drop out from mathematics at age 16, 17, and later. The impact of the programme of research by Professor Williams and his team involves changes to professional practice, learner approaches to learning, and improvements to the context and design of commercial texts.

Pathways to impact. Following the approach outlined in REF3a this case study illustrates both an instrumental change model based on the direct reporting of findings and a constructivist approach through working with stakeholders in schools, colleges, higher education, and publishing companies. This enabled both the translation and transmission of findings, and attention to research legibility enabled the use of those findings with various interested groups. User engagement was furthered through networking and the building of partnerships in the production of e.g. textbooks, and the development of new examinations. Specific pathways to impact include the:

(a) development of new assessment tools for Schools and Colleges via the Hodder-tests, MaLT project, together with the marketing and sales provided by Hodder Murray publishers;

(b) real world mathematical modelling research was used by the researchers in partnership with external organisations in the design of the new `Use of Mathematics' qualifications and assessment materials and teaching texts, and shaped the assessment and written support materials published in texts and on websites to support teachers (partnerships with and funded by AQA and Nuffield Foundation);

(c) Transmaths projects (2006-11) — and of particular relevance is the fourth ESRC `Follow-on' project which is devoted to social impact of the three earlier research projects — researched the effect of the new qualifications and teaching of `modelling' in `Use of Mathematics' and other qualifications in FE and in transition to HE. The findings from this work have been widely disseminated through the relevant professional bodies, notably the NCETM, and HE-STEM networks, inter alia, through a set of research briefings and edited HE-STEM books, and this has been confirmed by the former Director of the National HE-STEM programme [5.9]. This activity involved the research team in collaborative partnerships with professionals and publishers in originating new texts, the writing of qualifications and assessments (e.g. via QCA, AQA, and others) and the development and evaluation of new resources and courses. This has led to an improvement in the base of resources and qualifications and assessments that teachers and institutions can use to help provide for better education and to widen participation. Further research also has provided some supporting evidence and evaluation of this impact.

The productive nature of the interactions regarding the use of research evidence is illustrated by a witness statement: "at QCA I was the project leader for the development of Free Standing Mathematics Qualifications. The contract for development of these qualifications was awarded to the team at Manchester under Professor Julian Williams and his then colleague, Geoff Wake from your department. Geoff Wake developed the actual FSMQ units and the corresponding Level 3 textbooks that were funded by the Nuffield Foundation. These FSMQs have proved hugely influential. Professor Williams was also commissioned to write a special report as evidence to Professor Adrian Smith's inquiry (commissioned by the government) on post-14 mathematics education" [5.3].

Significance and reach. The impact of research on new educational provision is significant through the novel approaches to learning, and reach is evidenced in the take up of new qualifications and access to new texts:

(a) Materials were developed for classroom use through the MaLT project, and published by Hodder Murray [5.1]. The Hodder Murray website states that: "a tried and tested approach to standardised mathematics assessment for ages 5-14, developed for Hodder Education by the University of Manchester. Nationally standardised on over 12,500 pupils, MaLT can be used at any time during the school year — for screening, monitoring and tracking progress, as well as for individual diagnostic profiling. Available in both pencil-and-paper and interactive computer-adaptive (CAT) formats, these nationally standardised tests — for groups or individuals - also yield diagnostic information which will directly support individual and whole-class teaching" [5.1]. A representative from Hodder Murray confirms the quality and impact of the research: "the Mathematics Assessment for Learning and Teaching (MaLT) series developed for classroom use by Professor Williams and his team offers a ground-breaking approach to Mathematics assessment. The tools are designed to support better teaching and learning for understanding and using mathematics, and hence impact on how learners learn" and he goes on to evidence the wider commercial impact during the REF2014 audit period: "that Hodder Education sales (from publication in 2005 to June 2013) of printed test papers are approaching 350,000 copies, with a total value (including supplementary manuals and scorer/profiler CDs) of £534k. In addition, we have to date sold 382 interactive versions (value £118K) for pupils to take the tests on stand-alone computers and school networks" [5.2]. These innovative tools afford improved assessment practices; the detailed profiling of pupils test responses and specific diagnostics can support better teaching and learning [5.2].

(b) The real world mathematics modelling research was funded by Leverhulme and the Nuffield Foundation, and led to impacts on: (i) the development of new qualifications (Free Standing Mathematics Qualifications) with confirmation that "from 2000 to 2010 the figures for all levels 2 and 3 FSMQs totalled some 53,000, including permitted aggregations into the pilots of AS and A level Use of Mathematics, which were built up of some core level 3 FSMQs together with a synoptic final examination paper... they (i.e. the qualifications) have played an important role in influencing the thinking of policymakers about less academic routes of mathematical progression form level 2 to level 3" [5.3]. Additionally "the significant impact of this work is evidenced by the number of certifications in FSMQs and `Use of Mathematics' qualifications each year. In 2013 across all the units that have been developed at the three major levels of the National Qualifications Framework (i.e. Levels 1, 2 and 3) there were over 27,000 certifications awarded" [5.4]. The witness from QCA confirms that "unfortunately, the present government decided, for political reasons, not to continue with the Pilot A level Use of Mathematics" but he gives an account of how the government is reviewing qualifications, and states: "there is no doubt that the methods pioneered by the Level 3 FSMQs will still play a significant role in the discussions that are about to take place" [5.3]; (ii) the development of new resources accessible from the Nuffield website [5.5] where "the Nuffield Foundation supported the development at all levels by providing support materials at a bespoke website...The Foundation continues to support these qualifications in their drive to support the application of mathematics across disciplines" [5.4]. The research continues to influence professional development work for the development of the `Uses of Mathematics' A-level (A2) qualification, piloted in 2011-12 and rolling out in 2013; and, (iii) the production of new textbooks by Haighton, Howarth and Wake: AS: The Use of Maths Calculus (2004); AS: Use of Maths Algebra and Graphs (2003); and, AS: The Use of Maths Statistics (2003) (Nelson Thornes), and "these texts continue to sell in significant numbers even some ten years after their first publication, averaging some £14,000 worth of sales per year" [5.4].

(c)The fourth ESRC, `Follow-on', project was devoted to social impact of the three earlier ESRC funded research projects. This `Follow-on' project developed materials to disseminate the research findings and transmitted knowledge about the effect of the new qualifications and teaching of `modelling' in `Use of Mathematics' and other qualifications/courses in FE and in transition to HE. The findings from this work have impacted on members of relevant professional bodies, notably HE-STEM networks, through a set of research briefings [5.6] and edited HE-STEM books and other materials [5.7]. The Transmaths research influenced development work on modelling in new post-16 (AS and A2) and first year university curricula. This work is currently impacting on HE-STEM `Mathematical modelling and problem solving' courses currently being taken up under HE-STEM funding in seven universities, including 13 STEM Programmes in Mathematics, Physics and Chemistry, and several engineering schools [5.8, 5.9, 5.10]. Importantly the impact of the research is recognised by users in higher education: "the substantive results of the Transmaths projects showed how student learning of mathematics can be significantly affected by `connectionist' pedagogy, which has been important in our development of more active learning and problem solving in first year courses in science, technology, engineering and mathematics (STEM)" [5.8]. Another research user adds that "we have been able to make the case at policy level for investing in the development of Mathematical Modelling and Problem Solving (MMPS) courses to aid transition into STEM across the University of (name) in the Schools of Physics, Chemistry, Civil Engineering, Electrical Engineering, Mechanical Engineering and the Department of Applied mathematics. These programmes involve between them nine academic staff and approximately 700 first year students...this work has also been taken up by 7 other project partners from universities across the HE sector. Finally, it has to be emphasised that the main impact of this work is in helping to bring about a sea-change in HE policy towards courses that deliver key skills. We are increasingly looking for evidence and case studies showing how such skills can be taught within mainstream academic disciplines. The Transmaths research has hit this nerve well and in a timely fashion." [5.9]

Sources to corroborate the impact

All claims referenced in text

[5.1] http://www.hoddertests.co.uk

[5.2] Witness statement from Hodder Murray.

[5.3] Witness statement from QCA.

[5.4] Witness statement from higher education.

[5.5] http://www.nuffieldfoundation.org/free-standing-mathematics-activities-fsmas-2002

[5.6] Transmaths and TLRP Research Briefings, on project website: www.transmaths.org

[5.7] Grove, M. and Williams, J. (eds) (2013) The Transition into Higher Education in STEM Subjects: Case Studies of Research and Practice, Birmingham: National HE STEM Programme, 2013. eScholarID:194604

[5.8] Witness statement from the former Director of the National HE-STEM programme.

[5.9] Witness statement from higher education.

[5.10] Croft, A.C., Grove, M.J., Kyle, J., and Lawson, D.A. (eds) (2014) Transitions in Undergraduate Mathematics Education, London: Higher Education Academy.