Summary of the impact

Teaching and learning of computer architecture has been enhanced using highly interactive simulations with carefully constructed visualisations and animations. Computer scientists need to understand and observe how different parts of a modern computer system's architecture and organization fit together, interact and support each other. Unique educational simulation software has been designed, developed and evaluated with these requirements in mind. Since the software and teaching materials have been made public, numerous universities worldwide adopted it in their courses with claimed positive impact on student engagement, course popularity, grades, speed of delivery of curriculum, attendance and peer recognition of best practice.

Underpinning research

The research has two dimensions: 1) Design and integration of visualisation techniques for animating, tracing, controlling and highlighting software and hardware algorithmic mechanisms of modern computer architectures and 2) Design of suitable teaching and learning materials and the evaluation of the educational value of the visualisations using these materials.

So far there has not been a clear consensus on what constitutes good and educationally valuable computer assisted simulations and visualisations that can help engage and significantly enhance the learning experiences of students of modern computer architecture at higher education level. This research has been prompted by a need for an educational resource suitable for supporting the delivery of lectures and practical sessions in Computer Architecture and Operating Systems modules at Edge Hill University. It was undertaken by the module leader, Mustafa, who joined Edge Hill University as a Senior Lecturer in the Department of Computing in 2004 (where he was employed throughout the assessment period). It resulted in the development of educational simulation software that modelled and embodied the essential key features of typical computer systems. Besim Mustafa started the development of software in 2006 at Edge Hill University and a working version was integrated in the above modules during 2008. With the help of funding from HEA the initial evaluation of the educational value of the simulations was carried out, which resulted in preliminary encouraging results. In 2009 this work was made public through a web site dedicated to disseminating the results of the research by making the software freely available for educational purposes. Between 2009 and 2013 Mustafa carried out further research that resulted in the publication of eight papers presented in international conferences, samples of which are identified in section 3 below. During the past three years this research concentrated on and sought to identify and evaluate features of simulation and visualisation techniques that enhanced student engagement and deep learning. The results provided valuable feedback that is used to improve the educational value of the software.

The research work is built on the foundations laid down by two taxonomies as its theoretical bases: Engagement Taxonomy¹ and Bloom's Taxonomy². The former concerns itself with the attributes of visualisation and simulation software at different levels of engagement; the latter identifies levels of learning from surface to deep learning. The two taxonomies are used as the guiding principles in the development of the educational tool and the associated teaching material, their evaluation and the application of feedback to improve the tool. The research also investigated the unifying themes between the two taxonomies and used this work to offer a formal methodology for evaluating computer software and hardware algorithms using simulations and visualisations. There is no known record of research in this area and the researcher therefore claims some degree of originality in his work.

¹ Naps, T.L., Fleischer, R., McNally, M. et al (2003) Exploring the Role of Visualization and Engagement in Computer Science Education. ACM SIGCSE Bulletin 35 (2), June 2003.

² Bloom, B.S., Krathwohl, D.R. (1956) Taxonomy of Educational Objectives: the Classification of Educational Goals, Handbook I: Cognitive Domain. Addison-Wesley.

References to the research

Initial research in 2009 was funded by The Higher Education Academy (awarded to Besim Mustafa; "Evaluation of a system simulator as an effective learning tool in undergraduate computing modules in computer architecture and operating systems"; December 2008 - August 2009, £2,746). The following three peer reviewed conference publications are deemed to be 2* quality, as indicated by their acceptance by ACM (Output 1), IEEE (Output 2 and 3) and Springer (Output 3). All are submitted in REF2 and are available on request.

Details of the impact

Teaching and learning of computer architecture has been enhanced using highly interactive simulations with carefully constructed visualisations and animations.

The teaching and learning of computer organisation and architecture is an essential part of the education of computer engineers and computer scientists. Most degree level courses in these areas offer modules in computer architecture and operating systems as identified and recommended in a joint report on Computer Science Curricula (Other Source 1) by ACM and the IEEE Computer Society. However, these areas have traditionally been relatively difficult to teach and complex to grasp by the students through passive teaching methods, experimenting using commercial products and developing software requiring moderate to high programming expertise. As a result tutors turned to using simulation and visualisation methods of demonstrating the architectural mechanisms and algorithms to their students.

Much algorithm simulation software has been developed over the past few decades and these have achieved some success in demonstrating the inner workings of isolated computational algorithms. Nevertheless there has not been a consistent method of assessing the requirements for simulating and visualising complex systems that can both engage and provide deep learning experiences for the students of computer architecture. It is precisely this aspect that this research addresses: methods of simulating, with the help of appropriate visualisations and animations, the workings of a modern computer system and demonstrating the interplay between its different components in ways that provide a rich, consistent and engaging learning environment from basic to more advanced levels.

The research work has resulted in a unique set of tightly integrated simulations in a single software package with related teaching materials and eight conference publications mainly reporting on the work done to improve and evaluate the pedagogical value of the simulations with funding support from the Higher Education Academy. A dedicated website (Other Source 2) was established in 2009 in order to help widely disseminate this work and make the software freely available. Additionally, a reference to this site was deposited (2011) in the learning materials repository Jorum (Other Source 3) which is a JISC-funded collaborative venture to collect and share learning and teaching materials and is based at the University of Manchester. This helps to direct users to the project website, as do the deposits in the following repositories:

• WWW Computer Architecture Page (Other Source 4), maintained by a group from Computer Sciences Department at University of Wisconsin-Madison (USA) — deposited in 2010.
• MERLOT: Multimedia Educational Resource for Learning and Online Teaching (Other Source 5), which is a program maintained by California State University (USA) in partnership with higher education institutions, professional societies and industry — deposited in 2010.

It is clear that this publicly and freely accessible presence on the Internet afforded maximum reach to educational institutions, mainly universities and colleges of higher education, and individuals worldwide. There is clear evidence that the simulation software has been and continues to be in use by tutors in several higher education institutions in various parts of the world. In a recent survey conducted by Mustafa on the reach and impact of the simulation software on their teaching and their students, tutors from five universities clearly identified the impact on their students between 2011 and 2013. The table below is a summary of the results of this survey demonstrating the estimated annual reach of the impact in these five universities (total annual number of students 775):

Country	Institution	Course(s)	Academic Level(s)	No of Students (Annual cohorts, est.)	Tutor Position
Iran	University of Tehran	Operating Systems.	UG	45	OS Lab Administrator
Iran	Hamedan University of Technology	Operating Systems.	UG	25	University Lecturer
Scotland	Edinburgh Napier University	Computer Systems. Systems & Services.	SQF level 7 SQF level 8	120 160	Senior Lecturer
Taiwan	National Chiao Tung University	Computer Organization.	Junior College	65	Teaching Assistant
India	PVP Siddhartha Institute Of Technology	Computer Organization. Compiler Design. Digital Logic Design.	3rd year UG 3rd year UG 2nd year UG	120 120 120	Associate Professor

The tutors from the above five universities also identified the following as evidence of impact on their teaching and on their students: peer recognition of best practice in delivering the curriculum, reduction in time required to deliver the curriculum, improvement in student attendance, improvement in general module/course satisfaction, improvement in student learning experience, increase in popularity of the module(s)/course(s), increase in student engagement and improvement in student grades. Please see Section 5 for details of corroborators. There is also clear evidence that these impacts are on-going as the same tutors stated that they intended to continue using the software in their teaching and requested additional support.

Additionally, there is strong indication (via correspondences and references to the software's web site from external VLEs) that other educators in a growing number of countries such as Chile, USA, Brazil, Canada, UK, Serbia, Australia and Sri Lanka have also been using the software in their teaching. Furthermore, according to the software's web site statistics there has been an average of 1200 hits per month from over 100 different countries and an average of 50 downloads per month over the past 12 months, rising sharply to 350 in the months leading to August 2013. According to published site statistics on the Jorum Site (Other Source 6) the software has been viewed an average of 20 times per month over the past 6 months since the start of collection of their new statistics (March 2013).

Sources to corroborate the impact

The following are the sources of corroboration and the claims they can corroborate. This information has been gathered using a survey of the tutors who indicated through correspondence that they have been using the simulation software in their teaching.

Contacts

1. National Chiao Tung University, Taiwan
   Issues addressed: Peer recognition of best practice in delivering the curriculum; Reduction in time required to deliver the curriculum; Improvement in student attendance; Improvement in general module/course satisfaction; Improvement in student learning experience; Modules in which the software is used and annual cohort numbers.
2. Lecturer, Hamedan University of Technology, Hamedan, Iran
   Issues addressed: Reduction in time required to deliver the curriculum; Improvement in student attendance; Increase in popularity of the module(s)/course(s); Improvement in student learning experience; Increase in student engagement; Modules in which the software is used and annual cohort numbers.
3. Senior Lecturer, Edinburgh Napier University, Edinburgh, Scotland
   Issues addressed: Improvement in student learning experience; Increase in student engagement; Modules in which the software is used and annual cohort numbers.
4. OS Lab Administrator, University of Tehran, Tehran, Iran
   Issues addressed: Reduction in time required to deliver the curriculum; Improvement in student grades; Improvement in student learning experience; Modules in which the software is used and annual cohort numbers.
5. PVP Siddhartha Institute of Technology, India
   Issues addressed: Peer recognition of best practice in delivering the curriculum; Reduction in time required to deliver the curriculum; Improvement in general module/course satisfaction; Improvement in student learning experience; Modules in which the software is used and annual cohort numbers.

Other Sources

1. Computer Science Curriculum 2008: An Interim Revision of CS 2001. Report from the Interim Review Task Force. Association of Computer Machinery (ACM) and IEEE Computer Society, December 2008. Available on request.
2. www.teach-sim.com
3. www.jorum.ac.uk
4. http://arch-www.cs.wisc.edu
5. http://www.merlot.org
6. Website usage report for www.teach-sim.com — available on request.