Summary of the impact

Europeana is the official European Union internet portal for cultural heritage located in European heritage institutions. Based on their research on user centred evaluations of information systems and Digital Library development, Strathclyde researchers were selected to evaluate the user experience of Europeana version 1.0. A series of user studies involving 89 people in 4 countries led to design recommendations based on Strathclyde's research which were incorporated into a new Europeana user interface (Europeana v2.0) leading to significant improvements in search capability and information access for users. Consequently, the latest version of Europeana is used by almost 4 million unique visitors from 241 countries who now benefit from increased user experience

Underpinning research

Context

The usability of digital heritage resources is an essential factor for attracting and retaining users. Europeana, the official European Union Digital Library for cultural heritage, provides a single access point to cultural heritage stored in Europe for specialist users of cultural heritage material and the general public worldwide. Europeana requires high usability in order to allow easy access to Europeana's content from a wide range of users including children and those with limited computer experience. The Europeana development team commissioned an independent study in order to evaluate Europeana's accessibility to ensure it was easily usable. The study had particular focus on (a) the ease of use and intuitiveness of Europeana, especially for first-time visitors to the website; (b) identification of `future' needs as the younger generation becomes older; (c) styles of use of Europeana v1.0 for knowledge discovery amongst younger individuals; (d) user expectations, including how the trustworthiness of information is assessed; and (e) similarities and differences in groups of users from different countries.

Key findings

The Strathclyde team were chosen to develop and run the evaluation study based on their extensive research experience on user-centric evaluation methodologies and digital library design:

(1) User centric evaluation methodologies. Since joining Strathclyde in 2001, Ruthven has specialised in the design of interfaces for search systems and the user-centred evaluation of novel information access systems. Ruthven has written extensively on user-centred evaluation, both on evaluations he has conducted and on the development of appropriate methodologies for evaluation, e.g. [1,2,3]. A key contribution was the focus on task-based evaluation methodologies in which experimental participants are given simulated search tasks to complete on the novel interface. This allows evaluation on realistic tasks which can give a deeper understanding of likely usability failures. This understanding of how to create tasks and interpret the results of the task performance was critical to understand usability problems with Europeana that may not be apparent from initial usage and first impressions. He has also has developed new methodologies for using eye-tracking technology within information access evaluations such as those conducted in the Europeana study. The use of eye-tracking can highlight areas where important information, such as site navigation, is missed by users and leads to better layout of user interfaces and information presentation.

(2) Digital Library design. Simultaneously, Dobreva and Graham were working in Strathclyde's Centre for Digital Library Research (CDLR) on the design and construction of new Digital Libraries for Cultural Heritage. A particular focus of CDLR's research was on approaches to develop good meta-data to facilitate end-user searching and approaches for aligning meta-data from different sources [4]. As Europeana uses meta-data from over 2000 different heritage institutions, being able to evaluate the quality of meta-data and its impact on user experience was an essential requirement for the usability study and one of the reasons why Strathclyde was chosen to conduct the study.

Key Strathclyde researchers

Dr Milena Dobreva — Research Fellow at Department of Computer and Information Sciences at time of research from 2008 - 2012; Ms Emma Graham (née McCulloch) — Research Fellow at Department of Computer and Information Sciences at time of research 2008 - 2012; Professor Ian Ruthven — Professor of Information Seeking and Retrieval in Department of Computer and Information Sciences from November 2001 — present.

References to the research

References 1 - 3 best indicate the quality of the underpinning research. Reference 2 is included in the REF2 Submission for UoA 11

Other evidence for quality of research (grants, patents etc.).

Funding for usability study obtained under competitive call. Europeana Focus Group, M. Dobreva and I. Ruthven. 15/10/09-31/12/09. £13,661. Stichting European Digital Library.

Details of the impact

Process from research to impact

In 2009, the European Commission issued a call for academic research groups to conduct an independent evaluation to assess the usability of the Europeana interface. The Strathclyde researchers named in the previous section submitted a bid along with international colleagues. The Strathclyde-led team was selected to conduct the evaluation based on their experience in the construction of Digital Libraries, their work on user-centred evaluations of information access systems and their research on the interoperability of meta-data from different resources. The Strathclyde group had the main responsibility for designing the evaluation study, its conduct, analysis and reporting. The collaborating partners were Jonathan Sykes of Glasgow Caledonian University (whose research group provided the eye-tracking technology used in the evaluation), Pierluigi Feliciati of University of Macerata (who contributed to the study design and ran one of the evaluation phases) and Yurdagül Ünal of Hacettepe University (a visiting researcher at Strathclyde who helped organise the focus groups).

The evaluations conducted consisted of 4 task-based group evaluations with school age children in the Netherlands and Bulgaria (the latter a country with low use of Europeana material and hence an area of interest for Europeana), 1 task-based group evaluation with University students on a Cultural Heritage degree course in Italy, one focus group of specialists and members of the general public in the UK, and 12 single person adult evaluations using task-based evaluations and eye-tracking methods. This was one of the first Digital Library evaluations to use eye-tracking as an evaluation technique. This range of evaluation approaches was critical in making precise recommendations to Europeana from different groups of potential users.

The evaluation report was submitted to the Europeana Working Group who led the design and operation of Europeana on behalf of the European Commission in 2009. The report detailed the methodology of the evaluations and the main recommendations in terms of improvements that could be made to the Europeana user interface, meta-data construction and data collection. It also identified problems with end-user understanding of the Europeana collections and preferred methods of accessing heritage information. The report made 22 short, medium and long term recommendations for the improvement of Europeana, including recommendations on content, branding, usability, collection development and multi-lingual support. Each recommendation was associated with suggestions on how the recommendation could be implemented. Of particular concern was the lack of precision in searching, the often poor quality meta-data supplied by end-users and confusion over the aggregator model used by Europeana.

Types of Impact

Improvements to the Europeana interface:

The immediate beneficiary of our evaluation work was the Europeana development team who gained new insights into how their system was used in practice, new design recommendations for overcoming usability issues, and how unbalanced subject coverage could lead to user rejection of the system. The recommendations made in the report led to interface improvements in the Europeana interface in version 2.0 of Europeana (Source B and D). These changes included changing the size and layout of thumbnail representations to improve their visibility and thus make it easier for users to make decisions about which objects to explore and changing the layout of the initial Europeana pages. The latter change resulted from the eye-tracking data which demonstrated that parts of the navigation provided by Europeana were not noticed by users and that first-time and returning users tended to focus on different parts of the home and search pages. These observations would not have been possible without the use of eye-tracking techniques. These changes coincided with other changes including the addition of new cultural heritage institutions to the Europeana database (Source C).

Improved user experience:

The changes to Europeana made as the results of our recommendations have led to improved user experience by all users of Europeana as the changes have resulted in better presentation of search results and easier access to relevant content within Europeana. Due to the anonymous nature of searching on Europeana — there is no registration required — we cannot breakdown which target groups have benefitted most from these changes. However, it is likely that the changes have most benefitted those groups, such as children and less frequent computer users, who typically face most challenges in operating search user interfaces.

Wider impacts: The cultural heritage institutions that contribute data to Europeana also benefit from having increased accessibility of their material through an improved user search interface and clearer recommendations on how to provide meta-data for Europeana.

As the Europeana Foundation notes, `As Europeana represents a major investment by the European Commission in making cultural heritage available to the world, it is important that Europeana is usable by a wide variety of end-users. A particular target group are children as Europeana's high-quality resources are intended to be a key cultural resource for schools across the European Union' (Source A, 2013). The study gathered opinions on the difficulties encountered which help to understand better users' expectations within the content and functionality domains of Digital Libraries which would be of possible interest to all stakeholders in developing cultural heritage repositories.

Reach and Significance

Europeana is the official European Union Digital Library for cultural heritage located in European heritage institutions. It provides access to over 26 million digitised cultural objects including books, paintings, films and audio, through an aggregator model by which institutions provide brief descriptions of objects within their collections and meta-data describing these objects. 2,200 individual heritage institutions, such as the Rijksmuseum, The British Library and the Natural History Museum, from 34 different countries, contribute to this resource. End-users can search in any of 31 languages and are then presented with descriptions of the matching objects along with the meta-data and links to the institution that contains the cultural heritage object. Launched in 2009, the site now receives almost 4 million unique visits per year from over 240 different countries.

Through helping to provide a more usable interface, we have contributed to "Europeana's mission of making cultural heritage more accessible and increasing engagement with the cultural heritage of the European Union" (Source A). The changes made as a result of our recommendations impact the core Europeana site — the search results and initial home pages. As a result, all users of Europeana are affected positively by these changes and all data providers benefit from end-users being able to more easily find relevant material.

Sources to corroborate the impact

A. Statement from the Business Development Director/Deputy Director, The Europeana Foundation, provides evidence that the Strathclyde evaluation study led to direct changes in Europeana and to an improvement in user experience

B. http://www.europeana.eu/portal/whatsnew.html — announcement of improvements to the user interface

C. http://www.europeana.eu/portal/europeana-providers.html — list of all providers to content

D. http://pro.europeana.eu/web/europeana-v2.0 summary of planned improvements to the user experience