Reading in the Digital Age: Vision, Text and Cognition

Submitting Institution

University of Kent

Unit of Assessment

Modern Languages and Linguistics

Summary Impact Type

Cultural

Research Subject Area(s)

Medical and Health Sciences: Neurosciences
Psychology and Cognitive Sciences: Psychology
Language, Communication and Culture: Literary Studies


Download original

PDF

Summary of the impact

The impact of the research undertaken for the collaborative AHRC-funded project `Poetry Beyond Text: Vision, Text and Cognition' (2009-11) has been the generation of a new awareness of reading and viewing behaviours in the digital age among the creative artists who were invited to respond to the project's research and visitors to the project's travelling exhibition. The `Poetry Beyond Text' project studied artworks that combine visual art (patterns, painted images, photographs, digital images) with poetic text. The research identified psychological processes in the reception of such works, and analysed the creative and collaborative processes involved in their making. The impact of this research was achieved in three main areas: (1) the public understanding of specific art forms such as digital poetry, artists' books, and concrete poetry; (2) the practice of commissioned artists and writers; (3) the policy and public profile of partner non-HEIs, notably the Scottish Poetry Library. 1 and 3 were primarily within Scotland, but 2 was both national and international, including artists and writers from the UK, the United States, and Brazil. The pathways to impact were the project's Poetry Beyond Text exhibition, a website featuring an online gallery and educational materials, and a series of debates open to the public.

Underpinning research

The underpinning research was carried out by Dr Anna Katharina Schaffner (Lecturer in Comparative Literature, University of Kent, 2007-12; Senior Lecturer in Comparative Literature, Kent, 2012-present), Dr Kim Knowles (Postdoctoral Research Fellow, Comparative Literature, Kent, 2009-11), Dr Ulrich Weger (Psychology, Kent), Prof. Andrew Michael Roberts (PI, University of Dundee), Dr Jane Stabler (Dundee), Martin Fischer (Dundee), and Lisa Otty (Dundee).

Recent research in psychology has identified the typical eye-movement patterns, and the associated cognitive processes, that occur when we read texts or look at images. For the AHRC-funded `Poetry Beyond Text: Vision, Text and Cognition' research project (2009-11), Schaffner (Co-Investigator with responsibility for leading the project at the University of Kent) worked with a colleague in the Department of Psychology (Weger) and a postdoctoral researcher (Knowles, now a Lecturer in Film Studies at the University of Aberystwyth) in the Department of Comparative Literature at Kent, as well as with the research team at the University of Dundee led by Roberts, to investigate what happens when readers are presented with artworks that combine text and image. This interdisciplinary project drew its theoretical and methodological tools from the fields of literary theory, art theory, and experimental psychology. It took account in particular of: (1) the latest research on the history and theory of digital poetry, pattern poetry, and concrete poetry; and (2) recent behavioural studies into the perception, comprehension, and recollection of texts, images, and auditory materials.

The psychological approaches employed in the project included cognitive research relying on mental chronometry, eye-tracking, EEG scans, and subjective rating studies. The use of these experimental methods in the analysis of the ways in which poetry is read led to the development of a `reflective feedback loop', in which participants in the experiments (undergraduate and postgraduate students studying literature, philosophy, film, and fine art) were invited to act as co-researchers. The participants' cognitive processes were assessed through eye-tracking and EEG scans in the viewing of three different kinds of artwork: (1) text poetry accompanied by images (photographs or drawings); (2) concrete or pattern poetry; and (3) digital poetry (including images or text-shapes). Some participants were then asked to produce artworks of their own in response to the experiments. Experimental procedures included: (1) modifying the layout of the text; (2) adding or removing specific modalities (for example, the same text with and without photographs); and (3) studying how reading strategies (such as the pattern of looking at visual poetry or texts with images, or interacting with digital poetry), affect memory, interpretation, and perceived aesthetic value. The research undertaken at Kent focused primarily on the analysis of reading patterns related to concrete and pattern poetry, digital poetry, and text film.

The principal findings of this research were based on the collection and analysis of empirical data on how spatial parameters in literary texts are actually `read' and integrated into the interpretative process. One of the primary findings that emerged from the eye-tracking experiments conducted at the University of Kent is that when participants encounter linguistic signs that are arranged in pictorial but ordered fashion, the initial response is to adopt a classical reading approach. Only if that is not possible, or not informative, does the viewer opt for a more varied scanning approach. This corroborates the findings of an earlier experiment, the well-known Stroop test. This test established that linguistic information is automatically accessed, even when readers direct their attention to visual information. In other words, automatized codes (verbal meaning) cannot be ignored or inhibited easily, even while participants attend to the visual aspects of words, which in the case of the Stroop test were colours, and in the `Poetry Beyond Text' experiments spatially arranged linguistic signs.

These findings were disseminated in various forms: on the project website; in the travelling Poetry Beyond Text exhibition and the accompanying exhibition catalogue; at conferences; and in journal publications. In `Reading Space: New Cognitive Perspectives' (Knowles et al. 2012a), for instance, theoretical claims regarding the alleged function of spatial values in visual poetry were integrated with results from a series of eye-tracking and other empirical experiments, which tested how people read, understand, and respond to spaces between units of meaning in a selection of visual poems ranging from works by Stéphane Mallarmé, Guillaume Apollinaire, and e.e. cummings to concrete poems by, among others, Eugen Gomringer, Claus Bremer, and Ian Hamilton Finlay. The experiments emerged from, and responded to, the subject of space in literary theory and word-image discourse, seeking empirical answers to theoretical questions, and revealing new insights into the reception, cognitive processing, and interpretation of visual poetry, above all the fact that the classical reading approach is generally adopted even when linguistic signs are presented to the eye in a pictorial manner.

References to the research

AHRC-Funded Research Project:

AHRC `Beyond Text' Large Grant (£472,617) for the `Poetry Beyond Text: Vision, Text and Cognition' project (March 2009-February 2011). Researchers: Andrew Michael Roberts (PI, Dundee), Martin Fischer (Psychology, Dundee), Lisa Otty (Dundee), Mary Modeen (Fine Art, Dundee), Anna Katharina Schaffner (Comparative Literature, Kent), Ulrich Weger (Psychology, Kent), Kim Knowles (Comparative Literature, Kent).

Selected outputs:

1. Knowles, Kim, Schaffner, Anna Katharina, Weger, Ulrich, and Roberts, Andrew Michael (2012a), `Reading Space: New Cognitive Perspectives', Writing Technologies, vol. 4, pp. 75-106. ISSN: 1754-9035. [Accessible online via the link above.]

2. Roberts, Andrew Michael, Otty, Lisa, Fischer, Martin, and Schaffner, Anna Katharina (2012b), `Creative Practice and Experimental Method in Electronic Literature and Human Experimental Psychology', Dichtung Digital: A Journal of Art and Culture in Digital Media, vol. 42. [Accessible online via the link above.]

Details of the impact

Impact activities:

  1. Exhibition. The project's Poetry Beyond Text exhibition opened at the Visual Research Centre, Dundee Contemporary Arts (DCA), in March 2011. Selections from the exhibition were subsequently exhibited at the Scottish Poetry Library in Edinburgh (14 May-15 July 2011) and the Moray Art Centre, Scotland (9 August-30 September 2011). The entire exhibition was then presented at the Royal Scottish Academy, Edinburgh (12 November-18 December 2011). Visitor numbers were over 4,500. The exhibition gave the research team the opportunity to show to the general public the creative works that had been produced by the students participating in the project, as well as a number of artworks (including artists' books, prints, digital poetry, and experimental films) commissioned by the AHRC project team from professional artists. The commissioning of collaborative artworks as part of the research advanced creativity by bringing together poets, artists, sculptors, and printmakers, with the stimulation of an invitation to engage specifically with the aesthetic and technical issues arising from the interaction of visual and poetic forms. Creative artists involved in this aspect of the project included distinguished poets (John Burnside, Robin Robertson, Thomas A. Clark, and Deryn Rees-Jones) and visual artists (Will Maclean and David Bellingham), as well as the film-maker Sarah Turner. In a few cases, the collaborations built on existing connections; in most cases, however, new creative partnerships were established across media; for instance, with the digital poets Giselle Beiguelman and John Cayley. The opening advisers' workshop of the `Poetry Beyond Text' project (held at Kent in May 2009) led to Cayley's creative work responding to the output of pilot eye-tracking experiments; subsequently this work, under the title The Readers' Project, was included in the project exhibition at Dundee Contemporary Arts in March 2011.
  2. Project website. The `Poetry Beyond Text' project website was designed as a public resource, and includes accessible articles on genres and methods as well as information on the project, its findings, news, events, etc. The online gallery provides images of all the commissioned works, as well as some of the actual works (digital poems and videos). The website continues to be developed as a public resource.
  3. A series of public debates. Discussions of the process of collaborative creation have included numerous public events: Helen Douglas with Valerie Gillies (at the opening of the Dundee exhibition in March 2011); Deryn Rees-Jones with Marion Smith (at the Scottish Poetry Library on 18 May 2011); Jim Carruth with Murray Robertson and Michael Waight (at the Scottish Poetry Library on 7 June 2011). These events have shown how the project's collaboration with artists has prompted new insights into formal and thematic aspects of creative practice. One element of the project, known as `Chinese Whispers', was inspired by debates about `visual thinking', and involved art students being asked to respond in purely visual form to poems by Wallace Stevens (`Anecdote of the Jar') and Jim Carruth (`Seasons'). The resulting artworks (in a range of media) were then presented to poets, who were asked - without knowledge of the original poems - to respond in poetic form. This sequence of `translations' between verbal and visual media was repeated a number of times. A public discussion of the process held at the Scottish Poetry Library in June 2011 showed that this exercise had stimulated new creativity in all those involved (see below), as well as offering poets and artists an opportunity to exhibit their work.

Reach of the impact:

  1. The Poetry Beyond Text exhibition attracted considerable public interest, especially during its period at the Royal Scottish Academy, Edinburgh (autumn 2011), with over 4,500 visitors in total. Visitors with diverse interests have commented on the stimulating nature of these exhibitions, as shown, for example, in the visitors' books (see selected extracts in `Significance of the impact' below). The creative dimension to the project included emerging visual artists and poets, and also art students, whose careers and profiles have been enhanced by the opportunity to exhibit and to engage in a collective process of creative research.
  2. The website is a developing resource, offering access to the research activities and findings in a manner that is designed to appeal to the non-specialist.
  3. The public debates and workshops were held at non-HEI venues (Dundee Contemporary Arts; Scottish Poetry Library) in order to encourage the presence and participation of those outside academia. The events were held at the Scottish Poetry Library in Edinburgh (24 May 2011), at DG Arts in Dumfries (3 June 2011), and in Kilmarnock as part of the Ayrshire Council Imprint Festival (10 November 2011). At each session the participants included creative writers and artists, primary- and secondary-school teachers, and interested members of the general public. Specific statements of impact recorded in written feedback on these workshops included a primary-school teacher who commented: `it's inspired me to take this idea into the classroom'; a high-school teacher who wrote that s/he planned to `do the task [a cut-up exercise based on one of the project experiments] with my own students'; a poet who wrote: 'I will use these techniques to write concrete poems'; and an artist who wrote: `I'm going to pursue this creatively through my work'.

Significance of the impact:

  1. The `Poetry Beyond Text' project has contributed to the critical understanding of processes of reading in a range of genres and media, demonstrating that when participants encounter linguistic signs arranged in an orderly pictorial fashion, the immediate reaction is to adopt a classical reading approach. Only if that is not possible, or not informative, do they adopt a different scanning approach. The impact of these findings has been achieved above all in the field of the creative arts, with artists having been invited to respond to the research findings. The poet Deryn Rees-Jones, for instance, has commented on the impact of the project on her writing of the poem-photograph `The Wren's Egg', which became the title poem of the T.S. Eliot Prize-shortlisted, Poetry Book Society-recommended, and Times Literary Supplement `Book of the Year' collection Burying the Wren (2012). Rees-Jones observes that the `Poetry Beyond Text' project enabled her `to think visually about language' (5.7). In addition, the Scottish Poetry Library has commented that as a result of the project it was `able to bring interesting content and discussions to our wide range of library users and visitors', with `inspiring insight into both the creative process and the reading process' (5.8).
  2. One of the follow-up projects, `Archive of Reading', is a public resource, making accessible the experimental results gathered during the `Poetry Beyond Text' project. It is housed at the Scottish Poetry Library in Edinburgh, and was also funded by the Arts and Humanities Research Council. The archive was launched as part of the Poetry Beyond Text exhibition, held at the Scottish Poetry Library from 14 May to 17 July 2011. This archive brings together `readings' of different types recorded in various forms. It includes heat maps, eye-tracking videos, reported readings, and fixation maps, as well as original artworks and poems created as 'readings' of one another during the `Chinese Whispers' series. The `Archive of Reading' can be accessed via computer in the Scottish Poetry Library.
  3. Through the exhibitions and the website, the project has fostered a greater awareness of how readers vacillate between viewing and reading modes when confronted with hybrid structures that bring together textual and visual elements. Evidence that the project exhibition in particular has impacted significantly on the way in which a diverse international public thinks about the relation between text and image is to be found, for instance, in the comments made in the exhibition visitors' book of the Moray Arts Centre, Findhorn (5.6). These comments include: `I particularly enjoyed the poem of the dog's hair and the image of someone's back before they died - and the river flowing text and the strange "massiveness" of the "Wren" procession.' (Peter Fraser, Melbourne, Australia, 25 Aug. 2011); `Thank you for the inspiration.' (Erin O'Byrne, Vancouver, Canada, 8 Sept. 2011); `The possibilities of text! Some very affecting pieces.' (S. Leggett, 8 Sept. 2011); `Wonderful and innovative and thought-provoking.' (C. Roy, Canada, 16 Sept. 2011).

Sources to corroborate the impact

Quantitative indicators:

  1. Project exhibition visitor numbers: Visual Research Centre, Dundee Contemporary Arts, Dundee (4 March-1 April 2011): 412 visitors.
  2. Project exhibition visitor numbers: Moray Arts Centre, Findhorn (9 August-30 September 2011): 1,389 visitors.
  3. Project exhibition visitor numbers: Royal Scottish Academy, Edinburgh (12 November-18 December 2011): c. 3,000 visitors.
  4. Project-related events at the Scottish Poetry Library, Edinburgh (14 May-15 July 2011): Nick-e Melville and Brigid Collins.
  5. Project website, including numerous artworks produced in response to the research project.

Independent testimony:

  1. Feedback on the exhibition in the Moray Arts Centre, Findhorn, visitors' book. See section 4.3 above for selected comments from this visitors' book.
  2. Statements by the creative artists Deryn Rees-Jones and John Cayley, corroborating the impact of the `Poetry Beyond Text' project on their creative practice. (See section 4.1 above for selected quotations.)
  3. Statement from the Scottish Poetry Library, confirming the impact of the `Poetry Beyond Text' and the `Archive of Reading' projects on the Library and its audience (see 4.1 above).