Reading in the Digital Age: Vision, Text and Cognition
Submitting Institution
University of KentUnit of Assessment
Modern Languages and LinguisticsSummary Impact Type
CulturalResearch Subject Area(s)
Medical and Health Sciences: Neurosciences
Psychology and Cognitive Sciences: Psychology
Language, Communication and Culture: Literary Studies
Summary of the impact
The impact of the research undertaken for the collaborative AHRC-funded project `Poetry Beyond
Text: Vision, Text and Cognition' (2009-11) has been the generation of a new awareness of
reading and viewing behaviours in the digital age among the creative artists who were invited to
respond to the project's research and visitors to the project's travelling exhibition. The `Poetry
Beyond Text' project studied artworks that combine visual art (patterns, painted images,
photographs, digital images) with poetic text. The research identified psychological processes in
the reception of such works, and analysed the creative and collaborative processes involved in
their making. The impact of this research was achieved in three main areas: (1) the public
understanding of specific art forms such as digital poetry, artists' books, and concrete poetry; (2)
the practice of commissioned artists and writers; (3) the policy and public profile of partner non-HEIs, notably the Scottish Poetry Library. 1 and 3 were primarily within Scotland, but 2 was both
national and international, including artists and writers from the UK, the United States, and Brazil.
The pathways to impact were the project's Poetry Beyond Text exhibition, a website featuring an
online gallery and educational materials, and a series of debates open to the public.
Underpinning research
The underpinning research was carried out by Dr Anna Katharina Schaffner (Lecturer in
Comparative Literature, University of Kent, 2007-12; Senior Lecturer in Comparative Literature,
Kent, 2012-present), Dr Kim Knowles (Postdoctoral Research Fellow, Comparative Literature,
Kent, 2009-11), Dr Ulrich Weger (Psychology, Kent), Prof. Andrew Michael Roberts (PI, University
of Dundee), Dr Jane Stabler (Dundee), Martin Fischer (Dundee), and Lisa Otty (Dundee).
Recent research in psychology has identified the typical eye-movement patterns, and the
associated cognitive processes, that occur when we read texts or look at images. For the AHRC-funded `Poetry Beyond Text: Vision, Text and Cognition' research project (2009-11), Schaffner
(Co-Investigator with responsibility for leading the project at the University of Kent) worked with a
colleague in the Department of Psychology (Weger) and a postdoctoral researcher (Knowles, now
a Lecturer in Film Studies at the University of Aberystwyth) in the Department of Comparative
Literature at Kent, as well as with the research team at the University of Dundee led by Roberts, to
investigate what happens when readers are presented with artworks that combine text and image.
This interdisciplinary project drew its theoretical and methodological tools from the fields of literary
theory, art theory, and experimental psychology. It took account in particular of: (1) the latest
research on the history and theory of digital poetry, pattern poetry, and concrete poetry; and (2)
recent behavioural studies into the perception, comprehension, and recollection of texts, images,
and auditory materials.
The psychological approaches employed in the project included cognitive research relying on
mental chronometry, eye-tracking, EEG scans, and subjective rating studies. The use of these
experimental methods in the analysis of the ways in which poetry is read led to the development of
a `reflective feedback loop', in which participants in the experiments (undergraduate and
postgraduate students studying literature, philosophy, film, and fine art) were invited to act as co-researchers. The participants' cognitive processes were assessed through eye-tracking and EEG
scans in the viewing of three different kinds of artwork: (1) text poetry accompanied by images
(photographs or drawings); (2) concrete or pattern poetry; and (3) digital poetry (including images
or text-shapes). Some participants were then asked to produce artworks of their own in response
to the experiments. Experimental procedures included: (1) modifying the layout of the text; (2)
adding or removing specific modalities (for example, the same text with and without photographs);
and (3) studying how reading strategies (such as the pattern of looking at visual poetry or texts with
images, or interacting with digital poetry), affect memory, interpretation, and perceived aesthetic
value. The research undertaken at Kent focused primarily on the analysis of reading patterns
related to concrete and pattern poetry, digital poetry, and text film.
The principal findings of this research were based on the collection and analysis of empirical data
on how spatial parameters in literary texts are actually `read' and integrated into the interpretative
process. One of the primary findings that emerged from the eye-tracking experiments conducted at
the University of Kent is that when participants encounter linguistic signs that are arranged in
pictorial but ordered fashion, the initial response is to adopt a classical reading approach. Only if
that is not possible, or not informative, does the viewer opt for a more varied scanning approach.
This corroborates the findings of an earlier experiment, the well-known Stroop test. This test
established that linguistic information is automatically accessed, even when readers direct their
attention to visual information. In other words, automatized codes (verbal meaning) cannot be
ignored or inhibited easily, even while participants attend to the visual aspects of words, which in
the case of the Stroop test were colours, and in the `Poetry Beyond Text' experiments spatially
arranged linguistic signs.
These findings were disseminated in various forms: on the project website; in the travelling Poetry
Beyond Text exhibition and the accompanying exhibition catalogue; at conferences; and in journal
publications. In `Reading Space: New Cognitive Perspectives' (Knowles et al. 2012a), for instance,
theoretical claims regarding the alleged function of spatial values in visual poetry were integrated
with results from a series of eye-tracking and other empirical experiments, which tested how
people read, understand, and respond to spaces between units of meaning in a selection of visual
poems ranging from works by Stéphane Mallarmé, Guillaume Apollinaire, and e.e. cummings to
concrete poems by, among others, Eugen Gomringer, Claus Bremer, and Ian Hamilton Finlay. The
experiments emerged from, and responded to, the subject of space in literary theory and word-image discourse, seeking empirical answers to theoretical questions, and revealing new insights
into the reception, cognitive processing, and interpretation of visual poetry, above all the fact that
the classical reading approach is generally adopted even when linguistic signs are presented to the
eye in a pictorial manner.
References to the research
AHRC-Funded Research Project:
AHRC `Beyond Text' Large Grant (£472,617) for the `Poetry Beyond Text: Vision, Text and
Cognition' project (March 2009-February 2011). Researchers: Andrew Michael Roberts (PI,
Dundee), Martin Fischer (Psychology, Dundee), Lisa Otty (Dundee), Mary Modeen (Fine
Art, Dundee), Anna Katharina Schaffner (Comparative Literature, Kent), Ulrich Weger
(Psychology, Kent), Kim Knowles (Comparative Literature, Kent).
Selected outputs:
1. Knowles, Kim, Schaffner, Anna Katharina, Weger, Ulrich, and Roberts, Andrew Michael
(2012a), `Reading Space: New Cognitive Perspectives', Writing Technologies, vol. 4, pp.
75-106. ISSN: 1754-9035. [Accessible online via the link above.]
2. Roberts, Andrew Michael, Otty, Lisa, Fischer, Martin, and Schaffner, Anna Katharina
(2012b), `Creative Practice and Experimental Method in Electronic Literature and Human
Experimental Psychology', Dichtung Digital: A Journal of Art and Culture in Digital Media,
vol. 42. [Accessible online via the link above.]
Details of the impact
Impact activities:
-
Exhibition. The project's Poetry Beyond Text exhibition opened at the Visual Research
Centre, Dundee Contemporary Arts (DCA), in March 2011. Selections from the exhibition
were subsequently exhibited at the Scottish Poetry Library in Edinburgh (14 May-15 July
2011) and the Moray Art Centre, Scotland (9 August-30 September 2011). The entire
exhibition was then presented at the Royal Scottish Academy, Edinburgh (12 November-18
December 2011). Visitor numbers were over 4,500. The exhibition gave the research team
the opportunity to show to the general public the creative works that had been produced by
the students participating in the project, as well as a number of artworks (including artists'
books, prints, digital poetry, and experimental films) commissioned by the AHRC project
team from professional artists. The commissioning of collaborative artworks as part of the
research advanced creativity by bringing together poets, artists, sculptors, and printmakers,
with the stimulation of an invitation to engage specifically with the aesthetic and technical
issues arising from the interaction of visual and poetic forms. Creative artists involved in
this aspect of the project included distinguished poets (John Burnside, Robin Robertson,
Thomas A. Clark, and Deryn Rees-Jones) and visual artists (Will Maclean and David
Bellingham), as well as the film-maker Sarah Turner. In a few cases, the collaborations built
on existing connections; in most cases, however, new creative partnerships were
established across media; for instance, with the digital poets Giselle Beiguelman and John
Cayley. The opening advisers' workshop of the `Poetry Beyond Text' project (held at Kent in
May 2009) led to Cayley's creative work responding to the output of pilot eye-tracking
experiments; subsequently this work, under the title The Readers' Project, was included in
the project exhibition at Dundee Contemporary Arts in March 2011.
-
Project website. The `Poetry Beyond Text' project website was designed as a public
resource, and includes accessible articles on genres and methods as well as information on
the project, its findings, news, events, etc. The online gallery provides images of all the
commissioned works, as well as some of the actual works (digital poems and videos). The
website continues to be developed as a public resource.
-
A series of public debates. Discussions of the process of collaborative creation have
included numerous public events: Helen Douglas with Valerie Gillies (at the opening of the
Dundee exhibition in March 2011); Deryn Rees-Jones with Marion Smith (at the Scottish
Poetry Library on 18 May 2011); Jim Carruth with Murray Robertson and Michael Waight
(at the Scottish Poetry Library on 7 June 2011). These events have shown how the
project's collaboration with artists has prompted new insights into formal and thematic
aspects of creative practice. One element of the project, known as `Chinese Whispers', was
inspired by debates about `visual thinking', and involved art students being asked to
respond in purely visual form to poems by Wallace Stevens (`Anecdote of the Jar') and Jim
Carruth (`Seasons'). The resulting artworks (in a range of media) were then presented to
poets, who were asked - without knowledge of the original poems - to respond in poetic
form. This sequence of `translations' between verbal and visual media was repeated a
number of times. A public discussion of the process held at the Scottish Poetry Library in
June 2011 showed that this exercise had stimulated new creativity in all those involved (see
below), as well as offering poets and artists an opportunity to exhibit their work.
Reach of the impact:
- The Poetry Beyond Text exhibition attracted considerable public interest, especially during
its period at the Royal Scottish Academy, Edinburgh (autumn 2011), with over 4,500
visitors in total. Visitors with diverse interests have commented on the stimulating nature of
these exhibitions, as shown, for example, in the visitors' books (see selected extracts in
`Significance of the impact' below). The creative dimension to the project included emerging
visual artists and poets, and also art students, whose careers and profiles have been
enhanced by the opportunity to exhibit and to engage in a collective process of creative
research.
- The website is a developing resource, offering access to the research activities and
findings in a manner that is designed to appeal to the non-specialist.
- The public debates and workshops were held at non-HEI venues (Dundee Contemporary
Arts; Scottish Poetry Library) in order to encourage the presence and participation of those
outside academia. The events were held at the Scottish Poetry Library in Edinburgh (24
May 2011), at DG Arts in Dumfries (3 June 2011), and in Kilmarnock as part of the Ayrshire
Council Imprint Festival (10 November 2011). At each session the participants included
creative writers and artists, primary- and secondary-school teachers, and interested
members of the general public. Specific statements of impact recorded in written feedback
on these workshops included a primary-school teacher who commented: `it's inspired me to
take this idea into the classroom'; a high-school teacher who wrote that s/he planned to `do
the task [a cut-up exercise based on one of the project experiments] with my own students';
a poet who wrote: 'I will use these techniques to write concrete poems'; and an artist who
wrote: `I'm going to pursue this creatively through my work'.
Significance of the impact:
- The `Poetry Beyond Text' project has contributed to the critical understanding of processes
of reading in a range of genres and media, demonstrating that when participants encounter
linguistic signs arranged in an orderly pictorial fashion, the immediate reaction is to adopt a
classical reading approach. Only if that is not possible, or not informative, do they adopt a
different scanning approach. The impact of these findings has been achieved above all in
the field of the creative arts, with artists having been invited to respond to the research
findings. The poet Deryn Rees-Jones, for instance, has commented on the impact of the
project on her writing of the poem-photograph `The Wren's Egg', which became the title
poem of the T.S. Eliot Prize-shortlisted, Poetry Book Society-recommended, and Times
Literary Supplement `Book of the Year' collection Burying the Wren (2012). Rees-Jones
observes that the `Poetry Beyond Text' project enabled her `to think visually about
language' (5.7). In addition, the Scottish Poetry Library has commented that as a result of
the project it was `able to bring interesting content and discussions to our wide range of
library users and visitors', with `inspiring insight into both the creative process and the
reading process' (5.8).
- One of the follow-up projects, `Archive of Reading', is a public resource, making accessible
the experimental results gathered during the `Poetry Beyond Text' project. It is housed at
the Scottish Poetry Library in Edinburgh, and was also funded by the Arts and Humanities
Research Council. The archive was launched as part of the Poetry Beyond Text exhibition,
held at the Scottish Poetry Library from 14 May to 17 July 2011. This archive brings
together `readings' of different types recorded in various forms. It includes heat maps, eye-tracking videos, reported readings, and fixation maps, as well as original artworks and
poems created as 'readings' of one another during the `Chinese Whispers' series. The
`Archive of Reading' can be accessed via computer in the Scottish Poetry Library.
- Through the exhibitions and the website, the project has fostered a greater awareness of
how readers vacillate between viewing and reading modes when confronted with hybrid
structures that bring together textual and visual elements. Evidence that the project
exhibition in particular has impacted significantly on the way in which a diverse international
public thinks about the relation between text and image is to be found, for instance, in the
comments made in the exhibition visitors' book of the Moray Arts Centre, Findhorn (5.6).
These comments include: `I particularly enjoyed the poem of the dog's hair and the image
of someone's back before they died - and the river flowing text and the strange
"massiveness" of the "Wren" procession.' (Peter Fraser, Melbourne, Australia, 25 Aug.
2011); `Thank you for the inspiration.' (Erin O'Byrne, Vancouver, Canada, 8 Sept. 2011);
`The possibilities of text! Some very affecting pieces.' (S. Leggett, 8 Sept. 2011); `Wonderful
and innovative and thought-provoking.' (C. Roy, Canada, 16 Sept. 2011).
Sources to corroborate the impact
Quantitative indicators:
- Project exhibition visitor numbers: Visual Research Centre, Dundee Contemporary Arts,
Dundee (4 March-1 April 2011): 412 visitors.
- Project exhibition visitor numbers: Moray Arts Centre, Findhorn (9 August-30 September
2011): 1,389 visitors.
- Project exhibition visitor numbers: Royal Scottish Academy, Edinburgh (12 November-18
December 2011): c. 3,000 visitors.
- Project-related events at the Scottish Poetry Library, Edinburgh (14 May-15 July 2011):
Nick-e Melville and Brigid Collins.
- Project website, including numerous artworks produced in response to the research project.
Independent testimony:
- Feedback on the exhibition in the Moray Arts Centre, Findhorn, visitors' book. See section
4.3 above for selected comments from this visitors' book.
- Statements by the creative artists Deryn Rees-Jones and John Cayley, corroborating the
impact of the `Poetry Beyond Text' project on their creative practice. (See section 4.1 above
for selected quotations.)
- Statement from the Scottish Poetry Library, confirming the impact of the `Poetry Beyond
Text' and the `Archive of Reading' projects on the Library and its audience (see 4.1 above).