Idea Transcript
Introduction to Psycholinguistics Lecture 7: Spoken Sentence Comprehension Pia Knoeferle Computerlinguistik Universität des Saarlandes
Serial principle-based theories ! Garden-path theory: Resolve ambiguity by choosing one analysis on the basis of structural criteria " Minimal attachment # Do not postulate any potentially unnecessary nodes # Example: VP-attachment of the PP is preferred # “John [hit [the girl] [with a book]].” # “John hit [the girl [with a book]].”
" Late closure: attach to open clause and attach low # If grammatically permissible, attach new items into the clause or phase currently being processed (i.e. the phrase or clause postulated most recently) # “Maria said to John she arrived yesterday.”
P. Knoeferle
Introduction to Psycholinguistics
2
Parallel weakly interactive theories ! Referential Theory of Sentence Comprehension (Crain & Steedman, 1985; Altmann & Steedman, 1988) " Alternative analyses are initially offered in parallel " Syntax proposes analyses autonomously " Immediate selection through weak interaction (constrained by semantics and context) " Hypothesis: For PP-attachment ambiguity # NP-attachment can be facilitated when a preceding linguistic context introduces more than one NP referent # VP-attachment is facilitated if there is only one NP referent introduced
P. Knoeferle
Introduction to Psycholinguistics
3
Linguistic referential context ! NP-supporting context " “A burglar broke into a bank carrying some dynamite. He planned to blow open a safe. Once inside he saw that there was a safe with a new lock and a safe with an old lock.” " NP-attached target # “The burglar blew open [the safe [with the new lock]] and made off with the loot.”
" VP-attached target # “The burglar [blew open [the safe] [with the dynamite]] and made off with the loot.”
P. Knoeferle
Introduction to Psycholinguistics
4
Linguistic referential context ! VP-supporting context " “A burglar broke into a bank carrying some dynamite. He planned to blow open a safe. Once inside he saw that there was a safe with a new lock and a strongbox with an old lock.” " NP-attached target # “The burglar blew open [the safe [with the new lock]] and made off with the loot.”
" VP-attached target # “The burglar [blew open [the safe] [with the dynamite]] and made off with the loot.”
P. Knoeferle
Introduction to Psycholinguistics
5
Linguistic referential context: results Mean reading times (centisec.) per for sentence under two conditions of referential context
VP-attached target (minimally attached)
Reading Time (csec.)
320 310 300 290 280
VP-attachment NP-attachment
270 260 250 240 230 VP-supporting context
NP-supporting context
NP-attached target (non-minimally attached)
Referential Context
1 safe P. Knoeferle
2 safes Introduction to Psycholinguistics
6
Unrestricted interaction ! Lexical constraint-based theory (e.g., MacDonald et al., 1994) " Parallel activation of multiple “constraints” # Lexical information, frequency, context, structural constraints, and world knowledge interact
" Unrestricted interaction of relevant constraints " PP attachment ambiguity # Lexical or discourse factors can affect attachment preferences # Initial analysis not proposed by syntax, but determined by free interaction of constraints # Constraints associated with noun phrase, verb, or preposition can influence attachment $ E.g., Fred ordered [a pizza [with pepperoni]].
P. Knoeferle
Introduction to Psycholinguistics
7
Eye-movements: characteristics ! How do we know about cognitive processes? " Off-line measures (e.g., self-paced reading) " Online measures (e.g., eyetracking, event-related potentials: provide high temporal resolution of comprehension processes
! Eye-movements: Used in many disciplines of cognitive science (e.g., spatial attention and scene perception, auditory and reading comprehension, memory, imagery, and consumer behaviour) ! Types and characteristics of eye-movements in information processing " Saccades: # Rapid movements of the eyes (high velocity) # Duration of saccade depends on distance # Latency of a Saccade: at least 150-175 ms (planning and executing of saccade)
" Pursuit eye-movements: when eyes follow moving target " Vergence eye-movements: when we move our eyes inward toward each other as when fixating a nearby target " Vestibular eye-movements: when eyes rotate to compensate for head and body movement in order to keep the same direction of vision P. Knoeferle
Introduction to Psycholinguistics
8
Eye-movements: characteristics " Fixations # Time during which we take in new information # Ca. 200 - 300 ms duration # Fovea (central 2 degree of vision), parafoveal (5 degree on either side of fixation) and peripheral (region beyond parafovea)
P. Knoeferle
Introduction to Psycholinguistics
9
Eye-movements ! Eye-movements and attention " In complex information processing tasks such as reading or spoken comprehension there is a tight coupling between attention and the locus of the eyes " Important: Attention in complex tasks typically precedes a saccade to a given location
! Measuring eye-movements: so far no “standard” kind of measurement " Some examples: " Corneal reflections # Measuring characteristics that behaved differently under head movement and eye-rotation # The relative positions of the pupil and the corneal reflection $ Changes when the eye rotates around its vertical and horizontal axes $ Does not change when head moves and eye is stable
" Video-based pupil monitoring " Infrared Purkinje image tracking (SMI, dual Purkinje) # Light bouncing off the eye produces a series of reflections # 1st and brightest: Corneal reflection; second image reflected of rear of cornea; another two by front and rear of the lens # Four images all have different motions in relation to eye-rotation
P. Knoeferle
Introduction to Psycholinguistics
10
Eye-movements ! Example: Corneal reflection
Rotation around horizontal axis of eye P. Knoeferle
Introduction to Psycholinguistics
Rotation around vertical
Head moves and eye is Stable: relation between corneal Reflection and pupil does not change 11
Eye-movements during text comprehension ! Can be used to investigate incremental cognitive processes during text reading ! Characteristics " Fixation duration: ca. 200-250 ms " Mean saccade size: 7-9 letter spaces " Purpose of saccade: bring new text region into foveal vision (reading on basis of parafoveal or pheripheral information near impossible) # Most saccades from left to right, but sometimes the eyes go back to previously read text (regression): Long regressions of more than 10 letter spaces caused by difficulty with certain text
! Factors affecting fixations " Content words are fixated about 85% of the time " Function words are fixated about 35% of the time, and are more likely to be skipped " As the length of a word increases, the probability of fixating it increases # 2-3 letter words only fixated 25 % of the time; words with 8 letters or more almost always fixated
" Frequency of words
! Eye-movement measures in reading, e.g., " First pass, second pass, regression path, total time P. Knoeferle
Introduction to Psycholinguistics
12
Eye-movements in spoken sentence comp. ! Relatively recent development during the mid-90s ! Relies on close time-lock between the words people hear in an utterance, and the objects they look at ! What do we measure? " Eye-movements to objects in the environment / on a scene: Frequency or proportion (as opposed to fixation durations in reading) " Recently, however, also other measures have been used: gaze duration (“gaze” is the sum of consecutive fixations within one region) " Less well-developed “linking hypothesis” # Linking hypothesis: What does a fixation “mean” # In reading a very elaborate linking hypothesis $ E.g.,first pass is usually associated with “early” comprehension processes such as the structuring of a sentence $ Second pass or regression path duration are “later measures”, and usually indicate semantic processing or structural revisions
# In spoken sentence comprehension $ A relatively coarse-grained linking hypothesis $ Close time-lock between the words people hear, and the objects they looks at (after people hear “pear”, they are going to look at a pear if there is one in the environment around them) P. Knoeferle
Introduction to Psycholinguistics
13
SMI Eyelink ! Equipment: 2 Pcs, 1 head-mounted camera ! Operator PC: saves data ! Subject PC: controls stimulus presentation ! Eyelink I (Saarbruecken): " Sampling rate of 250 Hz (every 4 ms) " Accuracy between 0.5 degree and 1 degree " Computes head-position and position of the eye
! Eyelink II " Sampling rate of up to 500 Hz " Corneal reflection
P. Knoeferle
Introduction to Psycholinguistics
14
Visual referential context ! Constrast between objects in a visual scene influences structural disambiguation online Tanenhaus et al., 1995
Put the apple on the towel in the box.
Ort oder Ziel?
P. Knoeferle
Introduction to Psycholinguistics
15
Incremental semantic interpretation More visual referential ambiguity " Influence of visual contexts on # determination of reference to entities # Properties of objects (small, tall) Sedivy et al., 1999
Pick up the tall glass and put it below the pitcher.
P. Knoeferle
Introduction to Psycholinguistics
16
Anticipatory eye-movements ! Anticipatory eye-movements: eye-movements to an object in a scene before it has been named ! Do verb selection restrictions allow anticipation of as yet unmentioned postverbal argument/ its referent in the scene ! What is anticipated? ! Why is an object anticipated? ! “The boy will move the cake.” " train, ball, toy car and cake can be moved
! “The boy will eat the cake.” " highly restrictive: only the cake is edible P. Knoeferle
Introduction to Psycholinguistics
17
Anticipatory thematic role-assignment ! Case-marking and world-knowledge (Kamide et al., 2003)
Der Hase (subj)
frisst gleich den Kohl.
The hare (subj)
eats soon
Den Hasen (obj)
frisst gleich der Fuchs.
The hare (subj)
eats soon
P. Knoeferle
the cabbage.
the fox.
Introduction to Psycholinguistics
18
Summary ! Comprehension of written sentences " What has research on written sentence comprehension revealed? # Preferred attachment # Effects of linguistic referential context on preferred structuring of the sentence
! Eye-movements " General # Types of eye-movements, eye-movements and attention, ways of measuring
" In reading # Characteristics, factors affecting fixation, types of measures
" In spoken sentence comprehension # What do we measure?, linking hypothesis
! Spoken sentence comprehension in visual worlds " Referential visual context, incremental comprehension, anticipatory processing
P. Knoeferle
Introduction to Psycholinguistics
Second Half: Lab Demo
19