PDF hosted at the Radboud Repository of the Radboud University [PDF]

PDF hosted at the Radboud Repository of the Radboud University Nijmegen

The following full text is a publisher's version.

For additional information about this publication click this link. http://hdl.handle.net/2066/133272

Please be advised that this information was generated on 2019-04-05 and may be subject to change.

From Beat to BCI A musical paradigm for, and the ethical aspects of Brain-Computer Interfacing

Rutger J. Vlek

The research presented in this thesis was carried out at the Donders Institute for Brain, Cognition and Behaviour, Centre for Cognition, Department of Cognitive Artificial Intelligence of the Radboud University Nijmegen, the Netherlands, with financial support of the BrainGain Smart Mix Programme of the Netherlands Ministry of Economic Affairs and the Netherlands Ministry of Education, Culture and Science. ISBN 978-94-6284-001-0 Cover design by: Rutger Vlek Printed by: Ipskamp Drukkers, Enschede, The Netherlands Copyright © Rutger Vlek, 2014

ii

From Beat to BCI A musical paradigm for, and the ethical aspects of Brain-Computer Interfacing Proefschrift

ter verkrijging van de graad van doctor aan de Radboud Universiteit Nijmegen op gezag van de rector magnificus prof. dr. Th.L.M. Engelen, volgens besluit van het college van decanen in het openbaar te verdedigen op dinsdag 9 december 2014 om 12.30 uur precies door Rutger Jan Vlek geboren op 20 oktober 1982 te Groningen

Promotor: Prof. dr. ir. P.W.M. Desain Copromotoren: Dr. J.D.R. Farquhar Dr. W.F.G. Haselager

Manuscriptcommissie: Prof. dr. R.G.J. Meulenbroek (voorzitter) Prof. dr. D.K.J. Heylen University of Twente Dr. H.T. van Schie

iv

This work is dedicated to Robert Moog (1934-2005), electronic music pioneer and technological innovator.

"(...) if the human brain was simple enough for us to understand, we would still be so stupid that we couldn’t understand it." Jostein Gaarder, Sophie’s World, 1991

Contents 1

Introduction

1

1.1

Let there be music! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1

1.2

On subjectivity, concepts and terminology . . . . . . . . . . . . . . . . . . . . .

4

1.3

Music and scientific research, the big questions . . . . . . . . . . . . . . . . . .

4

1.4

BCI, motivation and methodology . . . . . . . . . . . . . . . . . . . . . . . . .

5

1.5

Definition of BCI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6

1.6

BCI measurement modalities . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6

1.7

BCI brain signatures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7

1.8

Online detection and interpretation . . . . . . . . . . . . . . . . . . . . . . . . .

8

1.9

BCI applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

8

1.10 Performance of a BCI system . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.11 BCI and ethics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.12 BCI and music . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.13 Outlook on following chapters . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2

3

4

Shared mechanisms in perception and imagery of auditory accents

15

2.1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

2.2

Experiment and analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2.3

Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

2.4

Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

Sequenced subjective accents for brain-computer interfaces

31

3.1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

3.2

Materials and methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

3.3

Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

3.4

Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

Ethical issues in BCI research, development, and dissemination vi

51

5

6

4.1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

4.2

Case scenario ‘Jane’ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

4.3

Case scenario ‘Nigel’ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

4.4

Case scenario ‘Ben’ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

4.5

Case scenario ‘Thomas’ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

4.6

Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

A note on ethical aspects of BCI

63

5.1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

5.2

BCIs for locked-in patients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

5.3

Informed consent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

5.4

Team responsibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

5.5

Communication with the media . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

5.6

Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

General discussion

75

6.1

Subjective accenting and BCI . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

6.2

BCI and ethics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

References

83

Brief Summary

103

Korte Samenvatting

107

Publication list

111

Thanks

115

Curriculum Vitae

117

Donders Series

119

vii

viii

1. Introduction 1.1

Let there be music!

For most of us, music is an inherent part of daily life. It is something we like, or sometimes hate. It touches us to tears or gives us cold shivers. It makes us aware of an incoming phone call. It is vital to a good movie or dance performance. It stimulates our shopping behaviour, or sets the mood in our favourite bar. It is part of celebrating birthdays and is part of the ritual when putting young children to bed. We may not realise this, but almost all of us are experts in processing the vast complexity of sound patterns that music consists of. To convince you of this complexity, let us take a moment to consider the ingredients of a typical musical piece. The most simple sound is a sine wave oscillation (Figure 1.1a). This oscillation has a certain amplitude, which we perceive as loudness, and a frequency, which we perceive as pitch. Beyond the fundamental frequency, a sine wave has no harmonics (oscillations at other frequencies, that co-occur with, and relate to the fundamental frequency). Now we move on to a piano, and play a note with the same fundamental frequency as the sine wave (Figure 1.1b). We perceive the same fundamental frequency, but additionally become aware of the presence of harmonics generated by the strings and resonance of the piano’s sound board. We perceive the piano as a different timbre from the sine wave, even though they share the same fundamental frequency. Now a series of the same notes is played on the piano, nine notes in total, the first, fourth and seventh of which are played louder (see Figure 1.2a). We perceive the louder notes as accented compared to the others (unaccented or non-accented). The regular pattern of accents, repeating every three notes, induces a sense of rhythm and meter. The difference between rhythm and meter is difficult to explain with only this example, but becomes clear when looking at Figure 1.2. The figure shows two patterns of accented and unaccented notes. While both (a) and (b) consist of a pattern repeating every three (equidistant) beats - a 3-beat meter - they have a different temporal subdivision or rhythm. While concepts of beat, meter and rhythm can become much more complex in practice, this explanation should suffice for the sake of the thesis. The interval between the beats that make up the 3-beat meter is referred to as tempo, which is expressed in beats per minute (BPM). While at first glance tempo seems merely a numerical property of music, in practice it is an important dimension of musical expression. Because temporal expectations do not scale over time, and we are far more accurate in judging time intervals that fall around the so called preferred rate of about 600 ms (Van Noorden and Moelants, 1999), a rhythm performed at a different global tempo can drastically change the perceptual structure. Next to the overall 1

Chapter 1. Introduction (a) Sine wave (f=440Hz) magnitude

amplitude

1 0 −1

0

(b) Piano note A4 (f=440Hz) magnitude

1 amplitude

1

0

1

0

−1 (c) Piano chord Aminor (f=[440|523.3|659.3]Hz) magnitude

amplitude

1 0

1

0

−1

frequency (Hz) (d) Chopin piano concerto magnitude

amplitude

0.1 0 −0.1

0

0.01

0.02

0.03

0.04

0.05 time (s)

0.06

0.07

0.08

0.09

0.1

1

0 20

440 20000 frequency (Hz)

Figure 1.1: Waveforms (left column) and normalized frequency spectra in the range of human hearing (right column) of (a) a sine wave at 440Hz, (b) a piano note with a fundamental frequency of 440Hz, (c) a piano A-minor chord with fundamental frequencies at 440, 523.3, and 659.3Hz, and (d) a fragment of Chopin’s piano concerto no.1 in E minor op.11

tempo, momentary increases (accelerando) or decreases (ritardando) in tempo can draw the listeners attention to specific parts of a composition and are often linked to the melodic phrase structure (Todd, 1985). On a larger time scale, small fluctuations in tempo throughout a song can help communicate a sense of ’energy’ (listen for example to a few Beatles songs and pay attention to differences in tempo between verses and choruses, and beginning and ending sections of a song). On a smaller time scale local rhythmic patterns are often performed non-metronomically, yielding expressive timing patterns that may induce a sense of swing or groove. A piano is a polyphonic instrument and can thus produce several notes at the same time, making up a chord (see Figure 1.1c for its waveform and harmonic content). Several chords after each other make up a chord progression. A typical musical composition contains several sections, each with their own chord progression. On top of the chords a melody may be played, or the instruments of an orchestra might be accompanying the piano, making it a piano concerto (see Figure 1.1d for its waveform and harmonic content). We can buy ourselves a ticket and thoroughly enjoy the performance, without being aware of the vast complexity of the sound patterns and perceptual processes dealing with them. At this point it might sound like the process of enjoyment is merely triggered by bottomup processing of the sound that enters our ears, but this is in fact not the case. While 2

Chapter 1. Introduction

(a)

(b)

time accented note

unaccented note

beat

Figure 1.2: Two different series of accented and unaccented notes, both following a 3-beat meter.

sounds are processed in a bottom-up fashion after perceiving them, top-down processes are active at the same time, drawing our attention towards certain elements in the sound, actively building temporal expectations (Desain, 1992) even for novel pieces, and linking our perception to memories and emotions (Huron, 2006). Interestingly, one could even enjoy part of the sensation without a ticket to a concert, and without sound. Can you imagine your favourite song? Have you ever had a song "stuck" in your head? This is also known as an ear worm, and is a form of music processing without external stimulus. Musical hallucinations are another form, and see Sacks (2008) for a wealth of similar musical phenomena. Music imagery turns out not to be so much different from active listening, and even electroencephalography (EEG) traces of the different tasks show commonalities (Schaefer et al., 2011b). Properties of imagery even suggest the possibility of a generic cognitive mechanism active throughout various modalities (e.g. visual, tactile, and auditory) (Kosslyn et al., 1995, 2001; Kraemer et al., 2005). Beside purely imagining a musical piece, it is also possible to subjectively change (or enhance) the perception of sounds. This becomes clear in the so-called "clock illusion", where a 2-beat meter is nearly automatically induced on a series of identical clock pulses. A clock physically making the sound "tick-tick-tick-tick" is subjectively transformed into "tick-tock-tick-tock". While detecting pure music imagery from EEG is hard because of the lack of a time lock, processes like these are ideal for event-related potential (ERP) studies (Brochard et al., 2003; Snyder and Large, 2005; Zanto et al., 2006; Schaefer et al., 2010). A 2-beat meter is nearly automatically induced, with a little effort we could also subjectively transform the clock pulses into a 3-beat ("tick-tock-tock...") or 4-beat meter ("tick-tock-tocktock..."). One could describe this process as "imagined accenting" or "subjective accenting". When instead of a regular imagined pulse a more complex temporal pattern is imagined we speak of "subjective rhythmization". 3

Chapter 1. Introduction

1.2

On subjectivity, concepts and terminology

Composers and performers of classical music often communicate about music with terms, such as "molto vivace" (very much lively), "con brio" (with vigor), or "grazioso" (gracefully). In other genres, terms like "groovy", "fat", or "oomph" are used. While this kind of terminology has a surprising effectiveness in communication among musicians, it is almost impossible to directly link these terms to quantifiable measurements. The highly subjective nature of music complicates matters for scientific discussion and well-grounded universal terminology, but also for science a consistent vocabulary is not that easy to achieve. While a psychologist might talk about subjective accenting as a "fluctuating level of attention" to auditory events, and about "expectancy" or "anticipation" of certain events in a sound pattern, a neuroscientist might describe the same phenomenon as resonance of a group of neurons with both exogenous and endogenous input. Alternatively, a cognitive neuroscientist might be talking about modulation of the N1/P2 complex of the auditory evoked potential. How to bring the worlds of musicians, composers, psychologists and neuroscientists together? How does a concept like attention, that has been defined before we were capable of EEG and fMRI (functional magnetic resonance imaging) measurements, relate to modern neurophysiological observations and insights? I am unable to answer this, but an update of concepts and terminology to fit the recent knowledge of the field of cognitive neuroscience is highly relevant to progress in our understanding of musical behavior and the brain.

1.3

Music and scientific research, the big questions

Several scientific disciplines are involved in music research, among them are psychology, neuroscience, and artificial intelligence. The following paragraphs provide a glance at the kind of questions and mysteries that motivated me (and possibly many of my colleagues) to get involved in music research. • It is amazing how much information is processed, how does the brain do that? What areas are involved in the processing, and are these specialized for music, or shared with other tasks (Levitin, 2006)? How are bottom-up and top-down processes combined into the experience of perception of music? • How come music can elicit emotions (Juslin and Sloboda, 2010), when certain patterns of sounds occur? Are emotions triggered by the sound patterns themselves, such that the same emotion is triggered every time the same piece is played? Or do emotions co-occur because music triggers certain memories or thoughts? 4

Chapter 1. Introduction • Why do we have music? What is the evolutionary advantage (Peretz and Zatorre, 2003)? Is it an artifact of other skills (Pinker, 1997; Huron, 2006)? Music and language (Patel, 2010) for instance share many properties, like time structure and intonation, but are very different on other grounds. • How come we can imagine music so well? Is it a simulation skill? How complex/detailed are the images, or are they just memories of a complex percept (Hubbard, 2010)? Such questions, again, are too big to answer in this thesis. The empirical work in this thesis focuses on traces of music imagery in electroencephalography (EEG). It is complemented by theoretical work, and attempts to answer to the following research questions: • Can music imagery (or more specifically: subjective accenting) be detected in EEG? • How does neuronal processing of music imagery (subjective accents) relate to that of perceived music (perceived accents), are mechanisms shared? • Can voluntarily imagined music patterns (subjective rhythms) be decoded from EEG and used for real-time control of applications (such as a Brain-Computer Interface)? • If such a technology would be developed and widely used, how would it impact society and what ethical concerns arise along the road?

1.4

BCI, motivation and methodology

Now it is the time to introduce you to the field of Brain-Computer Interfacing (BCI), as most of the work in this thesis was performed within the methodological framework common to BCI research. There were several motivations for this methodological approach: • First of all, the machine-learning methods common to BCI research are potentially more sensitive to relevant details in the EEG. Regularized machine-learning methods are more robust to outliers than typical ERP analyses based on averaging, and more suited to the multivariate nature of EEG recordings (e.g. multiple channels and samples). • Second, the use of single trial classification and machine-learning methods enforce much more caution for overgeneralization from a limited set of observations. • Third, there is benefit for two goals: while gaining insight in the brain with respect to music processing, steps are also made towards practical application of those insights in BCI. 5

Chapter 1. Introduction

1.5 1

Definition of BCI

A Brain-Computer Interface (BCI) is a system that allows its user to control a machine (e.g.

computer, automated wheel chair, artificial limb) using purely mental activity, without utilizing the peripheral nervous system (Dornhege et al., 2007; Van Gerven et al., 2009; Nijboer et al., 2011; Nicolas-Alonso and Gomez-Gil, 2012). Control with a BCI is effectuated when a user performs a specific mental task. A typical BCI combines neurophysiological measurement technology with machine learning software to detect patterns of brain activity that relate to this specific mental task. A user is often provided with a few mental tasks (e.g. imagined movement of left hand, right hand or foot) that the system is trained to detect. Once the system detects that the user has been performing one of the mental tasks, the corresponding actions are automatically triggered (e.g. move cursor left, right or click).

1.6

BCI measurement modalities

Implementation of a BCI requires brain activity to be measured. Technology to do so can be categorized as invasive, such as subdural or epidural electrocorticography (ECoG) or an implanted multi-electrode array (MEA), or non-invasive, such as electroencephalography (EEG), magnetoencephalography (MEG), functional magnetic resonance imaging (fMRI) or near-infrared spectroscopy (NIRS). All of these technologies provide some measurement of the brain’s activity. The choice for a specific method is often determined by a balance between factors such as health risks, user comfort, signal quality, portability and cost. While ’wet’ EEG, where conductive gel is used to improve the electrode’s connection to the surface of the scalp, is still the most popular non-invasive method for clinical BCI applications, low-cost, wireless, and ’dry’ alternatives are emerging rapidly, often aimed at less critical consumer applications, such as BCI games and entertainment for healthy users (see for instance Zander et al., 2011; Emotiv, 2013; NeuroSky, 2013). The following sections will focus on the details of EEG as a measurement modality, as this modality was used in the studies presented in this dissertation. EEG is typically recorded on multiple channels with electrodes electrically connecting to the skin on the head. These electrodes register the electrical potentials resulting from neuronal activation in the brain. While EEG’s temporal resolution is relatively high, compared to other methods, its spatial resolution is low. Any measurable response in EEG has to be attributed to larger groups of simultaneously firing neurons. Signals from other sources than the brain are also recorded with EEG, such as external electrical interference, 1 Sections 1.5, 1.6 and 1.9 are expanded versions of sections from Vlek et al. (2012), which is included in this thesis as chapter 4.

6

Chapter 1. Introduction and movement of the eyes and facial muscles. These have to be properly treated or corrected for to ensure the BCI is using only brain signals.

1.7

BCI brain signatures

In a typical BCI system the user performs a mental task that is known to elicit a very specific and measurable brain signal or signature. For an EEG-based BCI system two categories of signatures can be distinguished, namely those evoked by a combination of external (exogenous) stimuli and internal (endogenous) cognitive processes, and those induced solely by internal mental activity. Evoked responses typically are strongly time-, and phase-locked to a stimulus event and observed in a time course of EEG, while induced responses often are less strongly time-locked, oscillatory in nature, and more easily observed after the EEG has been transformed to the frequency or time-frequency domain. Although temporal and spectral features are often treated as two separate domains, it is important to realize that both are a reflection of the same neurophysiological mechanism: the electrical activity of collections of neurons. A variety of evoked responses have been used for BCI purposes, such as the steadystate evoked potential (SSEP) (Middendorf et al., 2000; Allison et al., 2008; Severens et al., 2013), P300 (Farwell and Donchin, 1988; Sellers and Donchin, 2006; Schreuder et al., 2010; Geuze et al., 2012; Van der Waal et al., 2012), error potential (EP) (Schalk et al., 2000), and readiness potential (RP) (Krauledat et al., 2004). The most popular evoked response in present BCI applications is the P300, which is typically elicited in an oddball paradigm, where low probability ’target’ stimuli (which elicit the P300 response) are mixed with high probability ’non-target’ stimuli. Induced responses have also been used successfully for BCI, such as the slow cortical potential (SCP) (Birbaumer et al., 1999), occipital alpha-band power or lateralization effects modulated by attentional mechanisms (Van Gerven and Jensen, 2009; Treder et al., 2011), and most notably the so-called sensory-motor rhythms (SMR) (Pfurtscheller and da Silva, 1999; Pfurtscheller et al., 2006) elicited by specific motor imagery tasks. The gamma-band has also been used successfully for BCI, but seems more relevant for invasive measurement technology (ECoG). When measuring the gamma-band with non-invasive EEG it is often difficult to disentangle gamma-band effects originating from the brain and those generated by muscles in the neighborhood of the sensors. This point is well illustrated by two recent studies, the first showing virtually no gamma-band effects left in the EEG after temporary chemical paralysis of facial muscles (Pope et al., 2009), and the second illustrating how frontal gamma-effects observed in the EEG originate from micro-saccades made by the eyes as part of a visual attention process (Yuval-Greenberg et al., 2008). 7

Chapter 1. Introduction In most BCI experiments, as well as in many other neurophysiological experiments, discrete conditions or classes of mental activity are discriminated. Most BCIs attempt to discriminate between a low number of classes of mental activity, often two. In case of a P300 BCI for instance, the two classes are brain responses to ’target’ versus ’non-target’ stimuli; in a motor imagery BCI the classes can be brain signals related to ’left hand imagery’ versus ’right hand imagery’. Intuitively, one can understand that categorizing mental activity in a few classes does not do justice to known properties and capabilities of the brain. We know the brain is highly plastic, constantly adapting to and learning from past experiences and observations, and constantly influenced by mental and physical conditions (e.g. emotions, stress, fatigue, effects of caffeine or level of blood-sugars). This leads to the obvious conclusion that task-related signals within the ’classes’ used for BCI are not at all uniform. This problem is known as ’non-stationarity’ (Roijendijk, 2009), and makes it difficult to generalize a robust detection scheme for a specific task-related signal. The problem even extends when comparing the brains of different individuals, each with their own physical and mental configurations and past experiences. At this level the problem is known as ’inter-individual variability’ and transfer learning (Roijendijk, 2009).

1.8

Online detection and interpretation

In common neurophysiological research the detection and interpretation of task-relevant components of the brain signals is done ’offline’, after the experiment has taken place. This allows for a wide range of analysis techniques to be experimented with, it allows for statistical properties of the entire dataset to be used, and computation time of a given analysis method is hardly a limitation (see Figure 1.3a). Although a similar approach is often used in the initial phases of BCI research (see Figure 1.3b), the realization of a true BCI requires an ’online’ (and as close to ’real-time’ as possible) approach to detection of task-relevant signals (see Figure 1.3c). In this online approach the detection is automated by means of machine learning techniques, which need to be causal: they have access to all previously collected data, but of course cannot look beyond the ’now’. Depending on the latency requirements of the given BCI, computation time of the machine learning and preceding signal-processing steps also needs to be taken into account.

1.9

BCI applications

BCI technology brings many promising applications in a variety of clinical and non-clinical areas. However, present clinical success with BCI is predominantly achieved with proto8

Chapter 1. Introduction online

offline

electrodes stimulus markers 135 4 3 5 4 32

Offline preprocessing

1000 500

(a)

subject

data slice

20

40

60

80

100

20

40

60

80

100

electrodes

120

apply method

140

140

40

60

subject

data slice

20

80

100

120

140

40

60

80

100

120

140

20

apply methods &

40

60

80

100

remove padding electrodes 1000

120

apply method

data !10000 dataset 160 180 0 stream

135 4 3 5 4 32

!500500

dataamp !1000 0 stream 10000

user

20

40

20

40

60

80

100

60

80

100

120

120

EEG recording 0

20

apply methods & remove padding

40

60

80

160

100

input 120

apply method

140

160

180

200

signal processing

analysis

preprocessing 60 80

100

120

80

100

120

80

100

datasets datasets datasets datasets 40

60

140 160 180 200 truncate/slice truncate/slice && truncate/slice truncate/slice && preprocess& & preprocess preprocess preprocess && feature 140 160 180extraction 200 feature extraction featureextraction extraction feature

apply 60 method

40

load

output 120

140

160

140

160

180

200

180

200

signal processing

classification classification classification classification

performance performance performance performance estimate estimate estimate estimate

machine learning

update state

1801000 200 online 500 10000 !500 500

slice & !1000 10000 classify 0

EEG

!500 !1000

200 20

single !1000 input 160 180 0 200 20 sample

data stream

!500 500

data !1000 0 feedback slice 0

20Online 40

store

output

Offline preprocessing

classify 1000 0

stimuli

140

stimulus markers

500

(c)

output 120

10000

!500 500

EEG recording input

!500 0

100

!500

!1000 00

!1000

80

1801000 200 online offline

!500 500

stimuli

load

update state

slice & !1000 0 classify 10000

EEG 20

160

apply 60 method

40

500

135 4 3 5 4 32

!500500

dataamp !1000 0 stream 10000

result

!500 500

Offline preprocessing

classify 1000 0

single !1000 input 160 180 0 200 20 sample

store

data stream

stimulus markersoutput

500

(b)

120

EEG recording input

apply methods & remove padding 1000

average & statistics

!500

!1000 00

0

datasets datasets datasets

stream

!500 !1000

truncate/slice & preprocess relevant signal

datasets

500

data 0 dataset

!500 500

stimuli

Online preprocessing

1000

EEG

dataamp 0 stream 1000

Online40 preprocessing 60 80

20

truncate/slice & !500 500 data !10000 preprocess & 40 140 160 180 0 200 20 stream feature extraction !500 140

single !1000 input 160 180 0 200 20 sample

100

120

140

160

device or application 180 200

140

160

180

classification 60

80

100

apply 60 method

80

100

40

120

output 120

signal processing machine learning update 140

160

1801000

200

control

state

200

500

data stream

Figure 1.3: Comparison of setupsoutputand methodology in (row a) typical EEG research, (row b) BCI slice & research, and (row c) BCI application. classify Key differences are found in the separation of online and classify offline parts, in the way data is treated, the role of the person as either ’subject’ or ’user’, and the final outcome. 1000

10000

500

!500 500

0

!10000 0

20

40

60

80

100

120

140

160

180

200

200 20

40

60

80

100

120

140

160

180

200

!500

!500

!1000

0

20

40

60

80

100

120

140

160

!1000 180 0

types in research laboratories. Invasive BCI systems have been reported specialized at real-time decoding of motor functions in non-human primates, sometimes even reproducing the decoded motor activity with a cursor or robotic arm (Wessberg et al., 2000; Carmena et al., 2003; Serruya et al., 2002). Eventually, such technology may develop into neuromotor prostheses for humans with motor disabilities. More recently, results of successful neural cursor control by a human with tetraplegia, implanted with a MEA in the motor cortex, were reported (Kim et al., 2008). Both invasive and non-invasive BCI approaches are also being explored for rehabilitation of gait in stroke patients (Buch et al., 2008; Rodriguez et al., 2011). Non-invasive BCI systems, mostly EEG-based, have been investigated for various applications, one of the most popular being communication. Spelling applications for instance, relying on the so-called P300 response, enable a user to ’mentally type’ symbols on a screen by focussing on a specific symbol in a matrix of randomly flashing symbols (usually letters, punctuation symbols and numbers) (Sellers and Donchin, 2006; Nijboer et al., 2008b). Variations to this application have also been presented for use in the auditory (Schreuder et al., 2010) and tactile (Van der Waal et al., 2012) domain, which may be relevant to users suffering from a progressive neurodegenerative disease affecting their vision 9

Chapter 1. Introduction in its later stages (e.g. ALS: amyotrophic lateral sclerosis). One of the key problems in these types of communication applications is speed. In order to achieve sufficient reliability of the system repeated measurement of the P300 response is necessary, resulting in reduction of speed of the application. Attempts have been made to overcome this lack of speed by adding artificial intelligence to the application in the form of automatic word completion from a dictionary, or even sentence based communication databases (Geuze et al., 2008). This increases throughput of language at the cost of verbal freedom. In addition to communication applications, successful prototypes for wheelchair control (Vanacker et al., 2007) or 3-D cursor control (McFarland et al., 2011) have also been reported. Beside current and potential clinical applications of BCI, applications are also being developed for game and entertainment purposes (Nijholt et al., 2009). Thanks to a very large group of potential users, this seems a clever route to increase research and development resources, from which clinical applications may benefit as a side-effect. Moreover, recent work by Münssinger et al. (2010) reports on a BCI painting application, and illustrates that even entertainment BCIs may have a certain degree of clinical relevance by improving a patient’s social and expressive potential. The fact that BCI technology for clinical use, while promising, still resides predominantly in research labs can be explained by several factors. BCI systems are still rather unreliable. They may work for some person, and may not work for someone else. They may suddenly stop working, due to changes in the environment or brain, to which most BCIs are not capable of adapting. The technology required for a BCI is also not as user-friendly and maintenance-free as required for intensive daily operation by an independent user. Finally, artifacts may be a factor contributing to initial success with a prototype followed by a compromised development into clinical application. Especially with EEG-based BCIs, signals originating from other sources than the brain, such as eye-movement and muscle activity, may - sometimes without being noticed - boost BCI performance. When the transition is made from healthy users (typically the first group to try a prototype) to patients, some of the performance previously achieved may get lost as a consequence of the patients physical condition. Much of the on-going BCI research is focussed at finding solutions for these issues.

1.10

Performance of a BCI system

Evaluating performance of a BCI is not as straight-forward as one initially might think. There are many different paradigms around, each with a certain speed, reliability, and type of output (e.g. letters, movement of a cursor, wheelchair commands). In order to compare different systems, their information transfer rate (ITR) is computed (Kronegg et al., 2003). 10

Chapter 1. Introduction This measure takes into account the variables of speed, reliability, and number of classes (underlying the type of output), resulting in a certain number of bits per minute.2 Although ITR is an important measure, other factors may also be taken into account when evaluating a BCI’s performance, such as user comfort and stability of performance over users, and within users over time.

1.11

BCI and ethics

While being involved in BCI research, and closely following its technological development and innovation, my interest was drawn towards ethical aspects of BCI. From an ethical point of view, at first glance BCI seems little more than a variation to EEG research, combined with an engineering aspect. However, BCI is an interdisciplinary field to which neuroscience, artificial intelligence, psychology and many other sciences contribute. A few ingredients of the interdisciplinary field of BCI make it a rather unique topic for ethical discussion (Nijboer et al., 2011; Clausen, 2011). One of these ingredients is the direct, closedloop (see the setup in Figure 1.3c) interaction between brain and machine. In some BCIs users receive feedback about their brain processes that they do not normally receive. This may lead to unforeseen effects on the brain, affecting for instance learning. Another ethical topic related to BCI is the work with locked-in patients as a target user group. Due to their physical condition, these people are unable to communicate in any direct way. Obtaining informed consent for BCI intervention, or taking actions based on the output of an experimental BCI, raises complex ethical questions. If the field of BCI continues to grow further, and produces useful applications to society, its general impact will grow with it. The ethical, legal and societal implications that can arise may well be of comparable size to other technological leaps forward, such as genetic modification, or the internet. Thus, a closer study of ethical issues in BCI seems required and some results of this study are included in the form of chapter 4 and 5 in this dissertation.

1.12

BCI and music

Music has met with BCI research in several occasions. Schreuder et al. (2010) have investigated a P300-based BCI system with auditory stimuli, exploiting our capacity to focus attention on sounds heard from different directions. Schaefer et al. (2011a) have investigated the decoding of tunes perceived by a subject from their EEG, and Schaefer et al. (2011b) 2

’Bits per minute’ is sometimes abbreviated as BPM, but as this conflicts with the musical definition of BPM as ’beats per minute’, the abbreviation of BPM for ’bits per minute’ is avoided in this thesis.

11

Chapter 1. Introduction observed involvement of attentional mechanisms, differentiating between the process of perception and imagery of tunes, as well as within these processes between different tunes. While not part of this thesis, and as of yet unpublished, my interest was also drawn towards experiments with the online detection of imagined rhythms, and the effects of different mental strategies used for imagination (Hessels, 2012). As a side-project, a connection between a P300-based BCI and an electro-mechanically controlled piano and synthesizer was made, resulting in Winifred Atwell’s ’Black and White rag’ played by a human player’s right hand and a BCI controlled left hand part. This experience was a nice diversion, but also a valuable lesson in the problems that arise when a BCI is taken out of the lab. Along the lines of creating music with BCI, see also a recent study by Miranda et al. (2011), where SSVEP (a visual SSEP) was used.

1.13

Outlook on following chapters

The first two chapters following this introduction describe EEG studies performed on the topic of subjective accenting, a musical cognitive task in which participants imagine the presence of a pattern of accents on top of the metronome pulses they perceive. This way a 2-, 3-, or 4-beat meter is subjectively induced. Instructing the subject to imagine something is a tricky issue. We seem to easily activate more modalities in such a task, not only imagining a drum sound as a pure auditory event, but also as a dynamic visual image of a drummer and possibly an imagined movement of our own arm hitting a drum. On the one hand this seems more or less involuntary, especially for musicians, as e.g. sensori-motor activation has been observed in expert musicians during passive listening in an fMRI scanner (Bangert et al., 2006). However, musical sounds that are quite abstract or are produced on an unknown instrument can still be recalled and imagined, and thus we focus in our studies on the aspect of pure auditory imagery. Chapter 2 describes a study on the detectability of subjective accenting in EEG, and the neurophysiological relationship between EEG observations during perception and imagery of accents. Shared information between perception and imagery EEG was investigated by means of principal component analysis and by means of single-trial cross-condition classification. Single-trial classification of accented and non-accented beats was successful from 500ms chunks of 64-channel EEG data, with an average accuracy of 70% for perception and 61% for imagery data. Cross-condition classification yielded significant performance above chance level for a classifier trained on perception and tested on imagery data (up to 66%), and vice-versa (up to 60%). The study revealed similarity in brain signatures relevant to distinction of accents from non-accents in perception and imagery, supporting the idea of shared mechanisms in perception and imagery for auditory processing. 12

Chapter 1. Introduction Chapter 3 is based on the same data and further investigates whether subjective accenting is a feasible paradigm for BCI and how its time-structured nature can be exploited for optimal decoding from EEG data. Several sequence classification approaches are presented and evaluated on their ability to decode the cyclic sequences of subjectively accented and non-accented beats that occur in a 2-, 3-, and 4-beat meter. Classification performances were compared by means of bit rate. The best scenario yielded an average bit rate of 4.4 bits/min over ten subjects. As it is important to not only study the detailed empirical questions but also consider what they mean and how they might influence society at large, the next two chapters consist of studies on philosophical aspects of BCI, considering present and future use of BCI, how BCIs could impact society, and what ethical issues may arise. Chapter 4 describes several case scenarios of BCI use, inspired by current experiences in BCI laboratories. With the help of the case scenarios, and a discussion group of BCI experts, ethical issues were identified and disentangled, thus making the debate of ethical issues in BCI accessible to a wider audience. Issues typical to the field of BCI relate to working with sensitive user groups, dealing with technological complexity and handling multidisciplinary teams. Ethical issues arise where there is a conflict of treatment and research interests. Managing the personal and public expectations of BCI is also important. Chapter 5 further elaborates on the ethical issues in BCI, and connects topics like ’lockedin syndrome’, ’informed consent’, ’shared moral responsibility in teams’, and ’media hypes’ with background knowledge from philosophy and medical sciences. This chapter illustrates that acquiring an ethically sound informed consent from a locked-in patient may be challenging due to the high expectations of the patient, the difficulty in communicating and the lack of alternatives. The chapter also touches on the complexity of shared moral responsibility in multidisciplinary teams typically involved in BCI research. The moral consequences are discussed of the expectations raised by media attention to promising novel BCI technology.

13

14

2. Shared mechanisms in perception and imagery of auditory accents Published as: Vlek, R.J., Schaefer, R.S., Gielen, C.C.A.M., Farquhar, J.D. & Desain, P. (2011). Shared mechanisms in perception and imagery of auditory accents. Clinical Neurophysiology 122 (8), 1526-1532.1

Abstract An auditory rhythm can be perceived as a sequence of accented (loud) and non-accented (soft) beats or it can be imagined. Subjective rhythmization refers to the induction of accenting patterns during the presentation of identical auditory pulses at an isochronous rate. It can be an automatic process, but it can also be voluntarily controlled. We investigated whether imagined accents can be decoded from brain signals on a single-trial basis, and if there is information shared between perception and imagery in the contrast of accents and non-accents. Ten subjects perceived and imagined three different metric patterns (2-, 3-, and 4-beat) superimposed on a steady metronome while electroencephalography (EEG) measurements were made. Shared information between perception and imagery EEG is investigated by means of principal component analysis and by means of single-trial classification. Classification of accented from non-accented beats was possible with an average accuracy of 70% for perception and 61% for imagery data. Cross-condition classification yielded significant performance above chance level for a classifier trained on perception and tested on imagery data (up to 66%), and vice-versa (up to 60%). Results show that detection of imagined accents is possible and reveal similarity in brain signatures relevant to distinction of accents from non-accents in perception and imagery. Our results support the idea of shared mechanisms in perception and imagery for auditory processing. This is relevant for a number of clinical settings, most notably by elucidating the basic mechanisms of rhythmic auditory cuing paradigms, e.g. as used in motor rehabilitation or therapy for Parkinson’s disease. As a novel Brain-Computer Interface (BCI) paradigm, our results imply a reduction of the necessary BCI training in healthy subjects and in patients.

1 Chapters 2 and 3 make use of a single EEG dataset, analysed from different angles and answering a different set of questions

15

Chapter 2. Shared mechanisms in perception and imagery of auditory accents

2.1

Introduction

Our sense for auditory rhythms, such as a pattern where every first beat out of two, three or four beats is accented, is generally well-developed (Michon and Jackson, 1985). These rhythmic structures in western music are usually stereotyped as a march (ONE-two), waltz (ONE-two-three) or common rock rhythm (ONE-two-three-four). It has been shown that this sense for rhythm is not only relevant for the perception and production of music (London, 2004), but also plays a role in speech (Vatikiotis-Bateson and Kelso, 1993) and in motor control tasks (Kelso, 1982). In these processes the underlying mechanism of entraining a mental oscillator by its coupling to periodic external stimuli may follow the theoretical accounts of complex dynamics (Large et al., 2002). Although in music, on top of the regular framework of beat and meter, rhythm is imposed which may be far from periodic and predictable, the importance of processing regular periodic stimuli is clear across modalities. This may point to general cognitive mechanisms and may explain why the use of auditory rhythms or cues has become increasingly popular in clinical environments for rehabilitation purposes. Motor rehabilitation has shown an increase in effect with the addition of external auditory cues, especially for gait rehabilitation (as in Roerdink et al., 2007), but also in bilateral arm training (see Latimer et al., 2010). Apparently the rhythmic processing adds something to the rehabilitation process. This is also seen in other types of time-structured therapies; used in, for example, dyslexia (Overy, 2003), aphasia (Belin et al., 1996), Parkinson’s disease (McIntosh et al., 1997; Willems et al., 2007), as well as a number of cognitive functions (Thaut, 2010). Interestingly, it is not necessary for the rhythmic cue to be externally presented, as patients have also been able to increase the efficacy of their rehabilitation while moving to their own internal or imagined rhythm to improve gait (Schauer and Mauritz, 2003). The ease at which such imagined rhythms can take place is nicely demonstrated by the so-called clock illusion or ‘tick-tock’ effect (Brochard et al., 2003). When a series of isochronous and equal sounding pulses is presented, such as the sound of a clock (‘tick-tick-tick-tick...’), the percept of a rhythmic pattern is usually automatically induced, consisting of subjectively added accents on every second beat (‘tick-tock-tick-tock...’). The mechanism of the brain inducing these accents is known as subjective rhythmization (Bolton, 1894; Fraise, 1982; London, 2004). As the mechanism of rhythm processing is not fully understood yet, we investigate the electrophysiology of simple rhythm processing, both with externally presented and internally generated accents. Several studies have looked into the perception of metric patterns and stimulus-induced responses in electroencephalography (EEG). These studies have shown, that both the perception of metric patterns (Snyder and Large, 2005) as well as the expectation of an accent is reflected in EEG-activity (Zanto et al., 2006; Jongsma et al., 2005; Snyder and Large, 2005). 16

Chapter 2. Shared mechanisms in perception and imagery of auditory accents Brochard et al. (2003) found that, for loudness deviations in a steady pulse train, subjects automatically exhibited different neuronal responses to deviants in even and odd positions, reflecting binary chunking. In a recent study Snyder and Large (2005) reported that (non phase-locked) gamma-band activity in EEG can reflect the metric structure of the stimulus and that at an omission of a stimulus this activity may persist. This suggests that a form of imaginary rhythm or internal clock is active. Subjective accents can also be added voluntarily, thus making it a deliberate process. Iversen et al. (2009) investigated this phenomenon and describe an effect in the upper beta-band of magnetoencephalography (MEG) measurements at subjectively accented versus non-accented tones. Studies investigating auditory imagery of rhythms or accents with EEG are scarce (but for exceptions, see Desain and Honing, 2003; Schaefer et al., 2010). A recent study by Navarro Cebrian and Janata (2010) investigated the effect of auditory imagery on the N100 component of the auditory event related potential (ERP) evoked by a target tone following a sequence of imagined tones. They reported a correlation between the N1 (a.k.a. N100) amplitude and the vividness of imagery, converging towards identical N1 amplitudes for perception and extremely vivid imagery. The relationship between mental imagery and perception and any similarities between neuronal structures involved in these processes, have been studied for different sensory modalities. Similarly, the relationship between imagery and actual motor activity has been studied. For a comprehensive overview on these relationships see (Kosslyn et al., 2001). Very similar neural activation patterns have been reported for actual and imagined movement tasks, in terms of mu and beta-band desynchronization over sensorimotor cortex (McFarland et al., 2000; Munzert et al., 2009). Strong support has also been reported for shared mechanisms in visual perception and visual imagery tasks. A study by Kosslyn et al. (1995) showed that during visual imagery the primary visual cortex is activated. Compared to the visual and motor domains, the number of studies focussing on the relationship between auditory perception and imagery is relatively small. Support for shared mechanisms in auditory perception and imagery comes from behavioural (e.g. Farah and Smith, 1983; Halpern et al., 2004) as well as clinical angles (e.g. Kasai et al., 1999; Shinosaki et al., 2003). Zatorre et al. (1996) reported evidence from a positron emission tomography (PET) study for activation of parts of the auditory cortex during perception as well as during imagination of music. A more recent fMRI study by Kraemer et al. (2005) reported activation of secondary and primary auditory cortices during silent gaps in familiar tunes, where subjects reported the experience of continuation of the tune in imagery. However, support for the idea of shared mechanisms is predominantly found in studies concerning timbre or pitch aspects of music (for an overview see Halpern, 2001; Hubbard, 2010). Several studies have identified that timbre and pitch aspects are represented in our ’auditory

17

Chapter 2. Shared mechanisms in perception and imagery of auditory accents mental image’, but interestingly, for loudness aspects this has never been shown (Hubbard, 2010). The idea of shared mechanisms in rhythm processing is supported by results of a recent study by Schaefer et al. (2010), where overlap was found between ERP responses to events in perceived and imagined rhythms. Recent developments in single-trial multivariate decoding of EEG signals, generally carried out in the context of brain-computer interface research (Dornhege et al., 2007; Van Gerven et al., 2009), can also be used to uncover patterns of brain activity that were previously not detectable. Single-trial multivariate decoding methods are often specialized in dealing with inter-trial variance and outliers, such that an optimal generalization and detection of the effect is possible. The aim of this study was to investigate whether it is possible to decode auditory accents from brain signals on a single-trial level, in both perceived and imagined auditory accenting patterns. Furthermore, we test the hypothesis that similar brain structures are involved in perception and imagination of accenting patterns, superimposed on a train of auditory beats. In a perception condition, subjects listened to a stimulus where the accents were physically different from the non-accents. In an imagery condition, subjects were listening to identical stimuli without accents, while they were instructed to imagine the accents. As pointed out by Hubbard (2010), a common problem in imagery studies is the lack of control for the process of imagery. In our study a behavioral task at the end of each imagery sequence guarantees a check on imagery processes. Differences between accented and non-accented beats were found with a principal component analysis and by means of single-trial classification, hereby expanding on the work previously reported with different data in a similar experimental design (Schaefer et al., 2010). Classification rates on both imagined and perceived accents are reported and comparison and interpretation is done for the discriminative signal properties in both experimental conditions. We hypothesize that similar brain structures are involved in imagery and perception of auditory accents, and test this hypothesis by classification of data with a cross-condition classification approach. This method allows to search for information shared between the conditions in the contrast of accented and non-accented beats on a single-trial level.

2.2 2.2.1

Experiment and analysis Experimental design and data acquisition

Ten subjects, five females and five males, aged between 22-34 years, participated in this study. One subject had a professional musical training, and six participants actively play a musical instrument. None of the subjects reported to be diagnosed with any neurological disorder or hearing deficiency. The experiment was undertaken with the understanding 18

Chapter 2. Shared mechanisms in perception and imagery of auditory accents and written consent of each subject, approved by the ethical committee of the faculty of social sciences at the Radboud University Nijmegen, and in compliance with national legislation and the code of ethical principles for medical research involving human subjects of the World Medical Association (Declaration of Helsinki). Subjects were seated in a comfortable chair in an electrically and acoustically shielded room at a distance of approximately 0.5 m from a 17” TFT computer monitor. Two speakers (Monacor, type MKS-28/WS), placed on each side of the monitor, were used to present auditory stimuli to the subjects (stimuli can be found online at http://www.nici.ru.nl/mmm/). A Biosemi active-electrode set (Ag-AgCl) with 64 electrodes was used in combination with an ActiveTwo AD-box to measure EEG at a sampling frequency of 2048 Hz. No further filtering or processing was done at the stage of recording. Simultaneously with the EEG, an electro-oculogram (EOG) was made to exclude eye movements as a possible source of information during EEG classification. Two pairs of auxiliary electrodes were placed. One pair was positioned above and below the left eye to measure eye movements in vertical direction. The other pair was positioned on the temples to measure horizontal eye movements. The stimulus sequences consisted of three phases, a perception phase, a fade and an imagery phase. A metronome was playing throughout the whole sequence (see Figure 2.1). In the perception phase of the sequence, an accent was superimposed on the metronome every two, three or four beats, thus creating binary, ternary and quaternary patterns. The metronome played at 120 BPM (beats per minute), resulting in inter-onset-intervals of 0.5 seconds between successive ticks. The rate of 120 BPM is chosen to avoid overlap of the expected perceptual EEG responses, such as the auditory evoked potential (AEP), which can have components as late as 400 ms (Burkard et al., 2007), and to stay close to a tempo that is easy to track by human subjects (Fraise, 1982). The sound was presented at a peak level of 57dB(A) for all subjects. In the perception phase, accents were added with the general MIDI sound ‘high woodblock’. This accent increased the peak loudness of the stimulus to 65dB(A). During the fade phase, as a transition from the perception to the imagery phase, the accents were played less loudly, decreasing the peak loudness of the stimulus to 61dB(A). In the imagery phase the accent was no longer added. A sample sequence is illustrated in Figure 2.1, showing a sequence of 3-beat patterns. At the start of each sequence, a white fixation cross of 3 cm was shown on the monitor. The appearance of the cross indicated the start of a sequence to the subject and served as a fixation point for the eyes throughout the sequence. After a random delay in the range between 1.0 and 1.8 s after the onset of the fixation cross, the pattern started. The accented pattern was first played for three measures, which is indicated as the perception phase in

19

Chapter 2. Shared mechanisms in perception and imagery of auditory accents perception

fade

imagery

probe response accent metronome

0.5s

time

Figure 2.1: The structure of a single sequence in the experiment is shown, in this case for a 3-beat pattern. The sequence started with three repetitions of a ternary metric pattern (perception phase), followed by one repetition (fade phase), where the intensity of the superimposed accent was reduced by 4 dB. Then the subject had to imagine the accenting pattern for five repetitions (imagery phase). At the end, an accented beat was presented to test whether the subject maintained the correct rhythm.

Figure 2.1. Subsequently, the pattern was played for one measure during the fade phase followed by five measures containing only the metronome, called the imagery phase. In the imagery phase subjects were explicitly instructed to imagine hearing the continuation of the accent pattern, and not to use any other strategies, such as counting, imagining bouncing balls or tapping hands to maintain the rhythm. During the experiment, subjects were visually observed to control for hand, head or other body movements to make sure that no artifacts would influence classification. To check whether the subjects did not lose track of the accenting pattern, a probe accent was sounded at the end of the sequence and the subjects had to answer the question whether this probe would have coincided with the accent in the pattern, if the accenting sound had not stopped playing. Probe accents were randomly placed on either accented or non-accented positions at the end of the sequence. This information was later used to check the subject’s answers. Each next sequence was started with a button press, giving the subject the opportunity to control the interval between sequences, and the opportunity to move freely between sequences. However, during the sequences they were asked to sit still and minimize any eye movements and eye blinks. A block in the experiment consisted of 12 sequences of 2-, 3-, and 4-beat patterns, giving a total of 36 sequences in a block. The order of beat patterns in a block was randomized before the start of the experiment. With 4 of these blocks per subject we gathered 12×4×5 = 240 cycles of each imagery pattern and 12 × 4 × 3 = 144 cycles of each perception pattern. Some of the cycles were rejected in further analyses, due to artifacts (see Section 2.2.2).

2.2.2

Preprocessing

The raw EEG signal was sliced in chunks of data around the markers indicating the presentation of a metronome tick. This means that the 2-, 3- and 4-beat cycles from the stimulus sequences are split into individual beats. A time window of -50 ms to 450 ms was chosen around each metronome tick where time 0 ms corresponds to the time of the tick. The time window was chosen to start 50 ms before each metronome onset, to capture possible 20

Chapter 2. Shared mechanisms in perception and imagery of auditory accents anticipatory responses to the coming event. We aim to detect contrasts in the imagined auditory image, not in any co-incident imaginary movement preparation that would be initiated much earlier. These data segments of 500 ms will be called trials. Bad channels were identified for each trial individually with an algorithm sensitive to four properties. Initially, any channel with a DC offset exceeding 30 mV was marked as ‘bad’, as well as channels exceeding 3500 µV2 of power in the 50 Hz band (45 to 55 Hz) or with a maximum derivative exceeding 200 µV/sample. Horizontal and vertical EOG channels were bandpass filtered between 0.2 and 15 Hz and decorelated from the EEG (Schlögl et al., 2007), thus removing eye drifts or blinks if present. The EEG signal, originally sampled at 2048 Hz, was temporally down sampled to a sample frequency of 128 Hz. Additionally, as a fourth property for identification of bad channels, within-trial variance was computed and channels exceeding a variance of 2000 µV2 were marked as ‘bad’. If - according to the four criteria - more than 20% of the channels in a trial were bad, the trial was excluded from further analysis. Trials from a sequence with a wrong answer to the probe accent at the end of the sequence, were also excluded. Trials coming from the first cycle of the perception or imagery phase of a sequence were rejected to avoid possible transient effects. For the remaining trials, bad channels were reconstructed by interpolation from the remaining good channels with a spherical spline interpolation algorithm (Perrin et al., 1989). The exact number of trials after artifact rejection and merging from different beat patterns is shown in Table 2.1 for each condition. For these trials the average number of bad channels that had to be reconstructed by interpolation was 1.5, with a maximum of 12 bad channels per trial. The interpolation step assures a stable number of good channels for the classifiers to work with, while avoiding rejection of channels throughout the whole data-set when channels are only occasionally bad. The remaining trials were re-referenced to a common average reference (CAR) and linearly de-trended. Trials were further processed using Fieldtrip (http://www.ru.nl/fcdonders/fieldtrip/) functions. A high pass filter with a 3-dB cut-off at 0.5 Hz and low pass filter with a 3-dB cut-off at 15 Hz were applied. Both filters were of a 6th order Butterworth type. A more common low-pass cut-off at 40 Hz was also tried, but only had a negative effect on classification performance (see Section 2.2.4), presumably due to additional noise without additional information about the classes.

2.2.3

Principal component analysis

For a better view on the neurophysiological response to perceived and imagined accents a principal component analysis (PCA) was performed (see Dien and Frishkoff, 2005, for an example of PCA on ERP data). Grand average ERPs over all subjects were computed, as well as ERPs per subject (the number of trials averaged to obtain these ERPs is shown in 21

Chapter 2. Shared mechanisms in perception and imagery of auditory accents Table 2.1: The exact number of trials per subject, after artifact rejection and merging accented and non-accented beats from different beat patterns for each condition. In the perception condition 144 cycles of each pattern were collected per subjected, but for analysis the first cycle is not used. The remaining 96 cycles result in a maximum of 96 × 3 = 288 accented and 96 × 6 = 576 non-accented trials, as each 2-, 3-, and 4-beat pattern contains one accented beat and one, two or three non-accented beats, respectively. Similarly, the maximum number of trials in imagery is 192 × 3 = 576 accented and 192 × 6 = 1152 non-accented trials. Due to the randomized position of the probe-accents a small amount of additional trials may be available after the five imagery cycles (e.g. see the additional accented beat available in Figure 2.1). If available, these trials were used for analysis and explain why the previously computed maximum is sometimes exceeded.

subject S1 S2 S3 S4 S5 S6 S7 S8 S9 S10

perception accented non-accented 284 564 226 452 276 558 260 535 276 556 249 502 243 471 266 514 281 567 278 559

imagery accented non-accented 638 1162 509 935 626 1150 590 1115 618 1143 546 1006 549 977 601 1061 632 1165 620 1149

Table 2.1). These ERPs were decomposed per experimental condition by means of PCA, resulting in spatial distributions and time-courses for each component, ordered according to magnitude of the eigenvalue (i.e. amount of variance in the data explained by the component). For the time-course of each component, differences between accented and non-accented beats were statistically tested using a cluster randomization test, a nonparametrical statistical test designed to deal with the multiple comparison problem present in EEG data, using physiologically motivated constraints to increase the sensitivity of the test (Maris, 2004; Maris and Oostenveld, 2007). A main advantage of the PCA is that its output is easy to visualize and interpret. However, this approach works with averaged ERPs and averaging is a fairly coarse way to obtain a generalization of the neurophysiological response to specific mental tasks. Alternatively, a different approach was pursued that has the potential to generalize more accurately over the individual trials.

2.2.4

Classification

A machine-learning paradigm was used to classify EEG data corresponding to accented and non-accented beats for the perception and imagery conditions, respectively. Classification of the trials was done using a regularized (L2 ) Logistic Regression algorithm 22

Chapter 2. Shared mechanisms in perception and imagery of auditory accents (Bishop, 2006). Given the experimental design, many more trials were available for the non-acccented condition than for the accented condition. In order to avoid a bias towards the non-accented class in the classifier’s output, the classifier’s loss-function was weighted per class to compensate for the unbalanced training data, such that both classes become equally important. To measure the performance of the classifier, 10-fold cross-validation (Bishop, 2006) was performed. The protocol for 10-fold cross-validation was performed multiple times, each with a different regularization parameter, and the best performance is reported1 . Since test-sets can be unbalanced and since we want to enforce equal importance of classes, balanced classification rates - defined as the average of per-class performance will be reported. For classification of perception data, 713 trials were used on average for training the classifier, and 79 for testing. For classification of imagery data, training sets contained on average 1601 trials, using 178 trials for testing. All available time-points per trial within the range of -50 to 450 ms around stimulus onset on all available EEG channels (64) were used as features for classification. In addition to the separate classification of imagery and perception data, ‘cross-conditional classification’ was performed. For this approach a classifier was trained on imagery data and tested on perception. A 10-fold cross-validation regime on the imagery data was used to find the optimal regularization parameter, and a classifier using this regularization parameter was retrained on all available imagery data. To avoid a structural preference of the retrained classifier to one of the classes in the new (perception) test set, calibration was performed by a restricted retraining of the classifier’s bias and gain (see the work of Shenoy et al., 2006) on a random set of 200 trials of perception data, while aiming for equal per-class performance. The trials used for calibration were not used for performance evaluation. In a similar fashion, cross-conditional classification was also performed in the other direction, training a classifier on perception and testing it on imagery data.

2.3

Results

For a better understanding of the classification performance on this data set, a characteristic of the signal relevant to discrimination of accented and non-accented beats is presented first. A PCA was performed on grand average ERPs and ERPs of a representative subject (S4), and the resulting spatial distributions and time-courses corresponding to the first 1 This approach to the selection of a regularization parameter has a potential of over-fitting, but was chosen for computational reasons. As a check, the reported performances were compared with those obtained with a double-nested cross-validation protocol with 10 outer and 5 inner folds, which is robust against this type of over-fitting. As performances did not significantly differ, it can be concluded that in this particular case the regularization parameter selection procedure does not lead to over-fitting.

23

Chapter 2. Shared mechanisms in perception and imagery of auditory accents Perception (GA) comp.1 (expl.var.:53.6%)

Perception (GA) comp.1

Imagery (GA) comp.1 (expl.var.:52.0%)

Imagery (GA) comp.1

15

0 −0.1

10 amplitude (µV)

0.1

15 0.2

5

0.1

0

0

−5

−0.1

−10 −0.2 −15

accented non−accented 0

Perception (S4) comp.1 (expl.var.:96.2%)

0.1

0.2 0.3 time (s)

0.4

10 amplitude (µV)

0.2

−15

(a)

Perception (S4) comp.1

0.1

0.2 0.3 time (s)

0.4

(b)

Imagery (S4) comp.1 20

10

0.1

0

0 −0.1

−10 accented non−accented

−0.2 −20

0

0.1

0.2 0.3 time (s)

amplitude (µV)

0.2 amplitude (µV)

−0.1

accented non−accented 0

Imagery (S4) comp.1 (expl.var.:93.1%)

0.2

0

0 −5 −10

−0.2

20

0.1

5

10

0

−10 accented non−accented

−0.2 0.4

−20

(c)

0

0.1

0.2 0.3 time (s)

0.4

(d)

Figure 2.2: A decomposition by means of PCA was made per experimental condition of grand average ERPs and ERPs from a representative subject (S4). The number of trials averaged to obtain these ERPs is given in Table 2.1. Panels (a) and (b) show the topographical distribution and time-course of the first PCA component of the decomposed grand average ERPs for perception and imagery, respectively. Panels (c) and (d) show the same for the decomposed ERPs of a single representative subject (S4). Areas contributing to a significant (p < 0.05) difference between accented and non-accented trials are marked by grey bars on the time axis. Similarity can be observed between the grand average and the single subject data, but more importantly, there is a large similarity for the perception and imagery conditions.

PCA component, i.e. the principal component with the largest eigenvalue, are shown in Figure 2.2. For the grand average, the first PCA component for perception (Figure 2.2a) explains 53.6% of the variance, while for imagery (Figure 2.2b) this is 52.0%. For subject S4 the first PCA component for perception (Figure 2.2c) explains 96.2% of the variance and explains 93.1% for the imagery data (Figure 2.2d). Areas in the time-courses contributing to significant differences (p < 0.05) are marked in grey on the x-axes in Figure 2.2. When looking at the topographical distributions of the first PCA component in the grand average data, a positive fronto-central distribution with a negative occipital counterpart can be observed for both the perception (Figure 2.2a) and imagery (Figure 2.2b) condition. Similar topographical distributions can be observed for subject S4, where perception (Figure 2.2c) shows a striking resemblance to the imagery (Figure 2.2d) condition. The distributions appeared to be relatively invariant over subjects, resulting in great similarity in the grand average distributions and single subject distributions, here illustrated by showing data for S4. In our work the first PCA component explains more than half of the variance, for one subject even much more. To see in how far other components can show a contrast between the two tasks (imagery versus perception), which is an interesting issue in itself, is better addressed by other methods that can simultaneously decompose both the 24

Chapter 2. Shared mechanisms in perception and imagery of auditory accents task and condition dimensions (Schaefer et al., 2013). Mapping these topographic distributions to the underlying cortical sources is not straight forward. Despite the apparent fronto-central focus, we interpret this topography as corresponding a pair of sources located in the auditory cortices. It has previously been shown (Mayhew et al., 2010) that due to the orientation of these sources current-dipoles their activation ‘projects’ onto the scalp as a positivity fronto-centrally and a negativity occiptally. Though the topographies appear to show motor, premotor and SMA activity this is not the case. Further, these distributions reflect time-domain activity, not induced activity as would be expected for motor involvement. In the time-courses of the PCA components, a significant (p < 0.001) difference between accented and non-accented beats was found around 180 ms for both perception and imagery (Figure 2.2). In the grand average data, the second positive peak at this latency, presumably the P2 component of the AEP, shows a larger amplitude for accented beats than for non-accented beats in the perception condition. In the imagery condition of the grand average, this effect does not become significant until 200 ms. In the data of subject S4 the effect appears to be reversed, showing a significantly (p < 0.01) larger positive deflection around 180 ms for non-accented beats than for accent ones in both imagery and perception condition. This illustrates the variability over subjects. For a more detailed decomposition of grand average data from a similar experimental design, and interpretation of responses to accented and non-accented beats with respect to their function in the overarching rhythm patterns, see Schaefer et al. (2010). In both the grand average data and subject S4 data, time-courses also show a significantly (p < 0.001) larger negative deflection after approximately 350 ms for accented beats, than for non-accented beats (Figure 2.2). This effect was found both for the perception and imagery conditions, but occurs slightly later in the grand average than in the data of subject S4. It persists until the end of the used time-window and possibly beyond, which may be important when interpreting the early effect between -50 to 50 ms. This early effect is presumably the same long-latency effect, carrying over from the trials where a non-accented beat is preceded by an accented beat. By dividing the non-accented beats into two groups, depending on whether they were preceded by an accented beat, we found a significant difference (p < 0.05) for both imagery and perception between these groups in the time interval between -50 and 50 ms for both the grand average and the single subject S4. This supports the idea that the strong negative deflection for accented beats starting around 350 ms persists beyond our time-window. Next, a classifier was trained to distinguish accented and non-accented beats, based on single-trial 64 channel time-domain data. Classifiers that implicitly build a spatial filter usually benefit from such high dimensional multi-channel data in which also channels

25

Chapter 2. Shared mechanisms in perception and imagery of auditory accents Imagery 1

0.9

0.9

0.8

***

***

***

***

0.7

***

***

Classification rate

Classification rate

Perception 1

*** ***

*** *** *

0.6 0.5

S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 AVG Subject

0.8 ***

***

***

***

0.6 0.5

(a)

***

0.7 ***

***

*** ***

***

***

S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 AVG Subject

(b)

Figure 2.3: Balanced classifier performances for the classification of accented versus non-accented beats per subject and as subject-average (AVG) for both the perception (panel (a)) and imagery condition (panel (b)). The error bars indicate the standard error within the 10-fold cross-validation. Significance levels from chance level (0.5) are indicated by * for p < 0.05 and *** for p < 0.001.

that are non-informative of the signal itself still contribute by characterizing the noise. Of course the high dimensionality comes with a risk of over-fitting that needs to be properly restrained by regularization as was done here. Its performance per subject is visualized in Figure 2.3. For the perception condition, the best subject (S4) reaches a classification rate of 74.9%, while on average subjects reach 69.6%(SD=5.1%). For the imagery condition, the best subject (S3) reaches 65.7% correct classification, while the average over subjects is 60.8%(SD=3.9%). These results illustrate that not only perceived accents, but also subjective or imagery accents, not present in the stimulus, can be decoded from brain signals at the level of single trials. Since the long latency (>350 ms) difference between accented and non-accented beats persists beyond the used time window, this could potentially influence classification performance. To check this, the data was processed and classified in exactly the same way as described, but the 500 ms time-window for a trial was now shifted forward 100 ms (originally starting at -50 ms and now at 50 ms). This way we avoid that the long-latency effect of the accented trials is leaking into the non-accented trials. Classifier performance did not significantly differ from the results achieved with the early time-window starting at -50 ms. This means that the classifier’s performance was not unjustifiably boosted by our initial choice of the time window. The preprocessing step, where EOG channels were de-corrrelated from the EEG, appeared to be a useful method for removing most of the eye artifacts in the EEG data. However, the method cannot guarantee that artifacts are completely removed. To make sure that eye artifacts possibly remaining are not providing the classifier with class-relevant information, classification was performed on the EOG channels alone. None of the subjects achieved a performance significantly different from chance level in this case. 26

Chapter 2. Shared mechanisms in perception and imagery of auditory accents Trained on Perception − tested on Imagery 1

0.9

0.9 Classification rate

Classification rate

Trained on Imagery − tested on Perception 1

0.8 0.7 ***

0.6 0.5

*

***

*** *

***

***

0.7

*** ***

0.6

***

S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 AVG Subject

0.8

0.5

(a)

**

***

*** ***

***

S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 AVG Subject

(b)

Figure 2.4: Balanced classifier performances for the classification of accented versus non-accented beats are shown for all subjects and as subject-average (AVG) for a classifier that is trained on imagery and tested on perception (panel (a)), and the other way around (panel (b)). The error bars indicate the standard error within the 10-fold cross-validation. Significance levels from chance level (0.5) are indicated by * for p < 0.05, ** for p < 0.01 and *** for p < 0.001.

The similarity between PCA components for the perception and imagery data in Figure 2.2 suggests that there is shared processing for perception and imagination. To investigate this further, cross-condition classification has been carried out. Figure 2.4 shows balanced classification performances for all subjects when a classifier was trained on perception (on average 792 trials) and tested on imagery data, and the other way around (where the training set on average consisted of 1779 trials). For a classifier trained on imagery and tested on perception data performance was 59.8% for the best subject (S3) and an average over subjects of 54.9%(SD=3.7%). Classification rates are significantly (p < 0.05) different from chance level for 6 out of 10 subjects. The classifier trained on perception and tested on imagery data yielded a performance of 66.4% for the best subject (S4), and an average performance of 58.8%(SD=5.8%) over all subjects. Classification rates are significantly (p < 0.05) different from chance level for 7 out of 10 subjects. For some subjects cross-conditional classification did not yield performances significantly different from chance level. This predominantly concerns subjects that also performed worse in imagery and perception (e.g. S9), which seems to point to a more general problem of signal strength.

2.4

Discussion

The results of this study show that it is possible to decode perceived and imagined accents from brain signals on a single-trial level with a mean accuracy of 69.6% for perception and 60.8% for imagery, using 500 ms of data. We described a method for cross-condition classification and showed that it is possible to classify imagery data significantly above chance with a classifier trained on perception data, and vice versa. The fact that cross-condition 27

Chapter 2. Shared mechanisms in perception and imagery of auditory accents classification is possible, indicates that trials from the perception and imagery condition contain shared information about the presence or absence of accents. Independent classification of EOG signals did not result in a performance significantly above chance for any of the subjects, which allows us to conclude that possible eye-artifacts were not unjustifiably boosting classifier performance. According to the AEP literature (Mulert et al., 2005; Burkard et al., 2007; Mayhew et al., 2010), the N1 and P2 components in the AEP may be influenced by intensity differences as are present between accented and non-accented stimuli in the perception condition. Considering that these stimuli also differ in harmonic content, the effect of timbre (or harmonic content) on P2 amplitude, reported by Meyer et al. (2006), may also be relevant. The N1 and P2 can be identified in the decomposed ERPs in Figure 2.2 at about 100 and 180 ms, respectively. Although no significant difference in the N1 component is observed, possibly due to relatively small intensity differences between accented and non-accented beats, the P2 is significantly different between perceived accents and non-accents (see Figure 2.2a and 2.2c). Surprisingly, the P2 in the imagery condition is also significantly modulated (see Figure 2.2b and 2.2d). This could reflect the imagined difference in timbre between accented and non-accented beats. Although some form of imagery process was guaranteed by the behavioural task in our experimental design, it is hard to tell what strategy subjects exactly used. Therefore, it is not unlikely that the inter-subject variability observed for the P2 modulation reflects the differences in imagery strategies. An important effect observed for the distinction of accented and non-accented beats, is a late (>300ms) effect, yielding a larger negative deflection for accented, than for nonaccented beats. Both timing and scalp distribution of this effect are consistent with the findings of Schaefer et al. (2010). Similar effects were also observed in a different musical task (Pearce et al., 2010), but a clear explanation of the underlying mechanism is not available yet. A connection to mechanisms for selective attention, with resulting ‘processing negativity’ (Naatanen, 1982) or contingent negative variation (CNV) (Chen et al., 2009) effects, is not unlikely, but highly speculative (mainly because both effects have usually been reported to occur at different latencies). As shown in Figure 2.2, signals relevant to the discrimination of accents from nonaccents have an almost identical topographical distribution over the scalp and reveal much resemblance in the time-course of the perception and imagery condition. In combination with positive results of cross-condition classification, this suggests that similar brain areas are involved in both perception and imagery. This is supported by the finding of shared brain areas for musical perception and imagery as reported by Zatorre et al. (1996) and Kraemer et al. (2005) and described by Halpern (2001) and Hubbard (2010). The finding of shared mechanisms for rhythm processing seems to complement existing literature on

28

Chapter 2. Shared mechanisms in perception and imagery of auditory accents shared mechanisms for timbre and pitch. Nonetheless, we are well aware that there are large differences between the two processes too, like the absence of motor planning in some instances of imagery. The importance of preparation for perceptual consequences of an upcoming production is limited for both perception and imagery, though maybe e.g. a vivid imagery of drumming may even include imagining the tactile sensations of the real act. Comparing the PCA analysis and single-trial classification, we conclude that both methods led to the same conclusion regarding information shared between perception and imagery data. Although the single-trial classification procedure potentially allows for a much better generalization of an effect than PCA on the averaged ERP, it is also more susceptible to noise and variance. Issues like non-stationarity in brain responses, in both background activity and responses related to the task, as well as noise sources outside the brain (from the environment or other parts of the body), cause performances to deviate from 100% correct classification. Independently of the neurophysiological or cognitive implications, our results will certainly be beneficial for research and applications in Brain-Computer Interfacing. Since subjective accents can be voluntarily added, it allows users to encode their intentions in their brain signals. The method described in this paper can be utilized to decode these intentions (for application of this approach see Vlek et al., 2011a). The fact that a classifier can be trained on perceptual data and later applied to imagery data adds extra convenience, because this takes much load off the subject (listening is much easier than imagining). A Brain-Computer Interface based on this auditory paradigm, may be able to augment or restore (communicative) abilities of patients (e.g. those suffering from amyotrophic lateral sclerosis or spinal cord injury). The commonalities between perception and imagery also have implications for cue based rehabilitation tasks. Based on these results, we might expect similar brain activations for self-generated rhythms in movement as in externally cued rhythms. Although a number of relevant aspects of cuing, such as tempo and stimulus complexity, were not investigated here, there is support for the notion that we can also deliberately generate the brain activity that would normally result from hearing a rhythmic pattern. The practical implications of these findings need to be further investigated, but considering the broad nature of functions apparently effected by rhythmic processing, the current results show promise in multiple clinical directions.

29

30

3. Sequenced subjective accents for brain-computer interfaces Published as: Vlek, R.J., Schaefer, R.S., Gielen, C.C.A.M., Farquhar, J.D. & Desain, P. (2011). Sequenced subjective accents for Brain-Computer Interfaces. Journal of Neural Engineering, 8(3), 036002.1

Abstract Subjective accenting is a cognitive process in which identical auditory pulses at an isochronous rate turn into the percept of an accenting pattern. This process can be voluntarily controlled, making it a candidate for communication from human user to machine in a Brain-Computer Interface (BCI) system. In this study we investigated whether subjective accenting is a feasible paradigm for BCI and how its time-structured nature can be exploited for optimal decoding from non-invasive EEG data. Ten subjects perceived and imagined different metric patterns (2-, 3-, and 4-beat) superimposed on a steady metronome. With an offline classification paradigm, we classified imagined accented from non-accented beats on a single trial (0.5s) level with an average accuracy of 60.4% over all subjects. We show that decoding of imagined accents is also possible with a classifier trained on perception data. Cyclic patterns of accents and non-accents were successfully decoded with a sequence classification algorithm. Classification performances were compared by means of bit rate. The best scenario yields an average bit rate of 4.4 bits/min over subjects, which makes subjective accenting a promising paradigm for an auditory BCI.

1 Chapters 2 and 3 make use of a single EEG dataset, analysed from different angles and answering a different set of questions

31

Chapter 3. Sequenced subjective accents for brain-computer interfaces

3.1

Introduction

In general, humans have a good sense for basic metric structures (Michon and Jackson, 1985), such as an auditory pattern where every first beat out of two, three or four beats is accented. These metric structures in western music are usually stereotyped as a march (ONE-two), waltz (ONE-two-three) or common rock rhythm (ONE-two-three-four). It has been shown that our sense for metric structures is not only relevant for the perception and production of music (London, 2004), but also plays a role in speech (Vatikiotis-Bateson and Kelso, 1993) and in motor control tasks (Kelso, 1982). The cognitive process responsible for inducing our sense for metric structures happens in a subconscious and fairly automatic way. This is demonstrated by the so-called clock illusion or ‘tick-tock’ effect (Brochard et al., 2003). Here, a binary accenting pattern is automatically induced in the brain when a series of isochronous sound pulses is presented. The name ‘clock illusion’ refers to one of the best examples of this mechanism where the sound of a clock, which sounds identical for every pulse (‘tick-tick-tick-tick. . . ’), is usually perceived with an induced subjective accent (‘tick-tock-tick-tock. . . ’). The mechanism inducing these accents is known as subjective accenting or subjective rhythmization (Fraise, 1982; London, 2004). Brochard et al. (2003) found that subjects exhibited different neuronal responses to loudness deviations at even and odd positions in a steady pulse train, reflecting binary chunking. Several studies have explored the perception of metric patterns and stimulus induced responses in EEG. These studies have shown, that both the perception of metric patterns (Snyder and Large, 2005) as well as the expectation of an accent was reflected in EEGactivity (Zanto et al., 2006; Jongsma et al., 2005; Snyder and Large, 2005; Desain and Honing, 2003; Schaefer et al., 2010). In a recent study Snyder and Large (2005) reported that (non phase-locked) gamma-band activity (GBA) in EEG can reflect the metric structure of the stimulus and that at an omission of a stimulus this GBA may persist. This suggests that a form of imaginary rhythm or internal clock is active. Subjective accents can also be added voluntarily, thus making it a deliberate process. Iversen et al. (2009) investigated this phenomenon and described an effect in the upper beta-band of MEG measurements at subjectively accented versus non-accented tones. The focus of this study lies with subjective rhythmization as a mental task for driving a Brain-Computer Interface (BCI) (Dornhege et al., 2007; Van Gerven et al., 2009). A BCI system allows a user to control an output device (for instance a speller, a cursor or an automatic wheel chair) with brain activity. By voluntary performing a specific metal task a user can use his pattern of brain activity to communicate an intended signal - as if it were a code. This activity is measured and in real-time (or as close as possible) decoded by a computer and turned into the control signal for an output device. An intuitive mental task, 32

Chapter 3. Sequenced subjective accents for brain-computer interfaces such as subjective rhythmization, could be a useful addition to the existing variety of BCI tasks, such as the P300 (Farwell and Donchin, 1988), steady state evoked potential (SSEP, Regan, 1977; Müller-Putz et al., 2005) and imagined movement paradigm (Pfurtscheller et al., 1997, 2006), some of which can be difficult to perform or require much attention and concentration. The introduction of new mental tasks could also be a way to overcome so called ’BCI illiteracy’ (Dornhege et al., 2007) of subjects for tasks commonly used. Recent developments in the domain of auditory BCIs predominantly yield systems that provide feedback in the auditory domain, but are based on (to BCI) rather conservative mental tasks, such as modulation of sensorimotor rhythms (SMR, Nijboer et al., 2008a), slow cortical potentials (SCP, Pham et al., 2005) or P300 responses to auditory events. Furdea et al. (2009) attached acoustically presented numbers to a five-by-five classic P300 spelling matrix (Farwell and Donchin, 1988), while in a similar way Klobassa et al. (2009) attached environmental sounds to a six-by-six matrix. Schreuder et al. (2010) reported using spatial hearing as an informative cue for evoking ERP responses (predominantly P300). Auditory stimuli were presented through five speakers sequentially and in addition to the spatial information thus provided, auditory stimuli were also acoustically different per speaker. Alternatively, systems have been reported related to the concept of auditory stream segregation (Hill et al., 2004; Kanoh et al., 2010). In these systems ERP responses to deviants in two streams of auditory stimuli elicitepd detectable differences in EEG, depending on the subject’s attention to one of the streams. With novel paradigms in the auditory domain, accessibility of BCIs may increase for specific groups of users, for instance to users with a visual impairment who are not capable of using a visual P300 speller. In order to assess the feasibility of subjective rhythmization as a task for BCI, we investigate whether subjective accents can be decoded from EEG on a single-trial basis, and more specifically, compare various approaches to decoding of subjective accents. Data segments are broken down into single accented and non-accented beats and classified. This approach is extended by a sequence classification algorithm. Aiming at an easy-to-use BCI, we also investigate the possibility of training classifiers on perception instead of imagery data. These approaches are compared to a more conservative approach where longer segments of data are classified at once.

3.2 3.2.1

Materials and methods Experimental design and data acquisition

Ten subjects, five females and five males, aged between 22-34 years (mean age 27), participated in this study. One subject (S9) had a professional musical training, and six participants 33

Chapter 3. Sequenced subjective accents for brain-computer interfaces (S1, S3, S5, S7, S9, S10) actively play a musical instrument. When asked, none of the subjects reported to be diagnosed with any neurological disorder or hearing deficiency. The experiment was undertaken with the understanding and written consent of each subject, approved by the ethical committee of the faculty of social sciences at the Radboud University Nijmegen, and in compliance with national legislation and the code of ethical principles for medical research involving human subjects of the World Medical Association (Declaration of Helsinki). Subjects were seated in a comfortable chair in an electrically and acoustically shielded room at a distance of approximately 0.5 m from a 17” TFT computer monitor. Two speakers (Monacor, type MKS-28/WS), placed on each side of the monitor, were used to present auditory stimuli to the subjects (stimuli can be found online at http://www.nici.ru.nl/mmm/). A Biosemi active-electrode set (Ag-AgCl) with 64 electrodes was used in combination with an ActiveTwo AD-box to measure EEG at a sampling frequency of 2048 Hz. No further filtering or processing was done at the stage of recording. Simultaneously with the EEG, an electro-oculogram (EOG) was made in order to be able to exclude eye motions as a possible source of information during EEG classification. Two pairs of auxiliary electrodes were placed. One pair was positioned above and below the left eye to measure eye movements in vertical direction. The other pair was positioned on the temples to measure horizontal eye movements. The stimulus sequences consisted of three phases, a perception phase, a fade and an imagery phase. A metronome was playing throughout the whole sequence (see Figure 3.1). In the perception phase of the sequence, an accent was superimposed on the metronome every two, three or four beats, thus creating binary, ternary and quaternary patterns, respectively. Throughout this article we will refer to the span of such a pattern with the term ‘cycle’, which could be considered equivalent to the musical term ‘measure’. The metronome played at 120 BPM (beats per minute), resulting in inter-onset-intervals of 0.5 seconds between successive ticks. The rate of 120 BPM is chosen to avoid overlap of the expected perceptual EEG responses, such as the auditory evoked potential (AEP) which can have components as late as 400 ms (Burkard et al., 2007), and to stay close to a tempo that is easy to track by human subjects (Fraise, 1982). The sound was presented at a peak level of 57dB(A) for all subjects. In the perception phase, accents were added with the general MIDI sound ‘high woodblock’. This accent increased the peak loudness of the stimulus to 65dB(A). During the fade phase, as a transition from the perception to the imagery phase, the accents were played less loudly, decreasing the peak loudness of the stimulus to 61dB(A). In the imagery phase the accent was no longer added. A sample sequence is illustrated in Figure 3.1, showing a sequence of a 3-beat pattern. At the start of each sequence, a white fixation cross of 3 cm was shown on the monitor.

34

Chapter 3. Sequenced subjective accents for brain-computer interfaces perception

fade

imagery

probe response accent metronome

0.5s

time

EEG classification: single-beat (trial) sequence long-segment

-50 450 ms

Figure 3.1: The structure of a single sequence in the experiment is shown, in this case for a 3-beat pattern. The sequence starts with three cycles of a ternary metric pattern (perception phase), followed by one cycle (fade phase), where the intensity of the superimposed accent was reduced by 4 dB. Then the subject had to imagine the accenting pattern for five cycles (imagery phase). At the end, an accented beat was presented to test whether the subject maintained the correct rhythm. The lower part of the figure schematically highlights the different ranges of data taken from the 64-channel EEG for each of the different classification methods described in Sections 3.2.3, 3.2.4 and 3.2.5, respectively.

The appearance of the cross indicated the start of a sequence to the subject and served as a fixation point for the eyes throughout the sequence. After a random delay between 1.0 and 1.8 s after the onset of the fixation cross, the pattern started. The accented pattern was first played for three cycles, which is indicated as the perception phase in Figure 3.1. Subsequently, the pattern was played for one cycle during the fade phase, followed by five cycles containing only the metronome in the imagery phase. In the imagery phase subjects were explicitly instructed to imagine hearing the continuation of the accent pattern, and not to use any other strategies, such as counting, imagining bouncing balls or tapping hands to maintain the rhythm. During the experiment, subjects were visually observed to control for hand, head or other body movements to make sure that no artifacts would influence classification. To check whether the subjects did not lose track of the accenting pattern, a probe accent was sounded at the end of the sequence and the subjects had to answer the question whether this probe would have coincided with the accent in the pattern, if the accenting sound had not stopped playing. Probe accents were randomly placed on either accented or non-accented positions at the end of the sequence and this information was later used to check the subject’s answers. Each next sequence was started with a button press, giving the subject the opportunity to control the interval between sequences, and the opportunity to move freely between sequences. However, during the sequences they were asked to sit still and avoid eye movements or blinks. A block in the experiment consisted of 12 sequences of 2-, 3-, and 4-beat patterns, giving a total of 36 sequences in a block. The order of beat patterns in a block was randomized 35

Chapter 3. Sequenced subjective accents for brain-computer interfaces before the start of the experiment. With 4 of these blocks per subject we gathered roughly 12 × 4 × 5 = 240 cycles of each imagery pattern and 12 × 4 × 3 = 144 cycles of each perception pattern. Some of the cycles were rejected in further analyses, due to artifacts (see section ‘preprocessing’).

3.2.2

Preprocessing

Bad channels were identified from the raw EEG signal for each trial with an algorithm sensitive to four properties. Initially, any channel with a DC offset exceeding 30 mV was marked as ‘bad’, as well as channels with a power exceeding 3500 µV2 in the 50 Hz band (45 to 55 Hz) or a maximum derivative larger than 200 µV/sample. Horizontal and vertical EOG channels were band-pass filtered between 0.2 and 15 Hz and decorrelated from the EEG (Schlögl et al., 2007), thus removing eye drifts or blinks if present. The raw EEG signal, originally sampled at 2048 Hz, was temporally down sampled to a sampling frequency of 128 Hz. Additionally, as a fourth property for identification of bad channels, withintrial variance was computed and channels exceeding a variance of 2000 µV2 were marked ‘bad’. If - according to the four properties - more than 20% of the channels in a trial were bad, the trial was excluded from further analysis. Trials from a sequence with a wrong answer to the probe accent at the end of the sequence, occurring on average in 10.3% of the sequences, were also excluded. For the remaining trials, bad channels were reconstructed by interpolation from the remaining good channels with a spherical spline interpolation algorithm (Perrin et al., 1989). The interpolation step assures a stable number of good channels for the classifiers to work with, while avoiding rejection of channels throughout the whole data-set when channels are only occasionally bad. Measures for bad channel identification are based on single trial data only, instead of all available data. This choice was motivated by the intention to use the same preprocessing pipeline in an online BCI system. Data was re-referenced to a common average reference (CAR) and linearly detrended. The same preprocessing was used for all subsequent analyses.

3.2.3

Single-beat classification

Data acquired during 2-, 3- and 4-beat cycles from the stimulus sequences were split into individual beats, which will be called trials from now on (see Figure 3.1). A time window from -50 ms to 450 ms was chosen around each pulse of the metronome, where time 0 ms corresponds to the time of occurrence of the pulse. Data collected during the first cycle of a beat pattern in the perception or imagery phase were excluded from analysis. After preprocessing, all trials were further processed using Fieldtrip (Fieldtrip, 2010). A highpass filter with a 3-dB cut-off at 0.5 Hz and low-pass filter with a 3-dB cut-off at 15 Hz 36

Chapter 3. Sequenced subjective accents for brain-computer interfaces were applied (both filters were of a 6th order Butterworth type). A more common low-pass cut-off at 40 Hz was also tried, but only seemed to have a negative effect on classification performance, presumably due to additional noise without additional information about the classes. All remaining time-points (64) per trial on all available EEG channels (64) were used as features for classification. Single beats were classified with a regularized (L2 ) Logistic Regression algorithm (Bishop, 2006) applied directly to the pre-processed spatio-temporal data matrices. Since the brain’s representation of subjective accents is not fully understood, two approaches to classification have been explored. The first approach makes a distinction between accented and nonaccented beats only, requiring a binary classifier. The number of trials remaining after preprocessing of perception data, was on average 264 (between 226 and 284) per subject for the accented class and 528 (between 452 and 567) for the non-accented class. For the imagery data on average 593 (between 509 and 638) trials were available for the accented and 1186 (between 935 and 1165) for the non-accented class. In the binary classification approach with this experimental design, many more trials are available for the non-accented condition than for the accented condition. In order to avoid a bias towards the non-accented class in the classifier’s output, the classifier’s loss-function was weighted per class to compensate for the unbalanced training data, such that both classes become equally important (Bishop, 2006). The second approach assumes that each beat has a unique representation, depending on the metric pattern as well as the position of the beat within the pattern. Classification under this assumption requires a 9-class classifier. Figure 3.2(b) shows how the nine classes (labels A to I) are distributed over the 2-, 3-, and 4-beat patterns. The multi-class classifier was constructed from 36 binary classifiers, one for each possible pair of classes - also referred to as a sub-problem - on a 1-against-1 basis (Bishop, 2006). Figure 3.2 illustrates the order and class-labels of single beats from the overarching beat patterns, for each of the two classification approaches. Figure 3.2(a) shows the accented and non-accented beats as ‘1’ and ‘0’, respectively. Figure 3.2(b) illustrates the nine different beats in the 2-, 3-, and 4-beat patterns. The total number of collected beat patterns led to about 88 perception trials (between 70 and 96, depending on the number of trials rejected in the preprocessing) and 187 imagery trials (between 140 and 216) for each of the nine classes for each subject. The number of trials per class is not structurally unbalanced in the 9-class approach. To measure performance of the classifiers, double-nested cross-validation (Bishop, 2006) was performed, using 5 inner folds for hyperparameter optimization and 10 outer folds for the final performance evaluation. As a precaution to overfitting, folds were constructed with the aim of keeping the trials in each fold consecutive in time (also known as in-order

37

Chapter 3. Sequenced subjective accents for brain-computer interfaces

2-beat 1 0 1 0 1 0 1 0 1 0 1 0 ..

2-beat A B A B A B A B A B A B ..

3-beat 1 0 0 1 0 0 1 0 0 1 0 0 ..

3-beat C D E C D E C D E C D E ..

4-beat 1 0 0 0 1 0 0 0 1 0 0 0 ..

4-beat F G H I F G H I F G H I ..

Time

(a)

Time

(b)

Figure 3.2: Two different ways of labeling the individual beats in a 2-, 3-, and 4-beat sequence are shown, corresponding with two different hypotheses on cognitive processing. Each character in a row represents a single beat within a specific beat pattern. Panel (a) shows the labeling when only a binary distinction between beats is made (labels 1 and 0 for accented and non-accented). This information is also relevant for sequence classification, since it can be interpreted as a ‘codebook’ that lists the possible ’code words’ as rows. Panel (b) shows the labeling when distinguishing nine different classes of beats (labels A to I).

cross-validation). Significance of the classifier performance was calculated using a t-test. Since test-sets in the binary classification approach can be unbalanced and since we want to enforce equal importance of classes, balanced classification rates (defined as the average of per-class performance) will be reported. As shown by Vlek et al. (2011b) it is possible to classify imagery subjective rhythmization data with a classifier that was trained on the perception data. This method, which we will refer to as ‘cross-condition classification’, was also used here as a variation to the binary classification approach. It also distinguishes imagined accents from non-accents in a binary fashion, but is trained to do so on perception instead of imagery data. A 10-fold cross-validation regime on the perception data was used to find the optimal regularization parameter, and a classifier using this regularization parameter was retrained on all available perception data. To avoid a structural preference of the retrained classifier to one of the classes in the new test set, calibration was performed by a restricted retraining of the classifier’s bias and gain (see the work of Shenoy et al., 2006) on a random set of 200 trials of imagery data, while aiming for equal per-class performance. The trials used for calibration were not used for performance evaluation.

3.2.4

Sequence classification

Next to classification of single beats we explored sequence classification, which is a technique popular in P300 spellers (Hill et al., 2008). EEG signals were sliced into individual beats, but classifier predictions on the slices were combined into predictions for a specific sequence of beats (see Figure 3.1). The 2-, 3-, and 4-beat patterns could each be interpreted as a unique and cyclic sequence of accented and non-accented beats (see Figure 3.2(a)). Given a classifier’s prediction for an individual beat, a prediction for each sequence of beats can be made by means of a sequence classification algorithm. The algorithm is in38

Chapter 3. Sequenced subjective accents for brain-computer interfaces formed about the order of accented and non-accented beats in each beat pattern through a ’codebook’ (see Figure 3.2(a)). At each new beat, an underlying binary classifier will deliver probabilities for each of the single-beat classes (accented or non-accented). Taking into account the possible codes in the codebook and the probabilities of all previous trials, predictions on the level of beat pattern classes are updated. Assuming trial independence, the probability for the ith row of the codebook matrix C (denoted Ci ) given the sequence of data, [X1 , X2 , . . .], is defined by the product of trial predictions as described in Equation 3.1. Pr(Ci |X1 , X2 , . . .) =

Y

Pr(Cin |Xn )

(3.1)

n=1,2,...

where Pr(Cin |Xn ) represents the trial prediction for the nth element (column index) in the ith row of codebook C, given the data Xn at trial n. The choice for a binary single-beat distinction, with classes accented and non-accented, was driven by the results of the single-beat classification (see Section 3.2.3). The imagery data was sliced in exactly the same way as for single beats: -50 to 450ms around stimulus onset. However, there was a difference with the previous procedure described in Section 3.2.3, in that the order of beats as they had occurred in the experiment was preserved, and that data from the first cycle of each beat pattern was included. Preprocessing and classifier training was done as described in Section 3.2.3. Performance of the sequence classifier was evaluated using test sets of (on average 13) imagery sequences left out of the training set. If bad trials, identified by the preprocessing pipeline, occurred somewhere in a sequence, class probabilities were set to chance level for this trial. In this way, bad trials are gracefully ignored, without negatively influencing the performance of the entire sequence. Additionally, sequence classification was performed on the basis of the single-beat classifier trained on perception data and tested on sequences of imagery data. Since results of the sequencing method described above are limited by the amount of available consecutive trials, a sequence simulation algorithm was developed. Using a Monte Carlo approach, this algorithm allows for extrapolation of the sequence classification performance curve by randomly sampling from the available data and applying the classifier. Similar to the sequence classification algorithm for real data (see Equation 3.1), the simulation algorithm as defined by Equation 3.2 is based on a product of per-trial predictions. Pr(Ci ) =

Y

Pr(Cin |s(Cin ))

(3.2)

n=1,2,...

The classifier is provided with Monte Carlo sampled data, denoted by s(Cin ), instead of real data for computing per-trial predictions. The Monte Carlo sampling process is defined

39

Chapter 3. Sequenced subjective accents for brain-computer interfaces by Equation 3.3, where s(c) represents a random element taken from the set of trials tr with the true class c. s(c) = rand({tr : class(tr) ∈ c})

(3.3)

The accuracy of the Monte Carlo simulation algorithm is computed as an average performance over 2000 simulated sequences per class.

3.2.5

Long-segment classification

To summarize the sequence classification approach, long segments of imagery data were explicitly broken down into individual beats and classified, after which classifier predictions were combined into sequence predictions. Explicit information about the time-structure of the mental task, such as the time interval between beats and the order of beats in each possible pattern, is in that case provided to the algorithm. As an alternative, we have investigated how well a classifier can deal with longer data segments, when no such topdown information is provided. We rely on the classifier to learn any time-structure beneficial to classification performance. For this, longer segments of data acquired during the imagery phase of the stimulus sequences were sliced and classified (see Figure 3.1). A segment consists of five cycles of each beat pattern, resulting in 5 s of data for the 2-beat pattern, 7.5 s for the 3-beat and 10 s for the 4-beat data after the start of the imagery phase. Since the classifier should work on equally sized data segments, these segments were all truncated to the shortest length of 5 s (corresponding to the five cycles of the 2-beat pattern). In addition to the preprocessing, described in Section 3.2.2, the data segments were filtered by a high-pass filter with a 3-dB cut-off at 0.5 Hz and by a low-pass filter with a 3-dB cut-off at 15 Hz. Both filters were of a 6th order Butterworth type. The resulting time-domain data on all 64 channels was fed to a regularized (L2 ) linear logistic-regression classifier. The 3-class problem (with classes 2-, 3-, 4-beat) was addressed with a 1-against-1 style multi-class classifier. Classification performances were obtained by a leave-one-out regime (Bishop, 2006), instead of cross-validation with 10 outer folds, since this provided a more reliable estimate of the performance with the smaller number of data segments available. The average number of long-segment trials available per class was 44 (between 36 and 48).

3.2.6

Bit rate comparison

It is difficult to directly compare classification performances of all methods described, because of different numbers of output classes and different duration of the slices of data 40

Chapter 3. Sequenced subjective accents for brain-computer interfaces required for each method. A comparison can more easily be made through bit rate. Using the definition of bit rate by Wolpaw, as described in Kronegg et al. (2003), the bit rate was computed for each of the classification methods. The bit rate is dependent on three variables: the number of classifications per second, the number of classes and the mean accuracy.

3.3 3.3.1

Results Single-beat classification

A grand average ERP over imagery data of all subjects was computed, consisting of the signals of 64 channels as a function of time from -50 ms to 450 ms relative to each beat. Principal component analysis (PCA) was used to spatially decompose this grand average. Figure 3.3 shows the spatial distribution and corresponding time-course per class of the first component, i.e. the principle component with the largest eigenvalue, explaining 52.0% of the variance. Figure 3.3 shows a characteristic of the signals for accented and non-accented beats, that are later provided to a classifier. Differences between accented and non-accented beats in the time-courses of the first component were statistically tested using a cluster randomization test, a non-parametrical statistical test designed to deal with the multiple comparison problem present in EEG data, using biophysiologically motivated constraints to increase the sensitivity of the test (Maris, 2004; Maris and Oostenveld, 2007). Areas in the time-courses contributing to significant differences (p < 0.05) are marked in grey on the time axis in Figure 3.3. The main difference between accented and non-accented beats appears to be in the late part of the auditory evoked potential (AEP) (Burkard et al., 2007) around 200 ms after stimulus onset, and in a negative deflection starting at 250 ms. Similar neurophysiological effects to an identical task are discussed in more detail in Schaefer et al. (2010) and in Vlek et al. (2011b). These effects follow a fronto-central distribution on the scalp, similar to scalp distributions observed for distinction of musical stimuli (Schaefer et al., 2011a). For the binary single-beat approach to the decoding of subjective rhythmization from brain signals, results are shown in Figure 3.4, where scaling of the y-axis ranges from chance level to maximum performance (100% correct). This scaling convention is used consistently for all performance figures. With the binary classification approach (Figure 3.4(a)), distinguishing perceived accented from non-accented beats, an average performance of 68.8% (SD=5.6%) is achieved. The best subject (S6) reached 74.3%. Of more interest to BCI is the classification of imagined accents, where the stimulus does not carry class information. Here an average classification rate of 60.4% (SD=4.2%) was achieved over subjects, while 41

Chapter 3. Sequenced subjective accents for brain-computer interfaces Imagery (GA) comp.1 (expl.var.:52.0%)

Imagery (GA) comp.1 15

0.2

0 −0.1 −0.2

10 amplitude (µV)

0.1

5 0 −5 −10 −15

accented non−accented 0

0.1

0.2 0.3 time (s)

0.4

Figure 3.3: A characteristic of the signal relevant to discrimination of imagery accented and nonaccented beats is shown. A grand average ERP over subjects was decomposed by means of PCA and the spatial distribution and corresponding time-course is displayed for the strongest component (explaining 52.0% of the variance). Areas in the time-course contributing to a significant (p < 0.05) difference between accented and non-accented trials are marked by grey bars on the time axis.

the best subject (S4) reached an accuracy of 66.8%. All subjects performed significantly (p < 0.001) above the chance level of 0.5. Alternatively, with the multi-class approach applied to imagery data (Figure 3.4(b)), we distinguish between nine beat classes. An average performance of 15.9% (SD=2.8%) was achieved, while a performance of 20.6% was obtained for the best subject (S7). All subjects performed significantly (p < 0.05) above the chance level of 1/9. Given the difference in chance level between the binary (1/2) and 9-class (1/9) approach, a comparison of performances is made by means of bit rates. Performance of the binary classification translates to an average bit rate of 4.4 bits/min (in the range of 1.0 to 10.0 bits/min) over all subjects, while the multi-class approach yields an average rate of 2.8 bits/min (in the range of 0.4 to 6.5 bits/min). Based on this finding, it was decided to construct the sequence classification algorithm (see Section 3.2.4) from a binary classifier. Its performance will be described in the following section (Section 3.3.2). Results of the cross-condition classifier can also be seen in Figure 3.4(a). An average performance of 58.2% (SD=4.8%) over subjects and 66.4% for the best subject (S4) is achieved. Results are significantly (p < 0.01) above chance for seven out of ten subjects. Beyond visual inspection of the subjects’ behaviour during the experiment, separate classification of the EOG channels was performed. In this way, we were able to verify that potentially remaining eye-artifacts (not completely removed by our decorrelation preprocessing step) were not turning into artifactual sources of classifier performance. None of the subjects performed significantly (p < 0.01) above chance when using only EOG channels. 42

Chapter 3. Sequenced subjective accents for brain-computer interfaces

Binary single−beat classification 1 perception imagery cross−condition

Classification rate

0.9 0.8

***

***

0.7 0.6 0.5

*** ***

*** ***

*** ***

S1

S2

***

***

***

***

S4

***

*** ***

S3

***

*** ******

S5

**

*** ***

S6 Subject

S7

*** ***

*** *** ***

S8

S9

***

S10

*** ***

AVG

(a)

9−class single−beat classification 1 perception imagery

Classification rate

8/9 7/9 6/9 5/9 4/9 3/9 2/9 1/9

** **

S1

**

** **

**

S2

** *

S3

S4

**

**

**

**

S5

**

**

S6 Subject

S7

** **

S8

**

** **

**

S9

S10

**

AVG

(b)

Figure 3.4: Panel (a) shows balanced classification rates for the binary approach to single beat classification of perception data, imagery data and with the method of training a classifier with perception, and testing it on imagery data (cross-condition). Panel (b) shows performances of the 9-class approach for the perception and imagery condition. The average over subjects is labelled AVG. The error bars indicate the standard error within the 10-fold cross-validation. Significance level is indicated by * for p < 0.05, ** for p < 0.01 or *** for p < 0.001, based on a t-test.

43

Chapter 3. Sequenced subjective accents for brain-computer interfaces

3.3.2

Sequence classification

Scaling up the recognition of single beats to the level where sequences of beats can be decoded, we obtained the following results. Classification of 2-, 3- and 4-beat sequences resulted in performances shown in Figure 3.5(a), yielding an average accuracy of 48.8% (SD= 8.7%) over subjects and 63.2% for the best subject (S7). A more detailed view of the sequence classification results for one of the best (S3) and worst (S9) subjects can be found in Figure 3.5(b). This figure shows how classification accuracy on real data (both solid lines) increases with each additional trial. Notice that it is not until the third trial, that the performance starts to increase above chance level. Sequencing on the basis of a binary classifier, trained with perception data, resulted in an average accuracy of 43.7% (SD= 5.2%) and 50.0% for the best subject (S3). Figure 3.5(b) also shows the result of the Monte Carlo simulation algorithm for sequence classification for one of the best (S3) and worst (S9) subjects. After 10 trials, the performance of the simulation algorithm deviates on average 5.4% from the performance on the real data. The Monte Carlo simulation approach will be used in Section 3.3.4 for extrapolation and exploration of other coding types.

3.3.3

Long-segment classification

In contrast with the sequencing approach, where top-down information about time-structure in the neural correlates is provided to the algorithms, the classification method for long segments is not provided with such information. Results of the classification of long segments of data are visualized in Figure 3.6. Seven out of ten subjects perform significantly (p < 0.05) above the chance level of 1/3. Over all ten subjects an average accuracy of 44.9%(SD= 7.6%) is achieved, with a maximum of 60.7% for the best subject (S7).

3.3.4

Bit rate comparison

For comparison of the reported results on imagery data, classification rates were translated to bit rates, visualized in Figure 3.7. With an average bit rate of 4.4 bits/min over subjects, the single-beat classification paradigm clearly performs best. From the classification methods capable of dealing with longer segments or sequences of data, the sequence classification method achieved the highest bit rate, with an average of 1.1 bits/min, compared to 0.7 bits/min for the long-segment classification method. Using the Monte Carlo approach, simulations were performed on sequence classification. First, we use the simulation for extrapolation of the sequences, that were otherwise restricted to the length of 10 beats. Secondly, we will illustrate the effect of different types 44

Chapter 3. Sequenced subjective accents for brain-computer interfaces

Sequence classification 1 Imagery Cross−condition

Classification rate

0.9 0.8 0.7

***

***

***

0.6 ***

***

**

**

0.5

*

**

**

*** ** *

***

*

0.4 S1

S2

S3

S4

S5

S6 Subject

S7

S8

S9

S10

AVG

(a)

1 S3: real data S3: Monte Carlo simulation S9: real data S9: Monte Carlo simulation

classification rate

0.8 0.6 0.4 0.2 0 1

2

3

4

5

6 7 trials

8

9

10

11

12

(b)

Figure 3.5: Panel (a) shows performances of the sequence classification of 2-, 3- and 4-beat patterns based on 10 trials (5 s) of data. For a classifier trained and tested on imagery data, the error bars indicate the standard error within the 10-fold cross-validation. For the cross-condition classification, a classifier was trained on the perception data, allowing all imagery data to be used for performance evaluation. For this regime, error bars thus indicate the standard error over all imagery trials. Significance level above the chance level of 1/3 is indicated by * for p < 0.05, ** for p < 0.01 or *** for p < 0.001, based on a t-test. The average over subjects is labelled AVG. Panel (b) gives an overview of sequence classification and Monte Carlo simulation performances, as a function of the number of trials. Performances of one of the best (S3) and worst (S9) subjects are shown. Long−segment classification 1

Classification rate

0.9 0.8 0.7 ***

0.6

***

0.5

***

* *

**

***

**

S10

AVG

0.4 S1

S2

S3

S4

S5

S6 Subject

S7

S8

S9

Figure 3.6: Performance of the classifier on long segments of time-domain data is shown. Subject S1 is performing below chance and is not visible with current scaling. The average over subjects is labelled AVG. Significance level above chance (1/3) is indicated by * for p < 0.05, ** for p < 0.01 or *** for p < 0.001, based on a t-test.

45

Chapter 3. Sequenced subjective accents for brain-computer interfaces Bit rate comparison 12 single beat single beat c−c sequence classification sequence classification c−c long−segment

Bit rate (bits/min)

10 8 6 4 2 0

S1

S2

S3

S4

S5

S6 Subject

S7

S8

S9

S10

AVG

Figure 3.7: Performance of the different classification approaches for imagery data, described in Sections 3.2.3, 3.2.4 and 3.2.5, was translated to bit rate and visualized for all ten subjects as well as the average over subjects. Cross-condition classification is abbreviated as ‘c-c’. The average over subjects is labelled AVG.

of coding. These conditions were not present in the experimental design, but simulating them may help in predicting the future possibilities of the subjective accenting paradigm. Simulations were performed per subject, since the Monte Carlo method is data-driven and thus subject specific. By computing the average bit rate curve over subjects, corresponding with simulated sequence classification of the 2-, 3- and 4-beat patterns over 40 trials, we found (Figure 3.8, solid line) that the optimum bit rate is reached within 5 trials. Furthermore, simulation allows us to illustrate the expected performance with patterns other than the standard 2-, 3-, and 4-beat. A positive shift in the average bit rate curve can be observed when using all possible phase shifted versions in addition to the three standard patterns (for instance a 2 beat pattern, started at the non-accented beat, as a separate class). In a BCI system this would allow the user to encode his or her intention with 9 instead of 3 classes. A higher bit rate is expected when selecting more optimal codes, such as cyclic codes of four beats with a Hamming distance of at least 2 bits to each other (being: 1001, 1010, 0101, 0110, 1100, 0011 and excluding 0000 and 1111). Such a type of coding would result in a 6-class output of the BCI system, achieving an optimal average bit rate of almost 3 bits/min.

3.4

Discussion

We have shown that it is possible to decode subjective accents voluntarily imposed on an auditory metronome from measured brain activity. A binary approach to classification of single beats, distinguishing accented and non-accented beats, gave a higher bit rate than a 9-class approach. The use of time-domain features resulted in better classification perform46

Chapter 3. Sequenced subjective accents for brain-computer interfaces 6 3−class (standard) 6−class (2−bit distance) 9−class (all phases)

bit rate (bits/min)

5 4 3 2 1 0 0

5

10

15

20 trials

25

30

35

40

Figure 3.8: Using a Monte Carlo approach, simulations were performed with the sequence classification algorithm. Performances were translated to bit rates and averaged over subjects. The bit rate curve of the standard 3-class situation, shows how the optimum bit rate is met within 5 trials. Furthermore, the simulations illustrate how different types of coding are beneficial to the expected bit rates, such as 9-class coding using all phase shifted versions in addition to the standard beat patterns, or 6-class coding using cyclic patterns of four beats with a Hamming distance of two bits.

ance than time-frequency features, such as the gamma-band activity described by Snyder and Large (2005), which was not found consistently and strongly enough to pursue further investigation for BCI application. The results of single beat classification translate into an average bit rate of 4.4 bits/min. This is better than the average bit rate of 2 bits/min reported for an auditory P300 BCI (Klobassa et al., 2009), and close to bit rates of approximately 5 bits/min reported for an auditory BCI based on the concept of auditory stream segregation (Kanoh et al., 2010). It has to be considered that translation of the classification performance to bit rate in the present study merely provides an estimate of the potential speed of an online BCI relying on this paradigm. The translation does not take into account the influence on bit-rate of factors, such as the inter-trial interval required for the subject to switch between different subjective accenting patterns or the time required to establish these patterns. These factors will be investigated in future research. Independent classification of EOG signals did not result in a performance significantly above chance for any of the subjects, and allows us to conclude that possible eye-artifacts were not boosting classifier performance. We have also shown that it is possible to decode subjective accents with a cross-condition classification method. This approach does not outperform conventional classification of subjective accents, but is nonetheless considered useful to BCI. Using perception of auditory accents during a BCI training regime takes away confusion for the user about what mental task to perform. This simplification of the training regime will contribute to correct execution of the (now well-defined) mental task. Once a classifier is trained on data 47

Chapter 3. Sequenced subjective accents for brain-computer interfaces gathered during this training regime, feedback could be used to guide the subject to an optimal performance of the less well-defined task of imagined accenting. Although single imagery accented and non-accented beats were sliced and classified successfully in this study, it has to be taken into account that these trials originate from data of the overarching beat patterns. The brain response leading to succesfull classification of a single accent may not exist without the presence of the overarching beat pattern. In other words, a subject may not be able to voluntarily decide per beat whether it will be accented or non-accented. Since this is crucial for BCI application of the single-beat approach, the assumption that subjects can do this needs to be validated in further research. In the case that this assumption would not be valid, an alternative is available through classification of sequences of beats. This method allows for combination of any number trials required to meet the desired accuracy of the BCI system. In a BCI application beat patterns instead of single beats could then be used to encode the users intention. We have shown that these patterns can be succesfully decoded with a sequence classification algorithm. With this algorithm, the achieved bit rates were considerably lower than for classification of single beats. This can be explained by suboptimal coding of the beatpatterns. As shown in the codebook (Figure 3.2(a)), the first two beats of all beat patterns are identical. Inherently, probabilities for the 2-, 3-, and 4-beat patterns are equal for these trials, when using a distinction of only accented and non-accented classes. A similar ambiguity between beat patterns occurs at other points in the sequence. This effect is reflected by the plateaus in the generally rising performance functions (see Figure 3.5(b)). While the theoretical maximum bit rate of the single beat classification approach at maximum confidence is

1 0.5/60

= 120 bits/min, the theoretical maximum of the sequence classification of beat

patterns has a ceiling of

log2 (3) 4·0.5/60

≈ 47.5 bits/min.

It has been mentioned that application of the single beat classification approach for BCI requires certain assumptions to be true, while the sequence classification approach is free of such assumptions, but a tradeoff in bit rate is inherent. By means of the Monte Carlo simulations, we were able to illustrate that alternative coding types can be found that boost the expected bit rate of the BCI system. These alternatives also have their own assumptions on what mental task subjects are able to perform. In both the 9- and 6-class coding types, the cyclic nature of the patterns persists, but it is assumed that subjects can voluntarily imagine a phase-shifted pattern, or imagine a more complex pattern consisting of multiple accented beats. Even in the standard 3-class coding, optimization may be possible by dynamically stopping the sequence classification process when the desired confidence of a prediction is reached. However, for rapid communication, this requires the subject to be able to stop imagining a pattern on demand and switch to the next one. For BCI applications of these types of coding, validation is first required. However, given the expected increase in bit

48

Chapter 3. Sequenced subjective accents for brain-computer interfaces rate, we believe this may be a fruitful direction for future research. Comparison of the performance of the Monte Carlo simulation algorithm and sequence classification on real data shows a relatively good fit. However, with this simulation algorithm trial independence is assumed, which may not be completely realistic given the brain’s properties. This is a possible reason for the Monte Carlo results to slightly deviate from classification results on the real data. For most subjects in this study, the sequence classification method yields slightly higher bit rates than the long segment classification method (see Figure 3.7). This seems to suggest that exploitation of the time-structure present in the neural correlates of the mental task is beneficial to classification performance. Finally, we conclude that the successful single-trial classification of both single beats and sequences of beats suggests that subjective rhythmization is a feasible paradigm for an auditory BCI. The paradigm can be made easier for the subject by using a cross-condition classification approach, such that the user’s task during the training part of the BCI merely consists of the perception of rhythmic stimuli.

49

50

4. Ethical issues in BCI research, development, and dissemination Four case scenarios Published as: Vlek, R.J., Steines, D., Szibbo, D., Kübler, A., Schneider, M.-J., Haselager, W.F.G. & Nijboer, F. (2012). Ethical issues in BCI research, development, and dissemination - four case scenarios. Journal of Neurologic Physical Therapy, 36(2), 94-99.

Abstract The steadily growing field of brain-computer interfacing (BCI) may develop useful technologies, with a potential impact not only on individuals, but on society as a whole. At the same time, this development presents significant ethical and legal challenges. In a workshop during the 4th International BCI-meeting (Asilomar, California, 2010), six panel members from various BCI laboratories and companies set out to identify and disentangle ethical issues related to BCI use in four case scenarios, that were inspired by current experiences in BCI laboratories. Results of the discussion are reported in this article, touching on topics such as the representation of persons with communication impairments, dealing with technological complexity and moral responsibility in multidisciplinary teams, and managing expectations, ranging from an individual user to the general public. Furthermore, we illustrate that where treatment and research interests conflict, ethical concerns arise. Driven by four case scenarios, we attempt to discuss salient practical ethical issues that may confront any member of a typical multidisciplinary BCI team. We encourage the BCI community to further identify and address pressing ethical issues as they appear in the current and near future practice of BCI research and its commercial applications.

51

Chapter 4. Ethical issues in BCI research, development, and dissemination

4.1 4.1.1

Introduction Definition

A Brain-Computer Interface (BCI) is a system that allows its user to control a machine (e.g. computer, automated wheel chair, artificial limb) using purely mental activity, without utilizing the peripheral nervous system (Dornhege et al., 2007; Van Gerven et al., 2009). Control with a BCI is initiated when a user performs a specific mental task. A typical BCI combines neurophysiological measurement technology with machine learning software to automatically detect patterns of brain activity that relate to this specific mental task. In most BCIs the user is provided with a small selection of mental tasks (e.g. imagined movement of left hand, right hand or foot) that the system is trained to detect. Once the system detects that the user has been performing one of the mental tasks, the corresponding actions are automatically triggered (e.g. move cursor left, right or click). The following subsections will provide a short overview of relevant aspects of the technology and current state of the art, leading up to the discussion of ethical aspects.

4.1.2

Measurement techniques

Implementation of a BCI requires brain activity to be measured. Technology to do so can be categorized as either invasive, such as subdural or epidural electrocorticography (ECoG) or an implanted multi-electrode array (MEA), or non-invasive, such as electroencephalography (EEG), magnetoencephalography (MEG), functional magnetic resonance imaging (fMRI) or near-infrared spectroscopy (NIRS). All of these technologies provide some representation of the brain’s activity. The choice for a specific method is often determined by a balance between factors such as health risks, user comfort, signal quality, portability and cost. While ’wet’ EEG, where conductive gel is used to improve the electrode’s connection to the surface of the scalp, is still the most popular non-invasive method for clinical BCI applications, low-cost, wireless, and ’dry’ alternatives are emerging rapidly, often aimed at less critical consumer applications, such as BCI games and entertainment for healthy users. BCI technology brings many promising applications in a variety of clinical and nonclinical areas. However, present clinical success with BCI is predominantly achieved with prototypes in research laboratories.

4.1.3

Applications

Clinical applications of BCI can roughly be divided into two categories. The first category aims at providing a technological alternative for a user’s lost function and can be con52

Chapter 4. Ethical issues in BCI research, development, and dissemination sidered a form of assistive technology. The second category aims at rehabilitation of the user’s own neural pathways in order to restore the lost function (see also Daly and Wolpaw, 2008). In the category of assistive technology, invasive BCI systems have been reported as specialized at real-time decoding of motor functions in non-human primates, sometimes even reproducing the decoded motor activity with a cursor or robotic arm (Wessberg et al., 2000; Carmena et al., 2003; Serruya et al., 2002). Eventually, such technology may develop into neuromotor prostheses for humans with motor disabilities. In humans, results have been reported of successful neural cursor control by a person with tetraplegia, implanted with a MEA in the motor cortex (Kim et al., 2008). Non-invasive BCI systems, mostly EEG-based, have also been investigated for various assistive applications, one of the most popular being communication. Spelling applications for instance, relying on the so-called P300 response, enable a user to ’mentally type’ symbols on a screen by focusing on a specific symbol in a matrix of randomly flashing symbols (usually letters, punctuation symbols and numbers) (Sellers and Donchin, 2006; Nijboer et al., 2008b). Variations to this application have also been presented for use in the auditory domain (Schreuder et al., 2010), which may be relevant to users with a progressive neurodegenerative disease affecting their vision in its later stages (e.g. ALS: amyotrophic lateral sclerosis). One of the key problems in these types of communication applications is speed. In order to achieve sufficient reliability of the system repeated measurement of the P300 response is necessary, resulting in reduced speed of the application (Zickler et al., 2011). Attempts have been made to overcome this lack of speed by adding artificial intelligence to the application in the form of automatic word completion or error correction from a dictionary (see for instance Ahi et al., 2011), or even entire graph-based sentence databases (such as described in Geuze et al., 2008). This increases throughput of language at the cost of verbal freedom. In addition to communication applications, prototypes for wheelchair control (Vanacker et al., 2007) or 3-D cursor control (McFarland et al., 2011) have also been reported. In the category of applications for rehabilitation, various results have been reported using an invasive or non-invasive BCI for the rehabilitation of gait in persons with stroke (Buch et al., 2008; Daly et al., 2009; Rodriguez et al., 2011). In these applications attempted or imagined movement is detected by a BCI and used to artificially move the user’s limb with a robotic aid, or by functional electrical stimulation (FES) of the muscles. This movement provides the user with proprioceptive and visual feedback of the limb following his or her intention, which in turn stimulates neural plasticity, and finally allows the user’s own neural pathways to regain control of the limb. Furthermore, BCI is being investigated as a therapy for various cognitive disorders. This is typically done by feeding back

53

Chapter 4. Ethical issues in BCI research, development, and dissemination fMRI measurements to the subject in real-time, allowing them to self-regulate a specified area of the brain. Initial studies have illustrated a wide variety of potential applications in treatment of pain, depression, schizophrenia, tinnitus, emotional disorders and memory (DeCharms et al., 2005; Weiskopf, ress; Zotev et al., 2011). Besides current and potential clinical applications of BCI, applications are also being developed for game and entertainment purposes (Nijholt et al., 2009). Thanks to a very large group of potential users, this seems a route to increase research and development resources, from which clinical applications may benefit as a side-effect. Moreover, recent work by Münssinger et al. (2010) reports on a BCI painting application, and illustrates that even entertainment BCIs may have a certain degree of clinical relevance by improving the user’s social and expressive potential.

4.1.4

Present research and development issues

The fact that BCI technology for clinical use, while promising, still resides predominantly in research laboratories can be explained by several factors. A key factor is the so-called signal-to-noise ratio in a given measurement modality, where the definition of ’noise’ includes any signal (including brain signal) that is not of interest to the discrimination of responses to the specific mental tasks used to control the BCI. Both speed and reliability of a BCI system are dependent on this signal-to-noise ratio. Adding to this, is the factor that the signal of interest typically varies greatly over subjects, as well as within subjects over time. This makes prediction of the quality of a BCI for a specific individual at a specific time very difficult, and may also lead to a frustrating experience for the user when the BCI suddenly stops working. It also means that most BCI systems require training or adjustment to each subject individually. Finally, artifacts may be a factor contributing to initial success with a prototype, but a difficult transition into clinical applications. Especially with EEGbased BCIs, signals originating from sources other than the brain, such as eye-movement and muscle activity, may (sometimes without being noticed) boost BCI performance. When the transition is made from healthy users (which is typically the first group to try a prototype) to persons with disability, some of the performance previously achieved may be lost as a consequence of the person’s physical condition. Much of the on-going BCI research is focused at finding solutions for the issues described above.

4.1.5

Ethical aspects

In the future, Brain-Computer Interfacing has the potential to impact not only individual users, but also society as a whole. The research and development of future BCI applications such as BCI computer games, neuroprostheses, online cognitive research, neuromarketing 54

Chapter 4. Ethical issues in BCI research, development, and dissemination or cognitive enhancement (Van Gerven et al., 2009), inevitably raises ethical and societal challenges, and a public debate on rights and restrictions is to be expected. Apart from being important in their own right, these ethical debates may substantially influence public acceptance of BCIs and related neurotechnologies. Nascent neuroethical debates have identified several topics of importance to BCI research, development and dissemination (see Clausen, 2009; Haselager et al., 2009; Tamburrini, 2009; Tamburrini and Mattia, 2011; Fenton and Alpert, 2008; Walter, 2010). Some of the ethical issues are well known in medical research and the medical device industry. However, there is also a category of issues that are relatively unique to BCI research. Some issues, such as the complicated process of obtaining informed consent from persons with locked-in syndrome, can readily be identified (Clausen, 2009), while other issues may be less obvious or concrete (e.g. privacy and mind-reading). One basic distinction that permeates ethical debates on BCI concerns the difference between research and treatment. Ethical issues depend, at least in part, on whether the aim is to apply approved technologies for treatment or to develop technologies. Especially in cases where treatment and research interests conflict important ethical concerns can arise (see e.g. Section 4.2 below). In an effort to triage issues according to technological imminence and ethical novelty (Farah, 2002), we set out to identify and disentangle ethical issues related to BCI use in four case scenarios, that were inspired by current experiences in BCI laboratories. Six panel members from various BCI laboratories and companies discussed each case scenario in the workshop on ‘ethical issues in BCI Research, Development, and Dissemination’, which took place at the 4th International BCI-meeting (Asilomar, California, 2010). Results of the discussion are reported in this article. We have chosen to discuss ethical issues, driven by the case scenarios, in an attempt to illustrate how issues relating to moral responsibility can, in different ways, confront researchers, clinicians, and developers with backgrounds in any of the various disciplines typically involved in BCI research or application. Since the case scenarios are kept close to the current (case 1-3) and potentially near-future (case 4) issues confronting BCI teams, they may also provide challenging material for analysis to those interested in the more theoretical ethical debate on neurotechnologies (such as BCI and deep brain stimulation).

4.2

Case scenario ‘Jane’ Jane is a 46-year-old housewife who has had the neurodegenerative disease amyotrophic lateral sclerosis (ALS) for 10 years. She lives at home where she is permanently supervised by a staff of caregivers. For one year she has been completely locked-in and thus cannot communicate in any way. She has a legal representative who enrolled her in 55

Chapter 4. Ethical issues in BCI research, development, and dissemination tests with non-invasive brain-computer interfaces. Until now the BCI researchers using non-invasive BCI say they can ‘see’ that Jane is making an effort, but that they are unable to reliably decode Jane’s brain activity. Jane’s husband, who is eager to communicate with his wife again, has read about invasive BCIs in the media and would like to try this method. He asks the BCI team if his wife could be considered for brain surgery. Jane’s case illustrates BCI research with the most vulnerable participants, namely those who cannot communicate. In such situations a ‘surrogate decision maker’ or legal representative is needed to represent the participant. Ethics in this regard are partially codified by law. There are different international regulations specifying when a person is considered to be legally incompetent and who should then be the legal representative. In the US and Netherlands a close relative is a preferred representative, whereas in Germany the preferred representative is a neutral person. Such differences between countries prevent a straightforward global discussion of the ethical aspects of BCI. Other legal issues include the liability of research teams. Currently, initiatives are underway to systematically gather information about international legal practices in relation to neuroscience and neurotechnologies (see e.g. Nadelhoffer, 2010). Furthermore, depending on national legislation, a conflict of interest may occur when close relatives legally represent a person who is unable to communicate. Caregivers and family members tend to underestimate the quality of life of persons with disability (Kübler et al., 2005b), and may choose not to let their loved one ‘endure’ additional BCI training. Alternatively, caregivers and family members may be so desperate to communicate with their loved one, that they would accept just about any intervention offered to them. The second topic raised in Jane’s case is how a BCI team should reply to the request from the husband of a person with complete locked-in syndrome (CLIS) to try a BCI intervention, assuming that the legal representative (if this is not the husband himself) also shows an interest in the BCI intervention. It is not uncommon for BCI research laboratories to receive such requests for intervention from people (when not yet CLIS) or from their legal representatives. Researchers may also have an interest to work with persons with CLIS, e.g. for understanding commonalities and differences between healthy user groups and user groups with disabilities, thus creating a situation of mutual interest between potential user or representative and researcher. Despite their mutual interest, both parties often do not share the same goal. Motivation from a research perspective can conflict with a therapeutic interest, sometimes leading to a ‘therapeutic misconception’ of the subjects participating in the research. Subjects then ‘fail to distinguish between clinical care and research and to understand the purpose and aim of research, thereby misconceiving their participation as therapeutic in nature’ (Beauchamp and Childress, 2008, p.129). When research is confused with therapy, not only may users or their legal representatives fail to 56

Chapter 4. Ethical issues in BCI research, development, and dissemination decide in best interest of the participant to the research, but also teams may be operating with an inadequate understanding of the person’s motives to participate in their research. From a broader perspective, there is also another ethical aspect to BCI treatment and research. At present, any BCI intervention would draw from scientific resources, for which responsibility lies with the researcher or research group. A moral aspect of this responsibility yields a balancing act between the freedom and wellbeing of the individual versus that of the public. Because of typical inter-individual differences between BCI users, resources spent on improving a system for a single user do not necessarily improve BCI for other users, whether these are individuals in the same disability group or in any of the other potential BCI target groups. Moreover, the balance between individual and public welfare may be skewed if pressure is put on access to experimental assistive technology by users who face the prospect of communication impairments or physical disabilities. A similar pressure was observed regarding experimental AIDS medication, where it led to public debate and lawsuits concerning the public right to experimental treatment (see e.g. Richman, 1989; Leonard, 2009).

4.3

Case scenario ‘Nigel’ Nigel is a 51-year-old research scientist who has had ALS for 11 years. He has used a P300 BCI home system for 4 years to communicate with his family members and for professional purposes with his lab members. In recent months he has dramatically reduced his use of the system and appears to be losing the capability to control it. The BCI team has noticed the decline and is trying hard to determine if the algorithms need to be adapted.

Unlike the first case scenario, Nigel is able to communicate. Persons with locked-in syndrome (LIS) like Nigel may also have a legal representative who must give the legally necessary informed consent. In addition, a person with LIS may still be able to give assent. Where this is possible, even by only signaling ‘yes’ via eye blinks or other muscle twitches, researchers must seek that assent in addition to the consent of the legal representative, and dissent should be respected (Haselager et al., 2009; World Medical Association, 2008). Second, the information regarding what to expect from a BCI system is crucial for giving informed consent. In this case scenario the subject should have been informed about a possible decline in BCI performance, but managing expectations of a BCI proves to be difficult. While a BCI system in principle yields numerous useful applications for a user, factors such as inter-subject differences and the brain’s plasticity make it hard to predict BCI success for a specific user. The multidisciplinary nature and typical size of a BCI research team 57

Chapter 4. Ethical issues in BCI research, development, and dissemination contributes to this problem. Various members of a team may assess expectations of a system differently. Haselager et al. (2009) point out the similarity with other interdisciplinary teams working in similarly demanding situations, such as intensive care units or teams in mental health care. Furthermore, BCI performance may be related to physical decline (Kübler and Birbaumer, 2008). If this is the case, informing the user with disability about BCI expectations implies that information is also provided regarding the progress of the disease, which the person may not (yet) want to know about (Borasio et al., 1998). Finally, overly enthusiastic media coverage about BCIs could heighten individual’s expectations of BCI, further undermining informed consent (Nijboer et al., 2011). A third issue that arises when visiting severely paralyzed or locked-in persons at home is that BCI training inevitably interferes with their (and their family’s and or caregiver’s) daily life and care. This is something to be included in moral considerations of the researcher. For example: a person with LIS with phantom limb pain reduced her morphine intake on days when she was training with a P300 BCI, because she knew morphine could reduce detection of the P300. Participants may opt to accept discomfort, changes of care schedules, or even pain to be able to work with the BCI. For this reason, the interference of BCI with daily life and care is a potential ethical concern. Information about the extent of interference with daily life is crucial for weighting positive and negative effects in a decision on BCI intervention. Participants in BCI studies often report feeling positively challenged by training. For example, subject H., who participated over two years in three BCI studies, stated that he looked forward to every training session, because he knew he could ‘work with his head’ and that he was mentally as healthy as other people (Nijboer et al., 2009). A recent paper showed that most participants were highly motivated throughout BCI training and mood was often good before training (Nijboer et al., 2010).

4.4

Case scenario ‘Ben’ Nine months ago Ben had a stroke resulting in paralysis of his right arm. He has regained some function in his arm after months of extensive motor rehabilitation, and Ben’s doctor asked him to enroll in a study that investigates whether BCI neurofeedback could accelerate Ben’s recovery. Ben has difficulty understanding what the doctor is asking of him (he has minor cognitive impairment), but he trusts his doctor.

Ben’s case refers to more recent BCI studies, which investigated the effects of neurofeedback training on the rehabilitation process after stroke (see for example Rozelle and Budzynski, 1995; Pfurtscheller and Neuper, 2006; Buch et al., 2008; Daly and Wolpaw, 2008; Prasad et al., 2009). The combination of acutely ill and vulnerable participants demands an 58

Chapter 4. Ethical issues in BCI research, development, and dissemination especially careful evaluation of risk and benefit, the process of consent, and the permissible treatment of control participants (Slyter, 1998). The question could be raised whether Ben, given his cognitive impairment, is really the most suitable subject for the study. In this situation, merely presenting Ben the option of BCI intervention may already carry some coercive weight. The issue of coercion is not unique to BCI, and has been empirically addressed in literature (Rain et al., 2003). Furthermore, this case emphasizes the need for clear inclusion and exclusion criteria for any study to be performed with participants in possible need of BCI. A second issue relates to possible side effects of BCI intervention and more specifically, neurofeedback training. No adverse side effects of BCI intervention and training have been reported, but no systematic research exists on this topic. Some BCI teams exclude study participants with epilepsy, which may be of relevance in studies involving flickering visual stimuli at specific frequencies and high contrasts, such as the stimuli sometimes used to evoke a P300 or SSVEP (steady-state visual evoked potential) response. In a neurofeedback study, Hoedlmoser et al. (2008) report an impact of sensorimotor rhythm (SMR) training on sleep and declarative learning. Their training paradigm is very similar to a paradigm regularly used in the BCI community (Nijboer et al., 2008a; McFarland et al., 2000). Regarding the use of the brain’s plasticity for obtaining or improving BCI control or rehabilitation, one of the panel members remarked that if we can do any good, we can certainly also do harm. The question here is: how different is mental training in BCI from any other forms of training that we are already very familiar with (e.g. learning to drive a car or subject preparations for psychological experiments), and are any of these differences cause for ethical concern? We recommend that in future clinical trials positive and adverse side-effects are systematically investigated. Beyond rehabilitation of gait after stroke, the principle of neurofeedback or self-regulation is being investigated for treatment of various neurologic or cognitive disorders as discussed in the 4.1 section. The increasing number of potential applications in this therapeutical area brings us into a different domain of ethical questions, regarding identity, and change of personality and self-perception (Tamburrini, 2009; Hildt, 2010). When for instance BCI is used for rehabilitation of an emotional disorder, one could wonder how this affects personality, albeit as a side-effect or as the main effect. A similar type of neurotechnology that is currently being investigated for a variety of diseases, such as psychiatric disorders, Alzheimer’s and Parkinson’s disease, is deep brain stimulation (DBS). The ethical debate surrounding DBS technology is addressing similar issues of identity and personality (Klaming and Haselager, 2010; Clausen, 2011).

59

Chapter 4. Ethical issues in BCI research, development, and dissemination

4.5

Case scenario ‘Thomas’ Thomas is a 30-year-old air traffic controller who was told by his boss that starting this month he would have to undergo attention training wearing a new neurotechnological tool that provides him with neurofeedback. Thomas does not know exactly what the device does but he feels that his attention has somewhat improved since he started training. The explanation that Thomas was given about the new tool was that it somehow reads his brain, and so he is sometimes afraid the tool can also read his thoughts. Also, last Monday Thomas got a lecture from his boss who said he could see that Thomas most likely had been drinking alcohol on Sunday night.

The scenario of Thomas (albeit somewhat futuristic) touches upon more philosophical topics such as extended (Fenton and Alpert, 2008) and enacted mind (Walter, 2010), but most importantly it raises concern for mind reading and privacy. As this case scenario illustrates, an employer may be able to gain more information than had been agreed upon when they require an employee to use a BCI system. Furthermore, the subject may be completely unaware of the extent of information that is being obtained from his or her brain. In this case the employer was able to discern the fact that the subject might have been drinking the night before, but more generally a BCI system may be able to reveal other psychological states, traits and mental health vulnerabilities (Farah, 2005). It may not be in an individual’s best interest to have this personal information available to others, especially to their employer, and workplace discrimination could be a concern. It could be seen as a violation of a person’s right to privacy. Furthermore, the case scenario raises concerns about social stratification and brain enhancement (Caplan and Elliott, 2004; Forlini and Racine, 2009). If brain enhancement in the future does become effective and popular, there could be pressure to enhance one’s brain to keep up with the competition. Barriers such as cost could prevent some people accessing this enhancement. This issue, though complicated, is not unique to BCI and is further discussed elsewhere (Farah, 2002, 2005; Haselager et al., 2009; Tamburrini, 2009).

4.6

Conclusion

BCI may develop useful technologies for humankind, but at the same time presents significant ethical and legal challenges. The challenges are due to several factors: BCI is a rapidly growing research area with potential future applications of great daily significance, both for medical and regular users, that is bound to attract the attention of media and commercial enterprises. 60

Chapter 4. Ethical issues in BCI research, development, and dissemination In four case scenarios we have identified ethical issues related to the application of BCI for users with disabilities and healthy users. Issues typical to the field of BCI relate to working with sensitive user groups, dealing with technological complexity and handling multidisciplinary teams. We illustrated that where treatment and research interests conflict, ethical concerns arise. Managing the expectations of this novel technology is important on different levels, varying from a single user and his/her legal representative to the public (via the media). We encourage researchers to facilitate the ethical and public debate and keep expectations in line with achievements since the future success of both BCI research and commercial application will rely on public acceptance of the technology. Practical recommendations from the panel members consist of creating more awareness of ethical aspects in the BCI field, which involves reaching members of all disciplines in a typical BCI multidisciplinary team. To facilitate discussion and sharing of information on ethics, we recommend the organization of an ethics workshop or discussion group in future field-specific conferences. Moreover, we suggest inviting ethicists and philosophers to participate in this dialogue. We hope that these recommendations, as well as this report, will contribute to the ability to identify and address pressing ethical issues as they appear in the rapid progress in BCI research and commercialization.

61

62

5. A note on ethical aspects of BCI Published as: Haselager, W.F.G., Vlek, R.J., Hill, N.J. & Nijboer, F. (2009). A note on ethical aspects of BCI. Neural Networks, 22(9): 1352-1357.

1

Abstract This paper focuses on ethical aspects of BCI, as a research and a clinical tool, that are challenging for practitioners currently working in the field. Specifically, the difficulties involved in acquiring informed consent from locked-in patients are investigated, in combination with an analysis of the shared moral responsibility in BCI teams, and the complications encountered in establishing effective communication with media.

1 Rutger Vlek played a key role in generating the main idea, structure and several sections of the paper, and co-authored all other sections.

63

Chapter 5. A note on ethical aspects of BCI

5.1

Introduction

Brain-Computer Interfacing (BCI) is a challenging and fast growing field of research, holding great promise for fundamental research and the development of a variety of applications, ranging from neurofeedback and neurostimulation to neurocontrol of actuators (e.g. for the purpose of communication and movement). Like other new and promising developments in research areas like genetics, neuroscience and AI, BCI provides cause for considering its potential philosophical, ethical and societal consequences. Especially over the last few years, there has been an enormous growth in publications in the area of neuroethics (Farah, 2005, 2007; Fins and Shapiro, 2007; Freeman, 2007; Fukushi et al., 2007; Glannon, 2007; Greely, 2007; Illes, 2005, 2007; Roskies, 2007; Wolpe, 2007). Various definitions of neuroethics have been offered, one of the more straightforward ones being the following: “a discipline that aligns the exploration and discovery of neurobiological knowledge with human value systems” (Illes, 2007, p.537). For BCI, the ‘alignment’ mentioned in the definition specifically concerns the practical application of neurobiological knowledge, with a focus on the development of technologies that mediate (facilitate, enhance) that application. Within neuroethics, many different topics could potentially be relevant to BCI: mind-reading and privacy; mind-control and the suppression/stimulation of (un)wanted impulses; personhood and the ownership of mind; elective enhancement and social stratification, to name but a few. Undoubtedly this list could be expanded. However, in this paper we will not try to give a comprehensive review of these major topics of neuroethics, but restrict ourselves to a domain that we think is of particular relevance to practitioners currently working in the field of BCI: the process of acquiring informed consent from locked-in patients. We will also discuss two problems that are not often mentioned but that nonetheless can have substantial impact on acquiring informed consent: sharing moral responsibility in BCI teams, and maintaining effective communication with the media. Adequately shared responsibility in interdisciplinary BCI teams is a prerequisite for good communication with patients and the presentation of BCI research within the public media is an important factor in the creation of reasonable expectations about the possibilities and limits of BCI. Earlier this year Clausen (2009) observed that BCIs “pose ethical challenges, but these are conceptually similar to those that bioethicists have addressed for other realms of therapy.” (e.g. liability, side effect, ‘policy of normalizing’, risks). Moreover he suggested that bioethics is well-prepared to deal with the issues that arise with BCI technologies. We agree, and therefore, throughout this paper, we will attempt to extract valuable insights from bioethical discussions of issues encountered in clinical research in general, as well as from practical experiences of medical teams such as Intensive Care Units. 64

Chapter 5. A note on ethical aspects of BCI

5.2

BCIs for locked-in patients

The locked-in syndrome is often ill-defined in BCI research and, between varieties of the locked-in syndrome, different ethical issues regarding informed consent may be present. The classical locked-in syndrome (LIS) can be defined as lack of voluntary motor control except for vertical eye movements and blinking, combined with preserved consciousness, whereas if any other remnants of voluntary motion other than those mentioned are present, one should consider the condition as an incomplete LIS (Bauer et al., 1979). A total lack of voluntary motor control, including all eye movements with intact cognition and sensory processing, is referred to as total or complete LIS (Bauer et al., 1979; Kübler and Birbaumer, 2008). Several studies have shown that incomplete LIS and classical LIS patients can use a BCI based on the electroencephalogram (EEG) for communication (Birbaumer et al., 1999; Kübler et al., 2001, 2005a; Neuper et al., 2003; Nijboer et al., 2008b; Sellers and Donchin, 2006; Vaughan et al., 2006). However, only two patients were able to use the BCI independently for the purpose of private communication without BCI experts being present at their home (Kübler et al., 2007; Vaughan et al., 2006). In addition, as long as reliable muscular functions such as eye movements or a minimal thumb movement are available, muscular based communication systems (e.g. with an infrared eye movement sensor) are probably more efficient than BCIs at their current level (Neumann and Kübler, 2003). Furthermore, to this date no complete locked-in patient has been able to use a BCI (Kübler and Birbaumer, 2008). Possible reasons for this are published in Birbaumer (2006), Hill et al. (2006) and Kübler and Muller (2007). Invasive Brain-Computer interfaces, which measure brain signals directly from the surface of the brain or from within the brain may provide better signal quality and dimensionality, in contrast to the above mentioned noninvasive EEG-based BCIs (Huggins et al., 2007; Kübler and Muller, 2007; Leuthardt et al., 2004). The electrocorticogram (ECoG) is recorded by placing an array of millimetre-scale electrodes epidurally or subdurally on the surface of the cortex inside the skull, whereas intracortical electrodes penetrate the grey matter and measure spike or field potentials from small numbers (tens, hundreds) of cells. ECoG-based BCIs can provide accurate control over a computer cursor in as little as 21 min in healthy subjects (Leuthardt et al., 2004). However, a complete LIS patient, who had been implanted with an array of ECoG electrodes did not achieve BCI control (Hill et al., 2006). Encouraging results come from the studies performed by Cyberkinetics Inc. with Brown University, in which tetraplegic patients are shown to be able to operate simplified computer interfaces via neural spiking recorded by intracranial electrodes (Hochberg et al., 2006; Kim et al., 2008). Significantly, this included one incompletely locked-in patient with amyotrophic lateral sclerosis (Kim 65

Chapter 5. A note on ethical aspects of BCI et al., 2008). Despite these and other promising results, there is no current out-of-the-box BCI application that might be offered to LIS patients. It is clear, however, that as soon as a BCI application effectively helps an individual who participated in a BCI study, one can speak of the BCI as not only a research tool, but also as a therapeutic intervention to maintain health and quality-of-life. Regarding clinical research in general, the 35 articles of the Declaration of Helsinki (World Medical Association, 2008), constitute a widely-accepted set of ethical rules governing work with human subjects. Emanuel et al. (2000) have grouped the ethical requirements of clinical research under the following diverse headings: social and scientific value, scientific validity, fair subject selection, favourable risk-benefit ratio, independent review, informed consent, and respect for subjects. Naturally one can expect to encounter BCIspecific issues in many if not all of these categories (in addition to some of the issues unique to BCI, as mentioned above). In the next section, we will focus specifically on the process of acquiring informed consent.

5.3

Informed consent

In medicine, informed consent to an intervention is the process, dialog and invitation for the fully informed patient to participate in choices about his/her healthcare (Liesegang, 2007). Faden and Beauchamp (1986) defined informed consent as the autonomous act by a patient or research subject to expressly permit a professional person to perform a medical action on a patient or to include a person in a research project. It implies that a discussion has to take place about basic elements, including the nature of the decision procedure; reasonable alternatives to the proposed intervention; the relevant risks, benefits and uncertainties related to each alternative; assessment of the understanding of the patient; and the acceptance of the intervention by the patient (Chenaud et al., 2007; Liesegang, 2007). For participation in research, the basic elements of informed consent are required. In addition, the patient should be informed about the purpose of the research, the expected duration of the subject’s participation, a description of the procedures to be followed and an identification of all the procedures which are considered experimental. Moreover, the patient should receive a statement describing the extent, if any, to which confidentiality of records identifying the subject will be maintained. For consent to be considered informed, patients and their relatives must have a realistic picture of the procedure being offered. During initial patient contact, expectations of the BCI should be clarified and misunderstandings carefully resolved: for example, do the patient and family erroneously believe that BCI is a treatment for a disease, or for the lockedin state itself (Neumann and Kübler, 2003)? Do they understand the difference between 66

Chapter 5. A note on ethical aspects of BCI the pathology of cases they may have heard about in the media, and their own? A stroke patient should have rather different expectations of BCI than a spinal-cord-injury patient whose cortex is essentially intact, for example. Do the patients realize the extent of media exaggeration and gloss, or are they under the mistaken impression that BCI has already allowed communication by people who cannot otherwise communicate at all? Researchers should take all necessary steps to verify the patients’ understanding of the issues discussed, perhaps by asking “comprehension” questions if appropriate, or by having patients give their assessment of risks and probable benefits. If the risks of the procedure under consideration are high, then to what extent is a patient’s decision based on reason, and to what extent on desperation, given that no known treatment offers hope? The latter mode of thinking can only weaken the ethical basis of consent. Is it possible to establish what level of pain or risk the patient would not accept? Then, there is the question to what extent the patient is able to signal their consent. Some locked-in patients may not be able to express consent adequately without the help of others to serve as interlocutors and may be declared incompetent. Incompetence is a legal term and refers to a juridical declaration that a person cannot manage his or her affairs (Terry, 2007). In this case, a legal representative of the patient is appointed, and informed consent must be obtained from this person (World Medical Association, 2008, article 27). In addition, a patient may still be able to give assent (the ethically necessary, but not sufficient, expression of a legally “incompetent” person’s agreement to participate). Where this is possible, even if it is only by signalling “yes” via eye-blinks, researchers must seek that assent in addition to the consent of the legal representative, and dissent should be respected (World Medical Association, 2008, article 28). Terry (2007) points out that obtaining informed consent and/or assent is a process, rather than just a signature on a piece of paper. Clearly it is desirable to begin this process as early as possible if the patient’s competence is decreasing over time, and to make every reasonable effort to interpret whatever signals may remain. The weakness or unreliability of the patient’s signals leads to challenges in obtaining consent or assent: one must also establish to what extent a patient’s consent, needs, or requests for information have been correctly interpreted by researchers and physicians. As Neumann and Kübler (2003) point out, a patient’s idiosyncratic signs for ‘yes’ and ‘no’ may require some study before they can be recognized reliably. Furthermore, does a weak or ambiguous signal indicate physical inability to respond clearly, or does it reflect ambivalence, confusion or indecision in the mind of the patient? Or the reluctance to give a yes/no answer to a question whose nuances the patient is not equipped to go into? Researchers have a duty to make communication with patients as clear as possible. Yet, faced with ambiguous or infrequent responses, it is common to observe how those attempting to

67

Chapter 5. A note on ethical aspects of BCI communicate with a patient will compensate by projecting their own expectations selectively onto the patient’s movements and it can be difficult, even for a researcher or practitioner who is aware of this, to avoid asking “leading questions” unless a rigid protocol has been put in place. At the simplest level this involves ensuring that questions be formulated unequivocally and be issued one at a time, leaving enough time for the patient to consider and respond. Basic principles of experimental design must then assert themselves in order to avoid bias: questions must not be unfairly weighted with regard to the time allotted to them, and the criteria for judging whether a response has been made must be consistent across questions. If the patient does not have a “yes” and “no” modality available to him or her, but only a unary “yes” in contrast to remaining inactive, adequate steps have to be taken to distinguish a voluntary from an involuntary lack of response. In such a case, questions must be asked in positive/negative pairs, sometimes with the positive half presented first, and sometimes second, as would befit a well-designed trial-based experiment. Responses should be objectively verified, perhaps by using EMG traces, or by having colleagues score a video of the communication attempt while blind to the questions being asked. From these measures, can we quantify our certainty of the interpretations we make of a patient’s response, using the appropriate statistical methods? Finally, might an impartial, untrained observer entertain any doubt that the patient can hear and understand the questions, that the patient is cognitively capable of reaching the correct answer, or even that the patient is conscious, at any given moment? If so, as in any experimental procedure, a control is required: “stupid questions”, to which patient and researcher both know the obvious answers, should be mixed into the design, and must occur in sufficient numbers to provide the required statistical power. The purpose of such questions must of course be made clear, diplomatically, to the patient. Video and audio records of such conversations between researchers and patients are very important, particularly when it comes to asking consent. Consent must also be asked repeatedly at intervals during an ongoing study, since patients may, and have the right to, change their mind during the study, yet they may be unable or be hesitant to report this (Chenaud et al., 2007). Patients must also be well informed in advance of what is expected of them in a study – in particular, how long a study will last (Liesegang, 2007). Neumann and Kübler (2003) suggest that patients should be told that a first test training of 4-6 weeks will be conducted, in which multiple approaches to BCI will be tested, after which a decision will be made on the continuation of the training. Although the data of a patient should always be kept confidential, it is wise to anticipate that researchers will want to use videos and pictures of the patient in conference presentations and publications. It should be a policy in BCI research to ask patients, early on, to what extent information about them may be used for publications, conferences or press releases.

68

Chapter 5. A note on ethical aspects of BCI It is vital, but in practice very complicated, to carefully explain the risks, disadvantages and benefits of the BCI to the patient. Risks of an EEG-based BCI system consist of the possibility of skin infections after applying the electrodes. Invasive methods carry a higher risk, since craniotomy is required to implant the electrode grid, and subjects are put under general or local anaesthesia, depending on whether interaction with the subject is required to position the electrode grid correctly. Implants can cause tissue damage and the surgery itself can lead to infections, although ECoG electrodes, particularly those placed epidurally, run this risk to a lesser extent (Schalk et al., 2008). Infections may be a long-term risk since the technology currently available for human use requires cables to be lead out of the body at some exit point that is kept permanently open – ideally, wireless transmission would be used in order to maintain full integrity of the skin. The long-term benefits of the electrodes may be difficult to predict, since immune-system reactions, and restoration of neurons’ myelin encapsulation, might conceivably change the signal properties over time, something which is only beginning to be quantified in human subjects (Kim et al., 2008). Functional disadvantages of the BCI may be the time-consuming nature of the training, and the frustration that patients might experience when training does not go well (Neumann and Kübler, 2003). BCI researchers should warn patients that ‘bad training’ days will inevitably occur and reassure them that this does not mean the end of the world and it is very common in healthy subjects as well. A further (dis)advantage constitutes the many visits of BCI researchers to the house of the patient. Although this is often rated by the patients as a pleasant side-effect of entering in a BCI study, it may also inflict a further restriction to the privacy of the household, which is already crowded due to always present caregivers and visiting doctors, ergotherapists and physiotherapists. Lastly, it should be explained to locked-in patients for whom muscular-dependent communication is still possible, that BCI cannot be guaranteed to perform better than these. A major problem regarding communication on this topic is that the often mentioned or implicitly used method of risk-benefit calculation is not easily applicable, if at all. Hildt (Hildt, 2006, 2008, p. 135) suggested that “Only those uses in which considerable benefit can reasonably be expected and in which the expected benefits clearly outweigh the risks can be considered acceptable.” However, the scientific community has not yet established a reasonable expectancy of a considerate benefit of BCIs. Moreover, for people who are (on the verge of being) completely locked-in, the potential benefit of a BCI, lacking alternatives, means the difference between communicating and not communicating at all. Complete locked-in patients or patients who are on the verge of this state could well (and reasonably?) be inclined to accept any disadvantage or any risk associated with non-invasive or invasive BCI use to regain communication. Despite these difficulties with the risk-benefit method, it is difficult to formulate an alternative decision principle that does not involve

69

Chapter 5. A note on ethical aspects of BCI an attempt to weigh risks and potential benefits and that would not be troubled by the problems indicated above. A further issue is that, as Liesegang (2007) points out, alternatives should be mentioned to the patient (although for complete-LIS patients no alternatives currently exist). A final complicating factor in making a balanced decision is that expectations from patients and caregivers are almost always (too) high, mainly because science fiction stories are told in the media, sometimes partly induced by overly enthusiastic BCI researchers (we will return to this point below). Clearly, no straightforward ethical procedure can be recommended here, but repeated and careful conversations with patients should guarantee their maximal understanding of the BCI system and its limits and possibilities.

5.4

Team responsibility

A full understanding of the limits and possibilities of BCI systems may not only be a difficult thing to achieve for the patient, but also not completely straightforward for the researchers working in the interdisciplinary research projects on BCI. The source of the problem of team responsibility is that teams that include a wide variety of experts (in BCI from mathematicians, electrical engineers and computer scientists to psychologists, neuroscientists, surgeons and physicians) have to deal with a fragmentation in the understanding of the overall picture. This is of course aggravated by the fact that the technology is developing fast, as is the knowledge about what the technology applies to (i.e. the brain). Furthermore, very different perspectives can be preferred by scientists from different disciplines. A pragmatic and, in itself, respectable viewpoint from an engineering perspective might be that we do not need to understand how the brain works as long as we can measure relative differences between mental tasks to drive the BCI, whereas for a neuroscientist the understanding of brain functioning is essential. Finally, it is not just knowledge about, or perspectives on, but also responsibility for the effects of BCI that can become unclear due to teamwork. As Hamilton et al. (1961, nr.70) once stated succinctly: “plurality in the Executive (...) tends to conceal faults and destroy responsibility.” The influence of group dynamics on decision making is vividly illustrated by the Abilene Paradox (Harvey, 1974), where a family ends up having a bad dinner in a lousy restaurant in Abilene, Texas. Each member believes the others want to go and never questions this. It is vital to organize intra-group communication in a way that such suboptimal outcomes can be prevented. Especially in relation to clinical aspects of BCI applications, much can be learned from interdisciplinary teams working in similarly demanding situations, such as intensive care units (ICU), teams concerned with severe mentally disordered patients (Liberman et al., 2001), or chronic or progressively ill patients facing end-of-life care decisions. A critical 70

Chapter 5. A note on ethical aspects of BCI element in this type of interdisciplinary teamwork is that the different areas of expertise must be integrated into a practical “service delivery” for the patient, while at the same time “mechanisms for accountability” must be assured (Liberman et al., 2001, p.1334). It is noticeable that disagreements arise easily in such situations. In one ICU study (Breen et al., 2001) conflicts were observed to occur among staff in 48% of cases (equal to the percentage of conflicts between staff and family), whereas only 24% of conflicts among family members were reported. In medical teams, conflict between medical and nursing staff is often “frequent and bitter” (Tchudin, 2001, p.465). The character of these conflicts is mainly determined by the different viewpoints from the different types of experts, each having contact with the patient. It is pressured even more by difference in social hierarchy (educational level). Similar to medical teams, a BCI team usually has a hierarchical structure, potentially pressuring the influence of experts lower in rank in favour of their authorities. However, whereas nurses have an intensive patient contact, many of the co-working experts in the BCI team do not. This difference can potentially decrease the feeling of moral responsibility in those co-working experts, while their unique point of view may actually be crucial for the evaluation of a specific moral consideration. Among several recommendations to improve team functioning, many of which are reasonably obvious (e.g. regular team meetings, ensuring good lines of communication), two are worth mentioning in relation to acquiring informed consent. First, ensure that appropriate members of the team are asked whether they should be present at a decision making meeting with patient and family. Second, have a ‘preconference’ of team members to develop team consensus and facilitate discussion of issues or conflicts that may occur within the team (Shanawani et al., 2008, p.780). Current ethical guidelines, such as formulated in the Declaration of Helsinki (World Medical Association, 2008), focus on a single scientifically/medically competent individual who carries the responsibility for the human subject. However, it is precisely this individualized type of responsibility that becomes problematic in a multidisciplinary research environment. It may well be possible that the scientist with the most encompassing perspective may not be the one actually communicating with the patient or family. In all, it is important that, within BCI projects, considerable attention is given to four general ethical issues regarding team work (Frey, 2007): how teams achieve justice in the distribution of work and the credits thereby attainable, assign responsibility for decisions that are made, especially those that may have far-reaching consequences for participants or patients, ensure reasonableness in allowing participation, resolution of conflicts and reaching consensus, and maintain honesty in communication and reporting results. It requires continuous effort and attention from the entire multidisciplinary BCI team to provide one or more well-informed, multidisciplinary competent individuals as contact persons for the

71

Chapter 5. A note on ethical aspects of BCI patient, capable of translating the multidisciplinary scientific content into understandable indications of risks and benefits for the subject.

5.5

Communication with the media

The difficulties involved in communicating about BCI deserve specific attention. Publicity about the possibilities and impossibilities of BCI will have implications for the expectations of patients, thereby influencing the process of acquiring informed consent. As Illes et al. (2005, p. 981) say: “A risk of public engagement is that of creating false hopes and expectations”. There are two major aspects of communicating with the press about BCI that are directly relevant to the topic of informed consent. First, there is the general issue of the (un)certainty of scientific knowledge. Researchers are thoroughly familiar with the vagaries of science, especially when it comes to recent developments and the process of acquiring new insights. For the general public, however, a ‘scientific finding’ is taken as a fact, as equivalent to ‘100% accuracy’ (Garrett and Bird, 2000). Explaining that certain statements for the moment do not go beyond being conjectures or hypotheses under investigation can be a long and arduous process. As every teacher knows, most students take years to go from a ‘tell me how it is’ attitude to the cautious questioning approach towards scientific findings that is characteristic of professional researchers. Even worse, uncertainty is generally not what the larger public seeks or appreciates about science (Bird, 2003). Although it is hardly possible to communicate effectively about the nature of science in general while announcing one’s recent research findings, this difference in perception of scientific certainty helps to emphasize that it is “extraordinarily important that scientists avoid over-hyping the significance of their findings” (Garrett and Bird, 2000, p.439). Rather, the accent should be put on the qualifications and limitations concerning the results reported. In other words, it is important to actively resist the temptation to go along with the public’s desire for certainty. The second problem in communicating with the media that is challenging in its own right, but that may also affect the process of acquiring informed consent arises due to the, at times large, gap between currently feasible and potentially possible applications in the medium or long term. Specifically the topic of mind reading (in connection to all kinds of spectacular applications) is bound to attract attention from the media. It is not hard to find headlines in media like BBC news and Science Daily such as ‘Paralysed man’s mind is ‘read” (BBCNews, 2007; ScienceDaily, 2008), ‘Brain fingerprints under scrutiny’ (McCall, 2004), ‘Towards zero training for BCI’ (ScienceDaily, 2008), and ‘Brain sensor for market research’ (Greene, 2007). Such reports mostly deal with future possibilities. This focus on what might be achievable with BCI is not objectionable in itself. If no one would expect 72

Chapter 5. A note on ethical aspects of BCI important progress within a reasonable amount of time, BCI would not be the fast growing field it is now. Also, in relation to BCI’s ethical implications, potentially problematic developments need to be identified before they arise, so that they may be dealt with properly. Therefore it is logical that future expectations do play a role in ethical analyses and in communication between scientists and journalists. The big question, of course, is what constitute reasonable expectations concerning which point (nearby, distant) in the future. It is precisely regarding these aspects that self-restraint and clarity are called for. When talking to the press about BCI it, therefore, would be advisable to be extremely reluctant to engage in speculations concerning anything beyond the near future (3-5 years or so) or depending on breakthroughs that, at present, are not foreseeable. The two problems indicated above are aggravated by the potential occurrence of misunderstandings or inadequate renderings in the media of a scientist’s statements. As most people who have been involved with media will know, it is not unusual that journalists come with a specific story in mind that they would like to tell their audience. Even if this is not the case, public media are generally more interested in what may be possible than in reporting scientists’ scepticism and reservations. It can be quite difficult to avoid seeing one’s words appear as part of an overall message that is not the scientist’s own. Avoiding this danger is, to a significant extent, the scientist’s responsibility, as Dennett (2003, p.17) has suggested: “We need to recognize that our words might be misunderstood, and that we are, to some degree just as responsible for likely misunderstandings of what we say as we are for the “proper” effects of our words. (...) Sometimes the likelihood of misunderstanding (or other misuse) of one’s true statements, and the anticipatable harm such misunderstanding could propagate, will be so great that one had better shut up.” However, such radical non-cooperation with the press can be undesirable for many different reasons, ranging from a sincerely felt general duty to inform the public to creating well-timed publicity for research project proposals that need funding. As Rose (2003, p.310) formulates it: “Researchers depend on grants for their work, and the higher the public visibility, the more likely one feels that one’s work is going to be noticed and the grant money flow in (...) It is no good announcing anything less than a major breakthrough”. Still, a responsible media strategy may consist in being as explicit as possible concerning the limitations of scientific ‘certainty’ in general and the current boundaries of BCI in particular. Restricting discussions of topics and illustrative cases to a short term future cannot guarantee against misinterpretations or exaggerated headlines, but it may help in taming the more extravagant claims or expectations of the media. For similar reasons, a certain amount of self-restraint concerning catchy but too promising titles of publications in scientific journals (or statements in them) is called for.

73

Chapter 5. A note on ethical aspects of BCI

5.6

Conclusion

To conclude, many practical ethical issues surround BCI in relation to both research and intervention. Acquiring an ethically sound informed consent from a locked-in patient may be challenging due to the high expectations of the patient, the difficulty in communicating and the lack of alternatives. However, more attention to strict and standardized policies like the ones suggested above could help to maximise the chance that patients get adequate information, have maximal comprehension and voluntarily enter the study. Similarly, a focus on the ethical issues regarding teamwork may help to achieve responsible functioning of everyone involved in BCI research. Prudence in what researchers communicate to the public media and watchfulness concerning how what has been said gets represented, would help to reduce unrealistic expectations. Research into BCI, as well as its applications in clinical settings, involves the exploration of a relatively new terrain, and is likely to become more important in the near future. A growing attention for the practical ethical challenges faced by scientists, clinicians, participants and patients is clearly called for.

74

6. General discussion 6.1 6.1.1

Subjective accenting and BCI Summary and conclusions

In the previous four chapters two EEG studies were presented on the topic of subjective accenting. This process allows subjects to add perceptual salience to specific tones in a sequence, tones that are objectively equal. The first study investigated whether subjective accents, voluntarily imposed by subjects in 2-, 3-, and 4-beat patterns on an auditory metronome, are detectable from 500 ms chunks of single-trial EEG by means of multivariate machine learning methods. Results show this is possible with classification rates significantly above chance level for perceived (70%) as well as for imagined (61%) accents. Moreover, a method was described for cross-condition classification and it was shown that this allows to classify imagery data with a classifier trained on perception data, and vice versa, thus revealing similarity in brain responses of heard and imagined accents. These results support the hypothesis of shared mechanisms in perception and imagery for auditory processing (see also Schaefer, 2011), which had been demonstrated before in the visual domain (Kosslyn et al., 1995). There is indeed a whole spectrum between pure top-down and bottom-up auditory processing (like hallucination, speech perception errors, selective listening) that indicates that listening is not a closed perceptual one-way module that simply translates sound into mental representation (Bregman, 1990). It is rewarding that a paradigm emerged for investigating these issues, and that the suggestion of shared mechanisms finds support in the observed brain responses for accents. The technique of crosscondition classification not only seems promising in this respect but also in a practical sense of experimental control: instructing a subject to listen is often much easier (for both the experimenter and the subject) than explaining how to imagine something. The second EEG study elaborated on the decoding of subjective accenting from EEG and investigated its feasibility for use in a BCI. Multiple algorithms were presented for decoding sequences of accented and non-accented beats (i.e. several cycles of 2-, 3-, or 4beat patterns). Decoding performances of these algorithms were compared by means of bit rate. The best scenario yielded an average bit rate of 4.4 bits/min over subjects, which whilst not high for auditory BCIs does prove the in principle feasibility of the paradigm.

6.1.2

Limitations and implications

Results of chapters 2 and 3 should be interpreted in the light of the following limitations. Both studies are based on a relatively small number of subjects (10). Also, it is unknown 75

Chapter 6. General discussion how the estimated BCI performances reported in chapter 3 will translate to a BCI based on a subjective accenting paradigm in the real world. Performances are expected to decrease slightly due to a bigger impact of non-stationarities in an online setting. Negative effects on the EEG, and thus on BCI performance, are also to be expected from a less well-controlled electrical environment (i.e. BCI outside a laboratory). As with any imagery study, it is difficult to control for the actual mental task that subjects perform. However, using probe tones it was assured that subjects were able to maintain an internal pattern of subjective accenting to the end of each sequence. It must also be mentioned that sequences of imagery trials were always preceded by sequences of perception trials (see chapter 3, figure 3.1). The experimental design and data handling did not leave room for any direct overlap of perception responses to imagery trials, but there may have been unforeseen ways in which subjects benefitted from the leading perception part. Any future BCI application will rely on the subject’s ability to voluntarily initiate different imagery patterns, so further investigation is needed to determine the importance of this leading perception part. Chapters 2 and 3 have illustrated that subjective accents are detectable, from EEG, when voluntarily imposed by subjects on a steady metronome. This result emphasizes that perception is very much an active process (Huron, 2006), where the percept is a combination of bottom-up (exogenous) input and top- down (endogenous) control. The difference in EEG waveforms relating to subjectively accented versus non-accented beats could be interpreted as originating from top-down modulation of the perceptual process. Thus, this work is also important to music cognition research, where the mechanism of top-down modulation (also in processing of melody and harmony) is a central topic (see the key questions of music research in section 1.3, point one). One hypothesis for the kind of top-down processes taking place is that perception utilizes a form of active tracking, building expectations for future development of sound patterns based on sound patterns so far perceived. Imagery may make use of the same structural mechanisms which is detected in the EEG signal as a common response component. Another hypothesis is that the process of imagery makes use of perceptual mechanisms by injecting information in the processing pipeline to simulate the imagined input. In this case the imagery can be seen as an endogenous substitute for the perception of exogenous stimuli. The fact that some phenomena (clock illusion, hallucinations) have a very life-like character, and that the resulting impression is hard to distinguish from a real percept provides intuitively good reasons to investigate this hypothesis further. The results presented in Chapters 2 and 3 can contribute to a better general understanding, and development of novel paradigms to investigate the central question of “what actually is auditory imagery?” (see the key questions of music research in section 1.3, point four).

76

Chapter 6. General discussion In general, the Machine Learning (ML) methods commonly used for BCI have also proven to be a valuable tool for research in neuroscience. Such ML methods impose very strict rules on data treatment (reducing the risk of over-fitting the data), while providing great sensitivity to multivariate details in the data. Such pattern based analysis techniques are already coming to the fore in fMRI analysis (O’Toole et al., 2007) and becoming increasingly common in analysis of electrophysiological data. Chapter 3 reported positive results on the feasibility of using subjective accents for a BCI. While visual P300-based BCIs are at present the most popular and reliable BCI method, some situations call for an alternative. For users with a visual impairment, or when the visual modality is occupied by other tasks, an auditory BCI based on subjective accents may provide a worthwhile alternative. The cross-condition classification method reported in chapters 2 and 3, using perception data to train a classifier for decoding imagery data, yields several implications for BCI. It is difficult to instruct imagery and control for what subjects are actually imagining, and whether the mental strategy they use remains stable over time. This is partly overcome when classifiers can be trained on perception data. An extra advantage in the case of subjective accents is that the EEG signals corresponding to perception are stronger, leading to a better quality of training data for the classifier. It may well be possible to extend the same approach to other mental tasks used for BCI based on imagined percepts.

6.1.3

Future direction of BCI

As a conclusion to this section on subjective accenting and BCI, I would like to share some considerations in the form of recommendations for future BCI research • BCI research has been quite slow to move beyond the ‘proof of concept’ stage. In the first stage fundamental knowledge (e.g. that motor imagery activates motor cortex in a measurable way) has been shown to be applicable, but often by only proving feasibility of a working BCI for a few of the population of subjects in a lab environment. Now the field is entering the stage of bringing these results into the real world. In real-world situations it can become clear that the internal mechanisms of the brain are not as simple or static as one may have hoped or as was implied by laboratory studies . The brain functions as a whole and can hardly be studied by looking only at a small part of it. Thus, studying a single brain response in isolation from other cognitive processes may by itself never provide us with the knowledge needed for robust real-life BCI applications. The P300 response for example is not at all static, but influenced by fluctuating attention levels and memory processes (Hohnsbein et al., 1995; Polich, 2007), to name only a few factors. This means we have to start modeling for 77

Chapter 6. General discussion instance attentional effects and perception effects, and use those to predict elements of other responses (e.g. a P300) with which they interact. • To be able to bring BCI out of the lab into the real world, the signal analysis methods also need careful examination. Many ‘proof-of-concept’ BCIs are constructed with methods that are not suited to fast, fully automated, and online use, sometimes they even rely on manual removal of trials with artifacts or outliers. In the real world, signal analysis methods can only use the data available so far and preferably use streamed (per sample) processing. Further, the analysis methods should be robust against outliers and non-stationarities, and suited to the type of classifier used. Ideally, artifacts should be automatically detected and where possible corrected for. Development and standardization of such online methods would provide a foundation to more online BCI. Large BCI research consortia like Tobi and BrainGain have taken-up this challenge (Tobi, 2013; BrainGain, 2013; BrainStream, 2013; OpenVibe, 2013; BCILab, 2013). This may result in systems that allow a faster turn-over of neuroscience and BCI research, as results become available during or immediately after an experiment. • As was shown in chapter 2, research results can yield both fundamental insights and potential practical applications. BCI is a great new tool for fundamental neuroscience research (Jensen et al., 2011) and the interaction of fundamental and applied research forms very fertile ground from which I foresee new insights to grow. In fact the quite rigid distinction between fundamental and applied research seems almost more an evolved cultural one, as understanding the underlying theory is the first step to developing a practical application, and attempting to apply a theory can reveal its shortcomings and initiate a new round of scientific hypothesis formation and testing. • While it seems safe to focus research efforts on the mental tasks known to work for BCI (e.g. P300 and motor imagery), a wider exploration of potential BCI paradigms is needed. Many of the present problems with BCI, such as BCI-illiteracy and signal non-stationarity, were identified from a very limited set of paradigms. Novel mental tasks may not suffer from these problems (or do so to a lesser extent) or have advantages for certain users, such as gaze-independence. There is a whole wealth of alternative mental activities (e.g. covert attention Bahramisharif et al., 2010) or signs thereof in measured brain activity (e.g. phase coherence) waiting to be explored in the context of BCI research. • While striving for success with BCI, we have to ask ourselves what exactly we are aiming for. Publication criteria are perhaps most easily met when aiming for high 78

Chapter 6. General discussion BCI performance for a few subjects. In the long run however, our focus should be on a robust BCI accessible to many people (Wolpaw et al., 2002). It is not unlikely that problems with inter-subject variability and non-stationarity in part have been a consequence of our evaluation methods. When we are aiming too much for high performance on a few subjects in lab settings, we may be losing real-world robustness over subjects and over time. • Rather than aiming for a BCI that works ‘out of the box’, we should exploit the brain’s capability to learn and adapt for control over a new BCI. Real-time feedback from the BCI is crucial for this process. Even with respect to the body, we are not capable of attaining control without feedback, as is illustrated by the fact that a person deprived of propriocepsis in the legs is hardly capable of walking, even though the pathway from brain to muscles is fully in tact (Sacks, 1985; Schalk, 2011). BCIs may even help tap this potential, like the emerging approaches for rehabilitation after stroke (Severens, 2013). I sincerely hope these considerations and recommendations can be of help to facilitate the transition of BCI into valuable technology for society.

6.2 6.2.1

BCI and ethics Summary and conclusions

Chapters 4 and 5 focused on ethical aspects of BCI. Chapter 4 discussed four case scenarios related to the application of BCI for users with disabilities and healthy users, inspired by current experiences in various BCI laboratories, and extrapolating along those lines. By identifying and disentangling ethical issues from these case scenarios, I hope to have made the ethical debate on BCI accessible to a wide audience. This is important because future success of both BCI research and commercial applications will rely to a substantial degree on public acceptance of the technology. Issues typical to the field of BCI relate to working with sensitive user groups, dealing with technological complexity and handling multidisciplinary teams. I illustrated that where treatment and research interests conflict, ethical concerns arise. Managing the expectations of this novel technology is also important on a variety of levels, varying from a single user and his/her legal representative in contact with medical staff to the general public addressed via the media. Chapter 5 elaborated on the ethical issues surrounding BCI and aspired to make connections to background knowledge in medical sciences and philosophy. Acquiring ethically sound informed consent from a locked-in patient may be challenging due to the high ex79

Chapter 6. General discussion pectations of the patient, the difficulty in communicating and the lack of alternatives. However, more attention to strict and standardized policies could help to increase the chance that patients receive adequate information, have maximal comprehension and voluntarily as well as knowingly enter the study. Similarly, a focus on the ethical issues regarding teamwork may help to achieve responsible functioning of everyone involved in BCI research. Prudence in communication with the public media would help to reduce unrealistic expectations. Given the current rate of progress and potential impact of BCI in both clinical settings and for healthy users, more attention for the practical ethical challenges faced by scientists, clinicians, participants and patients is called for.

6.2.2

Implications and future direction

In relation to both chapters on ethics, I would first like to emphasize the value of first-hand experience with BCI research when studying ethics, and would encourage young scientists to take interest in the philosophical aspects of the research they are doing. I believe this is of great importance to society, as researchers are best placed to communicate to the public about the nature and implications of their work. With chapter 4 I hope to have made the ethical debate surrounding BCI more accessible, and I look forward to the involvement of a wider audience on the topics discussed. Furthermore, I hope that both chapter 4 and 5 provide useful insights and recommendations that current BCI researchers, developers and clinicians can keep in mind, while working on the BCIs of the future. Although an ethical debate is valuable in itself, a strong point can be made when the debate is combined with a more elaborate empirical foundation. The following paragraphs provide suggestions for a practical approach to obtaining such an empirical foundation, focusing on some of the key topics from chapters 4 and 5. • Shared moral responsibility in multidisciplinary teams and technological complexity: The key here is to find out to what level different members (e.g. everyone from the doctor or experimenter directly involved with a patient or subject to the programmer who provided parts of the experimental software) of a multidisciplinary BCI team are aware of ethical issues and their moral responsibility in the work of their group. With a questionnaire (see e.g. Nijboer et al., 2011) awareness could be investigated, and information obtained on relevant procedures. Questions should be included along the lines of: who informs the subject? What information is exactly provided to the subject? Who is held morally responsible throughout intervention or an experiment? Is this a single person or is responsibility distributed? Did events occur in the past that required moral judgments? If so, how was this handled, and by whom? 80

Chapter 6. General discussion • Public expectation and (mis)conceptions:

A good approach here would be to organ-

ize a public discussion between experts (from both research and industry), potential users and a wider audience. Before and after the discussion takes place conceptions and expectations about BCI of both the audience and potential users could be measured by means of a questionnaire or short interviews. This allows us to observe (mis)conceptions and unrealistic expectations, as well as the impact of the discussion. Any misconceptions and unrealistic expectations persisting after the discussion will be very informative to the experts, and provide them feedback on their way of communicating. Another interesting topic would be to find out which ethical issues mainly determine public (non-)acceptance of BCI technology. • Conflicts between research and therapeutic interest: A wealth of information could be obtained from independent observation of present BCI research and interventions. An independent scientist could investigate interests and expectations of all parties involved, allowing conflicts of interests to become crystal clear. Moreover, it would be interesting to observe how such conflicts are dealt with, if they occur. Another approach, especially relevant to (near-)future applications, would be to make use of fictive BCI scenarios including conflicts of interest and ask researchers, clinicians and potential users for their moral judgments regarding these scenarios. This should help identifying situations that require extra attention. With the increasing potential and use of BCI for society, ethical issues will become more prominent to the general public. Current and future development in the area of cognitive enhancement may raise additional ethical concerns, and when such technologies become more widely used, attention to the legal aspects is also required. An appropriate answer needs to be found to questions such as: Can a person be held legally responsible for harm done to another person by his or her malfunctioning BCI or cognitive enhancement (e.g. Haselager, 2013)? Can an employer require employees to use cognitive enhancements to increase productivity? What about competition arising between employees willing, and those unwilling to use cognitive enhancements? Such questions encourage continuing involvement of ethicists with this topic, closely following on future developments.

81

"Want men kan niet vroeg genoeg leren dat we ons, als we prijs stellen op wat vreugd in dit leven, nu eenmaal moeten behelpen met de werkelijkheid." S. Carmiggelt, Duiven Melken, 1955

References

83

7. References Ahi, S., Kambara, H., and Koike, Y. (2011). A dictionary-driven P300 speller with a modified interface. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 19(1):6–14. Allison, B., McFarland, D., Schalk, G., Zheng, S., Jackson, M., and Wolpaw, J. (2008). Towards an independent brain–computer interface using steady state visual evoked potentials. Clinical Neurophysiology, 119:399–408. Bahramisharif, A., Van Gerven, M., Heskes, T., and Jensen, O. (2010). Covert attention allows for continuous control of brain-computer interfaces. European Journal of Neuroscience, 31:1501–1508. Bangert, M., Peschel, T., Schlaug, G., Rotte, M., Drescher, D., Hinrichs, H., Heinze, H.J., and Altenmüller, E. (2006). Shared networks for auditory and motor processing in professional pianists: Evidence from fMRI conjunction. NeuroImage, 30(3):917–926. Bauer, G., Gerstenbrand, F., and Rumpl, E. (1979). Variables of the locked-in syndrome. Journal of Neurology, 221:77–91. BBCNews (2007).

Paralysed man’s mind is ‘read’.

Retrieved June 23, 2008 from

http://news.bbc.co.uk/2/hi/health/7094526.stm. BCILab (2013). BCILAB - open source matlab toolbox for brain-computer interface research. Retrieved September 23, 2013 from http://sccn.ucsd.edu/wiki/BCILAB. Beauchamp, T. and Childress, J. (2008). Principles of Biomedical Ethics. Oxford University Press, 6th edition. Belin, P., Eeckhout, P. V., Zilbovicius, M., Remy, P., Francois, C., Guillaume, S., Chain, F., Rancurel, G., and Samson, Y. (1996). Recovery from nonfluent aphasia after melodic intonation therapy: a PET study. Neurology, 47:1504–1511. Birbaumer, N. (2006). Brain–computer-interface research: Coming of age. Clinical Neurophysiology, 117(3):479–483. Birbaumer, N., Ghanayim, N., Hinterberger, T., Iversen, I., Kotchoubey, B., Kübler, A., Perelmouter, J., Taub, E., and Flor, H. (1999). A spelling device for the paralysed. Nature, 398(6725):297–298. Bird, S. (2003).

The IPTS report:

Communicating scientific advice to the public.

Retrieved June 2, 2009 from http://ipts.jrc.ec.europa.eu/home/report/english/ articles/vol72/SCI3E726.htm. Bishop, C., editor (2006). Pattern recognition and machine learning. Springer. 84

7. References Bolton, T. (1894). Rhythm. American Journal of Psychology, 6:145–238. Borasio, G., Sloan, R., and Pongratz, D. (1998). Breaking the news in amyotrophic lateral sclerosis. Journal of Neurological Sciences, 160(1):S127–133. BrainGain (2013). BrainGain. Retrieved September 23, 2013 from http://www.braingain.nl. BrainStream (2013).

BrainStream TWiki.

Retrieved September 23, 2013 from

http://www.brainstream.nu. Breen, C., Abernethy, A., Abbott, K., and Tulsky, J. (2001). Conflict associated with decisions to limit life-sustaining treatment in intensive care units. Journal of General Internal Medicine, 16(5):339–341. Bregman, A. (1990). Auditory scene analysis. MIT Press: Cambridge, MA. Brochard, R., Abecasis, D., Potter, D., Ragot, R., and Drake, C. (2003). The "ticktock" of our internal clock: direct brain evidence of subjective accents in isochronous sequences. Psychological Science, 14(4):362–366. Buch, E., Weber, C., Cohen, L., Braun, C., Dimyan, M., Ard, T., Mellinger, J., Caria, A., Soekadar, S., Fourkas, A., and Birbaumer, N. (2008). Think to move: a neuromagnetic brain-computer interface (BCI) system for chronic stroke. Stroke, 39:910–917. Burkard, R., Don, M., and Eggermont, J., editors (2007). Auditory Evoked Potentials. Lippincott Williams and Wilkins. Caplan, A. and Elliott, C. (2004). Is it ethical to use enhancement technologies to make us better than well? PLoS Med, 1(3). Carmena, J., Lebedev, M., Crist, R., O’Doherty, J., Santucci, D., Dimitrov, D., Patil, P., Henriquez, C., and Nicolelis, M. (2003). Learning to control a brain-machine interface for reaching and grapsing by primates. PLoS Biology, 1(2):doi:10.1371/journal.pbio.0000042. Chen, Y., Huang, X., Yang, B., Jackson, T., Peng, C., Yuan, H., and Liu, C. (2009). An event-related potential study of temporal information encoding and decision making. NeuroReport, 21(2):152–155. Chenaud, C., Merlani, P., and Ricou, B. (2007). Research in critically ill patients: standards of informed consent. Critical Care, 11(1):110. Clausen, J. (2009). Man, machine and in between. Nature, 457(26):1080–1081. Clausen, J. (2011). Conceptual and ethical issues with brain-hardware interfaces. Current Opinion in Psychiatry, 24(6):495–501. 85

7. References Daly, J., Cheng, R., Rogers, J., Litinas, K., Hrovat, K., and Dohring, M. (2009). Feasibility of a new application of noninvasive brain computer interface (BCI): A case study of training for recovery of volitional motor control after stroke. Journal of Neurologic Physical Therapy, 33(4):203–211. Daly, J. and Wolpaw, J. (2008). Brain-computer interfaces in neurological rehabilitation. Lancet Neural, 7:1032–1043. DeCharms, R., Maeda, F., Glover, G., Ludlow, D., Pauly, J., Soneji, D., Gabrieli, J., and Mackey, S. (2005). Control over brain activation and pain learned by using real-time functional MRI. In Proc. Natl. Acad. Sci. USA, volume 102, pages 18626–18631. Dennett, D. (2003). Freedom evolves. New York Viking. Desain, P. (1992).

A (de)composable theory of rhythm perception.

Music Perception,

9(4):439–454. Desain, P. and Honing, H. (2003). Single trial ERP allows detection of perceived and imagined rhythm. In Proceedings of the RENCON workshop International Joint Conference On Artificial Intelligence (IJCAI). Dien, J. and Frishkoff, G. (2005). Event-related potentials: A methods handbook, chapter Principal components analysis of event-related potential datasets. MIT Press. Dornhege, G., del R. Millán, J., Hinterberger, T., McFarland, D., and Müller, K.-R., editors (2007). Toward Brain-Computer Interfacing. MIT Press. Emanuel, E., Wendler, D., and Grady, C. (2000). What makes clinical research ethical? Journal of the American Medical Association, 283:2701–2711. Emotiv (2013). Emotiv | EEG system | Electroencephalography. Retrieved June 12, 2013 from http://www.emotiv.com. Faden, R. and Beauchamp, T. (1986). A history and theory of informed consent. New York, NY: Oxford University Pres. Farah, M. (2002). Emerging ethical issues in neuroscience. Nature neuroscience, 5:1123–1129. Farah, M. (2005). Neuroethics: The practical and the philosophical. Trends in Cognitive Sciences, 9(1):34–40. Farah, M. (2007). Social, legal, and ethical implications of cognitive neuroscience: “neuroethics” for short. Journal of Cognitive Neuroscience, 19(3):363–364. 86

7. References Farah, M. and Smith, A. (1983). Perceptual interference and facilitation with auditory imagery. Perception and Psychophysics, 33(5):475–478. Farwell, L. and Donchin, E. (1988). Talking off the top of your head: Toward a mental prosthesis utilizing event-related potentials. Electroencephalography and Clinical Neurophysiology, 70:510–523. Fenton, A. and Alpert, S. (2008). Extending our view on using BCIs for locked-in syndrome. Neuroethics, 1(2):119–132. Fieldtrip (2010). FieldTrip, a Matlab software toolbox for MEG and EEG analysis. Retrieved April 28, 2010 from http://fieldtrip.fcdonders.nl/. Fins, J. and Shapiro, Z. (2007). Neuroimaging and neuroethics: Clinical and policy considerations. Current Opinion in Neurology, 20(6):650–654. Forlini, C. and Racine, E. (2009). Autonomy and coercion in academic “cognitive enhancement” using methylphenidate: Perspectives of key stakeholders. Neuroethics, 2:163–177. Fraise, P. (1982). The psychology of music, chapter Rhythm and tempo, pages 149–180. Academic Press. Freeman, A. (2007). Neuroethics: Defining the issues in theory, practice, and policy. Journal of Consciousness Studies, 14(3):118–121. Frey, W. (2007).

Ethics of team work.

Retrieved October 25, 2008 from http://cnx.

org/content/m13760/1.7/. Fukushi, T., Sakura, O., and Koizumi, H. (2007). Ethical considerations of neuroscience research: The perspectives on neuroethics in Japan. Neuroscience Research, 57(1):10–16. Furdea, A., Halder, S., Krusienski, D., Bross, D., Nijboer, F., Birbaumer, N., and Kuebler, A. (2009). An auditory oddball (p300) spelling system for brain–computer interfaces. Psychophysiology, 46. Garrett, J. and Bird, S. (2000). Ethical issues in communicating science. Science and Engineering Ethics, 6(4):435–442. Geuze, J., Desain, P., and Ringelberg, J. (2008). Re-phrase: chat-by-click: a fundamental new mode of human communication over the internet. In CHI ’08 extended abstracts on Human factors in computing systems, CHI EA ’08, pages 3345–3350, New York, NY, USA. ACM. 87

7. References Geuze, J., Farquhar, J., and Desain, P. (2012). Dense codes at high speeds: varying stimulus properties to improve visual speller performance. Journal of Neural Engineering, 9(1):016009. Glannon, W., editor (2007). Defining right and wrong in brain science. Washington, DC: Dana Press. Greely, H. (2007). On neuroethics. Science, 318(5850):533–533. Greene, K. (2007).

Brain sensor for market research:

people’s minds while they view ads.

A startup claims to read

Retrieved September 27, 2008 from

http://www.technologyreview.com/Biztech/19833/. Halpern, A. (2001). Cerebral substrates of musical imagery. Annals of the New York academy of sciences, 930:179–192. Halpern, A., Zatorre, R., Bouffard, M., and Johnson, J. (2004). Behavioral and neural correlates of perceived and imagined musical timbre. Neuropsychologia, 42:1281–1292. Hamilton, A., Jay, J., and Madison, J. (1961). The federalist papers. New York: New American Library. Harvey, J. (1974). The abilene paradox and other meditations on management. Organizational Dynamics, 3(1):63. Haselager, P. (2013). Did i do that? brain-computer interfacing and the sense of agency. Minds and Machines, 23(3):405–418. Haselager, P., Vlek, R., Hill, J., and Nijboer, F. (2009). A note on ethical aspects of BCI. Neural Networks, 22(9):1352–1357. Hessels, R. (2012). Neural correlates of mental strategies in subjective accenting - implications for bci. Master’s thesis, Artificial Intelligence Radboud University Nijmegen. Hildt, E. (2006). Electrodes in the brain: Some anthropological and ethical aspects of deep brain stimulation. International Review of Information Ethics, 5(9):33–39. Hildt, E. (2008). Theoretical and ethical issues on brain-computer interfaces. In 4th international brain-computer interface workshop and training course. Hildt, E. (2010). Brain-computer interaction and medical access to the brain: Individual, social and ethical implications. Studies in Ethics, Law, and Technology, 4(3):10.2202/1941– 6008.1143. 88

7. References Hill, J., Farquhar, J., Martens, S., Biessmann, F., and Schölkopf, B. (2008). Effects of stimulus type and of error-correcting code design on BCI speller performance. In Advances in Neural Information Processing Systems 21, pages 665–672. Hill, N., Lal, T., Bierig, K., Birbaumer, N., and Schölkopf, B. (2004). Attentional modulation of auditory event-related potentials in a brain-computer interface. In Biomedical Circuits and Systems 2004 IEEE International Workshop. Hill, N., Lal, T., Schroder, M., Hinterberger, T., Wilhelm, B., Nijboer, F., Mochty, U., Widman, G., Elger, C., Schölkopf, B., Kübler, A., and Birbaumer, N. (2006). Classifying EEG and ECoG signals without subject training for fast bci implementation: Comparison of nonparalyzed and completely paralyzed subjects. IEEE Transactions on Neural Systems and Rehabilitation Engineering,, 14(2):183–186. Hochberg, L., Serruya, M., Friehs, G., Mukand, J., Saleh, M., Caplan, A., Branner, A., Chen, D., Penn, R., and Donoghue, J. (2006). Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature, 442(7099):164–171. Hoedlmoser, K., Pecherstorfer, T., Gruber, G., Anderer, P., Doppelmayr, M., Klimesch, W., and Schabus, M. (2008). Instrumental conditioning of human sensorimotor rhythm (1215 hz) and its impact on sleep as well as delarative learning. Sleep, 31(10):1401–1408. Hohnsbein, J., Falkenstein, M., and Hoormann, J. (1995). Effects of attention and timepressure on P300 subcomponents and implications for mental workload research. Biological Psychology, 40(1-2):73–81. Hubbard, T. (2010).

Auditory imagery: Empirical findings.

Psychological Bulletin,

136(2):302–329. Huggins, J., Levine, S., Graimann, B., Chun, S. Y., and Fessler, J. (2007).

Toward

brain–computer interfacing, chapter Electrocorticogram as Brain-computer interface signal source, pages 129–145. Cambridge, Massachusetts, USA: The MIT Press. Huron, D. (2006). Sweet Anticipation: Music And the Psychology of Expectation. MIT Press. Illes, J., editor (2005). Neuroethics: Defining the issues in theory, practice and policy. Oxford: Oxford University Press. Illes, J. (2007). Empirical neuroethics - Can brain imaging visualize human thought? Why is neuroethics interested in such a possibility? Embo Reports, 8:S57–S60. 89

7. References Illes, J., Blakemore, C., Hansson, M., Hensch, T., Leshner, A., Maestre, G., Magistretti, P., Quirion, R., and Strata, P. (2005). International perspectives on engaging the public in neuroethics. Nature Reviews Neuroscience, 6(12):977–982. Iversen, J., Repp, B., and Patel, A. (2009). Top-down control of rhythm perception modulates early auditory responses. In Annals of the New York academy of sciences, volume 1169, pages 58–73. Jensen, O., Bahramisharif, A., Oostenveld, R., Klanke, S., Hadjipapas, A., Okazaki, Y., and Van Gerven, M. (2011).

Using brain-computer interfaces and brain-

state dependent stimulation as a tool in cognitive neuroscience.

Front Psychology,

2(100):doi:10.3389/fpsyg.2011.00100. Jongsma, M., Eichele, T., Quiroga, R. Q., Jenks, K., Desain, P., Honing, H., and van Rijn, C. (2005). Expectancy effects on omission evoked potentials in musicians and nonmusicians. Psychophysiology, 42(2):191–201. Juslin, P. and Sloboda, J. (2010). Handbook of music and emotion: theory, research, applications. Oxford University Press. Kanoh, S., Miyamoto, K., and Yoshinobu, T. (2010). A brain-computer interface (BCI) system based on auditory stream segregation. Journal of Biomechanical Science and Engineering, 5(1):32–40. Kasai, K., Asada, T., Yumoto, M., Takeya, J., and Matsuda, H. (1999). Evidence for functional abnormality in the right auditory cortex during musical hallucinations. Lancet, 354(9191):1703–1704. Kelso, J., editor (1982). Human motor behavior. Lawrence Erlbaum Associates. Kim, S., Simeral, J., Hochberg, L., Donoghue, J., and Black, M. (2008). Neural control of computer cursor velocity by decoding motor cortical spiking activity in humans with tetraplegia. Journal of Neural Engineering,, 5(4):455–476. Klaming, L. and Haselager, P. (2010). Did my brain implant make me do it? Questions raised by DBS regarding psychological continuity, responsibility for action and mental competence. Neuroethics, pages DOI 10.1007/s12152–010–9093–1. Klobassa, D., Vaughan, T., Brunner, P., Schwartz, N., Wolpaw, J., Neuper, C., and Sellers, E. (2009). Toward a high-throughput auditory p300-based brain–computer interface. Clinical Neurophysiology, 120(7):1252–1261. 90

7. References Kosslyn, S., Ganis, G., and Thompson, L. (2001). Neural foundations of imagery. Nature Reviews Neuroscience, 2:635–642. Kosslyn, S., Thompson, W., and Alpert, I. K. N. (1995). Topographical representations of mental images in primary visual cortex. Nature, 378:496–498. Kraemer, D., Macrae, C., Green, A., and Kelley, W. (2005). Musical imagery: Sound of silence activates auditory cortex. Nature, 434(158). Krauledat, M., Dornhege, G., Blankertz, B., Losch, F., Curio, G., and Müller, K.-R. (2004). Improving speed and accuracy of brain–computer interfaces using readiness potential features. IEEE Engineering in Medicine and Biology Society, 6:4511–4515. Kronegg, J., Alecu, T., and Pun, T. (2003). Information theoretic bit-rate optimization for average trial protocol brain computer interfaces. HCI International. Kübler, A. and Birbaumer, N. (2008). Brain-computer interfaces and communication in paralysis: extinction of goal directed thinking in completely paralysed patients? Clinical Neurophysiology, 19(11):2658–2666. Kübler, A. and Muller, G. (2007). Toward brain–computer interfacing, chapter An introduction to Brain-Computer Interfacing, pages 1–25. Cambridge, MA, USA: The MIT Press. Kübler, A., Neumann, N., Kaiser, J., Kotchoubey, B., Hinterberger, T., and Birbaumer, N. (2001). Brain–computer communication: Self-regulation of slow cortical potentials for verbal communication. Archives of Physical Medicine and Rehabilitation, 82(11):1533–1539. Kübler, A., Nijboer, F., and Birbaumer, N. (2007). Toward brain–computer interfacing, chapter Brain–computer interfaces for communication and motor control - perspectives on clinical applications, pages 373–392. Cambridge, Massachusetts, USA: The MIT Press. Kübler, A., Nijboer, F., Mellinger, J., Vaughan, T., Pawelzik, H., Schalk, G., McFarland, D., Birbaumer, N., and Wolpaw, J. (2005a). Patients with ALS can use sensorimotor rhythms to operate a brain–computer interface. Neurology, 64(10):1775–1777. Kübler, A., Winter, S., Ludolph, A., Hautzinger, M., and Birbaumer, N. (2005b). Severity of depressive symptoms and quality of life in patients with amyotrophic lateral sclerosis. Neurorehabilitation and Neural Repair, 19(3):182–193. Large, E., Fink, P., and Kelso, J. (2002). Tracking simple and complex sequences. Psychological Research, 66(1):3–17. 91

7. References Latimer, C., Keeling, J., Lin, B., Henderson, M., and Hale, L. (2010). The impact of bilateral therapy on upper limb function after chronic stroke: a systematic review. Disability and Rehabilitation, 32:1221–1231. Leonard, E. (2009). Right to experimental treatment: FDA new drug approval, constitutional rights, and the public’s health. Journal of Law, Medicine and Ethics, 37. Leuthardt, E., Schalk, G., Wolpaw, J., Ojemann, J., and Moran, D. (2004). Brain–computer interface using electrocorticographic signals in humans. Journal of Neural Engineering, 1:63–71. Levitin, D. (2006). This Is Your Brain On Music. Dutton Books. Liberman, R., Hilty, D., Drake, R., and Tsang, H. (2001). Requirements for multidisciplinary teamwork in psychiatric rehabilitation. Psychiatric services, 52(10):1331–1342. Liesegang, T. (2007). The meaning and need for informed consent in research. Indian Journal of Ophthalmology, 55(1):1–3. London, J. (2004). Hearing in time: psychological aspects of musical meter. Oxford University Press. Maris, E. (2004). Randomization test for ERP topographies and whole spatiotemporal data matrices. Psychophysiology, 41(1):142–151. Maris, E. and Oostenveld, R. (2007). Nonparametric testing of EEG- and MEG-data. Journal of Neuroscience Methods, 164:177–190. Mayhew, S., Dirckx, S., Niazy, R., Iannetti, G., and Wise, R. (2010). EEG signatures of auditory activity correlate with simultaneously recorded fMRI responses in humans. Neuroimage, 49(1):849–864. McCall, B. (2004).

Brain fingerprints under scrutiny.

Retrieved May 25, 2006 from

http://news.bbc.co.uk/2/hi/science/nature/3495433.stm. McFarland, D., Miner, L., Vaughan, T., and Wolpaw, J. (2000). Mu and beta rhythm topographies during motor imagery and actual movements. Brain Topography, 12(3):177–186. McFarland, D., Sarnacki, W., and Wolpaw, J. (2011). Electroencephalographic (EEG) control of three-dimensional movement. Journal of Neural Engineering, 7(3):doi: 10.1088/1741– 2560/7/3/036007. 92

7. References McIntosh, G., Brown, S., Rice, R., and Thaut, M. (1997). Rhythmic auditory-motor facilitation of gait patterns in patients with parkinson’s disease. Journal of Neurology, Neurosurgery and Psychiatry, 62:22–26. Meyer, M., Baumann, S., and Jancke, L. (2006). Electrical brain imaging reveals spationtemporal dynamics of timbre perception in humans. Neuroimage, 32:1510–1523. Michon, J. and Jackson, J. (1985). Time, mind and behavior. Springer, Berlin. Middendorf, M., McMillan, G., Calhoun, G., and Jones, K. (2000). Brain–computer interfaces based on the steady-state visual-evoked response. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 8:211–214. Miranda, E., Magee, W., Wilson, J., Eaton, J., and Palaniappan, R. (2011). Brain-computer music interfacing (bcmi): From basic research to the real world of special needs. Music and Medicine, doi: 10.1177/1943862111399290. Mulert, C., Jäger, L., Propp, S., Karch, S., Störmann, S., Pogarell, O., Möller, H., Juckel, G., and Hegerl, U. (2005). Sound level dependence of the primary auditory cortex: Simultaneous measurement with 61-channel EEG and fMRI. Neuroimage, 28(1):49–58. Müller-Putz, G., Scherer, R., Brauneis, C., and Pfurtscheller, G. (2005). Steady-state visual evoked potential (SSVEP)-based communication: impact of harmonic frequency components. Journal of Neural Engineering, 2(4):123–130. Münssinger, J. I., Halder, S., Kleih, S., Furdea, A., Raco, V., Hösle, A., and Kübler, A. (2010). Brain painting: evaluation of a new brain-computer interface application with ALS patients and healthy volunteers. Frontiers in Neuroscience, 4(182). Munzert, J., Lorey, B., and Zentgraf, K. (2009). Cognitive motor processes: The role of motor imagery in the study of motor representations. Brain Research Reviews, 60:306–326. Naatanen, R. (1982). Processing negativity: An evoked-potential reflection of selective attention. Psychological Bulletin, 92(3):605–640. Nadelhoffer, T. (2010).

The law and neuroscience blog.

Retrieved July 24, 2010

from http://lawneuro.typepad.com/the-law-and-neuroscience-blog/2010/06/call- for.html. Navarro Cebrian, A. and Janata, P. (2010). Electrophysiological correlates of accurate mental image formation in auditory perception and imagery tasks. Brain Research, 1342:39–54. 93

7. References Neumann, N. and Kübler, A. (2003). Training locked-in patients: a challenge for the use of brain-computer interface use. IEEE Transactions on neural systems and rehabilitation engineering, 11(2):169–172. Neuper, C., Müller, G., Kübler, A., Birbaumer, N., Dornhege, R., del R. Millán, J., Hinterberger, T., McFarland, D., Müller, K.-R., and Pfurtscheller, G. (2003). Clinical application of an EEG-based brain–computer interface: A case study in a patient with severe motor impairment. Clinical Neurophysiology, 114(3):399–409. NeuroSky (2013). Neurosky - brainwave sensors for everybody. Retrieved June 12, 2013 from http://www.neurosky.com. Nicolas-Alonso, L. and Gomez-Gil, J. (2012). Brain computer interfaces, a review. Sensors, 12:1211–1279. Nijboer, F., Birbaumer, N., and Kübler, A. (2010). The influence of psychological state and motivation on brain-computer interface performance in patients with amyotrophic lateral sclerosis - a longitudinal study. Frontiers in Neuroprosthetics, 4(55). Nijboer, F., Clausen, J., Allison, B., and Haselager, P. (2011). The Asilomar survey: Stakeholders’ opinions on ethical issues related to brain-computer interfacing. Neuroethics, DOI 10.1007/s12152-011-9132-6. Nijboer, F., Furdea, A., Gunst, I., Mellinger, J., McFarland, D., Birbaumer, N., and Kübler, A. (2008a). An auditory brain-computer interface (BCI). Journal of Neuroscience Methods, 167:43–50. Nijboer, F., Kleih, S., and Kübler, A. (2009).

Das technisierte Gehirn, chapter Gehirn-

Computer Schnittstellen fuer schwerstgelaehmte Menschen - klinische Moeglichkeiten, technische Grenzen und etische Fragen, pages 51–64. Paderborn: Mentis Verlag GmbH. Nijboer, F., Sellers, E., Mellinger, J., Jordan, M., Halder, S., Matuz, T., Furdea, A., Mochty, U., Krusienski, D., Vaughan, T., Wolpaw, J., Birbaumer, N., and Kübler, A. (2008b). A P300based brain-computer interface for people with amyotrophic lateral sclerosis. Clinical Neurophysiology, 119(8):1909–1916. Nijholt, A., Bos, D. P.-O., and Reuderink, B. (2009). Turning shortcomings into challenges: Brain–computer interfaces for games. Entertainment Computing, 1(2):85–94. OpenVibe (2013). Software for brain computer interfaces and real time neurosciences. Retrieved September 23, 2013 from http://openvibe.inria.fr. 94

7. References O’Toole, A. J., Jiang, F., Abdi, H., Pénard, N., Dunlop, J. P., and Parent, M. A. (2007). Theoretical, statistical, and practical perspectives on pattern-based classification approaches to the analysis of functional neuroimaging data. Journal of Cognitive Neuroscience, 19(11):1735–1752. Overy, K. (2003). Dyslexia and music. from timing deficits to musical intervention. Annals of the New York academy of sciences, 999:497–505. Patel, A. (2010). Music, Language, and the Brain. Oxford University Press. Pearce, M., Ruiz, M. H., Kapasi, S., Wiggins, G., and Bhattacharya, J. (2010). Unsupervised statistical learning underpins computational, behavioural, and neural manifestations of musical expectation. Neuroimage, 50:302–313. Peretz, I. and Zatorre, R., editors (2003). The Cognitive Neuroscience of Music. Oxford University Press. Perrin, F., Pernier, J., Bertrand, O., and Echallier, J. (1989). Spherical splines for scalp potential and current mapping. Electroencephalography and Clinical Neurophysiology, 72:184–187. Pfurtscheller, G., Brunner, C., Schlögl, A., and da Silva, F. L. (2006).

Mu rhythm

(de)synchronization and EEG single-trial classification of different motor imagery tasks. NeuroImage, 31:153–159. Pfurtscheller, G. and da Silva, F. L. (1999). Event-related EEG/MEG synchronization and desynchronization: basic principles. Clinical Neurophysiology, 110:1842–1857. Pfurtscheller, G. and Neuper, C. (2006). Future prospects of ERD/ERS in the context of brain-computer interface (BCI) developments. Progress in Brain Research, 159:433–437. Pfurtscheller, G., Neuper, C., Flotzinger, D., and Pregenzer, M. (1997). EEG-based discrimination between imagination of right and left hand movement. Electroencephalography and Clinical Neurophysiology, 103:642–651. Pham, M., Hinterberger, T., Neumann, N., Kübler, A., Hofmayer, N., Grether, A., Wilhelm, B., Vatine, J., and Birbaumer, N. (2005). An auditory brain-computer interface based on the self-regulation of slow cortical potentials. Neurorehabilitation and Neural Repair, 19(3):206–218. Pinker, S. (1997). How the mind works. Penguin books. Polich, J. (2007). Updating P300: An integrative theory of p3a and p3b. Clinical Neurophysiology, 118(10):2128–2148. 95

7. References Pope, K., Fitzgibbon, S., Lewis, T., Whitham, E., and Willoughby, J. (2009). Relation of gamma oscillations in scalp recordings to muscular activity. Brain Topography, 22:13–17. Prasad, G., Herman, P., Coyle, D., McDonough, S., and Crosbie, J. (2009). Using a motor imagery based brain-computer interface for post-stroke rehabilitation. In Proceedings of 4th international IEEE/EMB conference on Neural Engineering, pages 258–262. Rain, S., Williams, V., Robbins, P., Monahan, J., Steadman, H., and Vesselinov, R. (2003). Perceived coercion at hospital admission and adherence to mental health treatment after discharge. Psychiatric Services, 54(1):103–105. Regan, D. (1977). Steady-state evoked potentials. Journal of the Optical Society of America, 67:1475–1489. Richman, D. (1989). Public access to experimental drug therapy: AIDS raises yet another conflict between freedom of the individual and welfare of the individual and public. Journal of infectious diseases, 159. Rodriguez, M. G., Grosse-Wentrup, M., Hill, J., Gharabaghi, A., Schölkopf, B., and Peters, J. (2011). Towards brain-robot interfaces in stroke rehabilitation. In Proceedings of 12th International Conference on Rehabilitation Robotics. Roerdink, M., Lamoth, C., Kwakkel, G., van Wieringen, P., and Beek, P. (2007). Gait coordination after stroke: Benefits of acoustically paced treadmill walking. Physical Therapy, 87:1009–1022. Roijendijk, L. (2009). Variability and nonstationarity in brain computer interfaces. Master’s thesis, Artificial Intelligence Radboud University Nijmegen. Rose, S. (2003). How to (or not to) communicate science. Biochemical Society Transactions, 31(2):307–312. Roskies, A. (2007). Neuroethics beyond genethics: Despite the overlap between the ethics of neuroscience and genetics, there are important areas where the two diverge. Embo Reports, 8:S52–S56. Rozelle, G. and Budzynski, T. (1995). Neurotherapy for stroke rehabilitation: A single case study. Biofeedback and Self-Regulation, 20(3). Sacks, O. (1985). The man who mistook his wife for a hat, chapter 3. Touchstone Books. Sacks, O. (2008). Musicophilia: Tales of Music and the Brain. Knopf Doubleday Publishing Group. 96

7. References Schaefer, R. (2011). Measuring the mind’s ear: EEG of music imagery. PhD thesis, Radboud University Nijmegen. Schaefer, R., Desain, P., and Farquhar, J. (2013). Shared processing of perception and imagery of music in decomposed EEG. NeuroImage, 70:317–326. Schaefer, R., Farquhar, J., Blokland, Y., Sadakata, M., and Desain, P. (2011a). Name that tune: decoding music from the listening brain. NeuroImage, 56(2):843–849. Schaefer, R., Vlek, R., and Desain, P. (2010). Decomposing rhythm processing: electroencephalography of perceived and self-imposed rhythmic patterns. Psychological Research, 75(2):95–106. Schaefer, R., Vlek, R., and Desain, P. (2011b). Music perception and imagery in eeg: alpha band effects of task and stimulus. Journal of Psychophysiology, 82(3):254–259. Schalk, G. (2011). Brain-computer interfaces: The hope, the hype, the power and the pain. Lecture at BCI2011 conference, Utrecht, Netherlands. Schalk, G., Miller, K., Anderson, N., Wilson, J., Smyth, M., Ojemann, J., Moran, D., Wolpaw, J., and Leuthardt, E. (2008). Two-dimensional movement control using electrocorticographic signals in humans. Journal of Neural Engineering, 5(1):75–84. Schalk, G., Wolpaw, J., McFarland, D., and Pfurtscheller, G. (2000). EEG-based communication: presence of an error potential. Clinical Neurophysiology, 111:2138–2144. Schauer, M. and Mauritz, K. (2003). Musical motor feedback (MMF) in walking hemiparetic stroke patients: randomized trials of gait improvement. Clinical Rehabilitation, 17:713– 722. Schlögl, A., Keinrath, C., Zimmermann, D., Scherer, R., Leeb, R., and Pfurtscheller, G. (2007). A fully automated correction method of EOG artifacts in EEG recordings. Clinical Neurophysiology, 118:98–104. Schreuder, M., Blankertz, B., and Tangermann, M. (2010). A new auditory multi-class braincomputer interface paradigm: Spatial hearing as an informative cue. PLos ONE, 5(4). ScienceDaily Retrieved

(2008).

Towards

September

30,

zero

2008

training

from

for

brain-computer

interfacing.

http://www.sciencedaily.com/releases/

2008/08/080812213820.htm. Sellers, E. and Donchin, E. (2006). A P300-based brain–computer interface: Initial tests by als patients. Clinical Neurophysiology, 117(3):538–548. 97

7. References Serruya, M., Hatsopoulos, N., Paninski, L., Fellows, M., and Donoghue, J. (2002). Brainmachine interface: Instant neural control of a movement signal. Nature, 416:141–142. Severens, M. (2013). Towards clinical BCI applications. PhD thesis, Radboud University Nijmegen. Severens, M., Farquhar, J., Duysens, J., and Desain, P. (2013). A multi-signature braincomputer interface: use of transient and steady-state responses. Journal of Neural Engineering, 10(2):026005. Shanawani, H., Wenrich, M., Tonelli, M., and Curtis, J. (2008). Meeting physicians responsibilities in providing end-of-life care. CHEST, 133(3):775–786. Shenoy, P., Krauledat, M., Blankertz, B., Rao, R., and Müller, K.-R. (2006). Towards adaptive classification for BCI. Journal of Neural Engineering, 3:R13–R23. Shinosaki, K., Yamamoto, M., Ukai, S., Kawaguchi, S., Ogawa, A., Ishii, R., MizunoMatsumoto, Y., Inouye, T., Hirabuki, N., Yoshimine, T., Kaku, T., Robinson, S., and Takeda, M. (2003). Desynchronization in the right auditory cortex during musical hallucinations: A MEG study. Psychogeriatrics, 3(2):88–92. Slyter, H. (1998). Ethical challenges in stroke research. Stroke, 29:1725–1729. Snyder, J. and Large, E. (2005).

Gamma-band activity reflects the metric structure of

rhythmic tone sequences. Brain research, Cognitive brain research, 24:117–126. Tamburrini, G. (2009). Brain to computer communication: Ethical perspectives on interaction models. Neuroethics, 2:137–149. Tamburrini, G. and Mattia, D. (2011). Disorders of consciousness and communication : Ethical motivations and communication-enabling attributes of consciousness. Functional Neurology, 26(1):51–54. Tchudin, V. (2001). A companion to bioethics, chapter Special issues facing nurses, pages 463–471. Oxford: Blackwell Publishing. Terry, P. (2007). Informed consent in clinical medicine. Chest, 131(2):563–568. Thaut, M. (2010). Neurologic music therapy in cognitive rehabilitation. Music Perception, 27:281–285. Tobi (2013). Tobi: Tools for brain-computer interaction. Retrieved September 23, 2013 from http://www.tobi-project.org. 98

7. References Todd, N. (1985). A model of expressive timing in tonal music. Music Perception, 3(1):33–58. Treder, M., Schmidt, A. B. N., Van Gerven, M., and Blankertz, B. (2011). Brain-computer interfacing using modulations of alpha activity induced by covert shifts of attention. Journal of NeuroEngineering and Rehabilitation, 8(24). Van der Waal, M., Severens, M., Geuze, J., and Desain, P. (2012). Introducing the tactile speller: An ERP-based brain-computer interface for communication. Journal of Neural Engineering, 9(4). Van Gerven, M., Farquhar, J., Schaefer, R., Vlek, R., Geuze, J., Nijholt, A., Ramsey, N., Haselager, P., Vuurpijl, L., Gielen, S., and Desain, P. (2009). The brain-computer interface cycle. Journal of Neural Engineering, 6(4). Van Gerven, M. and Jensen, O. (2009). Attention modulations of posterior alpha as a control signal for two-dimensional brain–computer interfaces. Journal of Neuroscience Methods, 179:78–84. Van Noorden, L. and Moelants, D. (1999). Resonance in the perception of musical pulse. Journal of New Music Research, 28(1):43–66. Vanacker, G., del R. Millán, J., Lew, E., Ferrez, P., Moles, F. G., Philips, J., Brussel, H. V., and Nuttin, M. (2007). Context-based filtering for assisted brain-actuated wheelchair driving context-based filtering for assisted brain-actuated wheelchair driving contextbased filtering for assisted brain-actuated wheelchair driving. Computational Intelligence and Neuroscience. Vatikiotis-Bateson, E. and Kelso, J. (1993). Rhythm type and articulatory dynamics in English, French and Japanese. Journal of Phonetics, 21:231–265. Vaughan, T., McFarland, D., Schalk, G., Sarnacki, W., Krusienski, D., Sellers, E., and Wolpaw, J. (2006). The wadsworth BCI research and development program: at home with BCI. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 14(2):229–233. Vlek, R., Schaefer, R., Gielen, C., Farquhar, J., and Desain, P. (2011a). Sequenced subjective accents for brain-computer interfaces. Journal of Neural Engineering, 8(3):036002. Vlek, R., Schaefer, R., Gielen, C., Farquhar, J., and Desain, P. (2011b). Shared mechanisms in perception and imagery of auditory accents. Clinical Neurophysiology, 122(8):1526–1532. Vlek, R., Steines, D., Szibbo, D., Kübler, A., Schneider, M.-J., Haselager, W., and Nijboer, F. (2012). Ethical issues in bci research, development, and dissemination - four case scenarios. Journal of Neurologic Physical Therapy, 36(2):94–99. 99

7. References Walter, S. (2010). Locked-in syndrome, BCI, and confusion about embodied, embedded, extended and enacted cognition. Neuroethics, 3:61–72. Weiskopf, N. (in press). Real-time fMRI and its application to neurofeedback. NeuroImage, page doi:10.1016/j.neuroimage.2011.10.009. Wessberg, J., Stambaugh, C., Kralik, J., Beck, P., Laubach, M., Chapin, J., Kim, J., Biggs, S., Srinivasan, M., and Nicolelis, M. (2000). Real-time prediction of hand trajectory by ensembles of cortical neurons in primates. Nature, 408:361–365. Willems, A., Nieuwboer, A., Chavret, F., Desloovere, K., Dom, R., Rochester, L., Kwakkel, G., Wegen, E. V., and Jones, D. (2007). Turning in parkinson’s disease patients and controls: The effect of auditory cues. Movement Disorders, 22:1871–1878. Wolpaw, J., Birbaumer, N., McFarland, D., Pfurtscheller, G., and Vaughan, T. (2002). Braincomputer interfaces for communication and control. Clinical Neurophysiology, 113:767– 791. Wolpe, P. (2007). Virtual mentor. American Medical Association Journal of Ethics, 9(2):128–131. World Medical Association (1964/2008). Declaration of Helsinki: Ethical principles for medical research involving human subjects (6th revision). Retrieved June 24, 2009 from http://www.wma.net/en/30publications/10policies/b3/17c.pdf. Yuval-Greenberg, S., Tomer, O., Keren, A., Nelken, I., and Deouell, L. (2008). Transient induced gamma-band response in EEG as a manifestation of miniature saccades. Neuron, 58:429–441. Zander, T., Lehne, M., Ihme, K., Jatzev, S., Correia, J., Kothe, C., Picht, B., and Nijboer, F. (2011). A dry EEG-system for scientific research and brain–computer interfaces. Frontiers in Neuroscience, 5(53):doi: 10.3389/fnins.2011.00053. Zanto, T., Snyder, J., and Large, E. (2006). Neural correlates of rhythmic expectancy. Advances in Cognitive Psychology, 2(2-3):221–231. Zatorre, R., Halpern, A., Perry, D., Meyer, E., and Evans, A. (1996). Hearing in the minds ear: a PET investigation of musical imagery and perception. Journal of Cognitive Neuroscience, 8:29–46. Zickler, C., Riccio, A., Leotta, F., Hillian-Tress, S., Halder, S., Holz, E., Staiger-Sälzer, P., Hoogerwerf, E.-J., Desideri, L., Mattia, D., and Kübler, A. (2011). A brain-computer interface as input channel for a standard assistive technology software. Clinical EEG and Neuroscience, 42(4):222–230. 100

7. References Zotev, V., Krueger, F., Phillips, R., Alvarez, R., Simmons, W., Bellgowan, P., Drevets, W., and Bodurka, J. (2011). Self-regulation of amygdala activation using real-time fMRI neurofeedback. PLos ONE, 6(9):e24522.

101

102

Brief summary Most people are familiar with the experience of ‘having a song in one’s head’, they ’hear’ the song (or parts of it) internally and are fully aware that the song is not audible in the outside world. This form of musical activity is known as auditory imagery. For this thesis, research was done on ‘subjective accenting’. This is a slightly different form of auditory imagery, where there are sounds in the outside world presented to a person, but the person subjectively changes the way the sounds are perceived by imagining emphasis (accents) on some of them. The aim of the research in this thesis was to gain fundamental insights in ‘subjective accenting’, as well as to investigate the potential for practical application of this knowledge in a brain-computer interface (BCI). BCI is a technique that allows devices to be controlled by signals from the brain. For this to work, one has to select a mental task that, when performed by a user, generates such a unique and strong activation of the brain that it can be detected reliably and automatically (by a computer program) in measurements of brain activity over a short period of time. Once a specific mental task can be detected, it can be coupled to execution of commands on a computer, or actions of a device, thus forming a brain-computer or brain-machine interface. BCI potentially serves a multitude of applications, varying from entertainment (control of games) to assistive technology (e.g. hands-free control of a tv or telephone). This thesis can be divided into two main sections, both with a different angle to brain-computer interfacing. The first section focuses on investigation of a novel musical paradigm for BCI, using the mental task of subjective accenting. While users are listening to a series of identical and regular pulse sounds, they are asked to imagine a stronger emphasis (or accent) on some of these pulses, such that they subjectively turn the series of pulses into a 2-, 3-, or 4-beat meter. While subjects were performing this mental task, also known as subjective accenting, their brain activity was measured with electro-encephalography (EEG). Chapter 1 provides an introduction to music cognition, brain-computer interfacing and ethics of BCI, complemented by a brief overview of previous scientific work in these areas, as well as motivation and background for the remaining chapters. Chapter 2 describes a study on the detectability of subjective accenting in EEG, and the neurophysiological relationship between EEG 103

8. Brief Summary observations during perception and imagery of accents. Results show that single-trial classification of subjectively accented and non-accented beats was successful with an average accuracy of 61%, meaning that subjective accenting (or imagined accents) can be detected from the EEG. Aided by principal component analysis and single-trial cross-condition classification, shared information in the EEG was revealed between perceived and imagined accents. This finding supports the idea that the brain uses shared mechanisms for auditory perception and imagery. Chapter 3 describes a study investigating whether subjective accenting is a feasible paradigm for BCI and how its time-structured nature can be exploited for optimal decoding from EEG data. Several sequence classification approaches are presented and evaluated on their ability to decode the cyclic sequences of subjectively accented and non-accented beats that occur in a 2-, 3-, and 4-beat meter. Classification performances were compared by means of bit rate. The best scenario yielded an average bit rate of 4.4 bits/min over ten subjects. This means that in principle a BCI driven by the mental task of subjective accenting is possible. The second section of the thesis approaches brain-computer interfacing from an ethical perspective, considering present and future use of BCI, how BCIs could impact society, and what ethical issues may arise. Issues typical to the field of BCI relate to working with sensitive user groups, dealing with technological complexity and handling multidisciplinary teams. Ethical issues arise where there is a conflict of treatment and research interests. Managing the personal and public expectations of BCI is also important. Chapter 4 describes several case scenarios of BCI use, inspired by current experiences in BCI laboratories. With the help of the case scenarios, and a discussion group of BCI experts, ethical issues were identified and disentangled, thus making the debate of ethical issues in BCI accessible to a wider audience. Chapter 5 further elaborates on the ethical issues in BCI, and discusses topics like ‘locked-in syndrome’, ‘informed consent’, ‘shared moral responsibility in teams’, and ‘media hypes’ against background knowledge from philosophy and medical sciences. In the last chapter all results are discussed and recommendations are outlined for future directions of brain-computer interfacing research. The main conclusion of this thesis is that subjective accenting is detectable for individual beats in EEG measurement of brain-activity. 2-, 3- and 4-beat sequences of subjectively accented and non-accented beats can be automatically decoded in a way that makes brain-computer interfacing possible, although further research is required to achieve real-life applications with this novel paradigm. Furthermore, evidence was found for the presence of shared mechanisms in the brain for auditory perception and imagery. The unique nature of BCI brings forward a range of ethical issues. With the discussion of these ethical issues, I hope to have provided useful insights and recommendations for current BCI researchers, developers and clinicians, as well as having

104

8. Brief Summary made the ethical debate surrounding BCI more accessible to a wider audience.

105

106

Korte samenvatting De meeste mensen weten hoe het is om een ‘liedje in het hoofd’ te hebben, ze ‘horen’ dan het liedje (of een deel ervan) intern en zijn zich er volledig van bewust dat het in de buitenwereld niet te horen is. Deze vorm van muzikale activiteit wordt auditieve voorstelling of inbeelding genoemd. Voor dit proefschrift is er onderzoek gedaan naar ‘subjectief accentueren’. Dit is een variant van auditieve inbeelding, waarbij er wel geluiden vanuit de buitenwereld aan iemand worden aangeboden, maar waarbij diegene de subjectieve beleving van deze geluiden beïnvloedt door zich op sommige van de geluiden een klemtoon (accent) voor te stellen. Het doel van het onderzoek in dit proefschrift was om fundamentele inzichten te krijgen in ‘subjectief accentueren’, en om te onderzoeken hoe deze kennis praktisch toegepast kan worden in een brain-computer interface (BCI). BCI is een techniek die het mogelijk maakt om apparaten te bedienen met signalen uit de hersenen. Hiervoor wordt een mentale taak geselecteerd die, wanneer deze wordt uitgevoerd door een gebruiker, zo’n sterke en unieke activiteit in de hersenen teweeg brengt dat deze activiteit betrouwbaar en automatisch (door een computer) herkend kan worden in korte opnames van hersenactiviteit. Wanneer een specifieke mentale taak op deze manier herkend is, kan hiervan de koppeling gelegd worden met het uitvoeren van taken op een computer of de bediening van een apparaat. Op deze manier komt de brain-computer of brain-machine interface tot stand. Met BCI is een grote diversiteit aan toepassingen mogelijk, van entertainment (besturing van games) tot ondersteunende technologie (bijv. hands-free bediening van een tv of telefoon). Dit proefschrift bestaat uit twee gedeeltes van waaruit met verschillende invalshoeken naar brain-computer interfacing wordt gekeken. In het eerste deel wordt een nieuw muzikaal paradigma voor BCI onderzocht, waarin subjectief accentueren als mentale taak gebruikt wordt. Deelnemers luisteren hiervoor naar een serie identieke en regelmatige puls geluiden, terwijl hen wordt gevraagd zich een sterkere klemtoon (accent) voor te stellen op sommige van deze pulsen, op zo’n manier dat de serie pulsen subjectief verandert in een 2-, 3-, of 4-kwartsmaat. Terwijl deelnemers deze mentale taak, namelijk subjectief accentueren, uitvoerden, werd hun hersenactiviteit geregistreerd middels electroencephalografie (EEG). Hoofdstuk 1 biedt een introductie tot mu107

9. Korte Samenvatting ziek cognitie, brain-computer interfacing en de ethiek van BCI, aangevuld met een kort overzicht van eerder wetenschappelijk werk op deze gebieden, alsmede de motivatie en achtergrond voor de opvolgende hoofdstukken. Hoofdstuk 2 bevat een studie naar de detecteerbaarheid van subjectief accentueren in het EEG, en de neurofysiologische relatie tussen de EEG observaties tijdens perceptie en voorstelling van accenten. De resultaten hiervan laten succesvolle classificatie van accenten en niet-accenten zien op basis van een enkele waarneming, met een gemiddelde betrouwbaarheid van 61%. Dit betekent dat subjectief accentueren (of voorgestelde accenten) uit het EEG gedetecteerd kan worden. Met behulp van principal component analysis (eigenwaarde decompositie), en kruis-conditie classificatie op basis van een enkele waarneming, werd het mogelijk om gedeelde informatie in het EEG te identificeren tussen waargenomen en voorgestelde accenten. Deze bevinding ondersteunt het idee dat de hersenen gedeelde mechanismen gebruiken voor auditieve perceptie en voorstelling. Hoofstuk 3 bevat een studie waarin wordt onderzocht of subjectief accentueren een realiseerbaar paradigma voor BCI is, en hoe de inherente tijdsstructuur geëxploiteerd kan worden voor optimale decodering uit het EEG. Verschillende benaderingen worden gepresenteerd voor de classificatie van sequenties, en geëvalueerd naar hun vermogen om de cyclische sequenties van subjectieve accenten en niet-accenten, die inherent zijn aan 2-, 3-, en 4-kwartsmaten, te decoderen. Classificatie resultaten werden onderling vergeleken met behulp van bitsnelheid. Het beste scenario leidde tot een gemiddelde bitsnelheid van 4.4 bits/min over tien deelnemers. Dat betekent dat een BCI op basis van subjectief accentueren in principe mogelijk is. Het tweede deel van dit proefschrift benadert brain-computer interfacing vanuit een ethisch perspectief, waarbij de huidige stand van zake en toekomstig gebruik van BCI beschouwd worden, wat voor impact BCIs op de samenleving zouden kunnen hebben, en welke ethische bezwaren zich daarbij kunnen voordoen. Bezwaren die specifiek zijn voor het veld van BCI zijn gerelateerd aan het werken met kwetsbare groepen gebruikers, de omgang met technologische complexiteit en multidisciplinaire teams. Ethische bezwaren doen zich voor in geval van belangenverstrengeling tussen behandelings- en onderzoeksdoelen. Ook is het beheren van persoonlijke en publieke verwachtingen van belang. In hoofdstuk 4 wordt een aantal voorbeeld scenario’s geschetst, geënt op huidige ervaringen uit BCI laboratoria. Aan de hand van deze scenario’s, en met hulp van een discussiegroep van BCI experts, werden ethische kwesties ontrafeld en ontleed. Met deze aanpak wordt het debat over ethische kwesties in BCI toegankelijk voor een breder publiek. Hoofdstuk 5 gaat dieper in op ethische kwesties in BCI, en belicht onderwerpen als ‘locked-in syndroom’, ‘geïnformeerde toestemmingsverklaring’, ‘gedeelde morele verantwoordelijkheid in teams’, en ‘media hypes’ tegen achtergrondkennis uit de filosofie en medische wetenschappen.

108

9. Korte Samenvatting In het laatste hoofdstuk worden alle resultaten besproken en worden er aanbevelingen geformuleerd voor toekomstige richtingen van het brain-computer interface onderzoek. De belangrijkste conclusie van dit proefschrift is dat de hersenactiviteit bij subjectief accentueren in het EEG herkend kan worden op basis van een enkele waarneming. 2-, 3-, en 4-kwartsmaatsequenties van subjectieve accenten en niet-accenten kunnen automatisch gedecodeerd worden op zo’n manier dat brain-computer interfacing mogelijk is, hoewel verder onderzoek nodig is om toepassing van dit nieuwe paradigma in het dagelijks leven te realiseren. Daarnaast werd bewijs gevonden voor de aanwezigheid van gedeelde mechanismen in de hersenen voor auditieve perceptie en voorstelling. De unieke kenmerken van BCI brengen ook een aantal ethische kwesties met zich mee. Met het behandelen van deze ethische kwesties hoop ik te hebben voorzien in zinvolle inzichten en aanbevelingen voor huidige BCI onderzoekers, ontwikkelaars en clinici, en hoop ik dat het ethische debat rondom BCI voor een breder publiek toegankelijk is geworden.

109

110

Publication list Journal publications • Vlek, R.J., Van Acken, J.P., Beursken, E., Roijendijk, L. & Haselager, W.F.G. (2014). BCI and a user’s judgment of agency. In: Brain-Computer-Interfaces in their Ethical, Social and Cultural Contexts, Grübler, G. & Hildt, E. (eds), Springer, Dordrecht • Blokland, Y., Spyrou, L., Thijssen, D., Eijsvogels, T., Colier, W., Floor-Westerdijk, M., Vlek, R., Bruhn, J. & Farquhar, J. (2014). Combined EEG-fNIRS decoding of motor attempt and imagery for brain switch control: an offline study in patients with tetraplegia. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 22(2): 1-8 • Vlek, R.J., Steines, D., Szibbo, D., Kübler, A., Schneider, M.-J., Haselager, W.F.G. & Nijboer, F. (2012). Ethical issues in BCI research, development, and dissemination four case scenarios. Journal of Neurologic Physical Therapy, 36(2): 94-99 • Horschig, J.M., Vlek, R.J., & Desain, P. (2011). Brain-Computer Interaces with User Chosen Features - A Free BCI. Proceedings of the Master’s Programme Cognitive Neuroscience, 6(2): 5-26 • Schaefer, R.S., Vlek, R.J. & Desain, P. (2011). Music perception and imagery in EEG: alpha band effects of task and stimulus. International Journal for Psychophysiology 82(3): 254-259 • Vlek, R.J., Schaefer, R.S., Gielen, C.C.A.M., Farquhar, J.D. & Desain, P. (2011). Sequenced subjective accents for Brain-Computer Interfaces. Journal of Neural Engineering, 8(3): 036002 • Vlek, R.J., Schaefer, R.S., Gielen, C.C.A.M., Farquhar, J.D. & Desain, P. (2011). Shared mechanisms in perception and imagery of auditory accents. Clinical Neurophysiology 122 (8): 1526-1532 111

10. Publication List • Schaefer, R.S., Vlek, R.J. & Desain, P. (2011). Decomposing rhythm processing: electroencephalography of perceived and self-imposed rhythmic patterns. Psychological Research, 75(2), 95 - 106 • Haselager, W.F.G., Vlek, R.J., Hill, N.J. & Nijboer, F. (2009). A note on ethical aspects of BCI. Neural Networks, 22(9): 1352-1357 • Van Gerven, M., Farquhar, J., Schaefer, R.S., Vlek, R.J., Geuze, J., Nijholt, A., Ramsey, N., Haselager, P., Vuurpijl, L., Gielen, S. & Desain, P. (2009). The brain-computer interface cycle. Journal of Neural Engineering 6(4): 041001

Conference publications • Blokland Y., Vlek, R., Karaman B., Özin F., Thijssen D., Eijsvogels T., Colier W., Floor-Westerdijk M., Bruhn, J. & Farquhar, J. (2012). Detection of event-related desynchronization during attempted and imagined movements in tetraplegics for brain switch control. Proceedings of the 34th Annual International Conference of the IEEE EMBS, doi: 10.1109/EMBC.2012.6346835. Hilton Bayfront Hotel, San Diego (CA), USA. Poster presentation & paper • Schaefer, R.S., Vlek, R.J., & Desain, P. (2011). Music perception and imagery in EEG: posterior alpha modulation by task and stimulus. Neuromusic Conference, Edinburgh, UK Poster presentation • Horschig, J.M., Vlek, R.J., Farquhar, J. & Desain, P. (2010). Brain-computer interfaces with user-chosen features - a free BCI. FENS Abstract, vol.5, 201.6, 7th Forum of European Neuroscience, Amsterdam, Netherlands Poster presentation • Schaefer, R.S., Vlek, R.J. & Desain, P. (2010). Rhythm processing decomposed: EEG of perceived and self-imposed rhythmic patterns. Society for Neuroscience Meeting in San Diego, USA. Spoken presentation • Van den Broek, P., Vlek, R.J., & Desain, P. (2010). BrainStream: a MATLAB toolbox for definition and execution of Brain Computer Interfaces. 4th International BCI meeting 2010, Monterey, USA. Poster presentation • Vlek, R.J., Schaefer, R.S., Farquhar, J., & Desain, P. (2010). Subjective rhythmization as a paradigm for BCI. 4th International BCI meeting 2010, Monterey, USA. Poster presentation, won poster prize for most innovative BCI design 112

10. Publication List • Vlek, R.J., Schaefer, R.S., Gielen, C.C.A.M., Farquhar, J., & Desain, P. (2009). Subjective rhythmization as a paradigm for BCI. Berlin BCI Workshop, Berlin, Germany. Poster presentation • Schaefer, R.S., Vlek, R.J. & Desain, P. (2008). Subjective rhythmisation as a paradigm for an EEG-driven BCI. Cognitive Neuroscience Society Meeting in San Francisco, USA. Poster presentation • Farquhar, J., Blankespoor, J., Vlek, R. & Desain, P. (2008). Towards a Noise-Tagging Auditory BCI-Paradigm. Proceedings of the 4th International BCI Workshop and Training Course, Graz, Austria. Spoken presentation & paper

113

114

Thanks Many people have, in their own way, contributed to this thesis. First and foremost I would like to thank my promotor Peter Desain. Your enthusiasm and wealth of ideas (although overwhelming at times) was inspiring and stimulating, and has also indirectly shown me the value of realism and constructive dissent. Jason, not only have you taught me a great deal about signal processing and machine learning, I have also greatly enjoyed our meetings as they were fueled by true scientific curiosity and an authentic passion for elegant and creative solutions. Pim, from the beginning you have shown confidence in my skills, creating an atmosphere of freedom to think and write. I especially enjoyed the way in which we bridged the gap between our different backgrounds and turned it into a multidisciplinary advantage. Many thanks also to Pieter Medendorp, Ruud Meulenbroek, Dirk Heylen, Hein van Schie, and all members of the corona for reading the thesis and taking part in the defense. To my co-authors, Stan Gielen, Rebecca Schaefer, Femke Nijboer, Jeremy Hill, David Steines, Diana Szibbo, Andrea Kübler, and Mary-Jane Schneider, without exceptions you have all been great to work with and learn from! Starting to work at the Donders Centre for Cognition in the brand new CAI-BCI group felt like becoming part of a dream-team of super-qualified, young scientists, together treasuring a vast arsenal of skills, knowledge, experience and ideas. This was a stimulating experience and I feel sorry that all things must pass. Embedded in an environment with great technical support from the ERG and mechanical workshop, and the stimulating beverages served at the DE-café and enjoyed in the "binnentuin", I had a good time collaborating with an ever expanding team. Rebecca, I had an awesome time sharing an office with you and Qui-Gon Jinn, have learnt a great deal from your scientific, organizational and social skills, and loved the large doses of humor, music and cheerful EEG cap montage sessions. Alex, from across the hall you have been closely following my every move, and I have followed yours. You have an exceptionally creative gift for music, science and friendship. I hope to stay in touch, while you move around the globe. Marianne and Linsey, the numerous walks and cups of coffee or tea facilitated a wonderful personal friendship and much needed distraction from the scientific routines. I could not have done this without you! Jeroen (G.), 115

11. Thanks Yvonne, Max, Makiko, Philip, thank you for being there at exactly the right moment and making life in the CAI-BCI group more inspiring and enjoyable. Thank you for being such great colleagues! A special thank you to the interns that contributed to the research in this thesis, namely Bram, Ruud, Jörn, Roy, Evine and Jan, who joined me in a shared struggle with the tenacious reality of BCI research. Sven, your vegetarian curry, tube-amp discussions (and those about women), coconut compressor, mac-quarium, and the studio sessions with Rebecca have meant a lot to me. Please stay on my radar! Jeroen (K.), you understand better than anyone what life in academia means, and shared a lot of your experience with me, complemented by a healthy dose of concert and comedy visits. Boris, your fresh view on science, innovation and collaboration has been an inspiration, but I most admire your ability to make courageous decisions and find balance between so many aspects of life. Autumn (Jeroen, Jan, Marjan, Jens and Mats) & The Gathering (René, Hans, Marjolein, Hugo, Silje, Frank, Wouter, Sjoerd and Linda), in the middle of my PhD years, you have given me one of my most memorable holidays by touring Europe together. Pictures of us on a Swiss stage have been keeping me company during my last years at the university, keeping the spirit alive. I’m very grateful towards my family (Sara, Charles, Charlotte) for catching me when I was falling and keeping my feet on the ground. Finally, Mrrjolein, we met under difficult circumstances and you stood by my side with great love and patience in moments of utter vulnerability, showing me that, even in the hardest of times, life can be beautiful. I’m forever grateful that you have given me the chance to open up and rediscover where I want to be in life.

116

Curriculum Vitae Rutger Vlek was born on October 20th 1982 in Groningen, The Netherlands. After obtaining a propaedeutic diploma in Philosophy in 2001 he completed an MSc in Artificial Intelligence in 2007, both at the Rijksuniversiteit Groningen (RuG). Following up on the work done for his masters thesis, he continued to work in auditory research on tinnitus at the University Medical Centre Groningen (UMCG). In the fall of 2007 he started working with Peter Desain on brain-computer interfacing and music imagery as a part of the over-arching ’BrainGain’ programme, a Dutch consortium focusing on Brain-Computer Interfacing and neurostimulation. The result of this work is contained in this thesis. Next to his research activities he has played an important role in the design and maintenance of the EEG laboratory facilities, and contributed to the design and development of an opensource real-time software platform for brain-computer interfacing. On the 4th International BCI Meeting (2010, Monterey, USA) his research was awarded with a poster prize for most innovative brain-computer interface design. He has actively pursued musical activities his entire life, varying from restoration and design of mechanical and electronic musical instruments, to music arrangement, composition, production and engineering, and live performances throughout Europe. He is currently looking for new opportunities in the mixed field of music, science and technology.

117

118

Donders series 1. Van Aalderen-Smeets, S.I. (2007). Neural dynamics of visual selection. Maastricht University, Maastricht, the Netherlands. 2. Schoffelen, J.M. (2007). Neuronal communication through coherence in the human motor system. Radboud University Nijmegen, Nijmegen, the Netherlands. 3. De Lange, F.P. (2008). Neural mechanisms of motor imagery. Radboud University Nijmegen, Nijmegen, the Netherlands. 4. Grol, M.J. (2008). Parieto-frontal circuitry in visuomotor control. Utrecht University, Utrecht, the Netherlands. 5. Bauer, M. (2008). Functional roles of rhythmic neuronal activity in the human visual and somatosensory system. Radboud University Nijmegen, Nijmegen, the Netherlands. 6. Mazaheri, A. (2008). The influence of ongoing oscillatory brain activity on evoked responses and behaviour. Radboud University Nijmegen, Nijmegen, the Netherlands. 7. Hooijmans, C.R. (2008). Impact of nutritional lipids and vascular factors in Alzheimer’s disease. Radboud University Nijmegen, Nijmegen, the Netherlands. 8. Gaszner, B. (2008). Plastic responses to stress by the rodent urocortinergic Edinger-Westphal nucleus. Radboud University Nijmegen, Nijmegen, the Netherlands. 9. Willems, R.M. (2009). Neural reflections of meaning in gesture, language and action. Radboud University Nijmegen, Nijmegen, the Netherlands. 10. Van Pelt, S. (2009). Dynamic neural representations of human visuomotor space. Radboud University Nijmegen, Nijmegen, the Netherlands. 11. Lommertzen, J. (2009). Visuomotor coupling at different levels of complexity. Radboud University Nijmegen, Nijmegen, the Netherlands. 12. Poljac, E. (2009). Dynamics of cognitive control in task switching: Looking beyond the switch cost. Radboud University Nijmegen, Nijmegen, the Netherlands. 13. Poser, B.A. (2009). Techniques for BOLD and blood volume weighted fMRI. Radboud University Nijmegen, Nijmegen, the Netherlands. 14. Baggio, G. (2009). Semantics and the electrophysiology of meaning. Tense, aspect, event structure. Radboud University Nijmegen, Nijmegen, the Netherlands. 15. Van Wingen, G.A. (2009). Biological determinants of amygdala functioning. Radboud University Nijmegen Medical Centre, Nijmegen, the Netherlands.

119

13. Donders Series 16. Bakker, M. (2009). Supraspinal control of walking: Lessons from motor imagery. Radboud University Nijmegen Medical Centre, Nijmegen, the Netherlands. 17. Aarts, E. (2009). Resisting temptation: The role of the anterior cingulate cortex in adjusting cognitive control. Radboud University Nijmegen, Nijmegen, the Netherlands. 18. Prinz, S. (2009). Waterbath stunning of chickens - Effects of electrical parameters on the electroencephalogram and physical reflexes of broilers. Radboud University Nijmegen, Nijmegen, the Netherlands. 19. Knippenberg, J.M.J. (2009). The N150 of the Auditory Evoked Potential from the rat amygdala: In search for its functional significance. Radboud University Nijmegen, Nijmegen, the Netherlands. 20. Dumont, G.J.H. (2009). Cognitive and physiological effects of 3,4-methylenedioxymethamphetamine (MDMA or ‘ecstasy’) in combination with alcohol or cannabis in humans. Radboud University Nijmegen, Nijmegen, the Netherlands. 21. Pijnacker, J. (2010). Defeasible inference in autism: A behavioral and electrophysiogical approach. Radboud University Nijmegen, Nijmegen, the Netherlands. 22. De Vrijer, M. (2010). Multisensory integration in spatial orientation. Radboud University Nijmegen, Nijmegen, the Netherlands. 23. Vergeer, M. (2010). Perceptual visibility and appearance: Effects of color and form. Radboud University Nijmegen, Nijmegen, the Netherlands. 24. Levy, J. (2010). In cerebro unveiling unconscious mechanisms during reading. Radboud University Nijmegen, Nijmegen, the Netherlands. 25. Treder, M. S. (2010). Symmetry in (inter)action. Radboud University Nijmegen, Nijmegen, the Netherlands. 26. Horlings, C.G.C. (2010). A weak balance: Balance and falls in patients with neuromuscular disorders. Radboud University Nijmegen, Nijmegen, the Netherlands. 27. Snaphaan, L.J.A.E. (2010). Epidemiology of post-stroke behavioural consequences. Radboud University Nijmegen Medical Centre, Nijmegen, the Netherlands. 28. Dado - Van Beek, H.E.A. (2010). The regulation of cerebral perfusion in patients with Alzheimer’s disease. Radboud University Nijmegen Medical Centre, Nijmegen, the Netherlands. 29. Derks, N.M. (2010). The role of the non-preganglionic Edinger-Westphal nucleus in sex-dependent stress adaptation in rodents. Radboud University Nijmegen, Nijmegen, the Netherlands. 30. Wyczesany, M. (2010). Covariation of mood and brain activity. Integration of subjective self-report data with quantitative EEG measures. Radboud University Nijmegen, Nijmegen, the Netherlands. 31. Beurze, S.M. (2010). Cortical mechanisms for reach planning. Radboud University Nijmegen, Nijmegen, the Netherlands. 32. Van Dijk, J.P. (2010). On the Number of Motor Units. Radboud University Nijmegen, Nijmegen, the Netherlands. 33. Lapatki, B.G. (2010). The Facial Musculature - Characterization at a Motor Unit Level. Radboud University Nijmegen, Nijmegen, the Netherlands. 34. Kok, P. (2010). Word order and verb inflection in agrammatic sentence production. Radboud University Nijmegen, Nijmegen, the Netherlands. 35. van Elk, M. (2010). Action semantics: Functional and neural dynamics. Radboud University Nijme-

120

13. Donders Series gen, Nijmegen, the Netherlands. 36. Majdandzic, J. (2010). Cerebral mechanisms of processing action goals in self and others. Radboud University Nijmegen, Nijmegen, the Netherlands. 37. Snijders, T.M. (2010). More than words - Neural and genetic dynamics of syntactic unification. Radboud University Nijmegen, Nijmegen, the Netherlands. 38. Grootens, K.P. (2010). Cognitive dysfunction and effects of antipsychotics in schizophrenia and borderline personality disorder. Radboud University Nijmegen Medical Centre, Nijmegen, the Netherlands. 39. Nieuwenhuis, I.L.C. (2010). Memory consolidation: A process of integration - Converging evidence from MEG, fMRI and behavior. Radboud University Nijmegen Medical Centre, Nijmegen, the Netherlands. 40. Menenti, L.M.E. (2010). The right language: Differential hemispheric contributions to language production and comprehension in context. Radboud University Nijmegen, Nijmegen, the Netherlands. 41. Van Dijk, H.P. (2010). The state of the brain, how alpha oscillations shape behaviour and event related responses. Radboud University Nijmegen, Nijmegen, the Netherlands. 42. Meulenbroek, O.V. (2010). Neural correlates of episodic memory in healthy aging and Alzheimer’s disease. Radboud University Nijmegen, Nijmegen, the Netherlands. 43. Oude Nijhuis, L.B. (2010). Modulation of human balance reactions. Radboud University Nijmegen, Nijmegen, the Netherlands. 44. Qin, S. (2010). Adaptive memory: Imaging medial temporal and prefrontal memory systems. Radboud University Nijmegen, Nijmegen, the Netherlands. 45. Timmer, N.M. (2011). The interaction of heparan sulfate proteoglycans with the amyloid protein. Radboud University Nijmegen, Nijmegen, the Netherlands. 46. Crajé, C. (2011). (A)typical motor planning and motor imagery. Radboud University Nijmegen, Nijmegen, the Netherlands. 47. Van Grootel, T.J. (2011). On the role of eye and head position in spatial localisation behaviour. Radboud University Nijmegen, Nijmegen, the Netherlands. 48. Lamers, M.J.M. (2011). Levels of selective attention in action planning. Radboud University Nijmegen, Nijmegen, the Netherlands. 49. Van der Werf, J. (2011). Cortical oscillatory activity in human visuomotor integration. Radboud University Nijmegen, Nijmegen, the Netherlands. 50. Scheeringa, R. (2011). On the relation between oscillatory EEG activity and the BOLD signal. Radboud University Nijmegen, Nijmegen, the Netherlands. 51. Bögels, S. (2011). The role of prosody in language comprehension: When prosodic breaks and pitch accents come into play. Radboud University Nijmegen, Nijmegen, the Netherlands. 52. Ossewaarde, L. (2011). The mood cycle: Hormonal influences on the female brain. Radboud University Nijmegen, Nijmegen, the Netherlands. 53. Kuribara, M. (2011). Environment-induced activation and growth of pituitary melanotrope cells of Xenopus laevis. Radboud University Nijmegen, Nijmegen, the Netherlands. 54. Helmich, R.C.G. (2011). Cerebral reorganization in Parkinson’s disease. Radboud University Nijmegen, Nijmegen, the Netherlands.

121

13. Donders Series 55. Boelen, D. (2011). Order out of chaos? Assessment and treatment of executive disorders in brain-injured patients. Radboud University Nijmegen, Nijmegen, the Netherlands. 56. Koopmans, P.J. (2011). fMRI of cortical layers. Radboud University Nijmegen, Nijmegen, the Netherlands. 57. Van der Linden, M.H. (2011). Experience-based cortical plasticity in object category representation. Radboud University Nijmegen, Nijmegen, the Netherlands. 58. Kleine, B.U. (2011). Motor unit discharges - Physiological and diagnostic studies in ALS. Radboud University Nijmegen Medical Centre, Nijmegen, the Netherlands. 59. Paulus, M. (2011). Development of action perception: Neurocognitive mechanisms underlying children’s processing of others’ actions. Radboud University Nijmegen, Nijmegen, the Netherlands. 60. Tieleman, A.A. (2011). Myotonic dystrophy type 2. A newly diagnosed disease in the Netherlands. Radboud University Nijmegen Medical Centre, Nijmegen, the Netherlands. 61. Van Leeuwen, T.M. (2011). ’How one can see what is not there’: Neural mechanisms of grapheme-colour synaesthesia. Radboud University Nijmegen, Nijmegen, the Netherlands. 62. Van Tilborg, I.A.D.A. (2011). Procedural learning in cognitively impaired patients and its application in clinical practice. Radboud University Nijmegen, Nijmegen, the Netherlands. 63. Bruinsma, I.B. (2011). Amyloidogenic proteins in Alzheimer’s disease and Parkinson’s disease: Interaction with chaperones and inflammation. Radboud University Nijmegen, Nijmegen, the Netherlands. 64. Voermans, N. (2011). Neuromuscular features of Ehlers-Danlos syndrome and Marfan syndrome; expanding the phenotype of inherited connective tissue disorders and investigating the role of the extracellular matrix in muscle. Radboud University Nijmegen Medical Centre, Nijmegen, the Netherlands. 65. Reelick, M. (2011). One step at a time. Disentangling the complexity of preventing falls in frail older persons. Radboud University Nijmegen Medical Centre, Nijmegen, the Netherlands. 66. Buur, P.F. (2011). Imaging in motion. Applications of multi-echo fMRI. Radboud University Nijmegen, Nijmegen, the Netherlands. 67. Schaefer, R.S. (2011). Measuring the mind’s ear: EEG of music imagery. Radboud University Nijmegen, Nijmegen, the Netherlands. 68. Xu, L. (2011). The non-preganglionic Edinger-Westphal nucleus: An integration center for energy balance and stress adaptation. Radboud University Nijmegen, Nijmegen, the Netherlands. 69. Schellekens, A.F.A. (2011). Gene-environment interaction and intermediate phenotypes in alcohol dependence. Radboud University Nijmegen, Nijmegen, the Netherlands. 70. Van Marle, H.J.F. (2011). The amygdala on alert: A neuroimaging investigation into amygdala function during acute stress and its aftermath. Radboud University Nijmegen, Nijmegen, the Netherlands. 71. De Laat, K.F. (2011). Motor performance in individuals with cerebral small vessel disease: An MRI study. Radboud University Nijmegen Medical Centre, Nijmegen, the Netherlands. 72. Mädebach, A. (2011). Lexical access in speaking: Studies on lexical selection and cascading activation. Radboud University Nijmegen, Nijmegen, the Netherlands. 73. Poelmans, G.J.V. (2011). Genes and protein networks for neurodevelopmental disorders. Radboud University Nijmegen, Nijmegen, the Netherlands. 74. Van Norden, A.G.W. (2011). Cognitive function in elderly individuals with cerebral small vessel disease.

122

13. Donders Series An MRI study. Radboud University Nijmegen Medical Centre, Nijmegen, the Netherlands. 75. Jansen, E.J.R. (2011). New insights into V-ATPase functioning: the role of its accessory subunit Ac45 and a novel brain-specific Ac45 paralog. Radboud University Nijmegen, Nijmegen, the Netherlands. 76. Haaxma, C.A. (2011). New perspectives on preclinical and early stage Parkinson’s disease. Radboud University Nijmegen Medical Centre, Nijmegen, the Netherlands. 77. Haegens, S. (2012). On the functional role of oscillatory neuronal activity in the somatosensory system. Radboud University Nijmegen, Nijmegen, the Netherlands. 78. Van Barneveld, D.C.P.B.M. (2012). Integration of exteroceptive and interoceptive cues in spatial localization. Radboud University Nijmegen, Nijmegen, the Netherlands. 79. Spies, P.E. (2012). The reflection of Alzheimer disease in CSF. Radboud University Nijmegen Medical Centre, Nijmegen, the Netherlands. 80. Helle, M. (2012). Artery-specific perfusion measurements in the cerebral vasculature by magnetic resonance imaging. Radboud University Nijmegen, Nijmegen, the Netherlands. 81. Egetemeir, J. (2012). Neural correlates of real-life joint action. Radboud University Nijmegen, Nijmegen, the Netherlands. 82. Janssen, L. (2012). Planning and execution of (bi)manual grasping. Radboud University Nijmegen, Nijmegen, the Netherlands. 83. Vermeer, S. (2012). Clinical and genetic characterisation of autosomal recessive cerebellar ataxias. Radboud University Nijmegen Medical Centre, Nijmegen, the Netherlands. 84. Vrins, S. (2012). Shaping object boundaries: Contextual effects in infants and adults. Radboud University Nijmegen, Nijmegen, the Netherlands. 85. Weber, K.M. (2012). The language learning brain: Evidence from second language and bilingual studies of syntactic processing. Radboud University Nijmegen, Nijmegen, the Netherlands. 86. Verhagen, L. (2012). How to grasp a ripe tomato. Utrecht University, Utrecht, the Netherlands. 87. Nonkes, L.J.P. (2012). Serotonin transporter gene variance causes individual differences in rat behaviour: For better and for worse. Radboud University Nijmegen Medical Centre, Nijmegen, the Netherlands. 88. Joosten-Weyn Banningh, L.W.A. (2012). Learning to live with Mild Cognitive Impairment: development and evaluation of a psychological intervention for patients with Mild Cognitive Impairment and their significant others. Radboud University Nijmegen Medical Centre, Nijmegen, the Netherlands. 89. Xiang, HD. (2012). The language networks of the brain. Radboud University Nijmegen, Nijmegen, the Netherlands. 90. Snijders, A.H. (2012). Tackling freezing of gait in Parkinson’s disease. Radboud University Nijmegen Medical Centre, Nijmegen, the Netherlands. 91. Rouwette, T.P.H. (2012). Neuropathic pain and the brain - Differential involvement of corticotropinreleasing factor and urocortin 1 in acute and chronic pain processing. Radboud University Nijmegen Medical Centre, Nijmegen, the Netherlands. 92. Van de Meerendonk, N. (2012). States of indecision in the brain: Electrophysiological and hemodynamic reflections of monitoring in visual language perception. Radboud University Nijmegen, Nijmegen, the Netherlands.

123

13. Donders Series 93. Sterrenburg, A. (2012). The stress response of forebrain and midbrain regions: Neuropeptides, sexspecificity and epigenetics. Radboud University Nijmegen, Nijmegen, The Netherlands. 94. Uithol, S. (2012). Representing action and intention. Radboud University Nijmegen, Nijmegen, The Netherlands. 95. Van Dam, W.O. (2012). On the specificity and flexibility of embodied lexical-semantic representations. Radboud University Nijmegen, Nijmegen, The Netherlands. 96. Slats, D. (2012). CSF biomarkers of Alzheimer’s disease: Serial sampling analysis and the study of circadian rhythmicity. Radboud University Nijmegen Medical Centre, Nijmegen, the Netherlands. 97. Van Nuenen, B.F.L. (2012). Cerebral reorganization in premotor parkinsonism. Radboud University Nijmegen Medical Centre, Nijmegen, the Netherlands. 98. Van Schouwenburg, M.R. (2012). Fronto-striatal mechanisms of attentional control. Radboud University Nijmegen, Nijmegen, The Netherlands. 99. Azar, M.G. (2012). On the theory of reinforcement learning: Methods, convergence analysis and sample complexity. Radboud University Nijmegen, Nijmegen, The Netherlands. 100. Meeuwissen, E.B. (2012). Cortical oscillatory activity during memory formation. Radboud University Nijmegen, Nijmegen, The Netherlands. 101. Arnold, J.F. (2012). When mood meets memory: Neural and behavioral perspectives on emotional memory in health and depression. Radboud University Nijmegen, Nijmegen, The Netherlands. 102. Gons, R.A.R. (2012). Vascular risk factors in cerebral small vessel disease: A diffusion tensor imaging study. Radboud University Nijmegen Medical Centre, Nijmegen, the Netherlands. 103. Wingbermühle, E. (2012). Cognition and emotion in adults with Noonan syndrome: A neuropsychological perspective. Radboud University Nijmegen, Nijmegen, The Netherlands. 104. Walentowska, W. (2012). Facing emotional faces. The nature of automaticity of facial emotion processing studied with ERPs. Radboud University Nijmegen, Nijmegen, The Netherlands. 105. Hoogman, M. (2012). Imaging the effects of ADHD risk genes. Radboud University Nijmegen, Nijmegen, The Netherlands. 106. Tramper, J. J. (2012). Feedforward and feedback mechanisms in sensory motor control. Radboud University Nijmegen, Nijmegen, The Netherlands. 107. Van Eijndhoven, P. (2012). State and trait characteristics of early course major depressive disorder. Radboud University Nijmegen Medical Centre, Nijmegen, the Netherlands. 108. Visser, E. (2012). Leaves and forests: Low level sound processing and methods for the large-scale analysis of white matter structure in autism. Radboud University Nijmegen, Nijmegen, The Netherlands. 109. Van Tooren-Hoogenboom, N. (2012). Neuronal communication in the synchronized brain. Investigating the functional role of visually-induced gamma band activity: Lessons from MEG. Radboud University Nijmegen, Nijmegen, The Netherlands. 110. Henckens, M.J.A.G. (2012). Imaging the stressed brain. Elucidating the time- and region-specific effects of stress hormones on brain function: A translational approach. Radboud University Nijmegen, Nijmegen, The Netherlands. 111. Van Kesteren, M.T.R. (2012). Schemas in the brain: Influences of prior knowledge on learning, memory, and education. Radboud University Nijmegen, Nijmegen, The Netherlands.

124

13. Donders Series 112. Brenders, P. (2012). Cross-language interactions in beginning second language learners. Radboud University Nijmegen, Nijmegen, The Netherlands. 113. Ter Horst, A.C. (2012). Modulating motor imagery. Contextual, spatial and kinaesthetic influences. Radboud University Nijmegen, Nijmegen, The Netherlands. 114. Tesink, C.M.J.Y. (2013). Neurobiological insights into language comprehension in autism: Context matters. Radboud University Nijmegen, Nijmegen, The Netherlands. 115. Böckler, A. (2013). Looking at the world together. How others’ attentional relations to jointly attended scenes shape cognitive processing. Radboud University Nijmegen, Nijmegen, The Netherlands. 116. Van Dongen, E.V. (2013). Sleeping to Remember. On the neural and behavioral mechanisms of sleepdependent memory consolidation. Radboud University Nijmegen, Nijmegen, The Netherlands. 117. Volman, I. (2013). The neural and endocrine regulation of emotional actions. Radboud University Nijmegen, Nijmegen, The Netherlands. 118. Buchholz, V. (2013). Oscillatory activity in tactile remapping. Radboud University Nijmegen, Nijmegen, The Netherlands. 119. Van Deurzen, P.A.M. (2013). Information processing and depressive symptoms in healthy adolescents. Radboud University Nijmegen, Nijmegen, The Netherlands. 120. Whitmarsh, S. (2013). Nonreactivity and metacognition in mindfulness. Radboud University Nijmegen, Nijmegen, The Netherlands. 121. Vesper, C. (2013). Acting together: Mechanisms of intentional coordination.

Radboud University

Nijmegen, Nijmegen, The Netherlands. 122. Lagro, J. (2013). Cardiovascular and cerebrovascular physiological measurements in clinical practice and prognostics in geriatric patients. Radboud University Nijmegen Medical Centre, Nijmegen, the Netherlands. 123. Eskenazi, T.T. (2013). You, us & them: From motor simulation to ascribed shared intentionality in social perception. Radboud University Nijmegen, Nijmegen, The Netherlands. 124. Ondobaka, S. (2013). On the conceptual and perceptual processing of own and others’ behavior. Radboud University Nijmegen, Nijmegen, The Netherlands. 125. Overvelde, J.A.A.M. (2013). Which practice makes perfect? Experimental studies on the acquisition of movement sequences to identify the best learning condition in good and poor writers. Radboud University Nijmegen, Nijmegen, The Netherlands. 126. Kalisvaart, J.P. (2013). Visual ambiguity in perception and action. Radboud University Nijmegen Medical Centre, Nijmegen, The Netherlands. 127. Kroes, M. (2013). Altering memories for emotional experiences. Radboud University Nijmegen, Nijmegen, The Netherlands. 128. Duijnhouwer, J. (2013). Studies on the rotation problem in self-motion perception. Radboud University Nijmegen, Nijmegen, The Netherlands. 129. Nijhuis, E.H.J (2013). Macroscopic networks in the human brain: Mapping connectivity in healthy and damaged brains. University of?Twente, Enschede, The Netherlands 130. Braakman, M. H. (2013). Posttraumatic stress disorder with secondary psychotic features. A diagnostic validity study among refugees in the Netherlands. Radboud University Nijmegen, Nijmegen, The

125

13. Donders Series Netherlands. 131. Zedlitz, A.M.E.E. (2013). Brittle brain power. Post-stroke fatigue, explorations into assessment and treatment. Radboud University Nijmegen, Nijmegen, The Netherlands. 132. Schoon, Y. (2013). From a gait and falls clinic visit towards self-management of falls in frail elderly. Radboud University Nijmegen Medical Centre, Nijmegen, The Netherlands. 133. Jansen, D. (2013). The role of nutrition in Alzheimer’s disease - A study in transgenic mouse models for Alzheimer’s disease and vascular disorders. Radboud University Nijmegen, Nijmegen, The Netherlands. 134. Kos, M. (2013). On the waves of language - Electrophysiological reflections on semantic and syntactic processing. Radboud University Nijmegen, Nijmegen, The Netherlands. 135. Severens, M. (2013). Towards clinical BCI applications: Assistive technology and gait rehabilitation. Radboud University Nijmegen, Nijmegen, Sint Maartenskliniek, Nijmegen, The Netherlands. 136. Bergmann, H. (2014). Two is not always better than one: On the functional and neural (in)dependence of working memory and long-term memory. Radboud University Nijmegen, Nijmegen, The Netherlands. 137. Wronka, E. (2013). Searching for the biological basis of human mental abilitites. The relationship between attention and intelligence studied with P3. Radboud University Nijmegen, Nijmegen, The Netherlands. 138. Lüttjohann, A.K. (2013). The role of the cortico-thalamo-cortical system in absence epilepsy. Radboud University Nijmegen, Nijmegen, The Netherlands. 139. Brazil, I.A. (2013). Change doesn’t come easy: Dynamics of adaptive behavior in psychopathy. Radboud University Nijmegen, Nijmegen, The Netherlands. 140. Zerbi, V. (2013). Impact of nutrition on brain structure and function. A magnetic resonance imaging approach in Alzheimer mouse models. Radboud University Nijmegen, Nijmegen, The Netherlands. 141. Delnooz, C.C.S. (2014). Unravelling primary focal dystonia. A treatment update and new pathophysiological insights. Radboud University Nijmegen Medical Centre, Nijmegen, The Netherlands. 142. Bultena, S.S. (2013). Bilingual processing of cognates and language switches in sentence context. Radboud University Nijmegen, Nijmegen, The Netherlands. 143. Janssen, G. (2014). Diagnostic assessment of psychiatric patients: A contextual perspective on executive functioning. Radboud University Nijmegen, Nijmegen, The Netherlands. 144. Piai, V. Magalhäes (2014). Choosing our words: Lexical competition and the involvement of attention in spoken word production. Radboud University Nijmegen, Nijmegen, The Netherlands. 145. Van Ede, F. (2014). Preparing for perception. On the attentional modulation, perceptual relevance and physiology of oscillatory neural activity. Radboud University Nijmegen, Nijmegen, The Netherlands. 146. Brandmeyer, A. (2014). Auditory perceptual learning via decoded EEG neurofeedback: a novel paradigm. Radboud University Nijmegen, Nijmegen, The Netherlands. 147. Radke, S. (2014). Acting social: Neuroendocrine and clinical modulations of approach and decision behavior. Radboud University Nijmegen, Nijmegen, The Netherlands.

126

13. Donders Series 148. Simanova, I. (2014). In search of conceptual representations in the brain: towards mind-reading. Radboud University Nijmegen, Nijmegen, The Netherlands. 149. Kok, P. (2014). On the role of expectation in visual perception: A top-down view of early visual cortex. Radboud University Nijmegen, Nijmegen, The Netherlands. 150. Van Geldorp, B. (2014). The long and the short of memory: Neuropsychological studies on the interaction of working memory and long-term memory formation. Radboud University Nijmegen, Nijmegen, The Netherlands. 151. Meyer, M. (2014). The developing brain in action - Individual and joint action processing. Radboud University Nijmegen, Nijmegen, The Netherlands. 152. Wester, A. (2014). Assessment of everyday memory in patients with alcohol-related cognitive disorders using the Rivermead Behavioural Memory Test. Radboud University Nijmegen, Nijmegen, The Netherlands. 153. Koenraadt, K. (2014). Shedding light on cortical control of movement. Radboud University Nijmegen, Nijmegen; Sint Maartenskliniek, Nijmegen, The Netherlands. 154. Rutten-Jacobs, L.C.A. (2014). Long-term prognosis after stroke in young adults. Radboud University Nijmegen Medical Centre, Nijmegen, The Netherlands. 155. Herbert, M.K. (2014). Facing uncertain diagnosis: the use of CSF biomarkers for the differential diagnosis of neurodegenerative diseases. Radboud University Nijmegen Medical Centre, Nijmegen, The Netherlands. 156. Llera Arenas, A. (2014). Adapting brain computer interfaces for non-stationary changes. Radboud University Nijmegen, Nijmegen, The Netherlands. 157. Smulders, K. (2014). Cognitive control of gait and balance in patients with chronic stroke and Parkinson’s disease. Radboud University Nijmegen Medical Centre, Nijmegen, The Netherlands. 158. Boyacioglu, R. (2014). On the application of ultra-fast fMRI and high resolution multiband fMRI at high static field strengths. Radboud University Nijmegen, Nijmegen, The Netherlands. 159. Kleinnijenhuis, M. (2014). Imaging fibres in the brain. Radboud University Nijmegen, Nijmegen, The Netherlands. 160. Geuze, J. (2014). Brain Computer Interfaces for Communication: Moving beyond the visual speller. Radboud University Nijmegen, Nijmegen, The Netherlands. 161. Platonov, A. (2014). Mechanisms of binocular motion rivalry. Radboud University Nijmegen, Nijmegen, The Netherlands. 162. Van der Schaaf, M.E. (2014). Dopaminergic modulation of reward and punishment learning. Radboud University Nijmegen Medical Centre, Nijmegen, The Netherlands. 163. Aerts, M.B. (2014). Improving diagnostic accuracy in parkinsonism. Radboud University Nijmegen Medical Centre, Nijmegen, The Netherlands. 164. Vlek, R. (2014). From Beat to BCI: A musical paradigm for, and the ethical aspects of Brain-Computer Interfacing. Radboud University Nijmegen, Nijmegen, The Netherlands.

127