v\Am - UNT Digital Library [PDF]

Still screenshot of real-time audio/visual sequencer, 'Hexagrama' by Arístides ...... Anyone who uses software to displ

0 downloads 14 Views 18MB Size

Recommend Stories


Paralysis as “spiritual liberation” in Joyce's ... - UNT Digital Library [PDF]
metaphor that emanates from the literal fact of the Reverend James Flynn's physical condition the narrator ... The basic interpretation of a crucial term in the study of James Joyce, namely paralysis, has notably ...... readily apparent to readers ve

digital library digital library services services
Suffering is a gift. In it is hidden mercy. Rumi

IndraControl VAM 10.2, VAM 40.2
When you do things from your soul, you feel a river moving in you, a joy. Rumi

e-0146-6,64 - Adventist Digital Library [PDF]
Officers: President, T. F. Judd. Secretary-Treasurer, J. J. Dever. Ordained Ministers: T. F. Judd, Cyril Pascoe, Patavaki,. Ragopitu, Simi, Tati. Licensed Ministers: Are, Baugasi, Beni, J. J. Dever, H. A. Dickens, Ereman, R. A. Harrison, ...... Direc

Untitled - IDRC Digital Library
Stop acting so small. You are the universe in ecstatic motion. Rumi

Untitled - IDRC Digital Library
And you? When will you begin that long journey into yourself? Rumi

Untitled - IDRC Digital Library
If you want to become full, let yourself be empty. Lao Tzu

Digital Masonic Library
Don’t grieve. Anything you lose comes round in another form. Rumi

Untitled - IDRC Digital Library
The best time to plant a tree was 20 years ago. The second best time is now. Chinese Proverb

Untitled - IDRC Digital Library
Raise your words, not voice. It is rain that grows flowers, not thunder. Rumi

Idea Transcript


tA/v\Am: REAL-TIME AUDIO/VIDEO SCRUBBING TOOLS FOR ANALYSIS OF MULTIMEDIA AND G®¡ND Daniel Albert Tramte, B.M., M.M.

Dissertation Prepared for the Degree of DOCTOR OF PHILOSOPHY

UNIVERSITY OF NORTH TEXAS May 2015

APPROVED: Jon C. Nelson, Major Professor David Bard-Schwarz, Minor Professor Panayiotis Kokoras, Committee Member Joseph Klein, Chair of the Department of Composition Benjamin Brand, Director of Graduate Studies in Music James Scott, Dean of the College of Music Costas Tsatsoulis, Interim Dean of the Toulouse Graduate School

Tramte, Daniel Albert. tA/v\Am: Real-Time Audio/Video Scrubbing Tools for Analysis of Multimedia and G®¡ND. Doctor of Philosophy (Composition), May 2015, 150 pp., 107 figures, bibliography, 39 titles. tA/v\Am (the Audio/Video Analysis Machine) is an interactive analysis engine optimized for audio/visual mediums, such as film, video games, and music. I designed tA/v\Am to allow users to pace the playback speed of videos containing sub-title style analytical text, without affecting the pitch content of the audio. The software affords writers the opportunity to display the relevant sensory data (i.e., analytical text and sound/visual media) more efficiently than the paper format. It also serves as a flexible medium; the writer may, for example, compress extensive text into a short amount of time, causing the reader to pause or slow down the rate of the video and thus suspending him/her in the sensorial moment which the writer describes. G®¡ND for Alto Saxophone, Percussion & Electronics is an exploration of the tipping point between signal and noise. Through tablature notation, MIR tools, granular synthesis, and the deconstruction of the saxophone, I have assembled a palette of inordinately contrasting sounds and threaded them together based on action profiles obtained by computer assisted analysis. With them, I have set varying physical conditions of friction that dictates whether the sonic energy is to become focused to one resonant point, or distributed equally/randomly throughout the spectrum as noise. In my critical essay, I use the software to analyze independent video games, showing how tA/v\Am is a highly appropriate tool for such analysis as it is an analogous medium. I then show the software's capabilities as a multimedia platform in analyzing acoustic music, as well as my own electroacoustic work, G®¡ND. In doing so, I advocate for a media-driven analyses, and maintain that one can communicate nuanced ideas using minimal verbal/textual explanation.

Copyright 2015 by Daniel Albert Tramte

ii

ACKNOWLEDMENTS As I worked to develop tA/v\Am, I faced numerous technical obstacles; I cannot imagine how I would have succeeded in solving these problems were it not for the support and encouragement of my professors, colleagues, family, and friends. I thank my advisor Jon C. Nelson, the other members of my committee, David Bard-Schwarz and Panayiotis Kokoras, and Joseph Klein, chair of the composition division at UNT, for guiding me as I designed the software for tA/v\Am and wrote the score and electronic components for G®¡ND. I thank my colleagues who supported this project in various ways: Ermir Bejo, Joshua Harris, Joseph Lyszczarz, Stephen Lucas, Mark Oliveiro, Patrick Peringer, Josiah Wolf Oberholtzer, and the members of NoiseFold (David Stout and Cory Metcalf). I thank Patchwork Duo (Noa Even and Stephen Klunk) for their tireless efforts in learning, practicing, rehearsing, and performing G®¡ND. Finally, I thank my family, my parents Kenneth and Helen Tramte and my sister Corrie Liptack, for their love and support, and especially my wife Monica Hershberger, who helped immensely throughout the entire dissertation process.

iii

TABLE OF CONTENTS Page ACKNOWLEDMENTS ................................................................................................................ iii LIST OF FIGURES ....................................................................................................................... vi PART 1 CRITICAL ESSAY CHAPTER 1 INTRODUCTION .....................................................................................................2 Critique of the Paper Medium, and a Demand for an Interactive [Multi-]Medium for Analysis ..................................................................................................................................2 Defining and Using tA/v\Am ..................................................................................................5 CHAPTER 2 TECHNICAL DESIGN OF THE SOFTWARE......................................................16 tA/v\Am-as-Software ............................................................................................................16 Components ...........................................................................................................................16 Media players: YouTube, VLC, and QuickTime ..................................................................18 Basic tA/v\Am User Interface ...............................................................................................20 Basic Required Functionality for tA/v\Am. ..........................................................................22 Technical Means for Achieving this Apparatus. ...................................................................22 Synchronous Granular Synthesis ...........................................................................................24 Video Playback ......................................................................................................................26 CHAPTER 3 tA/v\Am ANALYSIS OF VIDEO GAMES ............................................................33 Analyzing Interactive Multimedia using an Interactive Multimedia Platform .....................33 My Body is Not My Own: Filtered Entities and Enduring Sonic Objects in Limbo ............35 Bracketing Time in Braid ......................................................................................................44 tA/v\Am Conception .............................................................................................................50 Down the Rabbit-Hole: Bracketing Aggregated Temporal Trajectories ...............................56

iv

The Interactive Experience of Video Games and tA/v\Am...................................................63 CHAPTER 4 tA/v\Am ANALYSIS OF MUSIC...........................................................................68 Analysis of Beat Furrer's Voicelessness (the snow has no voice): 'Spring Theory' ..............69 Electroacoustic Music Analysis ............................................................................................82 Analysis of G®¡ND ...............................................................................................................83 Composer Sketches History (Tracking Changes: Versions/Iterations of the first page) .......91 CHAPTER 5 CONCLUSION......................................................................................................104 Looking Forward .................................................................................................................107 BIBLIOGRAPHY ........................................................................................................................109 PART II SCORE Notes for Performance ...................................................................................................... 114 G®¡ND for Alto Saxophone, Percussion & Electronic[s/-technician] ............................. 115

v

LIST OF FIGURES Page Figure 1.1.

Potential misconception of the title for the Audio/Video Analysis Machine. .........5

Figure 1.2.

An interactive analysis-displaying apparatus...........................................................6

Figure 1.3.

Still screenshot of real-time audio/visual sequencer, 'Hexagrama' by Arístides García. ......................................................................................................................7

Figure 1.4.

Fixed video containing the moving image and the theorist's analysis of the content. .....................................................................................................................8

Figure 1.5.

Theorist discussion corresponds with event. ...........................................................8

Figure 1.6.

User imports theorist's analyzed video into tA/v\Am software. ..............................9

Figure 1.7.

User plays media at normal speed while listening, viewing, and reading text. .....11

Figure 1.8.

User adjusting playback rate to 20% in order to take in the dense information. ...11

Figure 1.9.

User increases playback rate. .................................................................................12

Figure 1.10.

tA/v\Am may be set in reverse...............................................................................13

Figure 1.11.

User may pause tA/v\Am while continuing to listen to the audio. ........................13

Figure 1.12.

User scrubbing through media with cursor. ...........................................................14

Figure 2.1.

The three components of tA/v\Am. .......................................................................17

Figure 2.2.

Importing media. ....................................................................................................18

Figure 2.3.

YouTube rate settings (on left); VLC player rate settings (on right). ....................19

Figure 2.4.

Max patcher for controlling playback rate. ............................................................21

Figure 2.5.

Synchronous granulator proposed by Barry Moon. ...............................................24

Figure 2.6.

Granulator slaved by video clock. .........................................................................25

Figure 2.7.

Vade optimization proposed by Anton Marini. .....................................................27

Figure 2.8.

Calculating 'nominal rate'.......................................................................................30

Figure 2.9.

Phasor driven audio/video media. ..........................................................................31 vi

Figure 2.10.

Calling image frames based on progress point of phasor~. ...................................32

Figure 3.1.

The perspective of the video game player. ............................................................34

Figure 3.2.

The perspective of the tA/v\Am user. ....................................................................34

Figure 3.3.

Character confronted with pond.............................................................................36

Figure 3.4.

Character jumps on floating body; theorist discusses audio filtering techniques. ..............................................................................................................37

Figure 3.5.

Character drowns rapidly; user pauses video to read analyst’s text. .....................37

Figure 3.6.

Theorist points to the sonogram on right, showing the filtered [sounding] audio. ......................................................................................................................38

Figure 3.7.

Theorist provides technical explanation of low-pass filter. ...................................38

Figure 3.8.

Extracting audio components.................................................................................40

Figure 3.9.

Outward temporal zooming. ..................................................................................41

Figure 3.10.

Quickly moving narrative with slowly moving text; player/character trips wire. .42

Figure 3.11.

Player/character re-spawns. ...................................................................................42

Figure 3.12.

User watches narrative at 200%. ............................................................................43

Figure 3.13.

Player/character solves puzzle using dead body. ...................................................43

Figure 3.14.

Theorist explains the player's experience of controlling a character. ....................44

Figure 3.15.

Braid map annotated with solfège syllables...........................................................47

Figure 3.16.

As Tim moves leftwards, the music and sounds play in reverse. ..........................47

Figure 3.17.

Audio transposes according to Tim's velocity, like a DJ scratching a record. ......48

Figure 3.18.

A reversal of a leftward motion cancels out, playing music forward. ...................49

Figure 3.19.

Minimal annotations. .............................................................................................50

Figure 3.20.

Tim meets Greeter..................................................................................................50

Figure 3.21.

Text stops scrolling as Tim stops walking rightwards. ..........................................51

Figure 3.22.

Text continues scrolling as Tim walks rightwards. ...............................................51 vii

Figure 3.23.

Tim's rudeness alters Greeter's dialogue. ...............................................................52

Figure 3.24.

Tim tries to apologize. ...........................................................................................52

Figure 3.25.

Tim rudely walks past Greeter. ..............................................................................53

Figure 3.26.

Theorist tells user to reverse playback. ..................................................................53

Figure 3.27.

Theorist explains the goal. .....................................................................................54

Figure 3.28.

Two streams of text. ...............................................................................................54

Figure 3.29.

Theorist directs user's attention to the appropriate text. ........................................55

Figure 3.30.

Theorist tells user to experience the media again forward in time. .......................55

Figure 3.31.

The normal way to cross the gap. ..........................................................................57

Figure 3.32.

Tim fixes phase of clouds by reversing time. ........................................................57

Figure 3.33.

Tim riding the cloud rightwards. ...........................................................................59

Figure 3.34.

Theorist tells user to reverse media in order to reverse a reversal. ........................59

Figure 3.35.

Tim running rightward without the aid of the clouds. ...........................................60

Figure 3.36.

Tim runs across green cloud, stopping just before he reaches the other side. .......61

Figure 3.37.

Time frozen in place, hovering in air, waiting for the next green cloud................61

Figure 3.38.

Spectral spikes in sonogram...................................................................................62

Figure 3.39.

Player/character hears a c-minor seventh chord when he approaches the machine guns. .........................................................................................................64

Figure 3.40.

The boy trips the laser, causing his death, and a harmonic move to the dominant.65

Figure 3.41.

Instead of resolving to the tonic, the ii7—V sequence repeats. ..............................65

Figure 3.42.

The dominant chord resolves when the player/character solves the puzzle. .........66

Figure 3.43.

The experienced progression. ................................................................................66

Figure 4.1.

First and last pages of Voicelessness; staves labeled with capital letters; systems labeled with roman numerals. ...................................................................70

viii

Figure 4.2.

Each staff is listed sequentially and the pitches are compiled into a single sonority. ..................................................................................................................71

Figure 4.4.

'Stretched-out spring' model...................................................................................72

Figure 4.5.

'Compressed spring' model. ...................................................................................72

Figure 4.6.

Question concerning the first sonority posed via tA/v\Am. ..................................73

Figure 4.7.

Reading A of first sonority in Voicelessness. ........................................................74

Figure 4.8.

Reading B of first sonority in Voicelessness..........................................................76

Figure 4.9.

In B major: I—ii—v—I—iv—I—flat-II—I. .........................................................76

Figure 4.10.

User scrubs through, experiencing the I—ii—v—I progression rapidly. ..............77

Figure 4.11.

User scrubs through the middle of the piece, hearing and seeing a iv—I plagal cadence. ..................................................................................................................77

Figure 4.12.

User scrubbing through the flat-II—I cadence. .....................................................78

Figure 4.13.

'Crimped ends' model. ............................................................................................79

Figure 4.14.

Three phases of the spring models: Stretched—compressed—crimped................80

Figure 4.15.

Couprie's analysis of Parmegiani's Ondes Croisée. ...............................................82

Figure 4.16.

Electronics: mixer notation. ...................................................................................84

Figure 4.17.

Defining the 'INHALE' gesture..............................................................................85

Figure 4.18.

Defining the 'EXHALE' gesture. ...........................................................................85

Figure 4.19.

The problematic beat: BEATq (beat in question). .................................................86

Figure 4.20.

BEATq is longer during the second and third iterations than the first. .................87

Figure 4.21.

Alternate approach: filmed performance with score, electronics, and annotations..............................................................................................................88

Figure 4.22.

First iteration of the repetition. ..............................................................................89

Figure 4.23.

INHALEq becomes extended. ...............................................................................89

Figure 4.24.

Gated gesture. ........................................................................................................90

ix

Figure 4.25.

Compiled list of INHALE/EXHALE durations. ....................................................90

Figure 4.26.

First three measures of Aaron Cassidy's Asphyxia. ...............................................92

Figure 4.27.

First version of opening five measures of G®¡ND. ...............................................93

Figure 4.28.

Second version of opening five measures of G®¡ND: consolidated saxophone notation. ..................................................................................................................94

Figure 4.29.

Third version of opening five measures of G®¡ND: tablature saxophone notation. ..................................................................................................................95

Figure 4.30.

Fourth version of opening five measures of G®¡ND: gray extension lines. .........95

Figure 4.31.

Fifth version of opening five measures of G®¡ND: consolidated percussion notation. ..................................................................................................................96

Figure 4.32.

Sixth version of opening five measures of G®¡ND: filled-in extension lines. ......96

Figure 4.33.

Seventh version of opening five measures of G®¡ND: re-composed second saxophone gesture. .................................................................................................97

Figure 4.34.

Eighth version of opening five measures of G®¡ND: centered meter text and added grace notes. ..................................................................................................97

Figure 4.35.

Ninth version of opening five measures of G®¡ND: final iteration. .....................98

Figure 4.36.

Composer sketches in tA/v\Am. ............................................................................98

Figure 4.37.

Quickly flashing pages like a digital flip book. .....................................................99

Figure 4.38.

Many states of the score. ......................................................................................100

Figure 4.39.

User may slow tA/v\Am to study the quickly flashing pages. .............................100

Figure 4.40.

tA/v\Am animates the changes. ...........................................................................101

Figure 4.41.

Final state of the G®¡ND as viewed through tA/v\Am. ......................................102

Figure 5.1.

Large gap in Super Meat Boy...............................................................................104

x

PART 1 CRITICAL ESSAY

1

CHAPTER 1 INTRODUCTION Critique of the Paper Medium, and a Demand for an Interactive [Multi-]Medium for Analysis Historically, writing has been the primary mode for analyzing art and music. Analysts generally turn to the written word (books, journal articles, etc.) in their efforts to find and explicate meaning within works of art. While much writing has transferred into the digital medium—online journals and e-books, for example—it clings to the format of its predecessor: words (and images) on a white canvas. Unfortunately, words and images may be either insufficient or inefficient in analyzing the sensual richness of multimedia and interactive art/music. When attempting to write about moving image, video games, or other forms of A/V art, one must make extensive use of verbose language to describe the visual aesthetics, sonic material, or the experience of empathizing with characters on screen, before one may begin writing an in-depth and informative analysis of the work. To be sure, analysts may incorporate static screen shots of the video and/or musical examples of the audio to demonstrate the media being described; however, these snapshots fall short of truly representing the experience of the media, and in many cases, certain audiences may not be able to read music notation or understand how to interpret spectrograms without hearing the audio to which it signifies. In an attempt to address these concerns, analysts might consider imposing their written analyses directly onto the visual medium in the form of subtitles. 1 In this way, all of the important data (sound, visuals, and analytical text) would exist in one place and would be presented efficiently. However, due to the fixed limits of the basic video medium and its

1

See Liu, Liao, and Cheng-Jun Peng, “Applying the Technology Acceptance Model and Flow Theory to Online E-Learning Users' Acceptance Behavior,” Issues in Information Systems 6, no. 2 (2005): 175-181.

2

playback, the subtitle format would likely fail to adhere to the viewer's preferred pace. Differing levels of familiarity, interest, and theoretical backgrounds of the media being analyzed make it desirable to provide the possibility of adjusting the speed of the video such that users may repeat, skip, or pause sections of the analysis. Analysts may also want to spend a large portion of an analysis highlighting a specific moment in a piece of A/V art. If the analysis is embedded within the A/V media, it may be necessary to pause playback of the image and sound in order for the viewer to read in-depth analyses of a small section. For any discussion of the audio to occur, the aural experience must be suspended until viewers are finished reading the text. In creating tA/v\Am (The Audio/Visual Analysis Machine), I propose a solution to these problems. Using Max with Jitter to process graphics while simultaneously processing digital audio signals, I built an audio/video player that gives the user complete control over the playback rate, direction, and cursor position without affecting the pitch content of the audio. This allows the user to control the pacing of the text and sensory data, as well as repeat playback at a comfortable rate. Thus, users may slow down, reverse, and pause/hold audio and video information in order to experience a single moment that may be critical to the analysis. Utilizing a granular synthesis module, the pitch and timbre of the audio are maintained as the user manipulates time. This provides analysts with a tool with which they can freeze audio to demonstrate the techniques being used at a particular point in time. While tA/v\Am may be used to analyze any medium containing audio and/or video, it is especially elegant in analyzing video games, since both are interactive audio/video formats. Those interested in analyzing video game music will be able to effortlessly show, for example, how a character's interaction with its environment affects the sounds or music within a game. tA/v\Am, however, also proves valuable to music theorists looking to provide interactive 3

audio/video musical examples. For example, theorists may use tA/v\Am to display audiovisual diagrams, encouraging readers/users to manipulate and hear for themselves the musical relationships that the theorist is illustrating. tA/v\Am may also be applied to the music theory classroom. Instructors wishing to demonstrate long-form tonal trajectories to students can simply generate videos, synchronizing audio with, for example, Schenkerian diagrams. 2 Then, the instructor can easily 'scrub' through the video, allowing students to hear the ursatz directly, and how it is articulated by the foreground/middleground structures. Finally, theorists may also use the software to enhance conference paper presentations, scrubbing through their own interactive analyses in front of an audience and zooming into moments by pausing the video for in-depth discussions on a sounding sonority. This binds the theoretical discussion to the audio example, solving the otherwise detached method of first presenting the theory, followed by the audio. In many ways, my goal in this research runs antithetical to the act of writing a dissertation—a textual medium [document] for doctoral students to contribute research to their field of study. With my dissertation, I seek to participate in the advancement of analytical mediums, accounting for current technological trends and asserting the importance of non-textual or fully mediated analysis. Indeed, I argue that as far as multimedia works of art are concerned, a high quality interactive musical example/diagram may be the most efficient and illuminating means of presenting an argument. Thus, the musical diagram should be the core of the analysis, with the text functioning as supplemental material. In some cases, text may not only be substantially suppressed, but may be completely removed. I assert that within an interactive a/v analysis environment, the need for additional text is minimal since the mediated analysis can speak for itself.

2

See audio/video example here: http://youtu.be/cvVCTTllVdA (accessed February 28, 2015).

4

Defining and Using tA/v\Am tA/v\Am is perhaps a slightly misleading title for the following research. The Audio/Video Analysis Machine implies some sort of apparatus that does analysis to works of audio/video mediums—a machine that accepts a specific format of multimedia data as an input and produces analytical data as an output. I have illustrated this potential misinterpretation below (figure 1.1).

Figure 1.1. Potential misconception of the title for the Audio/Video Analysis Machine.

tA/v\Am does not execute analyses itself; indeed, missing from figure 1.1 is an agent (i.e., the theorist) who produces analyses of audio/video works, as well as another agent (i.e., the user) who experiences the analysis. To extend the industrial metaphor of the machine, the theorist acts as a mechanic, modifying the machine. The user thus becomes the foreperson, whose duties are to operate the machine and oversee the analysis. The machine, tA/v\Am, is thus not an analysis-

5

producing apparatus, but rather, an interactive analysis-displaying apparatus as shown in figure 1.2. 3

Figure 1.2. An interactive analysis-displaying apparatus.

I have built tA/v\Am in the form of real-time audio/video scrubbing software, and indeed, when I use the term 'tA/v\Am,' I am often referring directly to the software; however, when one looks solely at the software, one will find that it resembles a media player similar to that of VLC player, QuickTime, or YouTube. The difference is that it has specialized controls and functionalities to serve a style of non-linear a/v scrubbing analysis. I thus prefer to use the term, tA/v\Am, to signify a wider reference: the theoretical methodology. In this way, tA/v\Am may be defined as the process of using any real-time audio/video scrubbing apparatus to create and/or experience a multimedia analysis that is operated by the user, whether or not it involves the

3

If one follows this metaphor through, one may note that the analysis is, in a way, 'stuck in production.' It never leaves the factory, and is never disseminated to the public. Therefore, I want to reiterate that by reassigning the analysis medium to a machine rather than a written document, I am promoting the reader to a level of active agency similar to that of a foreperson. This agent is given the freedom to oversee and manipulate his/her experience of the content produced and assembled by the mechanic (theorist).

6

particular software that I have created. Since the software I have built functions as a specialized media player, the theorist is tasked with creating analysis videos. These are videos containing analytical text superimposed onto the video being described. The user, then, imports these analysis videos into the software in order to scrub through them, read the text, and experience the audio/video content. To demonstrate the process of creating a tA/v\Am analysis video, let us take the following example of a real-time audio/visual sequencer by Arístides García (aka., lasal) called Hexagrama (figure 1.3).

Figure 1.3. Still screenshot of real-time audio/visual sequencer, 'Hexagrama' by Arístides García.

Theorists wishing to analyze this piece using tA/v\Am would have to use video editing software (Windows Movie Maker, iMovie, etc.) to add subtitle-style or scrolling text to the video, and render it as a fixed analyzed video. Figure 1.4 shows a screenshot of the theorist explaining the rotating motion of the lines in the video using superimposed text.

7

Figure 1.4. Fixed video containing the moving image and the theorist's analysis of the content.

While not always necessary, the topics of discussion should temporally correspond with the events that take place within the video so that one may read the analytical text while viewing and listening to the content being discussed (figure 1.5).

Figure 1.5. Theorist discussion corresponds with event.

8

Although the time it takes to read the analytical text will rarely map perfectly onto the duration of the event, the user will be able to set the ideal playback rate of the video once [s]he imports the fixed media into tA/v\Am, a flexible medium (figure 1.6).

Figure 1.6. User imports theorist's analyzed video into tA/v\Am software.

Thus the purpose of the tA/v\Am software is not to analyze fixed videos, but to transform fixed [analyzed] videos into something more flexible which users may interact with and experience. In other words, tA/v\Am is not an analyzer, it is a medium. This flexible analytical medium provides theorists with new and non-linear techniques to present their research on multimedia works. I demonstrate the various ways tA/v\Am may be employed by presenting my own analyses of works using tA/v\Am. In doing so, I often take the perspective of the tA/v\Am user, via imported screen shots. Consider the following example of my analysis of Ted Coffey's video work, Blue Cycle: Noise (2008) using the tA/v\Am platform. In figure 1.7, the user views and listens to Ted Coffey's work, while simultaneously reading the theorist's text. The hand-written text, written by Coffey, displayed in the center of the screen reads as follows:

9

noise is only ever an overwhelming, in some provisional [temporal, spatial, conceptual] context, of a particular subject's ability to re-/cognize pattern. The analytical text below (in subtitles), written by the theorist, describes the following concept: In Coffey's work, both the written and spoken word of his textual message are severely manipulated, written in chicken-scratch handwriting and spoken distortedly through heavy digital signal processing, making the message of the text difficult to comprehend. One cannot rely solely on either the visual or aural cues. Instead, one must rely on his/her ability to recognize patterns from both modes of perceptual input to discern the message. The catch here is that as Coffey defines the term 'noise' throughout the piece, he is in part describing this exact concept; thus Coffey is using a faulty medium (distorted spoken/written text) to demonstrate his message. As the following figures are presented from the perspective of the tA/v\Am user, we can see how the user operates tA/v\Am in order to grasp the concept described above. Displayed in the bottom right-hand corner of figure 1.7 is the speed at which the user watches the video. In this case, the user views the video at 100%, or 'normal speed.'

10

Figure 1.7. User plays media at normal speed while listening, viewing, and reading text.

Figure 1.8 shows the user slowing down tA/v\Am's playback rate to 20% in order to read the analytical text at a comfortable rate.

Figure 1.8. User adjusting playback rate to 20% in order to take in the dense information.

11

As the user increases the speed of the playback to 200% (double speed), the audio does not transpose as it would in a typical video player (figure 1.9). In other words, increasing the playback speed does not cause the speaking voice in Coffey's work to suddenly sound like a chipmunk. Rather, the granular synthesis algorithm in the tA/v\Am software allows the medium to be temporally manipulated independent of the pitch. Therefore, the user will hear the speaking voice at the original pitch, and the theorist can rely on the fact that the user experiences the medium with no pitch deviation from the original media.

Figure 1.9. User increases playback rate.

If the analytical text moves too quickly, the user may play the media in reverse, catching any information [s]he might have missed on the first pass (figure 1.10).

12

Figure 1.10. tA/v\Am may be set in reverse.

If a user finds it necessary to spend a substantial amount of time experiencing the media, while letting the theorist's concepts sink in, the user may pause the medium, setting tA/v\Am to 0% (figure 1.11). Again, as this medium is paused, the audio of that precise moment in the media continues to sound; thus the user will experience that sonority until [s]he presses 'play.'

Figure 1.11. User may pause tA/v\Am while continuing to listen to the audio.

13

Finally, the user may 'scrub' through the media by clicking and dragging the play-bar cursor (figure 1.12). This allows the user to experience the media intimately; since the granular synthesis algorithm processes the audio while scrubbing, the user will continue to hear the spectromorphological motion of the sonorities as [s]he moves the cursor over time.

Figure 1.12. User scrubbing through media with cursor.

In my dissertation, I borrow Coffey's meta approach by referring to the medium through my argument and referring to my argument through the medium. In the same way that Coffey uses a faulty medium (distorted communication) to demonstrate the content of his argument, I use the 'dissertation medium' to demonstrate the inefficiencies of analyzing the sensorial richness of dynamically-generated multimedia art and analysis. In this way, my dissertation is atypical. Much of my research is not even presented in this document. As an interactive video application,

14

I cannot import the interactive experience of the audio/video medium into this still and fixed written medium. I thus have to refer to the experience through the standard and inefficient methods that I critique, such as reproduced images of my own analysis videos, as I have done above. I empathize with music theorist David Bard-Schwarz, who encountered a similar issue in his recent book, An Introduction to Electronic Art Through the Teaching of Jacques Lacan: Strangest Thing. Bard-Schwarz points out: To refer to a dynamically-generated work of electronic art through such a remove seems not only to see the world as through a glass darkly, but to barely see at all. 4 Readers of this dissertation document will likewise view my interactive analysis videos “through a glass darkly,” despite the fact that I am using my dissertation as a tool to advance a possible solution for this very problem. Through technical descriptions of my multi-media platform (chapter 2), and analyses of videos games (chapter 3), acoustic music, and electroacoustic music (chapter 4), I seek to illuminate how both the content of my dissertation and my dissertation itself act as a demonstration in favor of the idea that new media calls for new modes of analysis.

4

David Bard-Schwarz, An Introduction to Electronic Art Through the Teaching of Jacques Lacan: Strangest Thing (London: Routledge, 2014), 4.

15

CHAPTER 2 TECHNICAL DESIGN OF THE SOFTWARE tA/v\Am-as-Software As I have explained, tA/v\Am exists primarily as a concept for multimedia analysis. Indeed, tA/v\Am-as-concept does not require specific software to be realized. Theorists may make analysis videos and request that their readers/viewers experience the videos through a media player of the theorist's choosing. Commercially available media players are not designed for the type of analysis videos I describe; such players have their own means of handling audio and their own user interface for time-controls. As we will see, many of these basic functionalities are highly unfavorable for certain situations that might arise when using these media players in this way. In the following chapter, I present my media player design (tA/v\Am-as-software) which accommodates for all of the possible analytical approaches that a theorist might take in a tA/v\Am-style analysis video. In doing so, I also show how I have built the software using Max 6, and document the challenges presented in the process. I begin by introducing the components, controls, and functions needed for a tA/v\Am-style analysis, explaining how these differ from other video players. With these features in place, we can then discuss in detail the algorithms required to achieve this media player.

Components The software is made up of three primary components, a video viewing window, a live FFT analysis window, and a play bar (figure 2.1).

16

Figure 2.1. The three components of tA/v\Am.

Component A is a video viewing window that displays the analysis videos designed by the theorist.

Component B is a real-time spectral analysis of the sounding audio (FFT spectrogram) to display the prominent frequencies, filtering techniques, and other alterations/modifications of the audio content. This component also includes an area that allows users adjust volume using their cursors, and load media. 5 To load media, users type the file name of the video into a text field. Figure 2.2 shows this text field as well as the code used to import .m4v files.

5

In an early version of tA/v\Am, the volume was controlled by the up and down arrow keys.

17

Figure 2.2. Importing media.

Component C is a play bar showing the progress of the media playback. Users interact with it by controlling the rate of playback using the arrow keys and/or clicking and dragging the cursor to scrub through the audio/video media. Component C also displays the rate at which the user chooses to play the media (in percentage) to the right of the play bar. Many other audio/video players have similar scrubbing and speed-control capabilities. Users can adjust the playback speed in digital players such as YouTube and VLC, both of which utilize a pitch-independent time stretching/skewing algorithm. Thus, while tA/v\Am shares many attributes with other audio/video players, it focuses more on interactive time control. It differs from the YouTube and VLC player in that it is designed and optimized to be manipulated in realtime throughout the duration of the video, whereas the primary function of YouTube and VLC is to play media at a rate of 100% (i.e., at a normal rate) while viewers experience the media remotely with minimal physical contact.

Media players: YouTube, VLC, and QuickTime Let us consider the default rates settable in YouTube and VLC player and compare them to those I will show in tA/v\Am (figure 2.3). In YouTube, one may set the rates to 25%, 50%, 100%, 125%, 150%, and 200%. VLC player allows users to set rates on a sliding scale, rather than fixed increments. This scale ranges from 25% to 400%. If using keyboard shortcuts, one 18

may even increase the speed to 6,400%.

Figure 2.3. YouTube rate settings (on left); VLC player rate settings (on right).

QuickTime player has many hidden functions, allowing advanced time control of the video. One may hold down on the pause/play button for approximately two seconds to open up a slide scale for adjusting the playback rate, which includes reversed rates. QuickTime also has the capabilities for fine scrubbing in addition to the typical play bar scrubbing. While these media players allow for various degrees of time control, some players, such as VLC player and YouTube are not capable of reverse playback. Missing from all of these media players, including QuickTime, is the ability to handle real-time rate adjustment using simple fingertip-ready controls, like buttons on the keyboard. And in those media players that are capable of reversing playback, doing so in real-time will often have undesirable effects. In QuickTime, for example, each modification of a reversed rate causes the video to restart from the end. An even more serious concern with these media players, however, has to do with how the audio is handled while the video is playing at any rate other than 100%. When these media players are pushed to extreme rates, such as 20%, the algorithm used to maintain the pitch-level of the audio (usually phase vocoder) introduces many unwanted artifacts. Additionally, most

19

media players set the audio to silence when paused. This is ideal for daily use of media, and indeed, it is an expected basic function. It is not, however, ideal for the non-linear and temporally free style of analysis that I propose. When the media is paused in tA/v\Am, the audio should continue to sound by freezing and sustaining the spectral content of the moment. In this way, the theorist may direct his/her users to aspects of the sounding sonority by referencing it in the analytical text.

Basic tA/v\Am User Interface As mentioned above, tA/v\Am users manipulate the software by typing/keying buttons on the computer as well as clicking and dragging the play bar cursor with the mouse or track-pad. The hotkeys are as follows: The spacebar is set to toggle between pause and play. Since tA/v\Am is unique in that users will often play the media at varying rates, however, I must qualify that it sets non-zero rates to 0%, and 0% to 100%. The delete button sets any rate to -100%, playing the media backwards. Finally, the user taps the left/right arrow buttons to nudge incrementally through speed rates (figure 2.4).

20

Figure 2.4. Max patcher for controlling playback rate.

The default speeds are quantified into thirteen fixed increments, and resemble the rates settable in youtube. Between rates 0% to ±100%, the speed at which one may watch the video is divided into nine discrete rates: 0%, ±20%, ±50%, ±75%, and ±100%. Users will spend most of their tA/v\Am experience in this range (between -100% to 100%) due to the fact that the additional sensory data will often cause the user to slow the media down in order to digest the increase of information. It thus does not make sense to incorporate as many gradations of fast rates beyond ±100%. Additionally, given that the play bar allows users to scrub through the media rapidly, there is little need for extreme speeds beyond ±200%. Thus, tA/v\Am only requires four more states: ±150%, and ±200%.

21

Basic Required Functionality for tA/v\Am. Based on the concerns addressed above regarding the design of tA/v\Am's controls, let us now review the two primary required functionalities:

• A video player that is synchronized with the audio. • A versatile audio time-scale modification (TSM) algorithm that can handle any rate, including negative rates, as well as a rate of 0%, or 'freeze.'

Technical Means for Achieving this Apparatus. Designing a real-time a/v scrubbing apparatus with pitch-independent time-scaling audio turns out to be a rather non-trivial task. 6 When a typical media player reads a video at half speed, it reads the audio samples at half the desired rate as well, and thus transposes it down an octave. For example, if a sine-tone of A 440 Hz (440 cycles per second) were to be read at half the speed, it would take two seconds to read through the 440 cycles, thus only producing 220 cycles per second (A 220 Hz—an octave lower). In my first attempt at creating tA/v\Am, I tried to counteract this transposition artifact by re-transposing it in the opposite direction in real-time. To do this, I used spigot~ (an object within the Max 6 programming environment) to extrapolate the audio from the video in order to process the audio in real-time. I then calculated the reciprocal of the rate of the video and used that value to pitch-shift the audio via gizmo~ (another Max object). Although this method successfully transposed the audio back to its original pitch-level, the sound quality was notably unsatisfying. If users were to set tA/v\Am to fast playback rates causing the audio to transpose upwards, the audio would have to be transposed downwards, and therefore would lose the

6

See Ivan Damnjanovic, Dan Barry, David Dorran, and Josh Reiss, “A Real-time Framework for Video Time and Pitch Scale Modification,” Proceedings of the 11th International Conference on Digital Audio Effects, Espoo, Finland, September 1-4, 2008.

22

fidelity of the high frequencies in the process. If users were to set tA/v\Am at low speeds causing the audio to transpose downwards, the audio would have to be transposed upwards, and would therefore become bit-crushed in the process. In the case of pausing tA/v\Am, the audio would have to be transposed infinitely, as the reciprocal of zero is not a real number. 7 In other words, pausing in this way produces no sound, and is therefore unacceptable. I thus abandoned spigot~ in favor for another solution: processing the audio separately, and synchronizing it with the video. 8 To do this, I needed a TSM (time-scale modification) algorithm to time-stretch/skew the audio without altering the pitch transposition. As I was aware of the technical challenges and difficulties of real-time TSM, I first sought after a third-party algorithm before I had attempted at one myself. Simon Adcock and Joe Jarlett created two Max objects called elastic~ and elasticx~. These objects allowed for real-time pitch and time stretching, utilizing the industry standard code developed by zplane. 9 Elasticx~ was a high quality, fully functional time/pitch scale modifier, which I purchased and implemented into tA/v\Am. While it was the operating TSM algorithm used in tA/v\Am for the first year of tA/v\Am's development, there were two fundamental issues with this object. Since it was a paid third-party algorithm, I was unable to disseminate tA/v\Am without purchasing a commercial license for each copy. The second issue I encountered was due to a/v asynchronicity. Rather than linking the audio and video to a single clock, the rate of each had to be set discretely. For example, if users played tA/v\Am at 200%, the tA/v\Am software would send a '2' message to the jit.qt.movie object to play the movie at 200%, while simultaneously sending a '2' message to the elasticx~ object to play the audio at 200%. This

7 8 9

The multiplicative inverse of 0/1 is 1/0, causing a situation in which tA/v\Am would have to 'divide by zero.' Damnjanovic et. al. came to the same conclusion. See “A Real-time Framework.” See “zplane” https://zplane.de/ (accessed January 21, 2015). Note: I use the past tense in this sentence, because these objects are no longer supported. See “Elastic Max” http://www.elasticmax.co.uk/ (accessed January 21, 2015

23

method therefore relied on perfect synchronicity between the audio clock and the video clock. While these two distinct clocks would maintain near-perfect synchronicity for minutes at length, the two would gradually become out of sync.

Synchronous Granular Synthesis To fix these issues, I had to develop my own TSM algorithm that could rely on a single clock. Rather than using phase vocoding algorithms, I implemented a simple 32-voice granular synthesis module, modeled after Barry Moon's 15th 'Baz Tutorial' (figure 2.5). 10

Figure 2.5. Synchronous granulator proposed by Barry Moon.

10

See Barry Moon, “Baz Tutorial 15 – Part 1,” uploaded May 21, 2009 https://www.youtube.com/watch?v=GR_bLnMNH-4 (accessed January 21, 2015); See also Curtis Roads, The Computer Music Tutorial (Cambridge, MA: MIT Press, 1996), 168-184.

24

In this method, Jitter acts as a global clock to which the audio is slaved. I calculate the progress point of the video and represent it as a decimal point (e.g., if the video is half-way finished, the software will send a '.5' to the granulator). The granulator will then continually playback grains of the sound file (150 ms in length), whereas the location of the desired sounding grain within the sound file is determined by the fraction sent from the video progress point (figure 2.6).

Figure 2.6. Granulator slaved by video clock.

This method proved successful in synchronizing the audio (slave) to the video (master) and also eliminated the dependency of elasticx~. Since the granular synthesis module is not quite as CPU-

25

efficient as the previously utilized phase vocoding algorithm, and since it is at this point dependent on the video clock, the video player began to reveal new issues.

Video Playback Jit.qt.movie runs on QuickTime, which is optimized for casual daily a/v playback, and thus becomes inadequate when manipulated in real-time. In order to reduce tA/v\Am analysis videos to a reasonable file size so as to conserve space and offload memory, one must implement “temporal compression” when rendering the video. Because of this, jit.qt.movie relies on key frame data. Key frames (or i-frames) are individual images separated by p-frames, which contain motion vector data to interpolate between the images. 11 As such, the QuickTime codec does not handle reverse playback well due to the fact that the key frames must be buffered and read in reverse, causing various issues such as freezes and erratic speed jumps. This issue was originally innocuous, as I was processing the audio separately with elasticx~. Once I utilized a granular synthesis module in lieu of elasticx~ and coupled it to the video clock, these erratic speed jumps became glaringly evident. In an attempt to compensate for these unwanted artifacts, I optimized the video using the 'Vade optimization techniques' proposed by graphics programmer, Anton Marini (figure 2.7). 12 Among many other useful advantages of the Vade optimization techniques, it offloads much of the processing from the CPU onto the GPU by using the OpenGL object, jit.gl.videoplane (textured plane), and reduces color/chroma information by using uyvy colormode, and

11

12

See “QuickTime 7: User's Guide,” p. 43, http://manuals.info.apple.com/MANUALS/0/MA104/en_US/QuickTime7UserGuide.pdf (accessed January 21, 2015); See also Tim Monroe, QuickTime Toolkit, Volume One: Basic Movie Playback and Media Types (San Francisco: Morgan Kaufmann, 2004). See “Jitter Movie Playback Optimization,” November 29, 2007 http://abstrakt.vade.info/?p=147 (accessed January 21, 2015).

26

converting it back to rgb. The Vade techniques helped to significantly reduce the a/v playback issues, but it did not eliminate them; although the video playback was optimized, it was still reliant on key frame data. Indeed, when tA/v\Am was pushed to extreme rates, such as -200%, the freezes were still present.

Figure 2.7. Vade optimization proposed by Anton Marini.

For a short period of time, I considered a playback method that would eliminate the need for playing media in reverse, since reversing media seemed to be the cause of the glitches. The concept is as follows: when the user loads the video media into tA/v\Am, the software imports and reads through two separate videos—a normal video, and a pre-reversed version of the same video. In this way, when the user sets tA/v\Am to reverse, rather than reading the data of a video backwards, tA/v\Am reads a pre-reversed video forwards. Thus, as the tA/v\Am user manipulates time backwards and forwards, the software crossfades between the two videos even though it looks like a single continuous video.

27

Implementing this concept was problematic. At any given point, there was always one active video and one inactive video, and the inactive one needed to be completely turned off. If the inactive video remained on, the jit.qt.movie object would still read through the video file in reverse, and although the inactive video would not be displayed, it still caused the stuttering effect that this method was designed to eliminate. Therefore, instead of using a simple crossfade via jit.xfade, the concept involved synchronizing the ons and offs of the two videos as the user crossed over the 'zero' rate mark. Meanwhile, the software had to keep track of the start times so that when one of the videos turned on, it began at the progress point where the other video left off. This playback system was fully implemented but discarded due to practical concerns. Either the theorist would have to render two separate analysis videos for each analysis module (one original video and one in reverse), or tA/v\Am would have to render a secondary video upon load of the original video. Either case is time-consuming, convoluted, and furthermore, it bloats the software in terms of memory and disk space. At this point, I sought to address the key framing issue by replacing the culprit, jit.qt.movie (QuickTime) with jit.gl.hap, which utilizes the Hap quicktime codec. 13 Hap was designed primarily for VJs (video jockeys) as a codec for real-time video manipulation, and is the video player engine for VDMX. The playback further compresses the video when sending the data to the GPU, and it does not rely on key framing in the same way as did QuickTime. Thus, the compression used by Hap successfully eliminated the unwanted artifacts when reversing media at extreme rates (e.g., -200%, -20%). During the process of designing tA/v\Am, it seemed as though for every problem solved, a new problem would arise. Implementing Hap made it possible to reverse media at extreme

13

See “Presenting Hap, A Family of Open-Source GPU Accelerated Video Codecs,” February 26, 2013 http://vdmx.vidvox.net/blog/hap (accessed January 21, 2015).

28

rates; however, it failed to smoothly handle changes of rates from positive to negative, or vice versa. Doing so would cause the media to freeze. Most of the time, Hap would only cause tA/v\Am to freeze for a small fraction of a second—an almost imperceptible amount of time. In extreme cases, however, the media would freeze for approximately one second, which was far worse than the temporal artifacts caused by reversing media with jit.qt.movie. I thus returned to jit.qt.movie but with a modified playback system so as to eradicate as many defects originating from the video as possible. Since the granulation module would choke up when the video choked up, it led me to believe that I could not rely on the video as a master clock. In my modified system I created an abstracted high-resolution clock using phasor~. By abstracted, I mean that this clock 'ticks' in the background, and both the video and audio are slaved to it. 14 One can thus map these values such that zero corresponds to the beginning of the a/v media, and one corresponds to the end of the media. Since it counts continuously at the audio-rate, it provides tA/v\Am with a continuous progress bar that exhibits both great stability and high resolution. The concept is as follows: upon load of the video, the software gathers video information (duration, timescale, and frames per second) in order to calculate the 'nominal rate' (in Hz). The 'nominal rate' is the rate at which phasor~ would require in order to count from zero to one in an amount of elapsed time such that it matches the length of the video (figure 2.8).

14

Phasor~ is an object that counts a floating point value from zero to one at the audio-rate.

29

Figure 2.8. Calculating 'nominal rate'.

Then, as the user manipulates the controls of tA/v\Am, this control data is sent to phasor~, which in turn drives the audio and video media (figure 2.9).

30

Figure 2.9. Phasor driven audio/video media.

To slave the video to phasor~, I use the 'frame' message to call/index frames of the video based on the progress point of the phasor (figure 2.10). Likewise, the audio is slaved to phasor~; as the phasor progresses, that progress data is sent to the audio module (recall figure 6) where it calls/indexes the appropriate grains.

31

Figure 2.10. Calling image frames based on progress point of phasor~.

This method has proven substantially more stable at playing back the media. It does not perfectly eliminate the key framing issue visually, but since the audio is not slaved to this faulty video playback clock, when it freezes, the audio does not drop out at all. The techniques used above to produce a stable real-time a/v scrubbing engine represent some of the current challenges in a/v processing. Live phase vocoding for 're-transposition' of original transposed audio signals proves problematic due to the artifacts introduced at extreme rate playback. Therefore, the audio and video must be processed separately, yet they must be tied to a single clock so as to avoid becoming out-of-sync. One cannot rely on the video clock within Jitter while using QuickTime because reversing rates causes temporal defects via key framing. Although Hap solves this issue, it produces further temporal artifacts, and therefore becomes unsatisfying. Finally, to preserve continuity in the audio, one must create a global clock to which frames from the video and grains from the audio are drawn. tA/v\Am currently utilizes this a/v processing concept, and at present, I believe it represents the most stable and robust real-time media scrubbing system that also achieves the most satisfying media quality. 32

CHAPTER 3 tA/v\Am ANALYSIS OF VIDEO GAMES Analyzing Interactive Multimedia using an Interactive Multimedia Platform I will now present several brief analyses of the independent puzzle-platform video games Limbo and Braid. Released in 2010 by the developer Playdead, Limbo is a 2D-sidescroller game with eerie and grainy greyscale graphics, in which the player controls a boy trapped in a horrifying world (limbo). Braid, designed by Jonathan Blow and released in 2008, requires its player to manipulate the flow of time to solve puzzles. Both games are frequently called 'art games,' or 'arthouse games' and have generated considerable discussion in popular media, including Roger Ebert's controversial article “Video Games Can Never Be Art,” as well as Scott Steinberg's response for CNN Tech, “Who Says Video Games Aren't Art?” 15 Video game music is receiving increasing attention from scholars as well. 16 In the following analyses of Limbo and Braid, I use my multimedia analysis apparatus (tA/v\Am) to analyze an interactive multimedia platform (video game), thus demonstrating how this new tool and medium may be used to shed more light on the topic of music and video games. Before we begin, it is important to acknowledge the different agents that act on different levels of abstraction. Take, for example, the following screen shot (figure 3.1):

15

16

Roger Ebert, “Video Games Can Never Be Art,” Robert Ebert's Journal, April 16, 2010, accessed December 1, 2014 http://www.rogerebert.com/rogers-journal/video-games-can-never-be-art; Scott Steinberg, “Who Says Video Games Aren't Art?” CNN Tech, August 31, 2010, accessed December 1, 2014. http://www.cnn.com/2010/TECH/gaming.gadgets/08/31/video.games.art.steinberg/. For example, see William Cheng, Sound Play: Video Games and the Musical Imagination (Oxford: Oxford University Press, 2014); Karen Collins, Playing with Sound: A Theory of Interacting with Sound and Music in Video Games (Cambridge, Massachusetts: MIT Press, 2013).; Kiri Miller, Playing Along: Digital Games, YouTube, and Virtual Performance (Oxford: Oxford University Press, 2012).

33

Figure 3.1. The perspective of the video game player.

This is how the video game player views and interfaces with the video game medium. The player controls his/her character, and since the goals of the player and character are generally shared, I refer to these two agents together as a player/character. The player/character meta agent, however, becomes bracketed into a lower level of abstraction once tA/v\Am is incorporated into the screen shots (figure 3.2).

Figure 3.2. The perspective of the tA/v\Am user.

In this scenario, we are no longer looking from the point of view of the player, but instead, from the point of view for the tA/v\Am user. The user views the actions of the 34

player/character and the theorist's analysis of those actions. While I will provide synopses of the low-level actions (those which occur within the video game) prior to discussing the analysis, I will often oscillate between the two levels in my demonstrations—even within the same sentence (e.g., “the theorist explains to the user how the player controls the character”). Each of the following analytical modules are excerpts from larger-scale analyses. These short modules stand on their own and most importantly, demonstrate tA/v\Am's unique capabilities.

My Body is Not My Own: Filtered Entities and Enduring Sonic Objects in Limbo In this analytical module from the game Limbo, I look closely at the ways in which sound designer Martin Stig Andersen manipulates the player's sense of character embodiment. I begin by demonstrating tA/v\Am's ability to use compressed text for what I call “temporal zooming.” Temporal zooming is similar to standard zooming in that some sort of sensory data lies out of the reach (or out of the scale) of perception, requiring an alteration of the medium in order to bring it in focus. Anyone who uses software to display PDF documents understands this experience. If a PDF page is zoomed at 100%, there is sometimes text that is still too small to read given the computer's screen size and resolution; thus, one must zoom to a higher percentage to read the text. In temporal zooming, rather than dealing with concerns of text size, the text rate is compressed, requiring one to slow down the speed rate. The following analysis demonstrates how by using temporal zooming in Limbo, we can better understand how the player's sense of character embodiment functions in this particular game. In the game, the boy (the player/character) approaches a pond that is too wide to cross (figure 3.3). He sees a dead body floating and decides to use it as a stepping-stone. Once he jumps onto the dead body, the body sinks slightly, causing the player/character to panic. Though there are not enough stepping-stones to make the following jump, he attempts the jump anyway 35

and drowns. When the player/character re-spawns and jumps onto the floating body again, he is not duped a second time. Instead, he realizes that the additional weight on the floating body will eventually reach an equilibrium and therefore poses no immediate threat. A secondary character approaches the pond from the right-hand side of the screen. This character's brain is infected with a white worm (or slug), which causes him to move leftwards, presumably against his will. When this character reaches the pond, he slowly slips in and drowns. Now, the player/character waits until the secondary character dies and then uses him as a second stepping-stone.

Figure 3.3. Character confronted with pond.

In tA/v\Am, when the boy first jumps onto the floating body, we see through the theoretical analysis how Andersen uses filtering techniques to create an embodied relationship between the player and character in the form of subtitles. See Figure 3.4.

36

Figure 3.4. Character jumps on floating body; theorist discusses audio filtering techniques.

The drowning sequence that occurs after the premature jump is rapid; the text flashes by in a fraction of a second, causing the user to pause the video. At this point, tA/v\Am is paused, so that the user may read the fast-paced text. The sound in that exact moment, however, continues to sustain without being transposed. Thus, the user may listen while [s]he views the action, and reads the text. Here, the theorist explains that cutting out high frequencies creates an under-water aural sensation for the player (figure 3.5).

Figure 3.5. Character drowns rapidly; user pauses video to read analyst’s text. 37

The theorist directs the user's attention to the sonogram. The user notes that at the very right side of the sonogram—which shows the spectral content of the current point in time—there is a marked cutoff in the high frequencies (figure 3.6).

Figure 3.6. Theorist points to the sonogram on right, showing the filtered [sounding] audio.

Now the theorist provides specific detail, such as the cutoff frequency at which the sound designer sets his low-pass filter. In the action, both the player/character and the secondary character drown simultaneously (figure 3.7).

Figure 3.7. Theorist provides technical explanation of low-pass filter. 38

Though the drowning sequence in the action lasts only a few seconds, tA/v\Am allows the theorist to 'temporally zoom' into the moment, ensuring that the user may experience the theorist's in-depth analysis over an extended period of time. The theorist achieves this by generating text that is too fast to read, and therefore causing the user to slow the video down so that [s]he may take the time to read the analytic text and listen to the respective sounds that the theorist is describing. Temporal zooming is a valuable and multifaceted tool for discussing sound in multimedia. Not only may a theorist use temporal zooming to highlight alterations in the spectral content of the audio as I have done above, but the theorist may also use it to unlace or bracket time-domain components of the sound. In Limbo, there is a recurring object called the circular saw. Early in the game, the saw is harsh and noisy, and flutters with a tremolo; however, when it presents itself in the late stages of the game, the harsh noise is filtered down to a single pitch, 'C.' Someone playing the game might not recognize that this sound is attributed to the object if not for the fact that the 'C' is treated with the same modulation (tremolo), which the player would recognize from earlier encounters in the game. Because the tremolo is a dynamic time-domain modulation, it disappears when the user pauses tA/v\Am. Theorists may thus use tA/v\Am to extract the time-domain functions from the pitch or timbre being modulated as I have done in the following figure (figure 3.8):

39

Figure 3.8. Extracting audio components.

In the figure above, knowing that the user will need to pause the video in order to read the large and quickly scrolling paragraph, the theorist directs the user's attention to the sustained sounds that occur when the user kills the tremolo feature by pausing the video. The concept of temporal zooming may also be employed in its opposite incarnation. Not only can the theorist compress text so that the user must 'zoom in' on a moment, but the theorist may also present text slowly, causing the user to 'zoom out.' This method is useful for displaying a narrative quickly, while analyzing the larger section as a whole. Let us return to the pond example. After the player/character re-spawns and solves the puzzle by using the newly deceased character as a second stepping-stone, the theorist explains that the player experiences conflicting sensory cues regarding embodiment and disembodiment between the player and character. See figure 3.9.

40

Figure 3.9. Outward temporal zooming.

Note that the action of the narrative moves quickly; the character traverses a substantial distance while the text makes minimal progress. By the next figure (figure 3.10), the player/character has already begun the next puzzle before the text has finished scrolling (which one may compare with the figure 3.9 above). This temporal zooming outward causes the user to speed up the video to a rate of 200% to witness the action efficiently. In the action, the player/character must cross a gap with a tripwire and a massive block hanging over his head. He then trips the wire, sending the block crashing down before he may escape.

41

Figure 3.10. Quickly moving narrative with slowly moving text; player/character trips wire.

After death, the player re-spawns again and tries to find another way across. The theorist foreshadows the action by reminding the user that the player/character must use dead bodies to solve the puzzle. See figures 3.11-3.12.

Figure 3.11. Player/character re-spawns.

42

Figure 3.12. User watches narrative at 200%.

The player/character drops the dead body into the trap, causing the block to fall while he stands at a safe distance away (figure 3.13).

Figure 3.13. Player/character solves puzzle using dead body.

In the final portion of this analysis module, the theorist explains that despite the conflicting [dis]embodiment cues, the character becomes an extension of the player's 'phenomenal body,' a lived-in body that does not necessitate conscious motor calculations in

43

order to function (figure 3.14). 17

Figure 3.14. Theorist explains the player's experience of controlling a character.

Bracketing Time in Braid In this analytic module, I examine how Braid designer Jonathan Blow creates a set of situations which require the player to bracket time and how tA/v\Am efficiently elucidates the role of the audio in these situations. In Braid, the player controls a young boy named Tim. Borrowing from and frequently referencing the classic video game Super Mario Bros., Blow sets the game as a 2D side-scroller, whereas the immediate objective is to move rightward and pass the obstacles at hand. Like Mario, Tim's ultimate goal is to reach a castle, finding and rescuing the princess. The mechanics of the game differ drastically from Super Mario Bros., however, because Tim is able to reverse time and fix his mistakes. 18 Thus unlike Mario, if Tim falls in a pit, just before death, he may rewind back to the point before he makes the failed jump,

17

18

On the “phenomenal body,” a concept borrowed from philosopher Maurice Merleau-Ponty, see Phenomenology of Perception, trans. Colin Smith (London: Routledge, 2005), 121. A 'physics mechanic,' 'game mechanic,' or 'mechanic' is video game lingo for a set of physic rules unique to a specific game.

44

recalculate the jump, and make additional attempts. Indeed, Tim cannot die; he gets as many chances as he needs. Though this physics mechanic is consistent throughout the game, Blow uses each 'world' within the game to introduce secondary physics mechanics. In the game's fourth world, not only is Tim able to reverse time to fix his mistakes, but his position within the X-axis of the map determines the temporality of the events that take place within the map. As Tim moves rightward, the other characters and objects proceed to run a predetermined course, forward in time; as Tim moves leftwards, the characters and objects rewind. Tim is able to change the predetermined course of action by coming into contact with these characters and objects. If Tim kills one of the monsters (a character exhibiting the traits of a 'goomba' from Super Mario Bros.), but moves leftwards and therefore reversing time, the monster will survive the kill. To make matters more complicated, these time-manipulating mechanics are often aggregated. In the fourth world, Tim may manipulate another character's temporal directionality by both reversing (or not reversing) his own actions, and also by moving left and right within the map. Tim may even reverse a leftward motion, causing these two mechanics to cancel out, resulting in a situation where Tim walks backwards towards the right of the map, while the objects and other characters move forward in time. The player is thus responsible for attaining mastery in all of these time-manipulation domains. 19 Indeed, in addition to discussing the role of sound in Braid, the primary thesis of the analysis is to show how the player must mentally 'bracket off' certain temporal strands in order to solve the puzzles. By this, I mean that when the player consciously reverses time, [s]he must ignore this reversal-action in order to focus on the

19

Before I demonstrate my use of tA/v\Am to analyze Braid, I must introduce yet one more time-manipulation mechanic, which is aggregated with the two introduced above: Objects that flash green are irreversible, i.e., they are immune to the time-manipulation techniques.

45

lower levels of the time-manipulation aggregate. If we return to tA/v\Am, we can see how Blow sets the music for this complex, multidimensional world. Here, the action is as follows: Tim needs to obtain the green key located under the overhang at the bottom left-hand side of the map in order to open the green gate that blocks him from the exit door on the top right-hand side of the map (figure 3.15). Since Tim is too tall to fit under the overhang, he must walk towards the gate, manipulating time so that the monster picks up the key. Tim then must jump into the pit, steal the key from the monster, and reverse time to get out. Once he has the key, he may walk through the gate, completing the puzzle. When the player/character first enters the map, Tim stands still, and thus time stands still as well. Jonathan Blow links the music with Tim's position in horizontal space. If Tim runs from left to right at a normal speed, the sound file associated with the map plays from beginning to end. In the case of this map, the sound file is a recording of a mechanical music box playing a rendition of Johannes Brahms's Lullaby (op. 49, no. 4). The tA/v\Am analysis below is able to demonstrate this concept concisely by simply annotating the map with solfège syllables. This explains how Tim's position in horizontal space acts as a cursor that scrubs through the sound file.

46

Figure 3.15. Braid map annotated with solfège syllables.

As the video continues, tA/v\Am users will see Tim walk under the solfège syllables and simultaneously hear the respective pitch in real-time. When Tim walks leftwards (figure 3.16), the user hears the melody in 'pure' retrograde, i.e., in addition to the order of pitches being reversed, the attack and decay of each note are reversed, such as the phenomenon created when a record is spun counterclockwise.

Figure 3.16. As Tim moves leftwards, the music and sounds play in reverse.

47

When, for example, Tim jumps into the pit to steal the key from the monster, the user hears Tim's horizontal velocity slow down as he reaches the ground and changes direction. As this happens, the audio, likewise, transposes and changes direction. This phenomenon is similar to that of a DJ scratching a record. The user can view the result of this effect in the spectrogram. The dips represent the frequency transpositions that occur when Tim slows down to change directions (figure 3.17).

Figure 3.17. Audio transposes according to Tim's velocity, like a DJ scratching a record.

As Tim reverses out of the pit towards the right, the user will hear the music played forward, because regardless of whether Tim is in reverse or not, the audio is always associated with Tim's position in horizontal space (figure 3.18). 20

20

Also important to note is the fact that when Tim is in reverse, Jonathan Blow adds a color filter, making the visuals look green, which acts as a metaphor for the concept of a red shift.

48

Figure 3.18. A reversal of a leftward motion cancels out, playing music forward.

Thus tA/v\Am allows the theorist to show more and tell less. All that is needed to indicate the concepts described above are a few strategically positioned solfège syllables and a few annotations (as shown below in figure 3.19). 21 Users will not need to read prose describing the game's mechanics or narrative, nor will they need descriptions of the visuals and sounds as is necessary in this document. All of this sensory data is provided to the user directly and in an economical fashion.

21

This analysis module contains yet one more annotation, briefly explaining that one should read these solfège syllables in moveable do. As Tim's speed changes, the pitch level transposes; therefore, the key of the Lullaby is not fixed.

49

Figure 3.19. Minimal annotations.

tA/v\Am Conception Braid played a major role in my thinking and conception of tA/v\Am. In the last level of the fourth world, when Tim reaches the castle and lowers the flag, a friendly dinosaur character called 'Greeter' comes out from behind a rock (figure 3.20).

Figure 3.20. Tim meets Greeter.

Greeter attempts to tell Tim something, but when Tim stops to listen, Greeter stops talking (figure 3.21).

50

Figure 3.21. Text stops scrolling as Tim stops walking rightwards.

The Player/Tim remembers that Greeter's message exists within the function of time, and since Tim's position in space dictates the progress of time for the objects and characters with whom Tim interacts, he must continue walking rightward in order to receive Greeter's message (figure 3.22).

Figure 3.22. Text continues scrolling as Tim walks rightwards.

As Tim progresses rightwards, Greeter proceeds to recite the iconic quote from Super Mario Bros.: “Sorry, but the Princess is in another castle.” Since the process of obtaining this message is contingent on Tim walking rightwards, however, it forces him to rudely walk past Greeter and thus ignore him. This rudeness interrupts Greeter, causing him to say “Hey, where are you

51

going?” See figure 3.23.

Figure 3.23. Tim's rudeness alters Greeter's dialogue.

Any attempt made by the player to politely listen to the Greeter will fail to correct the inevitable rude narrative. Walking back to the dinosaur causes the message to be erased (figure 3.24).

Figure 3.24. Tim tries to apologize.

Therefore, the only option is to advance the predetermined narrative by walking rightward (figure 3.25).

52

Figure 3.25. Tim rudely walks past Greeter.

In designing tA/v\Am, I took inspiration from Blow's elegant use of Braid's physical mechanics as a means to modify the narrative. I developed my application with the anticipation that theorists might use the textual annotations in creative ways. Indeed, in addition to using temporal zooming techniques in order to highlight individual moments, one may write text in reverse so as to cause the reader to watch the action in reverse. Sometimes it is desirable to analyze a scenario by describing an end result first, and then working backwards, examining the cause of that effect. In the following tA/v\Am analysis of LIMBO, the theorist first tells the user that the module should be experienced in reverse, and even says which button to press in order to do so (figure 3.26).

Figure 3.26. Theorist tells user to reverse playback.

53

Once in reverse, the theorist begins by stating the solution to the puzzle (which is to kill the aggressors by luring them into the same traps that have already killed the player/character). See figure 3.27.

Figure 3.27. Theorist explains the goal.

In this analytical module, there are sections that contain two streams of text: one that exists on the top, presented in reverse chronological order, and one on the bottom in chronological order (figure 3.28).

Figure 3.28. Two streams of text.

54

To ensure that the user does not get confused by the anti-chronological text once it appears, the theorist tells the user to ignore the text that is not applicable during a current point in time (figure 3.29).

Figure 3.29. Theorist directs user's attention to the appropriate text.

After raising a question concerning the number of deaths the player/character endured, the theorist tells the user to switch temporal directions in order to reveal the answer (figure 3.30).

Figure 3.30. Theorist tells user to experience the media again forward in time.

The user then takes a second pass through the analysis module reading unique text that describes the events in chronological order. 55

Down the Rabbit-Hole: Bracketing Aggregated Temporal Trajectories As a video game that embraces time-reversing manipulation, Braid seems like the ideal candidate for incorporating an analysis in a similar [reversible] style on a medium such as tA/v\Am; however, since the game already aggregates multiple time-reversing mechanics, tA/v\Am analyses become rather confusing once this new level of time-manipulation is added. Situations arise in which a tA/v\Am user may reverse time, causing triple, and even quadruple temporal negations of the narrative. While this may be undesirable for many practical purposes, I will take this opportunity, regardless, to analyze Braid in the form of an experimental theoretical text. In the following tA/v\Am analysis, we will enter deep into a rabbit-hole of conflicting aggregated temporal trajectories, whereas the user must mentally 'bracket time' in order to understand the analysis, much in the same way that the player must in order to solve the puzzles. In this analysis called Variations, the action unfolds as follows: Tim must collect a puzzle piece located above the two cannons on the right of the screen (figure 3.31). 22 These two cannons produce leftward-moving clouds that Tim must use as stepping-stones to cross. The clouds, however, are too far apart for Tim to make the jump. The solution to this world relies on a secondary physics mechanic: objects that flash green are immune to Tim's time-reversal controls. Thus when Tim pauses time, the green clouds emanating from the lower cannon continue floating leftwards, while the non-green clouds remain in place.

22

See audio/video example here: http://youtu.be/Rpe4MdFrD1M (accessed February 28, 2015).

56

Figure 3.31. The normal way to cross the gap.

Note that in the following image (figure 3.32), the green clouds have made leftward progress, while the non-green clouds are in the same position as the above image. Since the clouds are staggered out of phase at this point, Tim may hop between them to finish the puzzle.

Figure 3.32. Tim fixes phase of clouds by reversing time.

The puzzle is simple and the standard solution is explained, but the theorist here provides other 'solutions' to the puzzle. Most of these solutions, though, are inadvisable and unnecessarily

57

difficult to execute, or are indeed impossible to execute without 'glitching' the game. 23 In the next 'solution' provided by the theorist, when Tim enters the map, the tA/v\Am user will notice that the music is playing in pure retrograde, and both of the clouds are moving rightwards into the cannon. The user will also view the action of Tim climbing the fence and jumping onto a cloud to ride it towards the right of the screen (figure 3.33). Though the user sees Tim easily traversing the clouds to obtain the puzzle piece, the premise of the puzzle is wrong; Tim shouldn't be able to cross the gap so easily because in the game, the clouds are supposed to float leftwards. The user is thus no longer concerned with how to solve the puzzle, but instead becomes interested in learning how the player and theorist manipulated the world in order to create these impossible scenarios. One potential explanation is that Tim crossed the clouds in the standard way as was shown in the previous example and then rode a cloud leftwards towards the beginning of the map. From here, Tim then rewound time in order to reverse the direction of the cloud that he was riding so that it appeared to go rightwards, and it is only this last portion of the scene that the theorist shows. This is an unsatisfying answer, however, because the cloud Tim rode was flashing green, and according to the game's mechanics, objects that flash green are immune to time-manipulation, and therefore should always emanate outwards from the cannon.

23

In gaming, glitching refers to using bugs in the game to cheat. I am using the term falsely to describe the concept of modifying the raw video footage taken from a screen-recording of the player playing the game.

58

Figure 3.33. Tim riding the cloud rightwards.

Next, the theorist reveals the trick: Tim did not use the game's mechanics to reverse time; rather, the theorist used the raw screen-capture footage of Tim riding the green cloud leftwards, and then set it in reverse using a different video editing software before adding the analytical text and importing it into tA/v\Am (figure 3.34).

Figure 3.34. Theorist tells user to reverse media in order to reverse a reversal.

Once the tA/v\Am user presses backspace to play the video in reverse (as was instructed in the theoretical text), [s]he experiences the footage in forward motion. Now, the green clouds are

59

moving leftwards as one would expect, carrying Tim towards the beginning of the map, and the music plays in its prime form. 24 As soon as this temporal layer is 'bracketed' by reversing the medium in tA/v\Am, the temporal dissonance within the game is resolved and everything looks and sounds as it was designed. In the next example, we step one layer further into this rabbit-hole. When Tim enters the map, the user will view the action of Tim climbing the fence, and 'walk on air,' i.e., instead of using the clouds as stepping-stones, Tim simply walks rightwards through the middle of the screen, as if he can fly (figure 3.35).

Figure 3.35. Tim running rightward without the aid of the clouds.

The user is experiencing reversed footage of Tim reversing the following action: Tim walks rightwards across a green cloud, then pauses time when he reaches the end of the cloud, at which point he stands still in mid-air until the next cloud approaches under his feet, whereas he repeats the process until he reaches the cannons. Let us unpack this last statement. As Tim is on a green cloud, he moves rightward while the cloud moves leftwards, though

24

Opposite of 'pure' retrograde, i.e., from beginning to end.

60

just before he reaches the right-end of the cloud, Tim pauses time (figure 3.36).

Figure 3.36. Tim runs across green cloud, stopping just before he reaches the other side.

Since green clouds always move leftwards in this puzzle regardless of Tim's time-manipulation, when Tim pauses time, the next green cloud in the sequence floats towards him, giving Tim another opportunity to make rightward progress. In the following figure, we see Tim floating in the sky; everything is paused except for the green clouds and the tA/v\Am text (figure 3.37).

Figure 3.37. Time frozen in place, hovering in air, waiting for the next green cloud.

61

This process takes many iterations before Tim makes it to the cannons because he only makes rightward progress during the short periods of time in which there is a green cloud under his feet. Otherwise, he stands in mid-air, floating, waiting for the next cloud (figure 3.38). The user witnesses this tedious start-and-stop process. In the following figure, the sonogram illustrates how this would sound for the user.

Figure 3.38. Spectral spikes in sonogram.

Once Tim finally makes it to the cannons, he reverses time. When the user views this action in reverse, however, [s]he sees Tim walking backwards toward the left-hand side of the screen in a continuous fashion. The user does not experience the start-and-stop sensation because the pauses never counted towards Tim's forward-moving action, and therefore were truncated upon reversal. While this concept explains how a player may create the illusion of Tim walking backwards on air towards the left of the screen, it still does not explain the original narrative of Tim walking forwards on air, towards the right of the screen. To explain this phenomenon, the theorist simply stated that to achieve this illusion, one must then reverse the footage of the game before importing the movie file into tA/v\Am. Thus, like the prior example, the theorist screenrecorded the player's actions of the game as explained above, then reversed this raw footage such that Tim seems to walk forwards towards the right of the screen. This reversal of a reversal maintains the original forward/rightward motion, but since the initial reversal caused the pause

62

truncation artifact, the user instead views a continuous rendition of the forward/rightward motion. In order to reveal the trick, the theorist adds yet one more temporal layer, explaining the secondary reversal tactic using reversed text: “footage. reverse Then,” This causes a tertiary reversal, whereas the user reverses the media within tA/v\Am to negate the secondary reversal and view the original reversed scene. In this experimental analysis I have presented, tA/v\Am not only serves as a medium for the analysis, but also as an example—in itself—that supports the thesis of the analysis: One must master the act of mentally bracketing time. Missing from this document is the experience of interfacing with the tA/v\Am media, which much more elegantly demonstrates these dense concepts. Because the user has control over one of the time-manipulation levels via tA/v\Am, [s]he is able to bracket out that temporal strand. When reversing the media, one experiences intentionality towards this act and can therefore switch more easily between the multiple levels of abstraction than one can by reading prose on a textual document.

The Interactive Experience of Video Games and tA/v\Am Just as video games are interactive, so too is the music. When a video game player progresses through a game, the progression of the music that the player experiences is dependent upon varying factors, such as how long the player/character takes to solve a puzzle, how many times the player/character dies, and how the player/character interacts with the objects on the screen. Consider the following scene from Limbo as an example: The boy enters a puzzle with two machine guns, both of which are equipped with lasers that shoot the boy when he trips them. 63

One of the machine guns hangs from the ceiling by a mechanism that causes it to oscillate back and forth along a path when the laser is tripped. The other machine gun is set on the ground facing rightwards. This is the problematic gun, because the player/character cannot outrun it when [s]he steps off of the platform. The solution to the puzzle is to trip the machine gun hanging from the ceiling, and to lure it over the second machine gun, causing the hanging gun to shoot and destroy the second gun (figure 3.39). This puzzle emits a c-minor seventh chord when the player/character approaches it.

Figure 3.39. Player/character hears a c-minor seventh chord when he approaches the machine guns. When the boy trips the laser, the harmony progresses to an F-major chord. The ii7-V sequence creates the expectation of a resolution to B-flat (figure 3.40).

64

Figure 3.40. The boy trips the laser, causing his death, and a harmonic move to the dominant.

Because the puzzle is difficult, one can expect the player/character to die a number of times, thwarting resolution (figure 3.41).

Figure 3.41. Instead of resolving to the tonic, the ii7—V sequence repeats.

65

When the player/character successfully gets the hanging machine gun to shoot the horizontal machine gun, the player experiences the B-flat chord (figure 3.42).

Figure 3.42. The dominant chord resolves when the player/character solves the puzzle.

Had the player/character solved the puzzle on the first try, he would have experienced a standard ii—V—I progression, but instead, he becomes 'stuck in limbo.' The ii—V progression vamps repeatedly as the character dies and resolves only when the puzzle is finished (figure 3.43).

Figure 3.43. The experienced progression.

66

These two possibilities (ii—V—I as well as ||: ii—V :||—I) offer drastically different aural experiences. Indeed, any musical progression within a video game is contingent upon the player's decisions and his/her ability to execute the actions. This last statement is especially true for Braid. I have already pointed out how the act of playing the fourth world may be likened to scratching vinyl records on a turn-table. As the player/character moves back and forth, [s]he is, in a way, performing like a DJ, creating a new progression of sonic events. In an article entitled “Music To My Thumbs: Transcribing Braid,” the editors for the gaming blog, Rock, Paper, Shotgun also noticed this phenomenon. 25 They attempted to notate a possible musical trajectory by analyzing the moves that Tim must make to complete the puzzle. For the player, this transcription becomes the listened soundtrack for that one possible combination of moves—one out of nearly an infinite amount of possibilities. For better or for worse, tA/v\Am presents a similar phenomenon. When users are asked to manipulate time in order to experience an analysis, the user is also altering the experience the player may have had, and thus, experiencing a new formal variation of the topic being analyzed. However, unlike other mediums, tA/v\Am provides users with a substantially similar experience to the experience of playing a video game, and this is specifically due to the incorporation of audio, video, and most importantly, interactivity. Indeed, tA/v\Am is just like a video game. The difference is that in a video game, a player's goal is determined by the game designer (or in some cases, by the user); whereas in tA/v\Am, the user/player's (a tA/v\Am user as a video game player) goal is to obtain the theoretical message provided by the theorist.

25

“Music To My Thumbs: Transcribing Braid,” Rock, Paper, Shotgun, June 20, 2012 (accessed December 17, 2014) http://www.rockpapershotgun.com/2012/06/20/music-to-my-thumbs-transcribing-braid/

67

CHAPTER 4 tA/v\Am ANALYSIS OF MUSIC Because many acoustic music compositions exist in a static graphical format (i.e., textual and graphical symbols engraved on paper), it seems natural that theorists would choose to use a similar medium (paper) to articulate analysis. Indeed, a multimedia analysis engine like tA/v\Am might appear unnecessary; however, I believe that the premise that music exists mainly in a textual/graphical form is a faulty one. When we recall a piece of music, we understand it not necessarily according to its score, but rather, according to its sonic characteristics, i.e., our knowledge of a given piece may be strongest due to a recording or performance. In some cases, an experience of a piece is most vivid in one's visual memory, such as in the case of a performance that one may have seen and heard. A performer's knowledge and experience of a piece, moreover, may even be largely of a physical/bodily understanding that relies greatly on muscle-memory. Acoustic music is a truly multi-sensorial experience, and thus music analysis can greatly benefit from incorporating a medium that can represent as much sensory stimuli as possible. In its facility to display video and play audio, a multimedia platform like tA/v\Am helps to acknowledge these other modes of knowing/experiencing an acoustic composition. Multimedia musical analysis platforms have already proven successful as a means to analyze acoustic music. Pierre Couprie, for example, has developed a number of applications to help theorists analyze music in a multimedia format. 26 His software, iAnalyse, allows theorists to create videos, displaying the score synchronized with the audio. It also provides theorists with tools to annotate these videos with text, drawings, and imported graphics. iAnalyse is in fact the ideal software for generating the type of analysis videos that one could import into tA/v\Am.

26

“iAnalyse 4: aide à l'analyse musicale, création de guides d'écoute” by Pierre Couprie, 2014 (accessed February 5, 2015) http://logiciels.pierrecouprie.fr/?page_id=672

68

Using software like iAnalyse and—or with—tA/v\Am, one can create interactive audio-visual diagrams of acoustic music. In the following analysis of Swiss-Austrian composer Beat Furrer's 1987 solo piano work, Voicelessness (the snow has no voice), I demonstrate how theorists might use tA/v\Am in various ways to 'audio-visualize' musical concepts that they describe. In doing this, I use tA/v\Am not as the sole analysis medium as I did in my chapter on video games, but rather as a supplemental analytical tool enhancing the analysis articulated here in this document. This style of analysis is particularly suited to the music theory classroom, as well as to paper/research presentations at conferences.

Analysis of Beat Furrer's Voicelessness (the snow has no voice): 'Spring Theory' In Voicelessness, Furrer (b. 1954) makes novel use of a notational concept; instead of bracketing each staff into unique systems, he staggers the brackets such that a given staff may serve as the bottom staff of a system during one pass, and as the top staff of a system during a second pass. 27 Figure 4.1 displays the first and last page of this work. Each staff is labeled with a capital letter, and each system with a roman numeral. 28 When the pianist reads this notation, [s]he pairs and plays the first two staves (staves A/B) as a grand staff. After executing this system, before continuing to simply play the following two staves (staves C/D) paired into a system, the performer must play the second and third staves (staves B/C) simultaneously. Thus almost every staff is articulated twice, once paired with the musical material from the staff above (prior), and once with the material below.

27

28

With the exception of the first and last staves, and in one instance where three staves are bracketed into one system. Beat Furrer, Voicelessness: the snow has no voice (Vienna: Universal Editions, 1986), 1, 4.

69

Figure 4.1. First and last pages of Voicelessness; staves labeled with capital letters; systems labeled with roman numerals.

Each staff contains a limited number of pitches, which are articulated and repeated throughout. This provides us with a simple reduction method: we may ignore the rhythms and compile the 70

pitches of each staff into single sonorities (figure 4.2).

Figure 4.2. Each staff is listed sequentially and the pitches are compiled into a single sonority. While figure 4.2 illustrates a progression of sonorities as they appear in the piece, it is important to note that this is an abstract way of viewing the work. These sonorities do not enter subsequently as implied by figure 4.2; rather, they are presented to the listener in an overlapping, fourth-species counterpoint fashion as is illustrated in figure 4.3.

Figure 4.3. Each system is listed sequentially and the pitches are compiled into a single sonority.

71

Figures 4.2 and 4.3 represent two distinct ways of viewing the form of the work; however, I prefer to imagine the latter as a modification of the former. Let us now view the work like a spring (figure 4.4).

Figure 4.4. 'Stretched-out spring' model. Each staff represents a point on a spring. 29 In this stretched-out version, the physical width of the letters do not overlap; however, if one compresses the spring, the letters converge, overlap, and start to resemble the fourth-species counterpoint illustrated in figure 4.3 (figure 4.5).

Figure 4.5. 'Compressed spring' model.

29

Letter 'O' is a bit of an anomaly; it requires an added point because, as one can see in figure 1, it sits in the middle of a three-stave system.

72

In this way, we can see how each staff is paired with its alphabetical neighbor, double-backing onto itself through repetition. With this graph, one may use the system number to index the staves. For example, in system i, A overlaps with B. In system ii, C overlaps with B. Also note that in system xiv, N, O, and P all overlap. Not only do I consider both the stretched and compressed versions of the spring as a metaphor for the form of Voicelessness, but I find that differing tonal implications of the pitch material alter one's reading of the shape of the spring. In the first system, Furrer juxtaposes two dyads, pitches C—E and B—D-sharp. When I hear two major third dyads separated by a half-step for such an extended duration as in the beginning of this piece, I begin to question what I hear as being the tonal center. To ask and answer this question concerning my aural experience of the first sonority, I turn to tA/v\Am (figure 4.6).

Figure 4.6. Question concerning the first sonority posed via tA/v\Am.

When listening to the sonority, I come up with two possible readings. In my first reading (Reading A), I hear the C and E as members of a iv-chord that progresses to B and D-sharp in a quasi-phrygian cadence in e-minor (figure 4.7). 73

Figure 4.7. Reading A of first sonority in Voicelessness.

At first glance, this reading is fairly convincing, because it offers a simple explanation for interpreting the pitch material at hand and because Furrer presents the first sonority in a way that one would expect if one were to read the action as a phrygian cadence. Looking back to figure 4.2 in staves A, B, and C, we find that the C—E dyad progresses to a B—D-sharp, followed by a chord containing pitches found in a natural e-minor scale: E, F-sharp, G, and D-natural (read from the root of the scale upwards). Such a reading enforces a tonality of 'e-minor,' and suggests that the piece begins with a sense of 'C-ness' because the bass pitch of the active chord at the beginning is 'C.' That Reading A exhibits a 'C-ness' and that it fits the model illustrated in figure 4.2, one might prefer to interpret this piece in the abstract 'stretched-out spring' form because this model begins with a C-major dyad. Even though the 'compressed spring' model is articulated in the piece in a more literal fashion, it does not conform with Reading A, which illustrates the C— E dyad progressing to a B—D-sharp, as demonstrated by the 'stretched-out spring' model. With tA/v\Am, one can listen closely and compare this reading to what one hears in the score, as I suggest in figures 4.6 and 4.7 above. In figure 4.6, the user scrubs through a video

74

containing a recording of the piece synchronized with the score, while in figure 4.7, the user scrubs through a focused case scenario of a sounding diagram of Reading A. Hearing and seeing Reading A directly after hearing and seeing the original version allows the user to easily scrutinize how well the interpretation stacks up. Likewise, the user may then examine the alternate interpretation and decide which is more cogent. These interactive audio-visual diagrams thus function as a tool to engage aurally and visually with the analyzed music. In my alternate reading (Reading B), I hear the C and E as members of a Neapolitan sixth chord in B-Major (figure 4.8). At first, this reading does not seem to match up with Reading A, as it requires cross-voicing, and an increase of parenthetical/implied pitches to justify the harmonies—many of which are not found in Furrer's original score. It also requires one to imagine the first two dyads in reverse, i.e., rather than beginning with the C—E dyad as one finds in figure 4.2 or figure 4.4, the piece would need to begin in B-major for this reading to ring true. Indeed, this reading better fits the 'compressed spring' model. As figure 4.5 indicates, we see that at System i, Staff A overlaps with Staff B, meaning one could argue that the order of the two dyads should not matter because when the spring is compressed, the harmonies are juxtaposed. 30 Although Reading B fits the compressed spring model more nicely than the stretched one, it still takes a leap of imagination to argue that the B—D-sharp dyad precedes the C—E dyad, as neither model perfectly exhibits this data.

30

This is similar to serial theory in which a vertical sonority does not indicate pitch class order.

75

Figure 4.8. Reading B of first sonority in Voicelessness.

The question now becomes “Does Voicelessness begin with a sense of 'C-ness,' which would support Reading A and the 'stretched-out spring' model, or is the tonal center of this piece B, which would support Reading B and the 'compressed spring' model?” While Reading A fits more neatly than Reading B, when one scrubs through and listens to the entire piece via tA/v\Am, one finds additional tonal implications instigated by the bass of each sonority, suggesting that the piece in fact has a tonal center of 'B.' This can be illustrated by taking the following harmonic progression (see figure 4.9) of Voicelessness and incorporating it into tA/v\Am.

Figure 4.9. In B major: I—ii—v—I—iv—I—flat-II—I.

This formal analysis diagram of Furrer's entire piece can be synchronized with an audio recording and imported into tA/v\Am as seen in figure 4.10. 76

Figure 4.10. User scrubs through, experiencing the I—ii—v—I progression rapidly.

Because the visual diagram is a compressed representation of this lengthy work, it encourages the user to scrub through the media rapidly, experiencing the whole piece within, potentially, a fraction of a second.

Figure 4.11. User scrubs through the middle of the piece, hearing and seeing a iv—I plagal cadence.

77

This may be seen as an extreme version of the 'temporal zooming-outwards' technique discussed in context of video games, whereas slow text scrolling encourages users to increase playback speed in order to view the accelerated narrative.

Figure 4.12. User scrubbing through the flat-II—I cadence.

Because tA/v\Am allows theorists and users alike to listen to and view a piece within such a short period of time, one can more easily detect long-term tonal progressions. When I listen to Voicelessness through tA/v\Am in this way, the piece is clearly centered on B, articulated by three cadences: the first being a relatively standard ii—v—I (though the 'v' is minor) as seen in figure 4.10. The second cadence is a standard iv—1 'plagal' cadence, and occurs in the middle of the work (figure 4.11). The piece then ends with a non-standard final 'flat-II—I' cadence (figure 4.12). Hearing an entire work as a few short chord progressions via tA/v\Am, theorists gain a global perspective of the work, and with this new perspective, they can make better informed decisions regarding their analyses. I would thus like to propose one last alteration to the 'spring' model: In accounting for these newly discovered long-form chord progressions, I view the spring as having 'crimped ends' (figure 4.13)

78

Figure 4.13. 'Crimped ends' model.

In the 'crimped ends' model, the piece begins with staff B instead of staff A, and likewise, ends with staff R rather than staff S. Not only do the points on the springs at staff B and R contain the evidence supporting B as a tonal center, but they reveal a flaw in my original reduction method. When I reduced each staff to a single block sonority and juxtaposed them, I did not account for the order in which the pitches occurred within the system. If we recall back to figure 4.1, we can see that the first sounding pitches are the B—D-sharp dyad (supporting Reading B, showing a B-major triad progressing to a Neapolitan 6th), and the very last sounding pitch is a low B tone. Thus, the piece is framed by pitches supporting the theory that the tonal center is B. Although staves A and S are the staves that frame the work (as they exist in the first and last positions of the original spring model), the pitch material contained within these staves actually distorts and confuses our reading. We must, therefore consider them not as frames, but as 'tucked-in' or embedded within staves B and R. Although I believe that the 'crimped ends' model is the best version of the three proposed spring models for the data found in Furrer's piece, I do not want to argue that his work can be 79

reduced to a single still image. Instead, I wish to argue that as the piece reveals itself over time, the form of the spring animates through three phases: from stretched, to compressed, and finally crimped (figure 4.14).

Figure 4.14. Three phases of the spring models: Stretched—compressed—crimped.

Phase 1: When the piece begins, we experience the juxtaposed dyads for an extended period of time. The best tonal interpretation of this sonority is Reading A, whereas the C—E dyad (iv) progresses to a B—D-sharp (V) and resolves in a phrygian cadence in e-minor. Indeed, I hear this progression articulated on the local level, and therefore, I imagine the sequentiality of the juxtaposed dyads, which is illustrated in the 'stretched-out' spring model.

Phase 2: As the piece continues, the harmonies fail to remain consistent with Reading A. I then begin to hear more cues suggesting that the piece exhibits a tonal center of B, and that what I 80

heard before as a iv—V—i in e-minor is instead some sort of juxtaposition of a neapolitan sixth with a I chord progressing to a iv chord in B-Major (Reading B). Because I retrospectively recombined those first two chords into a single juxtaposition mentally, I began to imagine the spring as becoming compressed.

Phase 3: When I hear the piece finish, I finally have the contextual perspective to detect the true frames of the work. The final note confirms that the tonal center of the piece is 'B,' and it is this pitch that frames the work despite the fact that it is not found in either the first or last staves. Rather, they are found in the second staff (Staff B) and the second-to-last staff (Staff R). I thus view staves A and S as tucked in, while staves B and R extrude outwardly. As a multimedia tool, tA/v\Am demonstrates this 'spring theory' both visually and sonically. Through the use of sounding interactive diagrams, theorists can efficiently express their analyses of long-form harmonic trajectories. In certain situations, such as a conference paper presentation, theorists do not have the luxury of assuming complete familiarity with a piece on the part of an audience. Although theorists may present audio examples to highlight localized musical ideas, they are constrained by the short time allotment prescribed by the conference presentation slot, and they will certainly not be able to play much more than a short excerpt of the audio, let alone the entire piece. tA/v\Am thus allows theorists to present longform analyses within a matter of seconds. Most importantly, tA/v\Am encourages the audience or user to directly experience the theorist's analysis, and thus obtain a more 'aurally informed' opinion of it.

81

Electroacoustic Music Analysis In addition to developing iAnalyse as a tool for creating audio-visual analyses of acoustic music, Pierre Couprie has developed EAnalysis—a similar tool optimized for electroacoustic music. 31 This software functions like Sonic Visualizer in that it provides users with many ways of representing audio (e.g., transverse wave-forms and spectrograms), and lets them customize these graphics. EAnalysis, however, also hosts an interface with which the user may draw graphic scores in a style similar to that of Rainer Wehinger's iconic visualization of György Ligeti's Artikulation. 32 These graphic scores may then be paired with the audio and [optional] annotations, culminating in an audio-visual graphical analysis. Below is a screenshot in which Couprie uses EAnalysis to analyze Bernard Parmegiani's Ondes Croisée (figure 4.15).

Figure 4.15. Couprie's analysis of Parmegiani's Ondes Croisée.

31

32

“EAnalysis: Sound-Based Music Analysis” by Pierre Couprie, 2010-2013 (accessed February 5, 2015) http://logiciels.pierrecouprie.fr/?page_id=402 See György Ligeti and Rainer Wehinger, Artikulation (Mainz: Schott Music, 1970).

82

In Couprie's analysis, we see four separate representations of the audio. The top segment shows his graphic score of the entire form of the work compressed to fit the width of the screen. He uses symbols to represent various musical objects of parameters, such as the large yellow obtuse triangle that signifies the crescendo that builds to a climax near the end of the piece. Below this formal diagram, we see a more local segment of the piece. At this 'zoomed in' level, we can see how he represents individual sounds with various shapes and colors. The bottom two lines contain the spectrogram and transverse wave-forms of the work respectively. Since all of these various methods of representations are displayed on a single screen in real-time with the audio, it is a valuable tool for giving viewers an immense amount of visual information about a given electroacoustic work in a single video. Although theorists wishing to analyze electroacoustic music could create analysis videos via software like EAnalysis and import them directly into tA/v\Am. I take a different approach in my analysis of my own work, G®¡ND because the electronics are fully notated and performed in real-time by the percussionist. Additionally, tA/v\Am already incorporates a spectrographic representation of the audio, so some of the features of both tA/v\Am and EAnalysis become redundant. Thus I show two alternate ways in which one may analyze a single excerpt of electroacoustic music: 1) audio+score format with analysis narrative, and 2) a filmed demonstration of the performer executing the mixer notation with the score and annotations.

Analysis of G®¡ND G®¡ND is a live electroacoustic work for Alto Saxophone and Percussionist, in which the percussionist realizes the electronics by performing notated mixer motions (figure 4.16). Each fader on the mixer controls the amplitude of a specific sound, except for one fader, which functions as a continuous modulation controller. The thickness of the lines correspond to the 83

intensity, or amplitude of the sounds; thus the percussionist must move the faders up and down to increase/decrease the volume of that channel in rhythm, similar to a sound engineer in a recording studio, live-mixing the discrete channels of a song. The bottom three faders control extremely resonant tones, while the second, third, and fourth faders from the top control glitchy, granulated noisy sounds. In the figure below, we see that the two middle measures must be repeated three times. In this analysis, I am concerned with how listeners perceive the fluctuating lengths of the gestures in the material below, particularly because the repeat signs fall within the middle of a gesture.

Figure 4.16. Electronics: mixer notation.

With youtube channels like Incipitsify, p0lyph0nyXX, and Score Follower, the 'audio+score video' has become a conventional format for presenting new music to audiences. 33 These videos work particularly well in tA/v\Am because one may present users with all of the sensory data of the score and sounds simultaneously, with analytical prose written directly on the score. Using this format, we can begin to understand how in G®¡ND, gestures occasionally straddle repeat signs such that only part of the gesture lies within one looping segment, and that listeners may thus recognize that material has repeated but are unable to determine the cut points. This is due to the fact that certain components of a gesture become associated with the 'wrong'

33

See the following links: https://www.youtube.com/user/incipitsify (Incipitsify) https://www.youtube.com/user/p0lyph0nyXX (p0lyph0nyXX) and https://www.youtube.com/channel/UCsCyncBPEzI6pb_pmALJ9Tw (Score Follower), my own youtube channel.

84

resolution. First, let us define these gestures (figure 4.17). Anytime we see material that unfolds at a rate slower than a quarter-note (crochet), we will call this gesture an 'INHALE.'

Figure 4.17. Defining the 'INHALE' gesture.

When we see fader movement material changing at a faster-paced subdivision of a quarter-note, such as the sextuplet and quintuplet illustrated in figure 4.18, we will consider this an 'EXHALE.' These two gestures, when presented sequentially, form to create an Electric Breath gesture.

Figure 4.18. Defining the 'EXHALE' gesture. 85

Looking back at figure 4.16, there are three electric breaths. The first electric breath fills the entire first measure; the INHALE occupies the first three beats, and the EXHALE occupies the last three beats. The second electric breath occurs within the first five beats of the second measure, split into a two-beat INHALE, followed by a three-beat EXHALE. Starting in the last beat of this second measure is the third and final electric breath. The INHALE lasts five beats and spans across the fourth measure with a three-beat EXHALE. We will call the remaining material beginning with the pickup to the fifth measure and continuing through the sixth measure the 'gated gesture,' as the resonant material quickly snaps on and off. With tA/v\Am, I point to a problematic beat (figure 4.19). The last beat just before the end-repeat sign is supposed to be associated with the gated gesture as described above.

Figure 4.19. The problematic beat: BEATq (beat in question).

The problem with this beat is that it is stuck within a repetition loop, and therefore becomes aurally associated with the INHALE segment of the second electric breath (i.e., the one at the beginning of the repetition). I will refer to this beat in question as 'BEATq.' The repeat signs cause an alteration in the length of the INHALE at the beginning of the repeated material. Indeed, on the first pass, there is no BEATq anacrusis, and therefore the length of the INHALE 86

gesture only lasts two beats. On the second and third passes, the INHALE lasts three beats because BEATq is tagged onto the beginning as an anacrusis (figure 4.20).

Figure 4.20. BEATq is longer during the second and third iterations than the first.

Because of this, listeners will hear the same Electric Breath material three times—one time with a two-beat INHALE and two times with a three-beat INHALE. Thus when listening to the electronics in G®¡ND, one experiences variety despite the direct repetition. Using tA/v\Am in this way, users may read the analytical prose while viewing the score and hearing the audio in a style similar to the sounding diagrams in my analysis of Beat Furrer's Voicelessness. Since the entire score is synchronized with the audio, however, the theorist has the opportunity to write out extensive prose that the user may read while listening to the entire piece. This technique is also particularly advantageous for text that does not necessitate audio synchronization. If, for example, a theorist or musicologist is discussing the work in a more philosophical, historical, or political context, it is not always crucial that the content of the text constantly refers to the score. Yet, tA/v\Am still serves as a medium where all of this information can exist in the same place, whereas otherwise, one would have to follow along to a recording

87

with a score while simultaneously reading the analytical prose. Most importantly, the user has the advantage of controlling the pace of this data. Let us now take an alternate approach to analyzing the same material. In the tA/v\Am display window, Figure 4.21 shows the excerpt of the score, a filmed performance of the mixer, and a screen capture of the software designed to run the electronics. Rather than incorporating dense analytical prose, the score is annotated with minimal text.

Figure 4.21. Alternate approach: filmed performance with score, electronics, and annotations.

As the video plays, the user sees the annotations move corresponding to the audio. The annotations visually bracket out the segments, labeling them according to their function in the gesture (e.g., INHALE or EXHALE), and displays the duration of the segment, measured in the number of beats (figure 4.22).

88

Figure 4.22. First iteration of the repetition.

Comparing figure 4.22 with 4.23, we see how the length of INHALEq (i.e., the inhale in question) extends when it is repeated. The first time (figure 4.22), INHALEq is two beats in length. On the second and third iterations, though, we see how the last beat of the looping window, BEATq, becomes the anacrusis of INHALEq (figure 4.23).

Figure 4.23. INHALEq becomes extended.

89

This repetition technique makes the following gated gesture more effective, as listeners will have just become used to hearing BEATq as part of a breath gesture. At this point, the sudden cutoff of the gated gesture is unexpected, and thus alarming (figure 4.24).

Figure 4.24. Gated gesture.

At the very end of this tA/v\Am excerpt, we see that the theorist lists the entire INHALE/EXHALE sequence (figure 4.25).

Figure 4.25. Compiled list of INHALE/EXHALE durations.

90

If we examine the numbers alone, we see the following pattern: 3323 5333 5333 53... Notice that the first and second lines of this pattern do not exhibit repetition. Then, a 5333 pattern emerges between the second and third lines, as well as the beginning of the fourth. This causes the listener to expect that the pattern will continue throughout the fourth line. Instead, one hears the pattern interrupted by a harsh cutoff. Thus the repeat signs first serve to destabilize repetition by altering the length of the repeating material as described above. Since this is repeated three times, a new pattern emerges/stabilizes, but just as the listener begins to recognize this pattern, it is forcibly interrupted. I find the above implementation of tA/v\Am particularly informative, as it vividly shows what prose tries to tell, effortlessly illuminating a rather non-trivial theoretical concept. It brackets, it highlights, it emphasizes and it signifies by simply existing in its interactive audiovisual format. Furthermore, it shows the performer executing the excerpt, demonstrating how the software responds, and how the electroacoustics sound. It is rich in sensory data, and thus it is an ideal method for analyzing electroacoustic music. As I have shown, however, this implementation is not the only way to use tA/v\Am for theoretical analysis. tA/v\Am is a medium. Theorists may use tA/v\Am in any way to suit their needs. Before I conclude, I will demonstrate an unconventional use of tA/v\Am, exploring how this application may in fact be employed to do more than it was designed to do.

Composer Sketches History (Tracking Changes: Versions/Iterations of the first page) In the saxophone part of G®¡ND, I wanted to incorporate noisy elements such as the key

91

click sound. Rather than leaving the choice of key click sounds to the performer, I wanted to have control over the fingerings of those clicks for two reasons: so that the performer would not default to tapping keys diatonically, and so that I might transform sets of key combinations into more resonant sounds when the saxophonist eventually blows through the instrument. To achieve this goal, I first studied Aaron Cassidy's solo soprano saxophone piece, Asphyxia (1999-2000). In Asphyxia and much of his other work, Cassidy employs a compositional method called 'decoupling.' Decoupling is the process by which two or more parameters of a musical idea are notated separately, usually on different staves. In figure 4.26, we see how Cassidy decouples the mouth and fingers in the first three measures of Asphyxia. 34

Figure 4.26. First three measures of Aaron Cassidy's Asphyxia.

The performer must execute the fingerings written in the bottom staff as per usual, but rather than synchronizing each fingering with a mouthed articulation, the two actions are in rhythmic friction/dissonance with one another, or in other words, rhythmically 'decoupled.' The decoupling technique exhibits an important distinction from traditional notation. Cassidy is interested in notating, or 'choreographing' the actions that the performer is to execute, whereas in

34

Aaron Cassidy, Asphyxia for solo soprano saxophone (copyrighted by the composer, 2000), 1.

92

traditional notation, composers are primarily concerned with signifying the resulting sound of actions. Decoupling notation often causes a degree of unpredictability in the resulting sound. When certain fingerings are executed with a conflicting embouchure setting, the actions may produce an entirely different note, a multiphonic, or in some cases, silence. 35 Like Cassidy, I wanted to explore this area of experimental unpredictability by decoupling the fingers from the mouth. In figure 4.27, we see the first five measures of G®¡ND in one of its earliest stages. This is my very first attempt in creating a decoupled notational system modeled after Cassidy's Asphyxia.

Figure 4.27. First version of opening five measures of G®¡ND.

Since the rhythmic decoupling in G®¡ND is not as complex as in Asphyxia, I consolidated the Mouth and Finger staves into a single staff (figure 4.28). In doing this, I assigned the 'x' symbol for the keys because the process of fingering the pitches would result in noisy click sounds. The embouchure then floats below the staff (without ledger lines). The percussion part here requires two staves titled Left Hand and Other Limbs.

35

For example, if an extremely high pitch is notated in the fingers staff, combined with insufficient air support notated in the mouth staff, the high note will likely fail to speak.

93

Figure 4.28. Second version of opening five measures of G®¡ND: consolidated saxophone notation.

This notational system gave rise to several concerns. I found that the saxophone part was now more confusing, as it was unclear whether the 'x' note heads signified the action or the resulting sound. Adding to the confusion, many of the fingered pitches lay above a D5 (transposed); therefore the saxophonist would be depressing the octave key despite the fact that it has no effect on the resulting sound. This means that it would make more sense to transpose everything above a D5 down to the bottom octave, which would be a more accurate representation of how the saxophone would resonate. More notably, I found that playing these quarter tones in such a quick and un-patterned sequence proved unnecessarily difficult to execute, when my initial goal was only to avoid diatonic key clicks and to have control over the tone of these clicks. I thus chose to create a tablature by notating the actions of each finger (figure 4.29). Much in the way that a composer might notate the fingering of a multiphonic above the staff to signify the means of producing the sound, I began the process of formalizing a fingering system for every note. Each filled-in black note head represents the finger to be depressed. White note heads signify releases. I use thick black lines to represent keys that are held-down over an extended period of time (closed state) and a thin black line to show extended releases (open state). 94

Figure 4.29. Third version of opening five measures of G®¡ND: tablature saxophone notation.

After re-notating every note in the piece using this tablature system, I found that it was somewhat difficult to tell the difference between a beam and a depressed note-extension, as both were represented by a thick black line. I thus changed the extension lines to a gray color (figure 4.30).

Figure 4.30. Fourth version of opening five measures of G®¡ND: gray extension lines.

Meanwhile, in the percussion part, I was disappointed with the fact that the left hand took so much precedence over the 'other limbs.' I consolidated the percussion into a single staff like the saxophone tablature, which also allowed me to increase the size of the snare range (figure 4.31).

95

Figure 4.31. Fifth version of opening five measures of G®¡ND: consolidated percussion notation.

Comparing figure 4.31 with 4.32, we see that there are more extension lines in the latter. The problem with the former was that there were instances in the piece where individual fingers would change open/close states over a longer period of time. Reading the piece, it was difficult to determine which fingers these note heads corresponded to when there was so few horizontal lines used as a visual reference for height. I thus implemented horizontal lines to signify the open/close state of each finger for every point in time.

Figure 4.32. Sixth version of opening five measures of G®¡ND: filled-in extension lines.

In figure 4.33, we see that I simplified the second gesture in the saxophone part while keeping the same rhythm. I found the original gesture too visually convoluted.

96

Figure 4.33. Seventh version of opening five measures of G®¡ND: re-composed second saxophone gesture.

At this point in the notational process, the tablature is set. The following two figures show the final modifications of the piece. In figure 34, we see added grace notes in the percussion part and the meters consolidated to the center of the system.

Figure 4.34. Eighth version of opening five measures of G®¡ND: centered meter text and added grace notes.

In figure 4.35, we see added repeat signs, text, phrase markings, and even more material for the embouchure.

97

Figure 4.35. Ninth version of opening five measures of G®¡ND: final iteration.

With tA/v\Am, I created a similar 'composer sketches history' in an interactive audio/visual setting, showing the notational progress over time. I imported images of every version/iteration of the first page of the score and set them to flash before the viewer's eyes in order sequentially (figure 4.36).

Figure 4.36. Composer sketches in tA/v\Am.

The result is something of a digital flip book; users see every change in the score animate before their eyes over the course of a few seconds. In this way, tA/v\Am can store an immense amount of information about a piece economically, because the information can be encoded as an 98

extremely short interactive video (figure 4.37).

Figure 4.37. Quickly flashing pages like a digital flip book.

Throughout figures 4.27-4.35, I exhibited only nine iterations of the score. With tA/v\Am, one may easily display hundreds of versions of the score, showing the many states in-between. We can view this as having a high sample rate of snapshots of the score, and therefore having a higher resolution of the animated changes. For example, figure 4.38 shows the intermediary state of the score that took place after figure 4.32, but before figure 4.33. In figure 4.38, the note heads of second gesture (mm. 3-4) were simplified; however, the extension lines are not yet adjusted to accommodate this change.

99

Figure 4.38. Many states of the score.

When played at slower speeds, users can extend their experience of the flipping pages and track every minute detail of a composer's creative process, revealing the immense quantity of information over the course of hours if they so wish (figure 4.39).

Figure 4.39. User may slow tA/v\Am to study the quickly flashing pages.

100

As in a flip book, tA/v\Am shows the score assembling by itself. Instead of having to compare the changes by looking back and forth between the figures presented here, objects simply appear in place, such as does the repeat sign in figure 4.40.

Figure 4.40. tA/v\Am animates the changes.

At the end of the video, users will see the final version of the piece, at which point they may pause it, scrub backwards, or let it loop again (figure 4.41).

101

Figure 4.41. Final state of the G®¡ND as viewed through tA/v\Am.

tA/v\Am may thus be employed in musical analysis in a variety of ways. Whether as an interactive sounding musical diagram, an audio+score+prose platform, a filmed performance with live annotations, or as a digital flip book to display a composer's sketch history, tA/v\Am provides music theorists with a valuable multimedia outlet for conveying analysis. The analyses that I have presented throughout this chapter demonstrate how tA/v\Am simultaneously brings “the reader” into contact with the theorist's analysis and most importantly, the multimedia. As I have argued, music, even purely acoustic music like Furrer's Voicelessness, is not simply a collection of graphical symbols awaiting textual analysis and explanation. Instead, music is visual, aural, and tangible, and thus deserving of a medium that can account for its many facets. In the case of pieces that involve live electronics, in which another medium (the speaker) is added, a multimedia analysis platform becomes all the more useful. In my piece, G®¡ND, for example, the electronic part in the score only shows the actions that are to be executed by the performer in a tablature format; it does not show the resulting sounds. tA/v\Am's ability to

102

generate a symbolic representation of these sounds via the real-time FFT spectrogram helps to display this missing data, making the aural visible.

103

CHAPTER 5 CONCLUSION In an interview explaining how he designs levels in video games, Edmund McMillen (codesigner/creator of Super Meat Boy) advocates the value of non-textual communication in teaching the mechanics of a game to the player. 36 Rather than using text, he presents his players with a simple obstacle, say, an excessively large gap (figure 5.1).

Figure 5.1. Large gap in Super Meat Boy.

The only way to get across this gap is to hold down the 'run' button and jump. Since this problem is presented without instructional or tutorial text, the player must figure out the action for him/herself. McMillen asserts that it is better for the player to learn the action physically, because as he points out: “Most people will just skip the text, and they will forget.” 37 McMillen's words suggest that in many cases, players would rather learn through the act of discovery, rather than through reading. McMillen suggests, moreover, that players better retain this discovered or experienced knowledge. Indeed, when the player is presented with a similar issue later in the

36

37

Edmund McMillen in the documentary film Indie Game: The Movie. See Lisanne Pajot and James Swirsky, Indie Game: The Movie (2012). Ibid.

104

game, [s]he will be more likely to remember the action required to solve the puzzle, because [s]he has already spent the time cogitating it. Super Meat Boy is just one of many recently released games pursuing this textless method of learning (particularly within the independent game sphere). In indie games like Minecraft, No Man's Sky and Jonathan Blow's upcoming game, The Witness, the player spawns onto a map without any explanation. The goal of such games is thus to make sense of the environment, and gradually learn the mechanics through discovery. In fact, Blow criticizes mainstream industry game designers, charging that they often “eliminate the joy of discovery.” 38 Commercial game designers “don't just give you a simple situation and let you work it out, they explicitly tell you what to do,” he says. Blow prefers “that click that happens in your head between you see something and you don't quite understand it and suddenly you do understand it, and that is a fundamental part of human existence in the world.” At the end of chapter 3, I argued that tA/v\Am is like a video game platform, in which the theorist takes on the role of the game designer by crafting an analytical experience, and the user becomes the player, actively participating in discovering the theorist's ideas by scrubbing through the media. While I have not yet found a way to completely remove textual or verbal explanation in a tA/v\Am-style analysis, I find that this new medium is consistent with the trends set by these independent video game designers in its ability to suppress our reliance on text, while still being able to communicate complex and nuanced concepts. Furthermore, it is a multimedia platform that could potentially accommodate for a textless mode of communication, should the theorist wish to work in this direction. tA/v\Am is not an analysis machine, as its name perhaps suggests, but rather an open

38

Jonathan Blow in “Jonathan Blow: How Mainstream Devs are Getting it Wrong,” March 8, 2012, http://youtu.be/I1Fg76c4Zfg (accessed February 8, 2015).

105

medium, to be employed as the theorist sees fit. Throughout this dissertation, I have only begun to exploit tA/v\Am's potential uses for film, video games, and music analysis. The possibilities may in fact reach beyond what I have imagined. Throughout his book The Singularity is Near: When Humans Transcend Biology, futurist Ray Kurzweil optimistically advocates the advantages of full-immersion audio/visual/tactile technology in its capacity to convincingly transmit human knowledge. 39 In fact, he imagines a decentralization of the education system as institutions continue to make more content available online (e.g., MIT's OpenCourseWare), and he believes that students will eventually be able to attend class in a virtual-reality environment. While tA/v\Am is not yet streamable online, nor is it at the level of virtual-reality, it does represent a step in the direction towards achieving this goal. It incorporates the audio, the visual, and the tactile, but most importantly, it is a platform to transmit human knowledge. While many skeptics worry that Kurzweil's visions may render humans obsolete, I find that tA/v\Am is a healthy way to engage with technology in that it does not replace the human (theorist). 40 The theorist is still doing the thinking, still crafting the argument, but taking advantage of technology to get that argument across. Finally, because this document contains nearly as many images as it has pages, it demonstrates the inefficiencies of the paper medium, advocating for a medium that can elegantly accommodate for the rich experience of dynamic multi-sensory art. If we continue taking these steps, we might someday reach a point where we no longer have to 'look through a glass darkly,' and instead, simply look, listen, and touch the media.

39

40

See Chapter 6 in Ray Kurzweil, The Singularity is Near: When Humans Transcend Biology (London: Penguin Books, 2005). Steve Jones, Encyclopedia of New Media: An Essential Reference to Communication and Technology (New York: Sage Publications, 2002), 276-277.

106

Looking Forward tA/v\Am is still a prototype. It currently exists as an application designed in Max, and it contains the basic functionality required to turn fixed analysis videos into the type of interactive analyses described in this document. Because Max itself is a graphical interface for creating live multimedia applications, it comes with some undesirable computational overhead. Although processing power and memory space are increasing at a consistently accelerated rate, tA/v\Am is, and will continue to be computationally expensive due to the real-time granular synthesis module and the video buffering issues. Until tA/v\Am is ported to a lower level programming language, it will remain bloated. tA/v\Am currently lacks an interface for generating analysis videos. Because of this, it is not particularly user friendly, as users must rely on third-party software, such as iMovie and Windows Movie Maker, to create subtitled videos. An ideal design for tA/v\Am would incorporate an 'edit mode' and a 'presentation mode,' similar to software like Powerpoint or indeed Max. Theorists could then build their analysis videos in one software, eliminating the need to create analysis videos in advance using different software. tA/v\Am might also include an interface enabling the theorist to draw, as well as edit audio and score images to make musical analysis videos—something similar to that in Pierre Couprie's iAnalyse. When ready to present, the theorist or user could then toggle the software into presentation mode, at which point, the granulation and FFT engines would turn on, allowing the user to interface with the video by manipulating the playback rates as tA/v\Am was designed to do. Finally, tA/v\Am could benefit immensely by being available online as a web-based application. In this scenario, a theorist could create an account online, build analysis videos with tA/v\Am's 'edit mode,' and publish these videos for online streaming. Like youtube, anyone could search for and play a video. Unlike youtube, the video player would exhibit tA/v\Am's 107

functionality. Despite its current shortcomings, tA/v\Am has much potential, and I look forward to continuing to improve (and perhaps eventually market) the application.

108

BIBLIOGRAPHY Barthes, Roland. Image Music Text. Edited and translated by Stephen Heath. London: Fontana Press, 1977. Cheng, William. Sound Play: Video Games and the Musical Imagination. Oxford: University Press, 2014. Chion, Michel. Audio-Vision: Sound on Screen. New York, NY: Columbia University Press, 1994. Coffey, Edward John. “The Gentle Guerrilla: Composing Transmission in the Performing Arts.” PhD diss., Princeton University, 2006. Couprie, Pierre. “iAnalyse 4: aide à l'analyse musicale, création de guides d'écoute.” 2014. http://logiciels.pierrecouprie.fr/?page_id=672. Accessed February 5, 2015. ———“EAnalysis: Sound-Based Music Analysis.” 2010-2013. http://logiciels.pierrecouprie.fr/?page_id=402. Accessed February 5, 2015. Collins, Karen. Playing with Sound: A Theory of Interacting with Sound and Music in Video Games. Cambridge, MA: MIT Press, 2013. Cox, Christopher and Daniel Warner, ed. Audio Culture: Readings in Modern Music. London: Bloomsbury Publishing, 2004. Damnjanovic, Ivan, Dan Barry, David Dorran, and Josh Reiss. “A Real-time Framework for Video Time and Pitch Scale Modification.” In Proceedings of the 11th International Conference on Digital Audio Effects. Espoo, Finland (September 1-4, 2008). Dolar, Mladen. A Voice and Nothing More. Cambridge, MA: MIT Press, 2006. Furrer, Beat. Voicelessness: the snow has no voice. Vienna: Universal Editions (UE18942), 1986. Heidegger, Martin. Being and Time. New York, NY: Harper & Row, 1962. Holmes, Thom. Electronic and Experimental Music, Third Edition. New York, NY: Routledge, 2008. Husserl, Edmund. The Phenomenology of Internal Time Consciousness. Bloomington, IN: Indiana University Press, 1964. Klapuri, Annsi and Manuel Davy. Signal Processing Methods for Music Transcription. New York, NY: Springer, 2006. Kokoras, Panayiotis. “Towards a Holophonic Musical Texture.” In Proceedings of the International Computer Music Conference ICMC. Barcelona, Spain (2005).

109

Lerch, Alexander. An Introduction to Audio Content Analysis. Hoboken, NJ: John Wiley & Sons, 2012. Li, Tao, Mitsunori Ogihara and George Tzanetakis. Music Data Mining. Boca Raton: CRC Press, 2012. Liu, Su-Houn, Hsiu-Li Liao, and Cheng-Jun Peng. “Applying the Technology Acceptance Model and Flow Theory to Online E-Learning User's Acceptance Behavior.” In Issues in Information Systems, 6 no. 2 (2005): 175-181. Luft, Sebastian and Soren Overgaard. The Routledge Companion to Phenomenology. London: Routledge, 2011. Marini, Anton. “Jitter Movie Playback Optimization.” November 29, 2007. http://abstrakt.vade.info/?p=147. Accessed January 21, 2015. Merleau-Ponty, Maurice. Phenomenology of Perception. Translated by Colin Smith. London and New York: Routledge, 1962. Miller, Izchak. Husserl, Perception, and Temporal Awareness. Cambridge, MA: MIT Press, 1984. Miller, Kiri. Playing Along: Digital Games, YouTube, and Virtual Performance. Oxford: Oxford University Press, 2012. Monroe, Tim. QuickTime Toolkit, Volume One: Basic Movie Playback and Media Types. San Francisco, CA: Morgan Kaufman, 2004. Moon, Barry. “Baz Tutorial 15 — Part 1.” Uploaded May 21, 2009 https://www.youtube.com/watch?v=GR_bLnMNH-4. Accessed January 21, 2015. Müller, Meinard. Information Retrieval for Music and Motion. New York, NY: Springer, 2007. Roads, Curtis. The Computer Music Tutorial. Cambridge and London: The MIT Press, 1996. Sacher, Jan and Phillippe Kocher. “Ambisonic Spatialization Tools for Max/MSP.” In Proceedings of the International Computer Music Conference ICMC. New Orleans, USA. (Nov. 2006). Schaeffer, Pierre. In Search of a Concrete Music. Translated by John Dack. Berkeley, California: University of California, 2012. Schwarz, David. An Introduction to Electronic Arts Through the Teachings of Jacques Lacan. London and New York: Routledge, 2014. ———Listening Awry. Minneapolis, MN: University of Minnesota Press, 2006. Seamon, David and Robert Mugerauer. Dwelling, Space & Environment: Towards a Phenomenology of Person and World. Malabar, FL: Kreiger Pub Co, 2000. 110

Smalley, Denis. “Spectro-morphology and Structuring Processes.” In The Language of Electroacoustic Music, edited by Simon Anderson. London: MacMillon, 1986. Smith, Julius O. Mathematics of the Discrete Fourier Transform (DFT), Second Edition. W3K Publishing, 2007. Truax, Barry. “Real-Time Granular Synthesis with a Digital Processor." Computer Music Journal, no. 12 (1988): 14-26. Witten, Ian H., Frank Elbe, and Mark A. Hall. Data Mining: Practical Machine Learning Tools and Techniques. Burlington, MA: Morgan Kaufmann, 2005. Xenakis, Iannis. Formalized Music: Thought and Mathematics in Composition. Bloomington and London: Indiana University Press, 1971. Žižek, Slavoj. Looking Awry: An Introduction to Jacques Lacan through Popular Culture. Cambridge, MA: MIT Press, 1991.

111

PART II SCORE

112

113

FO R EV

R IE W

N

O

LY

LY

G®¡ND

for Alto Saxophone, Percussion and Electronic[s/-technician] (read ...'Electronics' or 'Electronic-technician')

N

by Dan Tramte written for Patchwork duo

O

PERFORMANCE NOTES

IE W

-The systems are bracketed into two parts: Alto Saxophone, and Percussion—both requiring the performer to realize multiple staves simultaneously. -An optional third performer (the Technician) may assume the responsibility for the mixer position, and thus execute the respective staff titled 'Mixer.'

EV

Alto Saxophone

Technician (optional)

R

-Medium (c. 3-5 seconds) regardless of note value

R

Percussion

-Short (c. 1-3 seconds) regardless of note value

FO

-Long (c. 5-7 seconds) regardless of note value

© Dan Tramte | 2014

114

Alto Saxophone

LY

The top half contains the fingerings to execute. Filled circular note heads followed by a thick gray line designates keys to be depressed. Open circular note heads followed by a thin gray line designate open keys. Likewise, the closed (reversed) oval note heads containing notenames in white text indicate the pinky-controlled keys to be depressed, and the open (reversed) oval note heads designate open pinky keys.

N

-For a majority of the work, the saxophonist is provided with a fingering tablature accompanied by an embouchure clef written directly below the dividing line.

IE W

O

***When executing these fingerings, some audible level of percussive 'tapping sounds' produced by the keys and pads will always exist. These tapping sounds are highly desirable, and should always be emphasized in G®¡ND. Even during the softest sections of the work (the very beginning for example), the performer should execute the fingerings such that the percussive tapping sounds are played as loud as possible.

The bottom half contains embouchure notation. There are three primary note heads: filled (ord. pitched tone), diamond (air sound), and accent (light slap). Arrows designate a transformation from one sound to another (often between air and pitch sounds). These transformations often contain a dotted tie, which helps to rhythmically show the transitioning. Stems with omitted note heads are rhythmic place-holders EXCEPT when marked with tenutos, in which case, they should be interpreted as diaphragm pulses.

R

EV

-Occasionally, a 'bite map' staff is combined with the above

FO

R

The vertical mouthpiece: a 'bite map' which illustrates where the performer is to bite on the reed. The saxophonist is to use the central incisors of the lower teeth to lightly bite the reed where indicated (with a thick, black line). This will cause the reed to bend, bifurcating the reed into two shorter segments. The shortened reed will thus vibrate at a much higher frequency (squeak) when the appropriate air pressure is applied.

115

N

LY

-Mm. 99-116 displays a normal Alto Saxophone 'sounding' part in addition to the Sax. Tab.

IE W

O

Though the Sax. Tab. alone is successful in representing sounds that a normal 5-line staff cannot show, it fails to clearly represent the partials to be emphasized during passages containing multiphonics. During these measures, the performer should consider this normal 5-line staff as the sonic goal, whereas the accompanying Sax. Tab. simply shows the actions required to articulate it.

EV

-At m. 165, the saxophonist is instructed to detach the mouthpiece from the saxophone and use the mouthpiece as an instrument in itself.

R

R

The horizontal mouthpiece (also paired with embouchure notation): The performer should set the reed in the mouth as per usual, and use the palm of the hand to open/close the hole on the other end of the mouthpiece (according to the open/filled note heads respectively). The high pitch generated by the mouthpiece will be muted when the hole is covered with the hand. The score also illustrates half-muting with a halffilled note head followed by a medium-thick gray line.

FO

The mouthpiece should not be connected to the neck of the saxophone. This staff is also sometimes paired with the bite map staff.

116

LY

-By m. 206, the saxophonist will have set away the mouthpiece and will use the rest of the saxophone (body + neck) as an

IE W

O

N

The saxophonist should blow a focused stream of air directly into the neck of the saxophone. Lips should be approximately 1 to 2 inches away from the hole of the neck. In mm. 209, 214, 216, & 218-219, the saxophonist is asked to lift the saxophone such that (s)he may blow a stream of air into finger hole #5 (the hole just under the pad controlled by the index finger of the right hand). This will create an airy tremolo effect when the performer is simultaneously asked to open & close key #5.

-The purposes of the decoupled multi-staff action notational method is as follows:

2) To clearly specify multiphonic fingerings.

EV

1) During quiet moments—of which the clicking of the keys/pads against the body of the saxophone are particularly audible—the written key combinations will result in controlled tapping sounds that avoid diatonicism. These fingerings are developed thematically and paired with different methods of soundproduction, thus requiring formalization.

R

3) To represent sounds that would otherwise be impossible to represent. In mm. 25-27 for instance, the fingering produces an unstable tone, which when executed at one dynamic (near niente) acts differently than when executed at a different dynamic (mezzo-piano). This unstable tone may indeed squeak, or at least threaten to squeak. Whether or not a squeak is achieved during a performance is less important than the performer's ability to establish the tense (or resonant, or muted, or full-bodied, or choked, or crisp, or fragile, etc.) state that the respective fingering TENDS to produce.

FO

R

4) Unpredictability (as implied above). Since G®¡ND is more of an action score than a sounding-result score, a semiotic dissonance between the score and the aural product will persist. Performers are generally more comfortable executing a score when they know they can regularly reproduce a sonic goal. When this sonic goal is removed from the score, the player may feel like a puppet, blindly moving muscles, unaware of whether or not (s)he is producing a sound that is anywhere near the sound the composer had in mind. The level of unpredictability in G®¡ND, however, is minimal compared to a score by, say, Aaron Cassidy, who has a significantly different musical objective. In G®¡ND, sounds are much more reproducible, but with slight variability. For example, m. 22 played correctly should result in what sounds like a mediated bisbigliando on pitch-class 'C' with airy terminations. What is not reliably determined is how the tone will arpeggiate through the partials by the beginning of the second beat. This should not be cause for concern, and indeed, it does not have a drastic effect on the overall gesture. The saxophonist's goal is to try to internalize these sounds (and the variability associated with them) during rehearsals. Over time, given sufficient practice, the saxophonist will likely create new semiotic links between the action notation and the aural result such that the two will be indistinguishable in the performer's mind.

117

LY

Percussion -The setup should resemble a drum set:

-Required instruments & objects: (1) Snare Drum (with a grainy-textured batter head coating)

N

(1) Tom-tom (mounted or floor) (1) Floor tom

O

Music stand

(1) Kick Bass Drum

Mounted tom Kick drum

IE W

(1) Wooden bar stool (or any smooth/stained wooden surface)

Mixer

(x) A box of assorted wind up toys (as many as possible) (1+ extras) Styrofoam block

(1+ extras) Super ball mallet

R

(1+ extras) Wooden stick

EV

(1+ extras) Styrofoam cup (not a mug as the symbol suggests)

(1+ extras) Sewing needle or chopstick

FO

R

(1) nanoKONTROL2 connected via USB to a computer running _G®¡ND.maxpat in Max 6 or Max Runtime 6. Computer should face the percussionist for monitoring purposes†

† This is only in the case that the percussionist plans to play the mixer part

118

Box of assorted toys

Snare drum

Floor tom

Bar stool

-The membranophone staff:

O

N

LY

The piece begins with a 2-line staff—though this should be viewed as a 1-space staff, whereas the space between the lines represents the head of the drum. The bottom line corresponds to the point of the rim nearest the performer's body, while the top line corresponds to the farthest point of the rim. This allows for the specification of a range of playing positions on the drum head in the form of an imaginary line from the front to back of the membrane; however, the the percussionist need not strictly constrain his/her playing to this line. Any 'diameter line' (rotated, a bit offset or even slightly curved) would suffice, and indeed may be necessary, especially in cases where multiple actions occur on the same membrane.

EV

IE W

The quantity of lines of the staff depends on the number of membranophones used within a given period of time. In the most extreme example, mm. 63-76, 79, & 83-84, the score calls for a snare drum, tom-tom, and floor tom, represented by a 4-line/3-space system. Any subset of these three drums is possible throughout the piece, therefore the performer must rely on the clef symbols to know which drum to actuate.

R

Since the bass drum is only ever struck with the kick pedal, it only requires a 1-line staff.

R

-Other notations:

FO

Crank a wind up toy to be set on a membranophone. Rhythms indicated in score are merely suggested. Some wind up toys will inevitably require more or less cranks than written.

119

Friction [noise] attacks

LY

-Produced by a styrofoam block, represented with a straight line, and always executed on the snare.

O

N

Rub a block of styrofoam across the drumhead membrane to create a stable stream of soft white noise. Typically, the dynamic written below the passage will correspond to the notated velocity of the styrofoam (i.e., fast passages spanning the entire diameter of the drumhead will be accompanied by a dynamic of f ); occasionally, however, the percussionist must adjust the pressure of the styrofoam against the drumhead to achieve the written dynamic. Higher pressure will increase dynamics, and vice versa.

IE W

The percussionist should not try to emphasize the beginning or end (or apply any other un-notated idiomatic interpretations) of these friction attacks. Usually the styrofoam block should rest on the snare head, and therefore remain in contact. The performer must avoid lifting/striking, and instead, move the block across the head with appropriate pressure.

Friction [noise—resonance] attacks

EV

-Produced by either a super ball or a styrofoam cup; always represented with a curvy line, terminating with an arrow.

R

The curvy lines indicate the gesture to be played on the surface. The loops were designed to designate where the performer must increase velocity, as such a gesture would require.

FO

R

-Super ball friction attacks may be applied to any membranophone.

Draw the super ball across the membrane such that it produces a resonant sound. In a few instances, such as m. 4, other objects are simultaneously resting on or exciting the same membrane, thus potentially dampening it. In these cases, attempt to produce some resonance, but do not be concerned if the attempt fails. Given the right pressure, a super ball may instead produce a noisy tremolo as it resonates at a slower frequency (< 20 Hz), bouncing rapidly across the drum. This is also desirable and the performer may use this technique when the membrane is muffled by other objects.

120

LY

-Styrofoam cup friction attacks are always executed on a wooden bar stool (or acceptable wooden surface substitution)

N

Lid-side down, with light pressure against the wooden surface and a gentle grip, slide styrofoam cup over the finished surface of the wood to cause the cup to resonate with a rather loud squeak. When not in use, do not set the cup away; rather, leave cup on the bar stool for quick and easy access.

IE W

O

-It is not expected that the super ball or the styrofoam cup will produce a purely steady resonating sound throughout the entire duration notated. Both of these techniques are naturally unpredictable. Indeed, G®¡ND is a study in the unpredictability and tension caused by the friction of two objects/bodies. The work is meant to straddle the tipping point at which noise becomes resonance.

Percussionist (continued)/Technician

EV

-Either the percussionist or an on-stage technician may perform the electronics using a nanoKONTROL2 mixer connected via USB to a computer running _GRIND.maxpat in Max 6 or Max Runtime 6. To monitor the electronic output, the computer screen must be visible to the percussionist/technician.

R

A) Granular scrub (controlled by the dials)

FO

R

-The mixer is split into two separate parts:

B) Sound generators (controlled by the faders)

(A more detailed breakdown of the mixer controls are available in the info section of the software, _GRIND.maxpat)

121

LY N O IE W EV

-Unpredictability

Each sound generator slowly mutates over time. This means that the sound will vary slightly depending on when the fader is lifted (and also depending on how much time elapsed since the percussionist/technician turned on the audio of the patch); however, one can expect for each sound generator to be fairly consistent at any given point.

R

-Sounds produced by the dials are set to relatively low levels; they are not balanced with the faders.

R

-Granular Scrub & Improvisation

FO

There are five sections in G®¡ND marked 'Granular Scrub,' which are short interludes designated for electronic improvisation. During these sections, the percussionist/technician should only use the dials of the nanoKONTROL2 mixer, and make sure to leave the faders set in the indicated position. A duration is listed above each dashed box designating how long the improvisation should last. These time periods are extremely flexible. The percussionist/technician should not count seconds or watch a clock, and instead, estimate.

122

LY

-Granular Scrub & Improvisation (continued)

N

At the end of each granular scrub section, the performer should cut off all sounds created by the dials. To do this, either set the three dials furthest to the left (dials 1, 2, & 3) to '0' by turning them all the way counterclockwise, or set the 'master dial' (dial 8) to '0.'

O

Brackets illustrated over the dials indicate the dials that are available for improvisation Dials marked 'X' may NOT be used in improvisation.

IE W

Dial 8 and occasionally dial 4 are marked with a preferred setting that is to be used during the duration of the improvisation.

In this case, set dial 4 to 10 o'clock and dial 8 to 12 o'clock Improvise on dials 2-3 & 6-7, not dials 1 & 5 (dials are numbered 1-8, left to right respectively)

EV

How to improvise -There are no rules for improvisation, only guidelines.

FO

R

R

• Search for an inspiring sound by navigating through the sound files using the rough scrubbing window dial. With the respective fine scrubbing point dial, explore the sound. Scrub through it at different speeds. Set the point somewhere within the sound and leave it there for a few seconds. • With the exception of Granular Scrub #5, there will always be at least one fader engaged, producing a synthetic sound. Search for granular scrub sounds that support the sound produced by the fader. Find complimenting sounds. Find dissonating sounds. • Turn dials back and forth rapidly, creating short, choppy sounds. • Try to create gestures similar to those found in the rest of the piece. • Do something drastic, such as finding one shrill and annoying sound to last for the entirety of the imrpov. section. • Consider silence • Perhaps do nothing!

-Granular Scrub #5 should contain lots of space and extremely quiet sounds. The following gesture at m. 173 should be incredibly frightening to listeners.

ALWAYS REHEARSE WITH EARPLUGS

123

124

FO R EV

R IE W

N

O

LY

125

FO R EV

R IE W

N

O

LY

126

FO R EV

R IE W

N

O

LY

127

FO R EV

R IE W

N

O

LY

128

FO R EV

R IE W

N

O

LY

129

FO R EV

R IE W

N

O

LY

130

FO R EV

R IE W

N

O

LY

131

FO R EV

R IE W

N

O

LY

132

FO R EV

R IE W

N

O

LY

133

FO R EV

R IE W

N

O

LY

134

FO R EV

R IE W

N

O

LY

135

FO R EV

R IE W

N

O

LY

136

FO R EV

R IE W

N

O

LY

137

FO R EV

R IE W

N

O

LY

138

FO R EV

R IE W

N

O

LY

139

FO R EV

R IE W

N

O

LY

140

FO R EV

R IE W

N

O

LY

141

FO R EV

R IE W

N

O

LY

142

FO R EV

R IE W

N

O

LY

143

FO R EV

R IE W

N

O

LY

144

FO R EV

R IE W

N

O

LY

145

FO R EV

R IE W

N

O

LY

146

FO R EV

R IE W

N

O

LY

147

FO R EV

R IE W

N

O

LY

148

FO R EV

R IE W

N

O

LY

149

FO R EV

R IE W

N

O

LY

150

FO R EV

R IE W

N

O

LY

Smile Life

When life gives you a hundred reasons to cry, show life that you have a thousand reasons to smile

Get in touch

© Copyright 2015 - 2024 PDFFOX.COM - All rights reserved.