Idea Transcript
Cognitive Studies/Psychology/Visual Studies 201: COGNITIVE SCIENCE IN CONTEXT Auditory Signal Detection (One-Day Experiment) Laboratory Module by Douglas R. Elrod, Cornell University
Introduction What is now known as Signal Detection Theory (SDT) got its start in radar research in the mid 1950's. Soon other fields, particularly psychology, had adopted its methods. Swets, Tanner, and Birdsall (1961) has a good overview of its history, and the mathematical details underlying it. We won't have time to cover its full scope, but using its framework, one can generate an "ideal" response pattern for any detection task, if one is given the various costs and benefits of the particular detection and non-detection outcomes possible. It uses two parameters, d' (d-prime) and (beta), to describe the performance of any binary signal-detection system. D-prime reflects the sensitivity of the detector, and is a criterion that reflects the tradeoff chosen between the goals of detecting the signal, when present, and not reporting the signal, when absent. By applying a probabilistic model to detection, it provided a better description of human perception, particularly for marginal cases when one performs better than chance, yet one doesn't "feel" that one is seeing/hearing the stimulus. Because of SDT, the use of forced-choice experiments, where one must produce an answer, even if one doesn't "consciously" perceive a stimulus, was given new support. Prior to this, it was often assumed that in that case, one just randomly chooses a response. SDT has been used to show that even in these cases some information is picked up from the environment.
A brief overview of Signal Detection Theory The statistical model
Figure 1. Probability diagram for the general Signal Detection Theory model
We look at one stimulus dimension, represented by the x-axis, horizontally across the page. In the absence of a signal, we assume that the value along this axis that is used to make a judgment is distributed in a "normal curve" around the mean of a "noise distribution", represented in Fig. 1 by the blue-shaded area (the bell curve on the left). (See the footnote for how the curve was constructed.) The presence of a signal shifts the resultant distribution to the right a certain amount, D (the Difference between the means of the two distributions), producing the pink-shaded signal+noise distribution shown on the right in Fig. 1. In the simplest case, which we assume here, the "signal+noise" distribution has the same shape as the "noise" distribution. In particular, (sigma), the standard deviation, which reflects the "spread" of a normal distribution, is the same between the two distributions (see Fig. 1). Signal Detection Theory assumes that, given this situation, we make our judgment of whether the signal is present, or not, by setting up a Criterion value, (beta). In Fig. 1, this criterion line separates the probability space into two parts. When a value is picked up that exceeds , we respond that the signal is present. When a value is picked up that doesn't exceed , we respond that the signal is absent. An important point is that the actual "comparison against " takes place "internally", because some noise comes from the perceptual system itself. This also means that the easiest way to think of is with no units -- as a "normally scaled" quantity. By convention, &beta is defined relative to the noise distribution, with 0 being at the center, and 1 being one standard deviation () to the right in Fig. 1 (in the direction of the signal). So a of 1.75 always means that there is a 4% chance of responding "Signal", when it is not present (a "false alarm") because 1.75 is the 96th percentile of a standard normal distribution. (To get a non-normalized version of one multiplies by an estimate of .) For simplicity, we will use the normalized version here. It also follows that, if we try to measure directly, in terms of the variability of external stimuli, we will underestimate it, because we aren't taking into account the "noise" (inefficiencies) of the perceptual system. Here we will focus on the more meaningful quantity, d' (d-prime) (see below). Depending on the task demands, we may move right or left. If we need to be conservative, and only respond that a signal is present when we are "really sure", it has been found that people act as though were moved in the "signal" direction ( right in Fig. 1). On the other hand, if it is more important to not miss any instances of a real signal, people tend to act as though were moved in the "noise" direction ( left in Fig. 1). Task demands don't change the essential "detectability" of the stimulus, though. We measure this, relative to a given detector system, by the parameter d'. This is defined to be D divided by . When the noise and signal+noise distributions are far apart, relative to their spread, then it is easier to distinguish whether the signal is present or not. In this case, d' is high. But when the two distributions overlap greatly, d' approaches zero, and one's pattern of responses will be less related to whether or not a signal is actually present. When comparing detector systems (on the same stimuli, for instance), d' can also be thought of as the "sensitivity" of the detector.
Possible response-outcomes to signals Table 1 enumerates the possible ways that a binary response can be related to the presence or absence of a stimulus. The colored patterns in the cells correspond to the regions in the probability diagram of Fig. 1. The blue patterned areas are situations where the stimuli were actually produced by noise (lack of signal), and the pink-patterned areas are where the stimuli were actually produced by a signal (along with ever-present noise, of course). The areas where the blue or pink pattern has lines going from upper left to lower right are those where a correct response was given (called "Correct Rejection" when noise leads to a "No Signal" response, and "Hit" when signal+noise leads to a "Signal" response). The areas where the blue or pink pattern has lines going from upper right to lower left are those where an incorrect response was given (called "False alarm" when noise leads to a "Signal" response, and "Miss" when signal+noise leads to a "No Signal" response).
Stimulus
No Signal
Noise
Signal+Noise
Correct Rejection
Miss
False Alarm
Hit
Response Signal
Table 1. Possible responses to stimuli, for Signal Detection. Referring back to Fig. 1, one should note that SDT predicts that, if d' is fixed, as one moves to the right to reduce False Alarms, one necessarily increases the proportion of Misses, and vice-versa.
The Two-Signal Variant: Signal Discrimination Discriminating between two signals can be seen as a variation on the classic signal detection problem. In the simplest case, signals A and B both have the noise that would be present in the "no signal" case added to them, as in Fig. 2.
Figure 2. Probability diagram for the Signal Discrimination Case The possible outcomes of an observation are then shown in Table 2.
Stimulus
Signal A+Noise
Signal B+Noise
Signal A
Correct Discrimination (A)
Mistake (A for B)
Signal B
Mistake (B for A)
Correct Discrimination (B)
Response
Table 2. Possible responses to stimuli, for Signal Discrimination. This model, then, can be used in the same way as the classic Signal Detection model. We will be applying it to the auditory domain to model discrimination between tones of various frequencies.
Reading Assignment Swets JA, Tanner WP, Birdsall TG, "Decision Processes in Perception", Psychological Review, 68(5), 301-340, 1961. NOTE: Read just pp. 301-313, which describes Signal Detection Theory. The later part is more technical.
Laboratory Exercise This laboratory exercise is designed to measure how sensitive we are to small changes in auditory frequency (which is experienced as "pitch"). It is adapted from Rosenblith & Stevens (1953), to use modern technology (such as computers), and be runnable in one day. As in their experiment, pure-tones are used, with rise and decay times of 40 ms. Covariates such as loudness are controlled for by using a within-subject design. Each participant experiences all auditory stimuli at one computer.
Instructions The Mac OS computers where the experiment can be run should each have a set of headphones attached. You should endeavor to sit at a computer as far away from other people as possible, to avoid distractions. If possible, arrange the moveable wall-sections to isolate the experimental area further. Setting up the Headphones If they aren't already, plug the headphones into the headphone jack on the left side of the monitor (this has a headphone symbol above it). Make sure that volume is set to a reasonable level. For standardization, check that the computer volume control (In "System Preferences:Sound: Output Volume") is set to maximum, and the left-right balance is set to the midpoint. Because the headphones will be plugged into the monitor jacks, the volume setting on the monitor will also affect the headphones. Set this to 50 (midway on the 0-100 scale). On the Panasonic monitors this is set with the "-" and "+" buttons, and on the Mitsubishi monitors this is set by pressing the "Menu" button to bring up the On Screen Display (OSD), then using the "-" and "+" buttons to change the volume, then pushing "Menu" several times until "OSD OFF" is highlighted, and then pushing "-" or "+" one more time. Running the experiment The experiment, in the basic form described here, should take about forty minutes to complete (plus the time for rest periods) for each participant. To run the experiment, first quit out of other applications, and disable the screen saver, if one is present. Make sure the monitor is set to 640x480 resolution (otherwise the instructions will either not fit on the screen, or will be displayed in a frame in the middle of the screen). Open the "Signal Detection" folder on the desktop, and then double-click on the "Auditory.script" file inside that folder. This should start SuperLab running the signal detection experiment. (If the computer is low on memory, you may find that the "Experiment Editor" display is empty at this point. You would then have to go to the File menu in SuperLab, and Open the "Auditory.script" from there.) Note: DO NOT MAKE ANY CHANGES within the SuperLab Experiment Editor display and DO NOT press on the buttons at the bottom of the window. When the SuperLab Experiment Editor window opens, go to the Experiment pull-down menu, and select Run. A different window will now open. This window allows you to enter the participant's name. Type in a unique name or designation for the participant and press the Run button in the window. Another window will appear. This window allows you to specify where the results of the laboratory exercise will be saved, and what will be the name of the file in which the results will be saved. You can select the hard disk (on the Desktop, for instance, in a folder labeled with the experimenter's name) for now, but you should remember to copy the results file to a backup location (a writable CD, for instance) as soon as possible. AND CorrectResponse="h") then "Correct Discrimination (H)" else IF(Response="h" AND CorrectResponse="l") then "Mistake (H for L)" else IF(Response="l" AND CorrectResponse="h") then "Mistake (L for H)" else IF(Response="l" AND CorrectResponse="l") then "Correct Discrimination (L)" Now, you ought to sort the columns by the AbsToneDiff column (make sure you are working on a COPY of your original data, so you can get back the original order the trials were run in, if necessary). Next, ignoring the "0" ToneDiff trials (which are the ones where the same tone was presented twice), for each of the 15 absolute differences, compute the four cells of Table 2. The number of trials in each condition goes into the cells (Excel has a "COUNTIF" function which helps with this). You ought to use an empty part of the spreadsheet laid out as in Table 3. Abs. ToneDiff Correct Discrimination (L) Mistake (L for H) Mistake (H for L) Correct Discrimination (H) 15
3
8
10
5
14
4
6
9
7
11
8
etc... 1
2
5
Table 3. Example of Response-Outcomes vs. ToneDiff table. Note that for each absolute value of ToneDiff, you use the data for the positive difference (where the second tone is higher than the first) for the "Mistake (L for H)" and "Correct Discrimination (H)" cells, and you use the data for the negative difference (where the second tone is lower than the first) for the "Mistake (H for L)" and "Correct Discrimination (L)" cells. Because 13 trials have a particular positive Tonediff, and 13 trials have a particular negative ToneDiff, the counts in the virtual 2x2 table for for each value of AbsToneDiff will add up to 26. Now, you can estimate a separate d' and B for each of the AbsToneDiff values using the formulas shown below (using the count data you computed above). d'-estimate = NORMINV("Correct Discrimination (H)"/ ("Correct Discrimination (H)" + "Mistake (L for H)")) - NORMINV("Mistake (H for L)"/ ("Mistake (H for L) + "Correct Discrimination (L)")) and -estimate = -NORMINV("Mistake (H for L)"/ ("Mistake (H for L) + "Correct Discrimination (L)")) where NORMINV is the inverse of the cumulative normal distribution function (specified to be the standard version, with 0 mean, and 1 standard deviation). (For the case where only one d' and are being estimated, these are the "maximum likelihood" estimators (MLE), see Dorfman & Alf (1968).) Because NORMINV is only defined between 0 and 1, excluding the endpoints, a zero in any of the four cells, "Correct Discrimination (L)", "Mistake (L for H)", "Mistake (H for L)", or "Correct Discrimination (H)" will produce an undefined d'-estimate. Because lim(NORMINV(x), x -> 0) = -∞ (negative infinity), and lim(NORMINV(x), x -> 1) = ∞ (positive infinity), this should be interpreted either as an infinite, or undetermined, estimate of d' (see the Special Cases below). Special cases: Zero cell counts The special case of one zero indicates that this particular tone-difference was too easy to distinguish to get a good estimate of d' (either by producing no errors, or nothing but errors in one of the columns of Table 2). The first leads to an estimate for d' of ∞, the second leads to an estimate of d' of -∞ (see below for the meaning of negative d' estimates). The special case of two zeros indicates either that the maximum-likelihood estimate of d' is undetermined (when the participant's responses are always "Higher" or always "Lower" -- these are cases when the -estimate is -∞ or ∞, respectively), or &infin (perfect correct responses), or -∞ (perfect incorrect responses). We don't need to consider the case of three cells with zero counts, because that can only occur when the subject is presented with only truly "Higher" pairs of stimuli, or only truly "Lower" pairs of stimuli. Interpretation of negative estimate of d': Although the parameter d' must be non-negative (given that it's merely the normally-scaled difference between the signal and noise distributions, see Fig. 1), our estimate of d' can be any positive, negative, or zero real value. Like large positive values, large negative d-prime estimates suggest that the detection system is sensitive to the signal, but in this case, in a way that tends to produce incorrect responses. In fact, it can be shown from the formula for our d' estimate that replacing correct responses by incorrect ones (and vice-versa) leads to a d'-estimate of the same absolute magnitude, but opposite sign. Therefore, a large negative d' estimate could result from either someone deliberately giving the "wrong" answer, or a misunderstanding about which key corresponds to which response ("lower" versus "higher", in this case). Analyses using d' and You can now test whether d' has the expected increase with AbsToneDiff by a linear regression. Treat d'-estimate as the dependent variable, and AbsToneDiff as the independent variable. Include a quadratic term, along with the linear one, to see if there is significant "nonlinearity" (a significant quadratic term means that you can reject that the data came from a simple linear relationship). You can also examine whether changes with ToneDiff. Other possible analyses: Does practice change d'? Perhaps we become more sensitive to stimulus distinctions after experiencing that area of stimulus space. This might be tested by grouping early trials together and comparing against later trials. Does practice change ? Do participants differ in their d' and ? One possible variable affecting this is years of musical experience. Because this was asked for in the experiment, you could test whether d' or is affected by this between-subject variable. A regression would be a logical way to check this. Is reaction time related to d' and ? Do "harder" judgments (lower d') take longer time, for instance? Advanced: Does your data provide support to, or against, Signal Detection Theory itself? The last part of Swets, Tanner, and Birdsall (1961) examines whether data at that time supported various aspects of the theory. Possible further lines of inquiry: Rosenblith & Stevens (1953) presented tones to one ear at a time. Does dichotic (both ear) presentation raise d' and thereby improve performance? Does "attention" affect d'? Perhaps it could "lower the noise", resulting in better sensitivity? Is "response confidence" correlated with d'? Signal Detection Theory seems to be "passive" relative to some conceptions of perception, such as J.J. Gibson's "active perception". Would the theory need to be modified to fit into such a framework?
Laboratory Report for the Signal Detection Module After you have completed the Laboratory Exercise, you will be ready to write a report in journal article format. Select a journal that represents an interest of yours. Follow its 'Instructions to Authors' and its general format in preparing your report. General guidelines can be found here.
Appendix Guide to deciphering Results Files from the Auditory Signal Detection Experiment Each participants' data will be stored in a file that can be opened using Microsoft Excel. It is also possible to read these files as text using other programs such as SimpleText or TeachText, or the statistics program StatView (which is particularly useful for statistical tests like t-tests and regression). The participant code is at the top of the file, followed by the script name ("Auditory.script"), and the date and time when the experiment was run. Then there's a blank line, and the data columns follow, headed by the column-names which are listed below. Each trial of the spreadsheet takes up one row of the spreadsheet, with successive trials going from top to bottom. The first several trials are the information screens. The Response value for "Intro1" is the number of years that the participant had musical training. The next 12 trials are the practice trials (which should be omitted in experimental analyses), and the ones after that are the experimental trials. The columns in the datafile, from left to right, are as follows: Trial Name: This consists of two numbers separated by a dash, for instance "250.00-251.00". The first number is the frequency, in Hertz (Hz), of the first tone that is presented in this trial, and the second number is the frequency of the second tone that is presented. Trial #, Event #, Response: These are generated automatically by SuperLab. We will generally ignore these columns. Response: The key that was pressed ("h" or "l"). Error Code: We will ignore this (it's always "C" in this experiment). Reaction Time: This is the time, in ms, between the appearance of the screen "Was the second tone higher (H) or lower (L) than the first?", and the participant hitting a key. F1-Whole, F1-Fract: These columns convey the information about the frequency of the first tone (sum them to get the correct number). F2-Wh1, F2-Wh2, F2-Fract: These columns convey the information about the frequency of the second tone (sum over them to get the correct number). Note: It's best to edit a copy of the data file, rather than the original data file itself, given the risk of deleting data! References: Dorfman, DD & Alf, E, "Maximum likelihood estimation of parameters of signal detection theory -- A direct solution", Psychometrika, 33(1), 1968, 117-124. Rosenblith, WA & Stevens, KN, "On the DL for frequency", The Journal of the Acoustical Society of America, 25(5), 1953, 980-985. Swets JA, Tanner WP, Birdsall TG, "Decision Processes in Perception", Psychological Review, 68(5), 1961, 301-340. This page maintained by Doug Elrod. Note: If you are curious about how the normal-curve diagrams were created, see Doug Elrod's Scalable Normal Curve for an explanation.