Idea Transcript
NEWS FEATURE
NEWS FEATURE
News Feature: How online studies are transforming psychology research The samples are large and diverse, but will this trend strengthen the field or merely introduce new sources of error? Amber Dance Science Writer
allows people to test their unconscious leanings: a deeply buried belief, for example, that boys are better than girls at math or that one religion outranks another. Nosek and his collaborators After six weeks spent busily coding, psychology When he landed in Seattle, he attended a wanted to study people’s responses. It was 1998 graduate student Brian Nosek finished his press conference about his new website, and “the Internet wasn’t being used for reproject in a cab on the way to the airport. Project Implicit (projectimplicit.org), which search,” recalls Nosek, now a professor at the University of Virginia in Charlottesville. “We had no idea if it would be of any success.” Within two days, 50,000 people had taken Nosek’s tests. Today, Project Implicit has gathered data from 18 million participants, enough to keep the organizers busy for their whole careers, should they wish. It has resulted in more than 100 publications on topics ranging from how the election of an African American president affected racial attitudes [not much (1)] to a slowly rising tolerance for gay people (2). Project Implicit is no longer alone on the Web; psychologists have taken their studies online in droves in recent years. The benefits are clear: easily accessible, large, diverse subject pools that extend beyond psychology students seeking course credit. But researchers must guard against online pitfalls. Participation is inherently biased toward those who choose to take online surveys or tests, sometimes for money. Psychologists must take care to understand their respondents, and make inferences only for the categories of people represented. However, researchers have no way to confirm who their online subjects are, and they have to watch out for dishonest or distracted participants who might introduce misleading data. Of course, every psychological research method and group of test subjects has its upsides and downsides. “There is no perfect method,” says Samuel Gosling of the University of Texas at Austin. Ideally, experts say, researchers should use multiple methods to confirm results. But online studies offer another potent and increasingly popular method to the mix.
Psychology studies done online allow for the collection of data from tens of thousands of subjects. Image courtesy of Dave Cutler. www.pnas.org/cgi/doi/10.1073/pnas.1520769112
WEIRD Science Picking the right pool of people to study has long been an issue in psychology research. “The psychology undergraduate has become the ‘model organism’ for psychological studies,” says Laura Germine, a postdoctoral fellow in psychiatry at Harvard Medical School and
PNAS | November 24, 2015 | vol. 112 | no. 47 | 14399–14401
as in-person pools (3). “Now, it’s pretty much accepted that people on the Internet are just people,” Germine says. Indeed, Hartshorne says he doesn’t know many researchers who have never done at least a tiny pilot study online. The Internet provides an easy way for psychologists to obtain the large, diverse, or specialized samples they need. For example, in one paper Germine and Hartshorne report a study of how cognitive ability varies with age (4). Such a study requires numerous people of all ages, and they simply don’t have the space or the experimenters to recruit so many subjects and test them in a laboratory, Hartshorne says. Online, they accumulated nearly 50,000 people who took IQ and memory tests. The researchers found that some abilities, such as mental processing speed, peak in the late teens. Others, such as vocabulary, peak around age 50. The authors compared their Internet scores to a smaller, standard dataset from in-person testing, and most of the findings held up across both testing groups, thus confirming the findings with multiple methods. In another case, Nosek used Project Implicit data for a study of how attitudes toward gender and career vary around the world. Study participants were given two kinds of words to categorize: gender-related words and careers in science or liberal arts. The subjects then had to sort, as quickly as they could, the two categories simultaneously. It turns out that most people sift more quickly and accurately if they have to bin masculine words such as “man” and “boy” together with science-related words like “physics” than if they have to match feminine words to science topics. Among more than half a million people who took that test, about 70% associated males with science more often than females, Nosek et al. reported in 2009 (5). The researchers accumulated data from 34 different countries, allowing them to analyze how those implicit attitudes correlated with education. In those countries where the subconscious stereotype was strongest, boys and girls had the biggest differential in performances on science and math. This result complemented previous findings that implicit World Wide Subject Pool stereotypes correlate with science and math Although researchers like Nosek have been scores on an individual level. conducting studies online since the days of dial-up, skepticism about the practice was Take this Quiz! widespread even just a few years ago. Germine Research online yields big datasets, fast. But recalls one comment when she presented at a running an online study is not quite as simple meeting in 2010: “ ‘How do you know that as posting a survey and sitting back while the your participants are not drunk and watching results roll in. Not many psychologists have porn?’ ” accused the questioner. There was trained in how to do an online study properly, a common perception, Germine says, that Hartshorne says. So he has become a bit of an “there’s some dark underbelly of society evangelist, traveling and speaking about the mistakes he’s learned the hard way, such as the and they’re on the Internet.” However, when Gosling tested his online re- time he posted a study for Japanese speakers, spondents for neuroticism and introversion, with monetary rewards, and ended up with characteristics linked to depression and so- participants who clearly didn’t understand cial isolation, he found they scored the same Japanese at all.
Massachusetts General Hospital in Boston. Retirees and young children are also fairly easy subjects to attract, adds Joshua Hartshorne, an incoming assistant professor of psychology at Boston College. The problem is that those volunteers typically come from societies that are Western, Educated, Industrialized, Rich, and Democratic: as psychologists say, they’re WEIRD. “There are giant holes in our knowledge about adults between 20 and 65, or about people not college-educated and living in the West,” Hartshorne says. Sometimes, it might not really matter who takes the test, so long as they have a human brain. For example, consider a classic test of attention: subjects are told to count how many times a group of basketball players in a video pass the ball, and many are so focused on the count that they fail to notice a man in a gorilla suit stroll across the screen. If you want to understand how the human brain ignores the gorilla, any study participants—so long as they can perform the task—may suffice. However, for other questions, demographics matter. If you want to know what kinds of voters support Donald Trump for president, you can’t ask a sample of all liberals. Can use of the online community fix the WEIRD problem? Yes and no. Internet research subjects are more diverse than psychology undergrads, but they are still a skewed participant pool. For example, in a 2004 review, Gosling et al. (3) compared the demographics of traditional samples, gleaned from a year’s worth of studies in the Journal of Personality and Social Psychology, to the population that used the personality-test website outofservice. com, run by Gosling’s collaborator. Sample composition varies depending on the school, its locale, and the sample selection procedures; but overall, in-person psychology subjects were 29% male. Online, that number rose to 43%. The Internet sample also had slightly more non-Whites, 23% compared with 20% in traditional studies, but with percentages of African Americans and Latinos still far below the United States census numbers. The online sample also had nearly double the proportion of people from outside the United States (3).
14400 | www.pnas.org/cgi/doi/10.1073/pnas.1520769112
First, researchers have to hook their potential participants. People love to take quizzes online, but researchers are competing with such popular frivolous offerings as “Which Disney princess are you?” “If you put up a website that’s boring, three people come in, then it’s just spiders,” Hartshorne says, referring to the automated webcrawlers that index sites. Some psychologists have reverse-engineered the clickbait traits of popular Facebook quizzes, and fit their studies to those parameters. For example, outofservice.com doesn’t say, “Take this standard personality test”; it trumpets, “Find your Star Wars twin” or offers up similar questions in a questionnaire labeled “All About You—A Guide to Your Personality” (3). For psychologists trained in traditional experimental settings, going online may be unsettling. “You’re giving up a lot of perception of control,” says Fred Sabb, a cognitive neuroscientist at the University of Oregon in Eugene. Under the watchful eye of a research assistant, subjects are likely to pay attention to their tasks. Far away on their computers or mobile phones, they might be concurrently watching a movie for all investigators know. Researchers have ways to make sure subjects are focused on their tasks in the laboratory, and these are even more important online. “Catch trials” are questions that anyone who is paying attention should ace, such as, “Have you ever had a heart attack and died?” “Manipulation checks” ensure the person understood the instructions. For example, the screen might offer up what looks like a multiple-choice question, but the instructions say to click on the question title, not an answer. Researchers can also time
Researchers must carefully monitor their respondent pool to ensure the integrity of the data they collect. Image courtesy of Dave Cutler. Dance
Dance
completing hundreds or thousands of studies, the workers may be more familiar with classic research materials than the average person, Chandler notes. That might skew results: for example, someone who takes a lot of IQ tests is likely to get better at them over time. Another downside, Kahan points out, is that Turkers connect online to exchange tips on the best studies and how to complete them quickly. Foreknowledge of an experiment could alter results. “I don’t want to have a lounge where people who are about to take the study mingle with people who have just finished,” he says. “It’s just bad social science hygiene.” Chandler thinks this problem is rare, but says he usually asks at the end of a study if participants saw the survey discussed anywhere. “The question is always, is the sample valid given the kind of inference you want to draw?” Kahan says. “With respect to a lot of things people are using MTurk for, the answer is ‘no.’” For example, he’d worry about studies of Turkers who claim to have psychological symptoms, based on Chandler’s report that about 10% of Turkers scored high for malingering: they claimed rare symptoms, perhaps in the hopes of qualifying for more work (12). Of course, the Internet will never work for many kinds of psychology studies. Germine points out that no survey can replace a face-toface clinical interview. Nor can Internet tests replace observational studies of people interacting in person, whether in real-life activities or in response to staged scenarios, although they can link up multiple online participants in games or cooperative tests. “There will never be a time, especially with behavioral research, where there isn’t a use for having people in person,” Nosek says. Despite these potential stumbling blocks, psychologists continue to plumb the Internet for greater and greater subject numbers. Hartshorne, for example, expects to publish a study soon with 700,000 participants, on how age affects learning of a second language. Learning another language can take decades, he points out. “I don’t have 30 years to wait for the longitudinal study to get done.” Thanks to the online community, he doesn’t have to.
6 Germine L, Dunn EC, McLaughlin KA, Smoller JW (2015) Childhood adversity is associated with adult theory of mind and social affiliation, but not face processing. PLoS One 10(6):e0129612. 7 Paolacci G, Chandler J (2014) Inside the Turk: Understanding Mechanical Turk as a participant pool. Curr Dir Psychol Sci 23(3):184–188. 8 Buhrmester M, Kwang T, Gosling SD (2011) Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspect Psychol Sci 6(1):3–5. 9 Paolacci G, Chandler J, Ipeirotis PG (2010) Running experiments on Amazon Mechanical Turk. Judgm Decis Mak 5(5):411–419. 10 Mason WA, Jones A, Goldstone RL (2008) Propagation of innovations in networked groups. J Exp Psychol Gen 137(3):422–433. 11 Mason W, Watts DJ (2012) Collaborative learning in networks. Proc Natl Acad Sci USA 109(3):764–769. 12 Shapiro DN, Chandler J, Mueller PA (2013) Using Mechanical Turk to study clinical populations. Clin Psychol Sci 1(2):213–220.
PNAS | November 24, 2015 | vol. 112 | no. 47 | 14401
NEWS FEATURE
humanity as a whole, but they still beat out university undergraduates, as well as many other online samples, for diversity (8, 9). Another advantage is that each Turker has a unique ID, which psychologists can use to prevent people from taking the same survey multiple times or to recontact study participants who meet a certain profile. (Chandler used these IDs to ask people repeat questions over time to trawl for liars.) And since MTurk “employers” can deny payment for a task poorly done, Turkers are motivated to follow instructions. “I think it’s probably the most well-studied sample out there right now,” Chandler says. It can be particularly good for researchers seeking specific populations, such as parents of toddlers, or people who are gay, he says. And its applications extend beyond simple online surveys. In one MTurk study, published in 2012, researchers investigated effective problem solving in groups. They assigned teams of 16 Turkers to explore a virtual desert for oil fields in a game called “Wildcat Wells.” As participants searched different parts of the landscape, the game shared their findings with just three other team members. Some played in a group with an efficient network, where this sharing distributed the information evenly throughout the team, whereas other groups dealt with an inefficient network in which small clusters of players mostly shared findings with each other. Previous research offline suggested that inefficient networks would get Rise of the Turk the best results, but the efficient networks Doing research online has gotten even easier performed best in the MTurk experiment since the 2005 launch of the Mechanical Turk, a (10, 11). service from Amazon. “MTurk,” as it’s affectionately called by users, allows people to post Turk Tomfoolery? tasks that require a human to perform: for It’s fast, easy, and cheap, but does MTurk example, identifying objects in pictures. It magnify the perils of online research? Dan was named for an 18th century hoax, a chess- Kahan, a professor of law and psychology at playing automaton that turned out to have a Yale University in New Haven, laments how human player concealed inside. often researchers turn to it without considering “Turkers” are the online users or “workers” that it may be unsuitable for their work. “You who perform those tasks, expecting about should wonder about the value of something 10 cents a minute in return. Psychologists you’re paying a few cents for,” he warns. Kahan cottoned on to MTurk’s possibilities around cautions that people on MTurk might be 2010, says Jesse Chandler, an adjunct pro- atypical precisely because they are attracted to fessor at the University of Michigan’s In- the site. stitute for Social Research in Ann Arbor and a Because so many psychologists are using survey researcher at Mathematica Policy Re- MTurk, and some individual Turkers are search. With the service, they can get surveys done by lots of people, fast, for as little as dimes, nickels, or even pennies per subject. No 1 Schmidt K, Nosek BA (2010) Implicit (and explicit) racial attitudes joke—Gosling once offered one cent for an- barely changed during Barack Obama’s presidential campaign and swering a two-question survey, and garnered early presidency. J Exp Soc Psychol 46(2):308–314. 2 Westgate EC, Riskind RG, Nosek BA (2015) Implicit preferences for 500 responses in just 33 hours (8). straight people over lesbian women and gay men weakened from Who’s willing to take surveys for pocket 2006 to 2013. Collabra 1(1). Available at collabra.org/articles/ change? As it turns out, Turkers are still a bit 10.1525/collabra.18/. Accessed October 28, 2015. WEIRD. They have above-average education, 3 Gosling SD, Vazire S, Srivastava S, John OP (2004) Should we trust web-based studies? A comparative analysis of six preconceptions but report fairly low income; Chandler suspects about internet questionnaires. Am Psychol 59(2):93–104. many are unemployed or underemployed 4 Hartshorne JK, Germine LT (2015) When does cognitive Millennials living with their parents. Most functioning peak? The asynchronous rise and fall of different abilities across the life span. Psychol Sci 26(4):433–443. Turkers reside in the United States or India, cognitive 5 Nosek BA, et al. (2009) National differences in gender-science where Amazon pays in cash, not merely store stereotypes predict national sex differences in science and math credit. Turkers certainly do not represent achievement. Proc Natl Acad Sci USA 106(26):10593–10597. the responses, because robot programs could be very fast and movie-watchers slow, and directly ask people if they were distracted. Because online studies net so many participants, researchers can liberally throw out suspect results while maintaining adequate sample size, Sabb points out. Participants are often anonymous, which can help recruitment. Germine recently performed a study of how childhood traumas, such as sexual abuse, affected adult thought processes. She was worried she might not get many subjects, but netted 30,000. People were probably more likely to participate because they did not have to identify themselves or their family members, Germine surmises. In an analysis of a subset of those participants, she discovered that parental abuse correlated with certain deficiencies, such as trouble inferring the thoughts and feelings of another person (6). Previous work had reported children who faced traumas had a hard time understanding other people’s emotions; Germine’s study showed those problems extend into adulthood. That anonymity comes at a price. People can certainly lie in person, but in the laboratory, it’s often easy to tell if a volunteer doesn’t qualify for a study. Online, people who want to take studies or are financially compensated for doing so might be inclined to lie to make themselves look like perfect candidates. However, studies indicate that people online lie about as often as they do in person (7).