The Great Market Research Debate:
Are Mobile Insights Better Than Online?
By Ben Leet, Managing Director Europe, Instantly Edward Appleton, Independent Insights Advisor and Author, “In The Moment – Perspectives on Mobile Market Research”
INTRODUCTION Momentum around using mobile technology for market research is gaining pace, but not as quickly as it should. That’s because increasingly consumers – our respondents – are living their lives on their mobile devices. In 2014, mobile internet usage overtook desktop internet usage, with 2 billion smartphone users predicted by 2015. If our industry wants to stay relevant, we have to follow them. However, many in our industry remain unsure of the benefit of mobile. Market research agencies worry that their clients are unwilling to adopt new methodologies, even while end clients cry out for innovative new ways to engage with their customers. At the heart of that debate is the age-old fear of change. This research was therefore designed with an open mind to prove or disprove that mobile gives more accurate insights than online. It’s important to note that Instantly is mode agnostic; we provide both mobile and online technology and panelists.
1
www.instant.ly
Are mobile insights better than online?
HYPOTHESES • Mobile methodology is preferable for shopper/shopability studies because mobile technology can capture insights closer to the moment of purchase, and it can deliver point-of-sale truth via photographic evidence, rather than relying on memory recall of an online respondent. • Mobile research delivers more granular responses in sensory evaluations on IHUTS than laptop/ desktop, because recall bias is eliminated. • Mobile delivers richer insights overall, including on open-ended questions.
METHODOLOGY The study was a three-stage process, run as a mini-diary. The target audience was shoppers that purchased a particular brand and flavor of chips. In the UK, Walkers Prawn Cocktail Flavour was chosen, and in the U.S., we chose Lay’s Cheesy Garlic Bread Flavor. More divisive flavors were chosen in order to elicit stronger sensory ratings.
Respondents were restricted to purchasing the chips from a supermarket/supercenter in order to maintain consistency across geographies and modes. The modes were kept mutually exclusive. Respondents were only exposed in either the mobile/tablet (mobile) cell, or desktop/laptop (online) cell. Through browser and operating system detection we were able to identify and eliminate cross over, such as online respondents using their mobile web browsers. Mobile respondents were geofenced via device GPS, ensuring that they were at the required supermarket during the shopper element of the diary. Online respondents, responding later and off location, had to rely on memory.
2
STAGE 1 – SCREENER Respondents completed a one-minute screener to eliminate non-considerers of the chosen chip flavours and selected purchase locations. Respondents were then given instructions to purchase the selected chips.
STAGE 2 – IN-STORE PURCHASE All respondents were asked a series of questions about shopability and findability. Mobile respondents completed this stage on their mobiles while in-store. Online respondents were asked to complete their next phase “as soon as possible” after the purchase.
STAGE 3 – CONSUMPTION Respondents were asked a series of questions about consumption, including sensory evaluations and purchase intent. They were instructed to answer the questions as close to consuming the chips as possible.
3
www.instant.ly
Are mobile insights better than online?
SAMPLE FRAMES AND FIELDWORK The target number of completes was a total of 600, or 150 completes per cell, mobile and online, in both the U.S. and UK markets. Given the diary nature of the study, a larger number of screened completes were needed to allow for natural drop-out rates over the course of the study. However, what was not anticipated was the extent to which mobile methodology experiences higher drop out relative to online. As a result, the mobile sample came up short and took longer to complete. The final sample frames were:
GOAL
ACHIEVED
TIME
MOBILE
n=150
n=111
2 WEEKS
ONLINE
n=150
n=255
< 1 WEEK
MOBILE
n=150
n=107
2 WEEKS
ONLINE
n=150
n=226
< 1 WEEK
As the chart below illustrates, the reason for the high mobile drop-out rates is that mobile respondents are far more impatient. For instance, in-store drop out occurred when respondents were asked to take a photo (they simply shut the survey down). And on the in-home test the majority didn’t even wait for the survey to load before shutting it down. The net result was a much lower start/complete ratio on mobile compared to online.
MOBILE
ONLINE
MOBILE
ONLINE
SCREENER STARTS
n=1,992
n=1,097
n=8,687
n=3.914
SCREENER DROP OUT %
15%
9%
18%
12%
IN-STORE DROP OUTS
238
37
197
53
PRODUCT TEST DROP OUTS
150
7
89
8
6%
23%
1%
6%
32%
41%
9%
12%
START/COMPLETE RATIO
CALCULATED IR
Other factors, such as interview length, locations, and survey experience, were all measured as almost identical across modes, and therefore had no bearing on the drop-off rate.
4
IN-STORE RESULTS Here, the difference in results between mobile and online was significant and staggering. In the U.S., the respondents were asked: “Where were the Lay’s Cheesy Garlic Bread Chips on the shelf?” with the options of answering “top,” “middle,” or “bottom.” From the online sample, 83% of respondents said that the chips were to be found in the middle. However, 57% of mobile responders said the chips were at the bottom.
So which should we believe? Very simply, mobile. Why? Because online relies on memory recall which biases the results of the study. We collected 300 photos, taken in real time from mobile respondents that conclusively prove that the majority of stores stock Lay’s Cheesy Garlic Bread Chips at the bottom of their shelves. It is therefore clear that online respondents “guessed” the answer to this question. In addition, when considering the question about the time it took to find the product in-store, the answers from online respondents conflicted with our survey length statistics. This provides further proof of memory recall bias when using online methodology.
Question: From the moment you entered the store, how long did it take you to find the Walkers Prawn Cocktail chips?
UK MOBILE
5
www.instant.ly
UK ONLINE
1-10 secs
28% 6%
11-30 secs
31% 16%
31-60 secs
15% 18%
1-2 mins
18% 31%
3-4 mins
5% 17%
5 mins
2% 7%
Longer
1% 5%
Are mobile insights better than online?
RESULTS – PRODUCT TEST An examination of the time elapsed between product tasting and survey complete revealed marked differences between online and mobile. This recency dynamic may well explain the high drop- out amongst mobile IHUT - mobile users are “in-the-moment” as well as impatient. This is an economic downside. On the up-side, the recency effect is highly likely to lead to enhanced accuracy or recall. Put another way, 44% of online respondents completed the IHUT survey at least 5 hours after having tasted the product. The impact on accuracy of recall is likely negative.
Question: When did you eat the chips?
ONLINE 1 in 3 UK IHUT completes based on product tasted more than 8 hours ago. Memory impacts on accuracy!
Question: Please tell us your opinion of the following aspects of the chips, using a scale of 1 - 5.
UK MOBILE
UK ONLINE
LAST 5 - 10 MINS
44%
6%
11-30 MINS AGO
11%
9%
30 MINS -1 HR AGO
7%
14%
ABOUT 2 HOURS AGO
14%
17%
3-4 HOURS AGO
4%
10%
5-8 HOURS AGO
4%
11%
MORE THAN 8 HOURS AGO
17%
33%
USA MOBILE
USA ONLINE
CRUNCHINESS
78% JAR
80% JAR
FLAVOR
53%
70%
SALTINESS
76%
68%
NUTRITIONAL INFO
69%
76%
VALUE FOR MONEY
69%
72%
AMOUNT OF CHIPS IN BAG
59%
56%
FRESHNESS
81%
77%
EASY TO OPEN
83%
81%
IHUT: MOBILE SURVEY CLOSER TO TASTING
IHUT: LITTLE DIFFERENCE ON SENSORY EVALUATION
6
IN-STORE RESULTS CONT. Unlike JAR, there was a sizable difference on top box purchase intent (PI). Across both countries, online respondents gave much higher “definitely would buy” ratings compared to mobile. This has significant implications for product owners seeking to optimize product positioning and supply controls, with online results suggesting a much more aggressive positioning and supply approach than mobile. For example, product owners following the online data would over-invest on positioning and product supply. This would throw out ROI metrics for the product, and could ultimately lead to the product being deemed an internal “failure”, whereas following the correct set of data with the resulting product strategy could lead to the product being deemed a success.
Question: How likely would you be to buy these flavor chips in the future?
IHUT: DIFFERENT TOP BOX PI SCORES
UK MOBILE
UK ONLINE
USA MOBILE
USA ONLINE
Purchase Propensity DEF. NOT
5%
7%
8%
6%
PROB. NOT
10%
7%
17%
9%
MIGHT/MIGHT NOT
14%
12%
16%
10%
32%
25%
34%
27%
39%
48%
25%
48%
PROB. WOULD BUY
DEF. WOULD BUY
With evidence showing that mobile respondents were much closer to the moment of consumption, it’s very possible that an element of satisficing contained within the online sample.
7
www.instant.ly
Are mobile insights better than online?
RICHNESS OF DIAGNOSTICS: MOBILE TRUMPS ONLINE Four open-ended questions were asked during the study. The results were not surprising. Average word counts in the open-ends were slightly less on the in-store section (respondents were probably busy shopping and less likely to type), but on the product test the word counts were higher for mobile (7.87 average words) versus online (6.72 average words). Much more significant was the quality of word count. Mobile delivered a richness in the responses that online lacked. This finding is consistent with many other Instantly studies. It is a clear sign that the next generation is more comfortable typing a response on a smartphone than on a desktop/laptop keyboard. Perception to the contrary suggests that we are in danger, as researchers, of being out of touch with our audiences. Researchers that think otherwise need to rip up their rule book.
Mobile diagnostics - fresher, more granular Dislikes (first 10) USA MOBILE
• Wasn’t cheesy enough • Not enough cheese flavor and I like white cheese • No, it was good • Nothing • Nothing • Nothing • Nothing • Nothing • Nothing • Nothing
• Garlic taste • I felt like the taste could be bolder. • I had pizza and breadsticks for dinner last night and I think eating these compared to actual Italian food creates a negative experience. Other than that they are good but definitely not my favorite. • They were a little greasy. • I wish there were more in the bag. • Nothing really enjoyed them • The garlic is too strong • It takes a few chips to actually taste the flavor • The cheese flavor was really fake • Couldn’t find anything I didn’t like.
MOBILE DELIVERS MORE ACCURATE AND RICHER INSIGHTS
RESPONSE USA MOBILE
6.72 7.87 WORDS
RESPONSE USA DESKTOP
WORDS
Dislikes (first 10) USA ONLINE
8
CONCLUSIONS In-store research clearly benefits from a mobile methodology. There is concrete proof that relying on respondents’ memories leads to inaccurate data – and potentially disastrous, costly decisions being made as a result. Mobile is much closer to the moment and is therefore the obvious methodology for any in-store research. For product testing, the conclusion is slightly less clear. Sensory ratings varied little across modes, and although different top box PI scores were registered, that is not conclusive enough to say that mobile should definitively be used. The clear advantages for mobile are that, again, it gets much closer to the moment, and it delivers much richer diagnostics when it comes to open-end questions. Mobile will, however, take longer to conduct, and cost more. For instance, this study would have commanded a 55% premium to conduct using mobile versus online, although the gap could have been closed on both counts with a widening of the location targeting. The key questions then become: Is it wise to spend money on an online study for in-store work when it is proven to be flawed and subject to inaccurate data? Or is it wiser to use mobile and get more accurate insights that are actionable and produce a much higher return on investment?
9
www.instant.ly
ABOUT THE AUTHORS Ben Leet has been in the research industry for over a decade, beginning by building new sales channels at several companies, including Decision Tree Consulting and Toluna, before being part of the seed team that founded Instantly Europe in January 2011. Ben now combines his market research experience with his sales and leadership skills to oversee Instantly’s European operations. Ben is a graduate of the Nottingham Business School with a Bachelor of Arts degree (Hons) in European business.
Edward Appleton is European Consumer Insights Manager at a major Multinational based in Berlin, Germany. A graduate of Oxford University, he has over 20 years research experience, on both client and supplier side – Group Manager Research at Nestle UK, European Research Manager at Rohm and Haas, Insights and Planning Director at Grey Communications Germany were positions that followed his start in research at Mass Observation UK. Edward is bi-cultural English/German, speaks fluent French and is a Member of the Market Research Society UK.
Instantly™ is the world’s largest audiences and insights platform, providing researchers and marketers with immediate access to consumers and automated insights tools to make faster, better decisions. Instantly is based in Los Angeles, with offices in the United States, Europe and Asia. For more information visit www.Instant.ly United States of America – Los Angeles 16501 Ventura Blvd, Suite 300 Encino, California 91436 T +(866) 872 4006
[email protected]
10
www.instant.ly
United Kingdom – London 88 Gray’s Inn Road London, WC1X8AA T +44 (0) 20 7400 3500
[email protected]
Germany – Frankfurt An der Welle 4 (Alte Oper) 60322 Frankfurt, Germany T +49 (0) 69 7593 7586
[email protected]