Gamifying. Not all fun and games - Breaking Blue · 2020-01-08 · gamification is weaker,...
Transcript of Gamifying. Not all fun and games - Breaking Blue · 2020-01-08 · gamification is weaker,...
Are We There Yet? Where Technological Innovation is Leading Research Proceedings of the Association for Survey Computing, Volume 7. Edited by T. Macer et al. Compilation © 2016 Association for Survey Computing
Proceedings of the Association for Survey Computing, Volume 7 1
Gamifying. Not all fun and games
Phil Stubington, Charlotte Crichton
Abstract
There has been a lot of attention on engaging online participants to address declining response rates and
the risks of poor quality data. Participant availability via panels has plateaued, while completion rates
fall and many surveys appear poorly designed. This raises real concerns about the representativeness of
participants who are recruited from online panels or client email lists and the quality of their data. This
paper reviews evidence from experiments we have run using gamification, some examples of the latest
techniques for gamification and considers the practical challenges of implementing gamified surveys
for agencies, clients and panel providers.
Keywords
Gamification; engagement; online research; response rates
1. Introduction
The challenge facing market research
Maximising the participant experience and thus response rates and data quality are not new topics and
engaging participants has always been vital to market research. Furthermore, there is a strong body of
evidence that suggests this task is becoming harder not easier over time. The clearest indicator for this
is that response rates are falling in most markets. According to analysis published by Pew Research, the
response rate of a typical telephone survey in the United States was 36 percent in 1997 and was just 9
percent in 2015. In the United Kingdom, a paper by Ipsos MORI as long ago as 2008 reported that the
response rate for the National Readership Survey had fallen from 73.4 percent in 1974 to 51.6 percent
(although the situation does now appear to have stabilised with only a further two percent decline since
then). Similar analysis of the UK Labour Force Survey (Barnes W, Bright G & Hewat C (2008)) has
shown a 21 percent decline in response rates over a fifteen-year period.
The move to on-line research should have altered the paradigm for response rates since (at least for the
panel providers such as Lightspeed GMI, Toluna, Research Now and SSI) participants are effectively
pre-screened concerning their willingness to cooperate by the act of signing up to panel membership.
However, online research still has to compete with the numerous other interesting things that the internet
has to offer; on-line samples often vary hugely in the manner in which they are recruited and there is a
wide variety of different approaches to contacting participants of varying efficacy. Furthermore, on-line
survey invitations have to convince the participant to cooperate without the intervention of an
interviewer who can answer questions concerning privacy, data protection or the purpose of the survey.
Phil Stubington, Charlotte Crichton
2 Are We There Yet? Where Technological Innovation is Leading Research
Data concerning on-line survey response rates is inconclusive, for many reasons including (but not
limited to) commercial sensitivity amongst the major panel providers, issues of calculating a response
rate for some types of recruitment such as river sampling1 and the wide variety of sample sources. In a
literature review of on-line response rates, Schonlau M, Fricker R & Elliott M (2002) commented,
“studies on the use of the Web as a response mode vary widely in terms of the nature of their target
populations, how participants are recruited, and whether any attempts at statistical adjustment are made
in the studies’ analyses.” The paper pointed out that even amongst broadly similar surveys conducted
by the U.S. Census Bureau, response rates ranged from 27 percent to 75 percent. A 2008 paper by
Baruch and Holtom reported an equally wide range of response rates in the UK (the average response
rate for studies that utilised data collected from organisations was 35.7 percent with a standard deviation
of 18.8 percent).
An admittedly unscientific web search by the authors of this paper suggests that response rates between
15 percent and 25 percent are increasingly regarded as the norm for on-line surveys, which would
suggest that on-line surveys have joined their telephone and face-to-face counterparts as being
challenged by poor response rates.
The industry’s response
Because of this trend, and the need to ensure high quality data more generally, a number of the major
panel providers and various research agencies have invested significant time and effort in testing the
impact of improving questioning techniques and/or enhancing the visual appearance of surveys.
The work of Jon Puleston of Lightspeed GMI is probably best known in this regard, but many conference
papers have now been presented on this topic, which between them provide a considerable body of (not
wholly consistent) evidence.
In industry parlance, these approaches have tended to be grouped under the umbrella heading of
‘gamification’. Mavletova (2014) summarised the main elements of a gamified survey as: (1) stating
clear rules and goals for the participants; (2) involving participants with a relevant and entertaining
narrative; (3) maintaining motivation by providing interesting and achievable tasks or quests; and (4)
giving feedback on the progress and rewards for accomplishing tasks and answering questions.
In practice, gamification is often conflated with the use of a variety of techniques to improve the
aesthetic of a survey. However, this need not be the case, since many gamification techniques can be
used for text questions (Cape (2016)). Equally, it is possible to improve the visual appearance of a
survey, for example using images to replace answer lists, without drawing upon game techniques. For
the remainder of this paper, for convenience, we use the term gamification in the widest sense to cover
both question wordings and the visual aesthetic of the surveys.
1 Where potential participants are recruited through pop-ups and promotions on various web sites with the survey
normally undertaken immediately
Gamifying. Not all fun and games
Are We There Yet? Where Technological Innovation is Leading Research 3
2. Our evidence to support gamifying surveys
Why we decided to run our own experiment
Gamification as a means to improve online surveys has been talked about for almost a decade and
numerous experiments looking at the impact of gamification on the participant experience have been
conducted.
As early as 2008, Puleston & Sleep reported the following benefits:
1. Less straight-lining: up to 80 percent lower levels in some experiments,
2. Lower neutral scoring: average 25 percent lower.
3. Lower dropout (if questions are designed ergonomically): able to reduce from 5 percent to 1 percent
in test experiments.
Gamification has received considerable exposure at industry conferences and a significant number of
papers have been presented concerning its effectiveness. Puleston (2012) described it as “the most
powerful and effective means we have ever come across to encourage participants to put more thought
and effort into taking part.”
Nevertheless, as researchers, we should always be aware of publication bias and at least one paper by
Koenig-Lewis, Marquet & Palmer (2013) has cast doubt on a number of the claimed benefits of
gamification including improved response rates and elements of the participant experience. Even its
leading advocate has commented, “There is also the difficulty of squaring off the objectives of a piece
of research with the objectives of a game. Often, when we have thought about putting some of these
ideas into practice, we have found the two can lead you in different directions” Puleston (2012).
Additionally, survey context is important and examination of the published case studies and conference
papers led to us to conclude that there are certain types of survey where the evidence base for
gamification is weaker, specifically:
B2B surveys (or mixed B2B and consumer surveys)
Global surveys involving developing markets
Surveys involving the need to provide accurate behavioural data rather than brand perceptions
(which by their nature are less tangible).
Since these surveys constituted a significant proportion of our online research, we felt that the evidence
base was still insufficient to recommend this approach to many of our largest clients.
Our experiment
Therefore, working with Lightspeed GMI, we commissioned an experiment based on an existing project
which includes a mixture of consumer and B2B interviews, is global in nature, and whose main objective
is to provide detailed behavioural information. In addition, we believed this survey provided a particular
challenge as the topic (purchase and usage of printer ink or toner cartridges) is of little intrinsic interest
to consumers and businesses. Additionally, to collect data on the widest selection of question types, the
normal survey questionnaire was expanded to include a typical customer satisfaction battery and a suite
of agree/disagree statements of the type normally used to allocate participants to segments.
Phil Stubington, Charlotte Crichton
4 Are We There Yet? Where Technological Innovation is Leading Research
Proponents of gamification advocate that more engaging surveys not only generate better data, they also
provide a better experience for the participant (Puleston et al). However, other experiments (Pashupati
& Weber Riley) have shown no statistically significant improvements in the participant experience. This
is clearly a key topic of interest for the panel providers and one we felt needed to be explored in our
experiment, since as well as increasing response rates for individual surveys, they have a considerable
interest in reducing panel attrition. Therefore, a number of questions were included in our experiment
to gauge participants’ reactions to the survey experience.
For the experiment, we created three versions of the survey. These were a baseline survey, an optimised
(visualised) survey and a gamified survey. The baseline survey reflected the current tracking
questionnaire in terms of both wording and presentation. The visualised survey included visualisation
and simplified questionnaire wordings in addition to features such as timers on question answers. Here
is an example of a question presented in the baseline and formatted versions:
The gamified version of the survey was the same as the visualised survey, but also included some quiz-
type questions interspersed at suitable places throughout the questionnaire. It should be noted that this
was very much ‘light touch’ gamification since the quiz type questions were discrete and did not interact
with the main question flow in a manner necessary to achieve a fully gamified questionnaire with a
coherent narrative structure.
Gamifying. Not all fun and games
Are We There Yet? Where Technological Innovation is Leading Research 5
In addition, all three survey designs included one standard open-ended question, one open-ended
question with an alternative wording recommended by a number of papers on gamification, and one
open-ended question with a timer. Including all three open-ended question formats in all three versions
of the survey allowed us to look their impact independent of the other aspects of the survey design.
We ran the experiment in three markets – the US, the UK and India - and conducted 100 interviews
using each version of the survey in each market (for a total sample of 900 interviews). In each of the
cells, the sample was evenly divided between consumers and businesses.
The remainder of this section discusses the key findings from this exercise, along with more recent
evidence where we have incorporated elements from the experiment into the original client
commissioned project, which was the basis of the survey design.
The results
One of the key questions in the survey is about which model of printer people own, since the client
requires the results to be segmented by printer platform in order to be useful. Typically, there is quite a
high level of non-response at this question and, if participants don’t know their printer model then they
have to be screened out of the survey. With the aim of improving participant engagement and thus the
number of participants selecting a printer at this question, in both the visualised and gamified versions
of the test we modified the question and turned this into a timed task. We told participants they had two
minutes to check which printer they owned and showed a timer on the screen counting down.
Participants did not have to use all the allocated time checking their printer and could click next at any
time before the countdown had finished to select their printer on the next page.
The result was a significant increase in response rates shown in the chart below. Similar results were
achieved on a second question towards the end of the survey where we asked people to go and find their
ink or toner cartridge packaging to check the brand. Again, with a time challenge we got notably more
cooperation.
Phil Stubington, Charlotte Crichton
6 Are We There Yet? Where Technological Innovation is Leading Research
The revised question format has since been imported into the main survey and dramatic improvements
in the completion rate provide compelling testimony to the effectiveness of improved questionnaire
design:
Proportion of participants identifying their printer model
Main survey 2015 Improved survey 2016
Germany 83 percent 95 percent
UK 82 percent 94 percent
United States 82 percent 94 percent
Russia 82 percent 87 percent
In the core section of the main survey, we needed to know the different types of cartridges that have
been bought over the past six months and the quantity of each purchased. Again, we used a more visual
approach to ask this question, showing both the usual text about the type of cartridge, an image of each
type of cartridge and the use of colour to denote whether we were asking about black or colour cartridges.
In the gamified version, we enhanced this question further by using a counter instead of a text box to
capture how many cartridges had been bought.
84%
83%
79%
89%
89%
88%
88%
93%
87%
Could name inkjet printer model
Could name laser printer model
Checked brand of ink or toner oncarton packaging
Baseline Visualised Gamified
Gamifying. Not all fun and games
Are We There Yet? Where Technological Innovation is Leading Research 7
In a manner similar to the screener question about the printer owned, this produced a significant increase
in the response rate.
One type of research that is particularly resistant to being redesigned is the tracking or continuous
survey, because of concerns that the data collected won’t be comparable with data collected from
previous waves. These fears are not unjustified, as Puleston (2012) noted, “the answer to this question
[what impact does it have on the data] is not inconsiderable. Often the results can be measurably
different.” The survey that formed the basis of this experiment is a tracking survey.
Therefore, we also needed to be confident not merely that we were getting a better response rate, but
that the results had not changed due to any redesign. Within the experiment, we were able to compare
data on purchase levels of cartridges in the previous six months between the different versions, and as
the chart below shows, they were very consistent across all three versions. Since the experiment has
been completed, we have incorporated the same techniques in the main survey and year-on-year
comparisons confirm that the main impact is an increase in response rates, with the tracking data itself
remaining within the margins of error of the previous wave.
Phil Stubington, Charlotte Crichton
8 Are We There Yet? Where Technological Innovation is Leading Research
The survey also included some rating questions of the type normally used in customer satisfaction
surveys. To gamify these we changed the questions from grid-type questions to a sliding scale with a
face that changed expression as the slider was dragged along the scale. By changing from a clearly
anchored scale to a more reactive scale, we got a wider distribution of responses across the scale as
demonstrated by a higher standard deviation. We believe the higher standard deviation indicates greater
consideration given to the response and in our view data that are more accurate. Similar tests were run
on the ‘agree’ – ‘disagree’ scales typically used for allocating participants to segments and the same
results were achieved. This is especially useful, since large numbers of participants, offering similar
ratings on these statements makes it extremely difficult for researchers to produce clear segments,
especially using techniques such as k-means cluster analysis.
The participant experience
At the end of the survey, we asked some feedback questions to participants about their experience of
taking part. We then compared responses to these questions across the three different surveys and we
found that participants were more satisfied with the experience of completing the visualised and
gamified versions. In addition to being more satisfied, participants also found the visualised and
gamified surveys easier to understand and valued the variety of question types, especially with the
gamified version.
Gamifying. Not all fun and games
Are We There Yet? Where Technological Innovation is Leading Research 9
There were some differences in the impact of visualisation and gamification between the three markets
covered; in particular, there was less impact from visualisation and gamification in India than in the
UK and the US. We believe that this is due to the participants in India having done fewer surveys and
therefore having less fatigue from unengaging surveys. We also saw slightly more of an impact,
particularly of gamification, in the UK rather than the US. This is a bit harder to explain given that
both sets of participants were equally experienced. Perhaps here in the UK we have a more light
hearted culture, or perhaps there was something a bit more British in style about the survey wording
and quiz questions.
Finally, we asked participants to rate the length of the questionnaire and compared it with the actual
time they spent completing it. This is an interesting topic, since on one hand the industry is being
pushed to shorten questionnaires, especially for those completing on a mobile device; whilst on the
other we are trying to introduce techniques to make participants’ take a more considered approach to
their answers. In our case, participants spent longer completing the gamified version, yet their ratings
for how acceptable the questionnaire length was were actually slightly better.
Conclusions
We know that many research teams do not believe that their surveys are suitable for gamification and
that this problem is particularly endemic for B2B surveys and trackers. What this exercise showed was
that even simply presenting the questions better can produce significant increases in response rates and
enhanced participant satisfaction. It is fair to say that from this particular experiment the impact of
gamification was less uniform. For some metrics (e.g. participant satisfaction), the gamified version
was clearly better than the visualised only version, whereas for others there was no measurable
difference. However, gamification can be taken a lot further than we did in this case and we might
describe our approach as gamification ‘light’. A fully gamified survey can be made more integral to
the survey theme, with gaming rewards being linked to full and thoughtful answers and there is plenty
of evidence to show that this can be highly impactful.
When it comes to trackers, the continuity issue cannot be dismissed completely. There is always the
possibility of data discontinuity when any aspect of tracking survey methodology is changed. Outside
of this experiment, we have seen the impact of changes in question wording on responses particularly
to attitudinal questions and even significant impacts from changing sample source between panel
providers. However, that is not sufficient reason to continue with bad practice and the evidence from
our experiment showed that the changes to tracking data could be manageable (indeed they certainly
appear to be less than the changes seen when many studies migrated from either telephone or face-to-
face to on-line). However, as with those changes, we would always recommend a period whereby both
options run in parallel and, if needed, subsequent calibration of the results.
Phil Stubington, Charlotte Crichton
10 Are We There Yet? Where Technological Innovation is Leading Research
3. Latest techniques for gamifying surveys
Our understanding of what works and what doesn’t work is improving at both a theoretical level and the
evidence base from experimental surveys is increasing. Time does not allow for a comprehensive review
of all the literature in this section, even less the more recent unpublished work from the panel providers,
instead our intention here is to highlight some relatively simple approaches, which we or others have
found successful.
Survey introductions and response rates
Much of the literature surrounding gamification cites improved completion rates, yet the issue of
response rates is equally important. To an extent, these are less susceptible to some of the gamification
techniques, but attention still needs to be paid to both the design and wording of survey invites otherwise
all our efforts will be wasted!
Adams and Hunt’s (2013) write-up of large scale field tests by the Financial Conduct Authority (FCA)
on different forms of written communication with customers provides some very interesting evidence
that changes to the layout, content and in some cases even the signatory can have a dramatic effect on
response rates. Cape (2016), drawing upon self-determination theory, suggests that web survey
invitations should reflect the following guidelines:
Autonomy – you are free to do this or not as you choose
Competence – you are good at this
Relatedness – people like you do this
Value – what you are doing has meaning
We recently had the chance to explore this whilst running a study on behalf of the FCA and our results
showed that even limited changes to the introduction script could produce significant improvements in
cooperation rates.
Version 1 Version 2
We are conducting a random survey of
organisations that are registered with the FCA
and would like to ask you some questions
relevant to consumer credit. Depending on your
answers the survey is likely to take about 20
minutes.
We are speaking with organisations that are
registered with the FCA. The main discussion
will be about forthcoming changes that will affect
your business. It is designed to improve the
FCA’s communications with you and is likely to
take about 15-20 minutes.
Cooperation rate = approx. 50 percent Cooperation rate = approx. 65 percent
Guessing games and quizzes
In addition to the use of game type questions to collect data, stand-alone guessing games and quizzes
can also be used to break up surveys and re-engage the participant with the survey they are taking. They
can include rewards for right and wrong answers or time pressure or a finite number of guesses to
encourage increased attention. A common theme can then be carried on through the survey with short
guessing games or quizzes at intervals throughout the questionnaire to prevent participant fatigue during
longer surveys.
Gamifying. Not all fun and games
Are We There Yet? Where Technological Innovation is Leading Research 11
As described in the previous section, we used a number of such questions in our experiment and they
appear to have resulted in higher satisfaction ratings from the participants.
Subject matter quizzes are also starting to be used as a screener component and may prove effective for
ensuring that only people who truly qualify for that survey go on to complete it. This idea seems
especially promising for B2B surveys being drawn from consumer panels since quiz questions can be
used to check whether participants’ knowledge of the subject matter is consistent with their claimed job
function.
However, it is important to use guessing games and quizzes with caution. Puleston (2012) comments
“we found these more elaborate approaches had mixed appeal and could have a somewhat corrupting
impact on the data. Also, participants got confused between fulfilling the game mechanics and the tasks.”
To minimise the chances of this occurring we would recommend the following guidelines:
Guessing games and quizzes need to complement the survey and not distract from it to be an
effective tool in encouraging participation and engagement
They need to be in keeping with the survey and visually reflect the look and feel of the survey
The time needed to complete the game needs to be proportionate to the over survey length and
purpose
If participants are rushing the survey to get to the next guessing game or quiz then these are not
effective additions to the survey and the survey would be better without them.
Guessing games and quizzes can also be used as part of the incentive structure and/or to encourage
participants to engage over a longer period (for example in a diary exercise). Telling participants that if
they check back at a later time they can see how they did in the guessing game can be an effective way
of breaking up a long survey and minimising participant fatigue.
Survey narrative and quests
Normally we write questionnaires following a structure, starting with broad information about the topic
before narrowing to more specific aspects, for example, following the purchase or decision-making
journey. However, this structure is not always communicated to the participants nor integrated into the
design of individual questions.
Phil Stubington, Charlotte Crichton
12 Are We There Yet? Where Technological Innovation is Leading Research
The idea here is that through applying a more overt narrative to a survey, participants can follow this
through the survey completing tasks (questions) which reflect aspects of this. Taking this a stage further,
quests revolve around wording of multiple questions to make them more interesting for participants.
Anecdotally this is one of the harder aspects of gamification to implement, for example, we were unable
to integrate it into our survey experiment. Nevertheless, creating a scenario and using this to position
questions can make the survey process more interesting and engaging for participants.
Puleston and Sleep (2011) found that turning a question about how much music artists were liked into a
task where the participants were asked to build a playlist of how much each artist should be played
resulted in more artists being evaluated. The standard question resulted in 83 artists being evaluated
whilst the questified version resulted in 148 artists being evaluated.
Applying rules to what participants can do
Rules can be applied to what participants do to encourage more considered answers to questions. This
is a relatively simple way of capturing more considered answers to survey questions. Puleston (2012)
lists a number of ways rules can be applied to questions:
Adding a specific scenario to a question
Adding a timer to the question
Restrictive rules where what the participant can do is constrained either by restricting the
number of words or placing a time limit on how long they can take to answer the question
Whittling down rules where the participant can only select a limited number of things
In our experiment, we found that these techniques appear to be especially successful in increasing the
proportion of participants providing a response to questions which require them to provide factual data
such as the model of printer owned.
Several research teams have also undertaken tests showing that gamification can produce a significant
increase in the level of detail given in open-ended questions. For example, Cape (2016) shows a doubling
of the number of characters in an open-ended reply, tested in two different languages. One of the most
effective approaches we have found is placing an open-ended question at the start of a quantitative
survey to allow the participants to tell their story. The results from a study undertaken amongst those
living with cancer illustrate this dramatically:
Cancer and its treatment can affect all aspects of someone’s life. Before we begin the survey
questions, we would be really grateful if you could tell us in your own words how cancer may have
affected the day-to-day life of the person you care for in terms of their ability to carry out practical
tasks or activities, and how it may have affected their relationships and emotional well-being. Please
also tell us about the impact caring for them had on your life.
Participation rate = approx. 85 percent wrote at least one full sentence
Average word count = 63 (across the entire sample)
Gamifying. Not all fun and games
Are We There Yet? Where Technological Innovation is Leading Research 13
4. Gamification – the reality
The situation
When we commenced our experiment, the big panel companies we work with told us that nine out of
ten surveys they were sending out were not optimised for engagement, with the survey experience being
worlds apart from people’s more typical experience online. Discussions with the panel providers in
preparing this paper suggest that this position may not have improved materially since. Based on our
discussions with clients, interviews with researchers and feedback from the panel providers we believe
there are a number of practical considerations that are significantly impeding the adoption of gamified
surveys. These considerations span everyone involved in the process – from the client, to the research
agency, to the panel providers. Resolving some of them goes right to heart of how our businesses are
organised.
One of the biggest issues when designing and implementing gamified surveys is understanding the
theory behind gamification and what is possible in terms of graphic design and survey programming.
Researchers also need a grounding in the implications of introducing gamification on response bias and
how tracking against legacy data might be impacted. A major problem in this regard is many of the
features of gamification are inherently bespoke to an individual survey. Puleston (2012) comments,
“there is no escaping the fact that it is a creative solution. There are no out-of-the-box techniques other
than perhaps point-based scoring mechanisms.” This means that quick fixes such as templating and
questionnaire libraries are not good enough solutions.
How the process of survey design makes gamification difficult
This is particularly an issue within research agencies where the process for designing a survey can make
it difficult for gamification and new techniques to be implemented. Smaller agencies may lack the staff
resource to keep up with developments, whilst larger agencies may struggle to communicate them to
large and often geographically dispersed teams. We also believe the situation is worse if the agency is
relying on external programmers from panel companies or a web survey specialist to provide the
expertise. Unlike improving the visual appearance of a web survey, which can and often is incorporated
at the programming stage without amending the questionnaire flow or wording, at least some elements
of gamification (for example the introduction of a narrative structure into the questionnaire flow) need
to be decided at the commencement of the questionnaire design process. This is where the problem often
starts!
The process for designing a survey is conventionally that a relatively junior researcher (for example at
SRE or Project Manager level) will write the first draft of the questionnaire, which is then reviewed, by
a senior researcher or director before it is shared with the client. Hopefully the junior researcher will
have been thoroughly briefed on the objectives of the study and in most agencies will either formally or
informally have access to a library of previous questionnaires or individual model questions. However,
anecdotal evidence suggests that this briefing will often not include a decision about whether or not to
gamify the survey and the implications thereof. As a result, even if the senior researcher is more familiar
with gamification, reviews the questionnaire and therefore has the opportunity to implement a more
engaging questionnaire, this is often too late in the process, and in some cases would require starting
again almost from scratch.
Phil Stubington, Charlotte Crichton
14 Are We There Yet? Where Technological Innovation is Leading Research
The immediate issue here is ensuring the entire pool of researchers is up to date with the latest techniques
and therefore the first draft of the questionnaire is gamified. The problem of knowledge about latest
techniques and gamification being held only by some individuals within an agency can be partially
solved by holding sessions to ensure that the knowledge is disseminated and creating documents to
support this. However, the challenge is compounded by the fact that many of the skills needed to
implement a gamified survey are outside the traditional research skill set. Puleston (2012) comments,
“designing survey games is a creative art form and akin to the skill required to design good advertising
- not something that everyone is capable of doing.” This might mean that agencies need to change their
quantitative research staffing structures in a similar way to how many leading qualitative teams now
include video and design specialists.
Convincing the client
Assuming the agency has managed to successfully design a gamified questionnaire, the next challenge
that is frequently experienced is reticence and uncertainty from clients. From our discussions with panel
providers, they report that migrating tracking studies with a legacy data set often dating back many years
is especially problematic and this certainly matches our experience. Tracking studies often provide
valuable management metrics to clients and senior management teams are frequently incentivised on
this basis, which creates an inherent conservatism amongst client research teams. In common with the
process inside the research agency, we believe it is vital to get the clients on-board at the beginning.
Attempting to gamify a survey, which has already been signed off by the client, will make the task
significantly harder, if not all but impossible.
It is important to understand what is driving this reluctance so that this can be overcome and we can
move the industry forward in survey design. There are a couple of straightforward actions, which can
be done to lessen the uncertainty. Firstly, running experiments such as those discussed in this paper and
sharing the results with clients demonstrates the benefits of gamifying surveys. For new studies, it will
be sensible to conduct a pilot phase and at the end of the survey gather feedback from participants on
both the experience of completing the survey and whether there were any areas where it was unclear
what was being asked. For existing studies, probably the most realistic approach is a parallel run where
the original or ‘traditional’ survey is run at the same time (but with different participants) as the gamified
survey.
Gamifying. Not all fun and games
Are We There Yet? Where Technological Innovation is Leading Research 15
The industry needs to work together
Constantly present in relation to gamification are the perceived issues of timings and costs. Online
research has given us the expectation that surveys can be run both quickly and cheaply and whilst
seemingly almost every industry conference includes at least one paper on gamification, it’s usually
followed by one on agile research or advocating that we need to undertake research faster! The reality
is also that many of the experiments on gamification (including ours) have been undertaken as stand-
alone projects to a much more relaxed timescale than a normal project would allow. Once in the field,
gamification does not increase the timings and costs of a study in the way that telephone and face-to-
face research do, but it can increase the time taken to get into the field through a more protected survey
design process and this needs to be managed by the industry as a whole.
Ironically, for our industry, we have also been very bad at sharing data on the participant experience,
which might convince more agencies and clients to migrate to using gamified approaches or at least
improved visual survey design. To the best of our knowledge, the major panel providers all collect data
about the participant experience yet the process of sharing this with the clients is patchy at best. If the
panel providers (and specialist web survey companies) were to provide benchmarked data on the
participant experience this might accelerate the process. For best effect, the industry should work
together to agree a standard suite of participant feedback questions so that clients can pool data collected
from multiple panel providers and in house. This would also allow research agencies to review their
own surveys and understand what is working and what is not.
In conclusion, we would argue that the debate in the industry needs to move on from discussing whether
gamification is worthwhile, to managing the practical implications of it. This initiative will need to cover
all parts of the industry from clients, through to research agencies and the panel providers. Industry
bodies such as the MRS and academic institutions also have a vital role to play ensuring that their
training courses reflect current best practice. Ultimately, this has to be a combined effort, since in
deciding whether to participate in a survey the participant is influenced at least as heavily by their
experiences as they are by whatever the new survey promises.
Phil Stubington, Charlotte Crichton
16 Are We There Yet? Where Technological Innovation is Leading Research
References
Adams P & Hunt S (2013) Encouraging consumers to claim redress: evidence from a field trial.
Financial Conduct Authority.
Barnes W, Bright G & Hewat C (2008) Making sense of Labour Force Survey response rates. Office
for National Statistics.
Baruch Y & Holtom B (2008) Survey response rate levels and trends in organizational research.
Published in Human Relations.
Cape P (2016) Gamifying questions using text alone. Paper presented at IIeX Europe March 2016.
Landy L (2008) Response Rates Finding Direction Bite Sized Thought Piece. Ipsos MORI.
Koenig-Lewis N, Marquet M & Palmer A (2013) The effects of gamification on market research
engagement and response. Swansea University School of Business.
Mavletova A (2014) A gamification effect in longitudinal web surveys among children and adolescents.
Published in the International Journal of Market Research Vol 57.
National Readership Survey (2014) NRS Response Rates.
Pashupati K & Weber-Raley L (2015) Gamification in Survey Research: Do The Results Support The
Evangelists? Paper presented at CASRO Digital Research Conference.
Pew Research Centre (2012) Assessing the Representativeness of Public Opinion Surveys. Pew
Research Centre.
Puleston J (2012) Online research, game on! Paper presented at NCRM conference, Web Surveys for the
General Population: How, Why and When?
Puleston J & Sleep D (2011) The game experiments: researching how gaming techniques can be used
to improve the quality of feedback from online research. Paper presented at ESOMAR Congress, 18–
21 September, Amsterdam.
Schonlau M, Fricker R & Elliott M (2002) Conducting research surveys via e-mail and the web. Rand
Corporation.
Wells C (2015) To Game or not to Game? An investigation of the impact of survey visualisation and
gamification. Published in the International Journal of Market Research Vol 58.
Gamifying. Not all fun and games
Are We There Yet? Where Technological Innovation is Leading Research 17
About the authors
Phil Stubington, Director, Breaking Blue
Phil is an accomplished researcher, having spent 30 years working in the sector, and is a member of the
Board at Breaking Blue. Phil’s particular expertise is in developing solutions to markets clients
previously considered “un-researchable”, with a focus on tracking studies concerning usage, market
sizing and market share. He also sits on our Quantitative Development team, which has a remit to
improve our research offer through adoption of better research design and techniques, as well as more
effective ways of presenting results. He is a full member of the Market Research Society and a member
of the Royal Statistical Society. He can be contacted at Breaking Blue, Priory House, Battersea Park
Road, London SW8 4BG; tel. 020 7627 7720; e-mail [email protected]
Charlotte Crichton, Research Manager, Breaking Blue
Charlotte is a skilled research manager, taking projects from commencement and questionnaire design,
through fieldwork to final reporting and recommendations. She is skilled in the handling and analysis
of large datasets, frequently working across many different markets simultaneously to co-ordinate
fieldwork and data processing. Although she regularly manages complex multi-methodology projects,
Charlotte specialises in online research. Charlotte manages the project on which our gamification study
was based and was one of the team involved in the experiment itself. She is a member of the Market
Research Society and the Social Research Association. She can be contacted at Breaking Blue, Priory
House, Battersea Park Road, London SW8 4BG; tel. 020 7627 7750; e-mail