1

voorwoordtowards a new metrics model

introduction

How do you measure transmedia? What types of metrics will help transmedia producers to better understand and subsequently compare and contrast the impact of a transmedia story? These are million-dollar questions begging for answers. Measuring engagement is important because it contextualizes what an audience size means and how a transmedia production is performing. It helps producers and storytellers to see what actually gets fans to engage. Measuring engagement, however, can be desirable from different perspectives. A producer or marketer will be interested in metrics to evaluate the campaign in terms of return on investment. A storyteller specifically wants to know how the audience responds to the narrative.

We believe that measuring engagement means placing audience size into the broader context of how the transmedia production as a whole is actually performing. Stakeholders in the production can see where, how and when fans engage, which allow them to make refinements. We aim to present an engagement model that can easily be integrated into the daily activities of a transmedia storyteller.

In this research, we choose to focus on the goals of the storyteller. We propose a model that can be used to create and give direction to a transmedia production team of writers and performers.

MOtivation

At the Fontys ACI Transmedia Storytelling Lab, students learn to design, develop and execute transmedia productions. In this process, the students experience all facets of transmedia

production. We discovered that the teams needed meaningful measuring tools to determine whether their campaigns performed as expected. They needed insights on audience behaviour to be able to give direction to their campaign and tap into the user need. Our goal is to design a model that can be used to create and give direction to a transmedia production.

A review of current models

User engagement definedIn their work, Attfield et al.[1] define user engagement from a broad perspective. Their definition reads: ‘User engagement is the emotional, cognitive and behavioural connection that exists, at any point in time and possibly over time, between a user and a resource’.This approach emphasises the emotional, cognitive and behavioural factors that, as a whole, define user engagement. All three factors are open to measurement.

Figure 1 The four components of Engagement (based on Forrester Research)

According to Forrester Research[2] cognitive engagement can be measured through awareness, interest and intention. It is important to measure the likelihood of a user advocating, for instance by forwarding content or inviting friends. Besides measuring influence, Forrestor Research stresses involvement and interaction. This ‘behavioural’ engagement focuses on the presence and action of the user, which can be measured by the number of visitors or the amount of time spent on a website.

How to measure transmedia experiences

2

Additionally, one could look at the click through ratio, the number of online transactions or the number of uploaded videos. Finally, emotional engagement attempts to measure the affection or aversion of a user. Intimacy is considered an emotional engagement factor and is measured by satisfaction rating or by sentiment analysis in blogs, comments, surveys and questionnaires.

What are points of interest for a transmedia storyteller?To Robert Bole, director of Broadcasting Board of Governors (US), it is clear that creators of ‘the new narratives’ need these analytics. He notes, ‘Rather than the old way of ‘more is better’ as a proxy for quality (more audience = bigger success), the digital world allows storytellers to hone their craft through nearly real-time feedback across multiple channels that can result in quick shifts, additions and changes to narratives depending on pockets of audience behaviour and interests’[3].

In his talk at the DC Digital Analytics Association Symposium in 2013, Bole defined six points of interest that need to be addressed to help understand engagement. The metrics and analytic tools that help understand the impact of stories that unfold across multiple platforms are, according to Bole,

� Narrative attribution � Conversion and pathways � Segmentation format feedback � Quantifiable impacts � Engaged loyalty � Audience-created enhancement

By measuring the narrative attribution, a storyteller can identify which story element (or media mix) converted the audience member into an engaged,

multi-channel user. Measuring the conversion and pathways helps determine ‘optimal’ paths for constructing narrative arcs. Segmentation format feedback measures the success of the format in engaging and propelling a narrative arc by audience type. Quantifiable impact is needed to measure audience participation across the narrative arc (The ‘What do people do?’ problem). Engaged loyalty means finding out if additional context and information leads to more informed, loyal and frequent users. Audience-created enhancement involves measuring how audiences add value through contributions to the narrative arc.

From points of view to actionable insightsAlthough we found several works on the needs of a storyteller, we found less information about transforming those needs into practical, actionable insights. We did, however, find a variety of models on the desired outcome of transmedia analysis. Mayfield’s ‘Power Law of Participation model’[4] is such an example. The model shows twelve stages of participation, ranging from the rather inactive ‘reader’ to that of a ‘lead’, a highly engaged and active audience member (see figure 2).

Figure 2. ‘Power law of Participation model’ (Mayfield)

a review of current models

3

voorwoordthe need for actionable insights

Mayfield’s work relates to the ‘roles’ that Henry Jenkins describes in his work[5]. There are at least two significant roles that users can play to push a campaign forward. Jenkins mentions two of those roles: lurking and spoiling. Lurking could be defined as a stage where users look at the content produced but are not actually interacting with it or sharing it. Lurking plays an important part in the evaluation phase. ‘Lurkers’ are those who are interested but not ready (or may never be ready) to start interacting with the transmedia campaign. Another important role is the spoiler. ‘Spoilers’ are a very active group of people that seek to know everything that can be known about a campaign. Often, they group together on forums or Facebook groups, out of sight of the campaign creators They also create blogs related to a campaign.

Pratten[6] defines five levels of increasing engagement: attention, evaluation, affection, advocacy and contribution. In his work, Pratten proposes measurements aiming specifically at measuring engagement in a transmedia world. His five levels of increasing engagement expand Forrester Research’s earlier-mentioned four measures of engagement, by adding ‘affection’, meaning ‘affinity towards the world’.

Figure 3. Measuring Engagement (based on Pratten)

Having this kind of information about the segmentation of the participants in a transmedia production is very useful for a storyteller. He or she will be able to measure and to adjust his story accordingly, based on these insights.

Conclusion on current models

Our students found various models and methods to measure engagement (not all of which were mentioned in this paper). A range of models is available to either classify levels of engagement or to create typologies of ‘the engaged’. We need to fill the gap between the existing models to find a way of gathering meaningful insights that can be used to give direction to a transmedia production in a practical way. Based on the research performed by our students on existing models, we want to make sure that our new metrics model:

� Measures engagement and classifies the user � Transforms ‘raw’ data into meaningful insights � Tracks the user journey � Tracks the connection between different

channels (on-and offline)

4

introducing our new metrics model

We see three important aspects of a transmedia production:

� A storyworld that is to be discovered � An audience that is involved � A period of time

We consider audience members who interact with the world to be engaged users. By tracking the behaviour of individuals, we can map how they discover the world and how they interact with it over time. We record each time a user touches something in the storyworld. By listing all of these ‘points of interaction’ and structuring them into chapters, scenes and beats, we can track the journey and the hotspots of engagement for individual users as they progress through the story.

Figure 4. The new metrics model: users interact with the storyworld over a period of time

How does the tracking work?Points of interaction are predefined by the creator (the storyteller) of a transmedia production in the form of acts, scenes, plot points, beats, and so on. An audience member, referred to as a user,

progresses in the storyworld by discovering these scenes and beats, interacting with characters, solving puzzles, sharing content with friends or viewing content on specific channels like YouTube or blogs. By tracking this individual user journey through the storyworld, the storyteller receives insights into the behaviour of that individual. With this information, the storyteller is able to analyse whether the transmedia production is performing the way it should. Is all the content discovered? Does the user progress? Does s/he need help? Now, actionable insights are at the storyteller’s disposal.

Interactions are tracked using the concept of ‘toggle switches’. If a user interacts with a certain point, the toggle switch changes from OFF to ON. Picture taking a tour through a house and switching on the light when entering a room, leaving a trail of lights in all the rooms visited.

Figure 4. Toggle switches change from OFF to ON at a point of interaction

Key benefit: comparing behaviour of individuals The key benefit of our conceptual model is that the behaviour of an individual audience member can be compared to the behaviour of others. In doing so, we can interpret the relative engagement of individuals compared to some or all others at certain points of interaction as a ratio. Tracking all of the behaviours of all individuals allows us to group them on different levels.

introducing our new metrics model

5

voorwoord

Examples of relative engagement measurement: � Compare the amount of interactions of User A

in [Scene 5] to all interactions in [Scene 5] � Show the progress of User B compared to the

most progressed user � List the number of users that interacted with

[Scene 7, Beat 2] compared to all users

Interpreting gathered data The gathered data about the user’s journey through the world (points of interest touched by a user at specific moments in time) is combined with that of other users. This also opens up the possibility for ways of segmentation and classification that the reviewed models offer. Segmentation is based on actual data of individual users, which is an asset over previous models. Now, users can be classified as ‘lurkers’, ‘contributors’ or ‘cognitively engaged’ relative to all the users in the storyworld.

expert evaluation

We asked leading experts to evaluate the clarity, completeness, effectivity, applicability and benefit of our model. Amongst them are Sam Ford (Peppercomm, New York, NY), Pamela Rutledge (Media Psychology Research Center, Boston, MA), Bart Robben (Elastique, Hilversum, NL), Egbert van Wyngaarden (Transmedia Desk, Munich, DE) and Soraia Ferreira (UT Austin, Porto, PT).

Since we focus on content engagement (story) rather than on return on investment (ROI) in the development of our conceptual model, we first wanted to know what respondents sought in a new metrics model for storytellers. As a cross-section of the various models, we

selected (1) loyalty measurement, (2) participation ratio, (3) target characteristics, (4) media channel effectiveness (5) content effectiveness and also (6) ROI measurement as conventional metrics used in defining the success of transmedia productions. Therefore, we asked our respondents to rank the importance of these metrics.

Our respondents chose content effectiveness as the most important focus of a storyteller’s measurement model. The second most important was the measurement of ROI. Loyalty measurement, participation ratio, target characteristics and media channel effectiveness seemed to be less important. These results are in line with our focus on the storyteller but they show the relative importance of ROI as well.

We also asked the respondents to evaluate our model. First, we asked them to rate the following items on a scale from 1 to 10: applicability, effectiveness, completeness and clarity.

Next, we asked our respondents to explain this numerical evaluation using semi-structured interviews. In general, we found that our respondents were positive about the applicability and clarity of the model. In addition, they liked the focus on the ability to measure user journeys. At the same time, they were sceptical about the effectiveness and completeness of our model.e model

expert evaluation

What you’ve designed here is very clear. It will work very well, especially in

a system like Conducttr.

Pamela Rutledge

6

voorwoord

Positive aspects of the modelOur respondents’ feedback on the positive elements of the new metrics model:

� ‘Most models process the use, engagement or participation of one medium, but this model combines all different media and their results’ (Welhuis).

� ‘It measures the journey, and also the overall journeys’ (Ferreira).

� ‘The model would be helpful for producers who are interested in understanding how each element of their story was successful, compared to the others, or to find out in what way each audience member wants to travel through their story and engage with some parts of the story’ (Ford).

Effectiveness of the modelIn terms of effectiveness, our respondents were critical about four things:

� Missing ability to respond to something unexpected (mentioned by Vegt and others)

� It doesn’t look at ways that people might engage in discussion about the story away from the story (mentioned by Ford and others)

� The missing ability to measure unfinished episodes (mentioned by Welhuis)

� The lack of focus on the returning visitor (mentioned by Robben)

expert evaluation

In the model the soft factors are missing, the things you can’t measure with numbers

but which do really have value for a transmedia project.

Egbert van Wijngaarden

7

voorwoord

Completeness of the modelRelated to the question ‘Why is the model not complete?’, our respondents highlighted three missed opportunities:

� No focus on the financial side of transmedia measurement (mentioned by Tan)

� No monitoring of the soft side e.g. emotions (mentioned by Wijngaard and others)

� No measurement of the physical world, e.g. live events (mentioned by Ferreira and others)

towards the future Adjustments to our model based on the insights of our Our next step is to adjust the model and develop it into an actual metrics tool. Based on the feedback of the experts, we think the model could be put into practice once some improvements are made. We are very pleased that the team at Transmedia Storytellerltd has taken an interest in our research. We would like to incorporate our findings into a metrics dashboard for Conducttrtm Pervasive Entertainment Platform created by Robert Pratten and his team.

‘Our plan is to incorporate this work into Conducttr so that we present a meaningful dashboard with actionable insights rather than a simple series of

charts. More of this later!’ Robert Pratten (Transmedia Storyteller)

There are some elements that a metrics model cannot cover, such as measuring the unexpected (Vegt) and engaging in discussions ‘away from the story’ (Ford). In the process of a transmedia production, monitoring the audience is key. Storytellers listen to their audiences through social media analysis, analytics or direct feedback

from users. Once a discussion or ‘something unexpected’ emerges, storytellers can tap into this audience participation and incorporate it into the storyworld. In our view, that would mean finding a way to connect a point of interaction (i.e., a toggle switch), to the transmedia production. For example: having a character participating in the discussion or inviting users to a gathering and have them interact with the storyworld. This way, ‘the unexpected’ becomes measurable.

When we started this research, we were keen on developing a model that would be applicable for transmedia producers. We were aiming for a model that would help create a transmedia production. Adding Toggle Switches along the way became a way of thinking for transmedia production teams. When creating scenes and beats, the teams will need to establish the point of interaction. In Conducttrtm, this means thinking in the trinity of triggers, conditions and actions.

Live events and real-life interactionsTransmedia storytellers like to combine on- and offline channels in their productions. Some respondents commented on the inability to measure the physical world. In Pratten’s concept of the storyworld of things makes the physical world measurable. If storytellers do not incorporate specific ways to measure attendance to these events or make the audience interact digitally in some way, measuring attendance, let alone engagement, becomes a difficult task. We watch the developments of connected devices and crossovers between real and virtual worlds with great interest. Producers can now dispose of ways of interacting with audiences in the real world. Technologies like NFC and RFID in iBeacons or gyroscopes in smartphones make it possible to

expert evaluation

8

voorwoord

interact with a storyworld in new ways. These techniques make the interactions through it measurable as well. All you have to do is make sure there is a Toggle Switch to go with it!

Emotional engagementBoth our students and our respondents address the importance of measuring emotions or emotional engagement. Storytellers apparently want to know what moves the audience, what enhances their flow, what involves them and keeps them involved.There are studies in various fields of interest that address the measurement of emotions. We are looking forward to meeting people who are involved in this field of research, which will allow us to exchange ideas that open up the possibility of incorporating emotion metrics in our model.

References

1. Attfield, S., Kazai, G., Lalmas, M., Piwowarski, B. (2011). Towards a science of user engagement (Position Paper). WSDM Workshop on User Modelling for Web Applications.

2. Forrester Research, (2007). Marketing’s New Key Metric: Engagement. Marketing Leadership Professionals. Forrester.

3. Bole, R. (2013) Going old school: telling digital stories in analogm media. Retrieved from: http://www.innovation-series.com/2013/06/14/going-old-school-telling-digital-stories-in-analog-media/, oktober 2014.

4. Mayfield, Ross. (2006) Power Law of Participation. Retrieved from: http://ross.typepad.com/blog/2006/04/power_law_of_pa.html, oktober 2014.

5. Jenkins, H., Ford, S., Green, J. (2013) Spreadable media. New York University Press. pp. 155-159.

6. Pratten, R. (2011) Getting Started with Transmedia Storytelling. CreateSpace Independent Publishing Platform.

towards the future

9

voorwoordabout this research

About this researchThis research is conducted by Eefje Op den Buysch, Head of the Fontys Transmedia Storytelling Lab and Hille van der Kaa, professor of the Fontys Future MediaLab. A team of students at the Fontys Transmedia Storytelling lab run the interviews.

Fontys’ Transmedia Storytelling Lab was designed for the research and development of transmedia productions and their value in the digital age. The professorship of Future Media aims to develop innovative media concepts. It puts focus on the influence of technology on storytelling.

Any questions and remarks are welcomePlease contact Ms. Eefje Op den Buysch+31 885077655 or +316 22452452 e.opdenbuijsch@fontys.nl

Fontys Academy for Creative IndustriesProf. Goossenslaan 15022 DM Tilburg – The Netherlandswww.fontysaci.com

About the Transmedia Storytelling Lab at Fontys Academy for Creative Industries

Fontys Transmedia Storytelling Lab is a workshop / research lab dedicated to telling stories. Stories unfolding across various channels. Stories that require the audience to get involved. Stories that capture the imagination through form, function and content.

Fontys’ Transmedia Storytelling Lab was designed for the research and development of transmedia productions and their value in the digital age. Various disciplines involved in (the production of) transmedia stories are addressed. Target development, for example, scenario’s, monitoring tools and models. Fontys’ Transmedia Storytelling Lab looks beyond the storyline, when developing a transmedia production. A transmedia production is regarded (depending on the target) as a component of a communication plan, a marketing target or an entertainment concept. The lab values transmedia concepts that are measurable as well as controllable.

Fontys’ Transmedia Storytelling Lab stimulates and facilitates research. (Practical) research takes place within the context of projects originating from the Transmedia Storytelling Lab. There are also opportunities for academic research.

www.fontysaci.com/research/tsl-labThank you for your interest,Eefje Op den Buysch and Hille van der Kaa

10

voorwoord

transmediastorytelling lab

october 2014

werkveldgidstransmedia storytelling lab