Evaluating the effectiveness of social media in promoting audience engagement

Jennifer BerktoldSenior Study Director

Westat

National Conference on Health Communication, Marketing, and Media, Atlanta, Georgia, August 11, 2009

The Boss of Me case study

“I don’t want no drama in my life,

even though we have a little bit, but

no more letting people control you.

That’s drama, because then you

become something that you’re not.”

-- Mary J. Blige

INTIMATE PARTNERVIOLENCE

Teens don’t achievetrue Intimacy

Partners are constantlybeing redefined making the word less relevant Reducing violence isour ultimate goal butstrategy should be prevention

EARLY RELATIONSHIP CONFLICTS

Formative years should be emphasized

Relationship is a more open term reflective of shifting expectations

Conflicts can be a frequent precursor to violence

Reframing the Issue

Five key strategies from research for reaching teens (ages 14 – 18, both genders, all races and ethnicities) on this issue:

2) Target the “drama” with humor and authentic teen language

3) Promote self-empowerment

4) Offer tools and resources to “dial down the drama”

5) Leverage teens’ organic media culture

1) Take an empathic and non-judgmental stance

The Strategic Approach

Being the boss of you means you’re in complete control of you at all times in your relationships. It means you refuse to submit to your own feelings of anger, fear, jealousy or insecurity. When you allow your emotions to negatively influence your actions, you lose control. And when you lose control, you’re not the “boss of” anything.

BOM (“Boss of Me”) is dedicated to helping young people stay in charge of their own relationship drama.

BOM provides tools to help teens maintain and manage their cool, instead of losing it, during a relationship conflict.

BOM or “Boss of Me”

Organizational Partners

Funder

Managing the Campaign

Brand Management

Evaluation

Evaluating BOM

Research goals

• Identify delivery mechanisms that engage young people with BOM

• Understand how young people engage with BOM– Gender differences

• Understand how the campaign impacts its target audiences

Intended uses of BOM evaluation results• Learning, best practices

• Communicate with the teen dating violence prevention community– Effective delivery mechanisms to reach teens– Engaging messages / content– Appeal to new audiences?

Basic model of an outcome evaluation

Products from activities

Web traffic

Number of radio shows developed

Number of ads

Number of events

Reach

Inputs OutcomesActivities Outputs

Resources

Money

Staff and volunteer time

Facilities

What is done

Website

Radio show

Ads / PSAs

Community events

Changes for participants

Increased knowledge

Attitudinal change

Behavior change

Outputs, Outtakes, Outcomes

Outtakes

• Awareness• Buy-in• Engagement• Participation

Outcomes

• Knowledge change• Attitudinal change• Behavior change

Outputs

• Products• Activities• Reach

New Logic Model, Simplified

Products from activities

Web traffic

Number of videos developed

Number of ads/ad buy

Number of events

Reach

Inputs OutcomesActivities Outputs

Resources

Money

Staff and volunteer time

Facilities

What is done

Tools development

Website development

Ads / PSAs

Community events

Contests

Changes for participants

Increased knowledge

Attitudinal change

Behavior change

Outtakes

Participation

Creative content

Conversation content

Tone of conversation

“Friending”

Texting opt-in

Approach

Heard of BOM but hasn’t engaged Radio ads / PSAs Outreach at school

Our challenge…

Engaged with BOM BOM411 site visits Social media friends Text messaging Community events User-generated content Hasn’t heard

of BOM

5-Point Evaluation Plan

• Monitoring• Content analysis• Web survey of teens• Pop-up survey at BOM411• Visits to community events

Web Monitoring

BOM Sites• BOM411• Facebook• MySpace

(2 sites)• Twitter• YouTube

Outside BOM• RSS (e.g., Google

blogs, Google news)

Data• Google analytics• Comment text• Comment analysis

• Articles/posts with BOM mentions

17

Content analysis

Subjects• Posted content on

BOM411• Posted content on

social networking sites

• User-generated content– Skateboard

designs– Rap lyrics

• Outside buzz

Measures (some examples)

• Type of conversation

• Affect • Attributes• Type of information

shared

Other Monitoring

Subjects

Radio

Cell phones

Voicemail line

Measures (some examples)

GRPs

Number of subscribers

Number of texts

Number of phone messages

Web survey of teens

Sample• Teens 14-18 in SF

Bay Area and Los Angeles

• Recruited from panel that agreed to take surveys online

• Parental consent obtained

• Knowledge Networks

Measures• Self-reported

exposure to campaign

• Awareness of relationship drama among peers

• Attitudes toward / relationship with BOM

• Self-reported attitude/behavior change

Pop-up survey at BOM411

Sample• Random selection

of visitors to BOM411

Measures• How found out

about website• Engagement with

website content (e.g., rating different parts of website, would you recommend to others)

Visits to community events

Sample• Random intercept

interviews

• Interviewer observation

Measures• How found out

about event• Why attending the

event• Activities performed

at the event• Description of event• Estimated

attendance

Timeline

Task Date

Site Launch August 24

Monitoring Launch thru February 2010

Content Analysis Launch thru February 2010

Other monitoring Launch thru February 2010

Pop-up survey September 2009, January 2010

Community events October 2009

Web survey of teens October /November 2009

Questions

Thank you!

Jennifer Berktold

Senior Study Director

Westat

jenniferberktold@westat.com