ADVANCED RESEARCH,

MEASUREMENT, &

EVALUATION

MELISSA D. DODD, PH.D., APR

@MELLYDODD

ASSISTANT PROFESSOR, ADVERTISING-PUBLIC

RELATIONS

NICHOLSON SCHOOL OF COMMUNICATION

UNIVERSITY OF CENTRAL FLORIDA

MELISSA.DODD@UCF.EDU

WELCOME TO CLASS!

VALUE

BARCELONA PRINCIPLES (2010, 2015)

1. Goal setting and measurement are fundamental

2. Measuring communication outcomes > outputs alone

3. Effects of PR on org performance can/should be measured

4. Measurement & evaluation require both quantitative and qualitative methods

5. AVEs are not the value of communication

6. Social media can/should be measured

7. Measurement and evaluation should be transparent, consistent, & valid

BOTTOM LINE:

• A fundamental, strategic, competitive differentiator

• The ability to better describe, understand, predict, and control

• Even if you don’t perform it, today’s climate calls for our ability to understand research and data (and the ability to distort it/truth)

DEFINING

• Research: Appropriate conceptualization and

methodologies allow valid and reliable collection of data.

Controlled, systematic, objective.

• Measurement: The ability to quantify data or summarize

qualitative data

• Evaluation: Was I successful? How successful was my

campaign or program? How can this work alongside other

organizational indicators of success?

• Metrics (KPIs): Specific tools we use to measure and

evaluate (Reach, impressions, share of voice, etc.)

• Insights: Data are applied meaningfully to specific

business scenarios

Source: American Evaluation Association

http://aea365.org/blog/john-lavelle-on-describing-evaluation/

HOW ARE RESEARCH, MEASUREMENT, & EVALUATION CONNECTED?

Begin at different points and hourglass funnel to using the same methods to

answer questions and analyze data. Diverge again when reporting outcomes.

BOTTOM LINE:

• Research methods are foundational to understanding proper measurement

and evaluation

University of Southern California & The Holmes Report

annual Global Survey (2016):

• Reach (68%)

• Impressions (65%)

• Content Analysis (64%)

• Advertising Value Equivalency (30%)

STEPS

1. Clarify organizational goals

2. Secondary/Benchmarking research

3. Set measurable objectives

4. Use the right research methods

5. Measure, Analyze, Evaluate

6. Present to CEO, senior leadership

CLARIFY GOALS & OBJECTIVES

WHAT ARE THE BUSINESS GOALS AND OBJECTIVES?

WHAT ARE THE COMMUNICATION OBJECTIVES?

ASK THE RIGHT QUESTIONS (in order):

• DEFINITION: What is it? How do I operationalize it?

• FACT: Does it exist? I what quantity? Do groups differ or did changes

occur over time?

• VALUE: How good or bad is it? How well was it done?

• POLICY: What should be done?

SECONDARY/BENCHMARKING

RESEARCH

• DO YOUR HOMEWORK

• UNDERSTAND WHAT HAS BEEN DONE AND HOW IT WAS

DONE

• REDUCES COSTS

• NEEDLESS REPLICATION

• PROVIDES POTENTIAL BENCHMARKS

• AGAINST WHICH PROGRESS CAN BE EVALUATED

SET MEASURABLE OBJECTIVES

1. AWARENESS (informational)

2. ACCEPTANCE (motivational)

3. ACTION (behavioral)

• MEET BUSINESS OBJECTIVES

• SPECIFIC AND RESULTS ORIENTED

• REALISTIC, MEASURABLE

Awareness

Acceptance

Action

To increase the number of senior public relations

professionals who know how to write correct objectives

to XX by December 15. (Informational Objective)

To increase the percentage of senior public relations

professionals who think measurement is essential to PR

practice by 15-20% by July 1. (Motivational Objective)

To increase the number of senior public relations

professionals who actually write measurable objectives

from 5% to 25% by June 1. (Behavioral Objective)

USE THE RIGHT METHODS

WHAT RESEARCH NEEDS TO OCCUR?

• Questions lend themselves to different methods of quantity and

quality

• Triangulated is best practice.

1. QUANTITATIVE: Large samples, generalizability

2. QUALITATIVE: Small samples, in-depth knowledge

3. TRIANGULATED: Secondary, Quantitative, Qualitative

RESEARCH METHODS 101

QUANTITATIVE

Probability Sampling

Surveys/Polls

Field Studies

Experiments

Content Analysis

QUALITATIVE

Nonprobability Sampling

Participant observation

Interviews

Focus groups

Case studies

MEASUREMENT

QUANTIFICATION: Assigning numbers to categories

LEVELS OF DATA:

1. CATEGORICAL

• Nominal

• Ordinal

2. CONTINUOUS

• Interval

• Ratio

ATTITUDINAL MEASUREMENT

LIKERT-TYPE SCALES

Strongly

Disagree

(1)

Disagree

(2)

Neither

Agree

nor

Disagree

(3)

Agree

(4)

Strongly

Agree

(5)

I am a PR champion.

I am a leader.

I am generally awesome.

I am a measurement god.

STRATEGIC ACTIONS

INPUTS: what you put into the

communication

• Secondary research

• Target audiences

• Strategic planning

OUTPUTS: the communication

produced

• Content, materials, activities

distributed to target audiences

• Categorized by Paid, Earned,

Shared, Owned (PESO)

OUTTAKES: what your target audiences took out of your

communication

• Unique visitors, Views, Likes, Followers, Fans

• Clickthroughs, Downloads, Comments

• Sentiment

OUTCOMES: evidence of effects of communication on target

audience

• Awareness, Attitudes, Actions

• Trust, Loyalty, Reputation, Relationship

• Return on Investment (ROI)

SURVEY METHOD

AWARENESS, ATTITUDES, ACTIONS = OUTCOMES

1. Determine the questionnaire content, scope, purpose

2. Identify the sample

3. Write questions (closed, open-ended, scales),

demographics

4. Distribute

5. Analyze data

• PURPOSE

What is the purpose of this survey?

What do I hope to learn? (RQs)

How will the data I collect influence decision-making?

For example: Reputation

Source: The Harris Poll http://www.theharrispoll.com/reputation-quotient/

• SAMPLING

WHO DO I WANT TO RESEARCH?

TARGET AUDIENCES

• Demographics, Psychographics

• Age, Geography, Income, Personality, Lifestyle

PROBABILITY v. NONPROBABILITY

• As related to research purpose

• Generalizability

• p<.05, 95% confidence

• Free online resources: Random Number Generators, Sample Size Calculator

• Magic number = 384

PROBABILITY

• Simple Random

• Systematic Random

• Nth

• 500/217 = 2.30

NONPROBABILITY

• Snowball

• Convenience

• WRITING QUESTIONS

INTRODUCTION

CLOSED-ENDED*

• Likert Scale items

OPEN-ENDED

• Other (Please specify)

• Why do you feel this way?

DEMOGRAPHICS

• ANALYZE DATA

Strongly

Disagree

(1)

Disagree

(2)

Neither

Agree

nor

Disagree

(3)

Agree

(4)

Strongly

Agree

(5)

I admire this company. X

This company is a good

place to work.

X

This company has high

quality products.

X

This company supports

good causes.

X

I intend to purchase from

this company.

X

PRESENTING DATA

FREQUENCIES, PERCENTS

• Categorical data

MEAN, MEDIAN, MODE, SD

• Continuous data

• Mean

• Standard Deviation

EVALUATION & REPORTING

Objectives – Were they met?

• Pre-Test, Post-Test

• Benchmarks

Performance - KPIs

Return-on-investment

• Per Capita Cost

• Break Even Point

Dashboard

Scorecard

Quantitative meets Qualitative

Insights

Source: https://www.targetdashboard.com/site/kpi-dashboard-best-

practice/

To increase the number of senior public relations professionals who know how to write correct objectives to XX by December 15. (Informational Objective)

To increase the percentage of senior public relations professionals who think measurement is essential to PR practice by 15-20% by July 1. (Motivational Objective)

To increase the number of senior public relations professionals who actually write measurable objectives from 5% to 25% by June 1. (Behavioral Objective)

GOAL Enhance measurement and

evaluation efforts

RESOURCES

International Association for the Measurement & Evaluation of Communication (AMEC)

Institute for Public Relations (IPR)

• Measurement Commission

• Public Relations Research Standards Center

International Public Relations Research Conference (IPRRC)

Public Relations Society of America’s “Business Case” Resources

Evaluating EPA’s Programs, Logic Model Training, Evaluation Tools, & Program Evaluation Reports https://www.epa.gov/evaluate

ONLINE CALCULATORS

Sample Size Calculator:

http://www.surveysystem.com/sscalc.htm

Random Number Generator: https://www.random.org/

CASE STUDY ACTIVITY