PR Measurement Summit 2016: Jim Macnamara's Keynote Speech Presentation
-
Upload
carma -
Category
Data & Analytics
-
view
256 -
download
3
Transcript of PR Measurement Summit 2016: Jim Macnamara's Keynote Speech Presentation
Latest International Developments In Standards And Models For Evaluation Prof.Jim Macnamara PhD, FAMI, CPM, FAMEC, FPRIA Associate Dean, University of Technology, Sydney - Link
Latest International Developmentsin Standards and Models for Evaluation
Breakthroughs at Last!
Professor Jim Macnamara PhD, FAMEC, FAMI, CPM, FPRIAProfessor of Public Communication, University of Technology SydneyVisiting Professor, London School of Economics and Political Science
Measurement and Evaluation – The Answer!
This Presentation■ Review some fundamentals
―Key principles
―The basis of evaluation frameworks and models – where do they come from?
■ New frameworks for evaluation―EU evaluation framework
―US Task Force on Standards
―UK Cabinet Office evaluation framework
―AMEC Integrated Evaluation Framework
■ A taxonomy of evaluation – mapping the stages, steps, metrics, and methods ―A tool to apply evaluation
This presentation
• Goal setting and measurement are fundamental for communication and PR
• Measuring communication outcomes is recommended versus only measuring outputs
• The effect on organizational performance can and should be measured where possible
• Measurement and evaluation require both qualitative and quantitative methods
• AVEs are not the value of communications
• Social media can and should be measured
• Measurement and evaluation should be transparent, consistent and valid
MEASUREMENT
• The taking of measures• The collection and analysis of data in relation to
a particular object, process, or condition• SO WHAT?
EVALUATION
• “Making a judgement” about the value or significance of something (Oxford and Merriam-Webster dictionaries)
• Evaluation is the systematic acquisition and assessment of information to provide useful feedback about some object” (Trochim, 2006, para, 3)
• “The systematic application of research procedures to understand the conceptualization, design, implementation, and utility of interventions (Valente, 2001, p. 106)
• Assessing results against objectives
SMART objectives
■ Contain numbers, percentages, dates
■ Design measurement in the planning stage
■ Be realistic
■ Linked to organisational objectives and goals
■ Achieved by a specific date
Three Types of Evaluation
FormativeProcessSummative
• Baseline (e.g., what is the current level of awareness)
• Precedents (have other similar programs worked in the past?)
• Pre-testing (will what we propose work?)
• CEA (is it the most cost-effective approach?)
• Insights / learning (what has been learned that can inform future strategy, policy, etc?)
• Impact (what has happened as a result of the communication?)
• Outcomes – intermediate and long-term (what attitude or behaviour change has occurred?)
• Monitoring and tracking outputs (e.g., are we reaching the audience?)
• Monitoring and tracking immediate outcomes (e.g., what response are we getting?
Why Stages?
Six stages of communication (W. J. McGuire, 1968, 1969)
ComprehensionPresentation Change/actionRetentionAttention Acceptance
Stages of communication
The AIDA model of advertising (Strong, 1925)
AWARENESS
INTEREST
DESIRE
ACTION
Stages of communication■ Exposure
■ Attention
■ Interest and/or liking
■ Comprehension
■ Cognition, particularly cognitive elaboration (thinking about the message)
■ Acquiring skills or knowledge required to deal with the issue (if necessary)
■ Attitude change, particularly to agreement (what McGuire calls “yielding”)
■ Storing information in memory (retention)
■ Retrieving information (i.e., recall)
■ Deciding to act in accordance with information (intention)
■ Action/behaviour
■ Cognitive integration of behaviour (e.g., reinforcement)
■ Encouraging others to behave similarly – what McGuire called “proselytizing” and what is commonly regarded in modern marketing as ‘advocacy’ (McGuire, 1999, 2001)
Frameworks/models of evaluation
The PII model of evaluation (Cutlip, Center, & Broom, 1985)
Frameworks/models of evaluation
Macro model of evaluation (Macnamara, 1992)
Frameworks/models of evaluation
Pyramid model of PR research (Macnamara, 2002, 2005)
OUTCOMES(Functional & organisational evaluation)
OUTPUTS(Process & programevaluation)
INPUTS(Formativeresearch)
Number who . change ..…..……………………………………………….. Quantitative surveys (large scale structured)behaviour …...…………………………………………….. Sales; Voting results; Adoption rates; Observation
Number who ….. changeattitudes ……….…………………………………………… Focus groups; Surveys (targeted) (eg Customer, Employee
Number who understand messages..……………………………… Focus groups; Interviews; Complaint decline; ExperimentsNumber who retain messages …….………………………………… Interviews; Focus groups; Mini-surveys; Experiments
Number who consider messages ……....….. ………………………… Response mechanisms (1800, coupons); InquiriesNumber & type of messages reaching target audience ……………………… Media Content Analysis; Communication Audits
Number of messages in the media ……………..….`…………………… Media Monitoring (clippings, tapes, transcripts)Number who received messages ………………...……………………….. Circulations; Event attendances; Web visits & downloads
Number of messages sent …………………………………..……………….. Distribution statistics; Web pages posted
Quality of message presentation …………………………………………………. Expert analysis; Peer review; Feedback; Awards
Appropriateness of message content …………………………………….………….. Feedback; Readability tests (eg. Fog, Flesch); Pre-testing
Appropriateness of the medium selected ………………………………………….…….. Case studies; Feedback; Interviews; Pre-testing (eg. PDFs)
How does target audience prefer to receive information? …………….…………..……… Academic papers; Feedback; Interviews; Focus groupsWhat does target audience know, think, feel? What do they need/want? ………………… Observations; Secondary data; Advisory groups; Chat rooms
& online forums; Databases (eg. Customer complaints)
or Shareholder Satisfaction); Reputation studies
Measurement Methodologies: (formal & informal)
Key Steps/Stages in Communication:
OUT-TAKES(Proposed by some as a 4th stage)
Frameworks/models of evaluation■ PR effectiveness yardstick (Lindenmann, 1993)
Measuring:Behaviour changeAttitude changeOpinion change
ADVANCED
Measuring:RetentionComprehensionAwarenessReception
INTERMEDIATE
Measuring:Target audiencesImpressionsMedia placements
OUTPUT
LEVEL #3
LEVEL #2
LEVEL #1
PUBL
IC R
LEAT
ION
S EF
FECT
IVEN
ESS
YARD
STIC
K
OUTCOMES
OUTGROWTHS
OUTPUTS
Frameworks/models of evaluation
Noble & Watson’s united model (Noble & Watson, 1999)
INPUT STAGEPlanning & Preparation
OUTPUT STAGEMessages & Targets
IMPACT STAGEAwareness & Information
EFFECT STAGEMotivation & Behaviour
Tactical feedback
Management feedback
Frameworks/models of evaluation
Communication controlling (DPRG/GPRA, 2000; DPRG/IPV, 2009)
Frameworks/models of evaluation■ The European Commission’s (EC) Better Regulation Guidelines uses the terms:
― Inputs―Outputs―Results― Impact
■ European Commission model (EC, 2015)
10 political priorities
Communication objectives
Activities Relevance Output Outtake Outcome
Recall indicators measuring the extent
to which the communication
activity is likely to have satisfied,
captured the attention of audiences, or raised
knowledge and awareness focussing
on the audience directly reached
Political matchOf the communication activities undertaken with the 10 political
priorities
• Organising events• Providing
information• Working with
information networks and desks
Reach indicators measuring the extent to which extent the
communication activity is likely to
have reached the right target audience
quantitatively and qualitatively
Engagement Trust indicators measuring the extent to which the communication
activity led to either a discernible action being taken or the
desired change in the target audience’s
perception
Inputs Results
Did the activity match the 10 priorities?
Did the activity reach the target?
Was the message received?
Did the activity meet the expectations of the target audience?
Did the action lead to the expected result?
Did the action change the perception of the EU?
Q.
Terms• Inputs / preparation
• Activities / implementation
• Outputs
• Outtakes
• Outcomes / outflows / outgrowths
• Results / effects / impact
Program theory and program logic models
Basic program model evolved since the 1970s (Kellogg Foundation, 2004, p. 1)
Program theory and program logic models
University of Wisconsin University Cooperative Extension Program (UWEX)(Taylor-Power & Henert, 2008, p. 5)
What are the standards for stages/terms?Program and PR evaluation models (1985–early 2000s)Basic
program logic model – UWEX (Taylor-Power & Henert, 2008)
Expanded program logic model – UWEX (Taylor-Power & Henert, 2008)
Classic Program Logic Model (e.g., Kellogg Foundation (1998/2004)
Cutlip, Center & Broom (1985) ‘PII model’
Macnamara’s Macro model’ (1992)
Lindenmann’s Effectiveness Yardstick (1993, 1997a)
IPRA Gold Paper on Evaluation (1994, pp. 10, 18–19)
Fairchild (1997, 2001); Fairchild & O’Connor, IPR Toolkit ( 1999)
Lindenmann Standards and Guidelines, 1997b, 1997c)
Noble & Watson’s ‘Unified Model’ (1999)
Grunig & Hon ‘relatinships’ model
DPRG/ GPRA (2000), DPRG/ICV (2009) models
Likely Performance Measurement Framework (2000)
Macnamara’s ‘Pryamid’ model (2000, 2002a, 2002b)
Lindenmann’s ‘guidelines' (2002/2003)
Input Inputs Inputs Inputs Inputs Inputs (hinted)
Input Input Inputs
Activities Planning
Output Outputs
Activities Participation
Outputs Implement-ation
Outputs Outputs (basic)
Outputs Output Outputs Output Outputs Output Outputs Outputs PR Outputs
Outtake Outtakes Outtakes PR Outtakes
Outgrowths
(inter-mediate)
Outcomes Outcomes Short-term Intermediate Long-term
Outcomes Outcomes (advanced)
Outcomes Outcome Outcomes Outcomes Outcome Outcomes Outcomes PR Outcomes
Impact Impact Results Business / organization outcomes
Impact Relationships Outflow Outgrowths Business / organization outcomes
Effect
9
14
41
125
1(Macnamara, 2016)
Theory of Change
INPUTS ACTIVITIES OUTPUTS OUTCOMES IMPACT
Communication• Objectives• Target audiences• Strategy
Actions
Short | Medium | Long term
The OrganisationOrganisation goals and objectives
Feedback loops to monitor and adjust strategy and tactics if required
Program Theory
INPUTS ACTIVITIES OUTPUTS OUTCOMES OUTCOMES OUTCOMES Short term Intermediate Long term OUTTAKES OUTCOMES IMPACT
Communication• Objectives• Target audiences• Strategy
Feedback loops to planning
Preparation Production Distribution Exposure/Reception Response Effects Results Organisation/Stakeholder Impact
ORGANISATIONOrganisation goals and objectives
Theory of ChangeProgram Theory
AMEC Integrated Evaluation Framework (AMEC, 2016)
INPUTS ACTIVITIES OUTPUTS OUTCOMES OUTCOMES OUTCOMES Short term Intermediate Long term OUTTAKES OUTCOMES IMPACT
Communication• Objectives• Target audiences• Strategy
Feedback loops to planning
Preparation Production Distribution Exposure/Reception Response Effects Results Organisation/Stakeholder Impact
ORGANISATIONOrganisation goals and objectives
Theory of ChangeProgram Theory
? ? ? ? ? ?
Inputs Activities Outputs Outtakes Outcomes Impact
INPUTS ACTIVITIES OUTPUTS OUTCOMES IMPACT
The OrganisationOrganisation goals and objectives
Feedback loops to monitor and adjust strategy and tactics if required
Short | Medium | Long term
Communication• Objectives• Target audiences• Strategy Theory of Change
Program Theory
INPUTS ACTIVITIES OUTPUTS OUTCOMES IMPACT
The OrganisationOrganisation goals and objectives
• Budget• Personnel• Expertise• Other resources
• Formative research• Creative design• Pre-testing• Planning• Production• Media relations
• Paid advertising• Media publicity• Publications (e.g.,
newsletters, reports)• Web sites• Social media posts• Events• Sponsorships• Community projects
Feedback loops to monitor and adjust strategy and tactics if required
Preparation Production Distribution Exposure/Reception Response Results
Short | Medium | Long term
• Recall• Awareness• Interest (e.g., follows,
likes, retweets)• Engagement• Satisfaction• Attitude change• Trust• Preference• Intentions• Inquiries / leads• Advocacy
• Reputation• Relationships• Public support• Sales / revenue• Donations• Cost savings• Staff retention• Customer retention• Complying behaviour
(e.g., give blood; drive safely, get fit)
• Quality of life / well-being improvements
Communication• Objectives• Target audiences• Strategy Theory of Change
Program Theory
INPUTS ACTIVITIES OUTPUTS OUTCOMES IMPACT
The OrganisationOrganisation goals and objectives
• Budget• Personnel• Expertise• Other resources
• Formative research• Creative design• Pre-testing• Planning• Production• Media relations
• Paid advertising• Media publicity• Publications (e.g.,
newsletters, reports)• Web sites• Social media posts• Events• Sponsorships• Community projects
Stakeholders, Publics, Society
Feedback loops to monitor and adjust strategy and tactics if required
• Sales/revenue data• Behaviour tracking• CRM data• Surveys• Cost benefit / ROI• Well-being metrics
• Surveys (awareness, reputation, trust)
• Interviews• Social media qual
analysis (likes, shares, comments)
• Net Promoter Score
• Media metrics (reach, impressions, OTS, TARPs, CPM)
• Content analysis (e.g., messages, tone/sentiment)
• Web site data• Social media stats• Attendance figures• Reader surveys
• Pre- surveys• Focus groups• Interviews• Baseline data
collection (e.g., databases)
• Pre-testing• Peer review / expert
review
EVALUATION METHODS
Preparation Production Distribution Exposure/Reception Response Results
Short | Medium | Long term
• Recall• Awareness• Interest (e.g., follows,
likes, retweets)• Engagement• Satisfaction• Attitude change• Trust• Preference• Intentions• Inquiries / leads• Advocacy
• Reputation• Relationships• Public support• Sales / revenue• Donations• Cost savings• Staff retention• Customer retention• Complying behaviour
(e.g., give blood; drive safely, get fit)
• Quality of life / well-being improvements
Communication• Objectives• Target audiences• Strategy Theory of Change
Program Theory
Questions