Evaluation Framework 2€¦ · Evaluation Framework 2.0. This updated Evaluation Framework is ....

Evaluation Framework 2.0June 2018

Acknowledgements:

Thanks are provided to the Evaluation Council, Directors of Communication, Engage Programme Board, the Cross-Government Insight and Evaluation forum, Professor Kevin Money (and associates), Dr Carla Groom, Dr Sarah Gates, Dr Cecile Morales and the Cabinet Office Insight and Evaluation team for their contributions to this guide.

Introduction from Alex Aiken, Executive Director of Government Communication

Evaluation Framework 2.0

This updated Evaluation Framework is provided to communicators across the wider public sector to assist in measuring the success of our work and appraising our activities. Evaluation remains a critical function for delivering effective communication activity, and this guide will help colleagues plan campaigns in a way that can be meaningfully evaluated. This will drive improvements across our profession, including our capability to provide impactful behaviour change and policy delivery. Ultimately this is about listening to stakeholders and the public so that we know which messages are landing, and how we can learn from that to make our communication more effective.

This Framework builds on the foundations created by the International Association for the Measurement and Evaluation of Communication (AMEC) and the Evaluation Framework that was a product of the Evaluation Council in 2016. These have been tailored to reflect our public service role and the latest campaign optimisation principles developed by the Engage1 programme, which brings data and science to the heart of our communication activity.

The new Framework edition is primarily aimed at paid campaign activity. It adds further guidance on calculating return on investment (ROI), recommends specific metrics for measurement depending on your campaign type, and enhances the guidance on measurement methods. It also introduces guidance on measuring reputation and the ethical use of data.

As all government communicators will know, successful evaluation depends entirely on setting meaningful C-SMART objectives. These are SMART objectives, with an additional C for ‘challenging’. New guidance is provided on the best practice for setting objectives in an OASIS plan to effectively evaluate communication activity and calculate the benefits.

Running a successful campaign requires clear objectives, underpinned by a theory of behaviour change that understands how communication activity will be effective. The GCS team have a number of publications and guides to assist campaign planning, the theory of behaviour change, and all elements of campaign planning. This guide should not be used in isolation, but will assist in effective evaluation from the outset of planning a campaign. All communication activity should consider evaluation and understand that measurement enables evaluation, which in turn becomes insight for future activity.

1 https://gcs.civilservice.gov.uk/about-us/improving-gcs/engage-programme

https://gcs.civilservice.gov.uk/about-us/improving-gcs/engage-programme

https://gcs.civilservice.gov.uk/about-us/improving-gcs/engage-programme

2


Evaluation Framework 2.0This Framework provides guidance for major paid-for campaigns and other communication activity. The Engage programme has identified three distinct types of funded campaign activity.

Behaviour change

The vast majority of government communication seeks to change behaviours in order to implement government policy or improve society. In the benchmarking categories that sit within the Engage programme, we also distinguish the main types of behaviour change: start, stop and maintain. This way we can start to learn about which methods, messages and channels are effective for certain types of behaviour change. Raising awareness will nearly always be part of an activity to change behaviour and should also be measured.

Recruitment

Recruitment is a specific form of behaviour change where people are encouraged to start an activity. This is a major concern for government and is vital to maintaining public services and protecting the country. Government invests a lot of money in recruiting people for important jobs (teachers, armed forces, nurses etc) and so this campaign type has been isolated because of its size. This is targeted at major employment campaigns rather than recruiting people to ‘register’ or ‘take part’.

Awareness

Some campaigns solely seek to raise awareness of an issue or to change people’s attitude. Raising awareness will almost certainly be an intermediary step for all communication activity, so behaviour change campaigns are also encouraged to measure awareness. The awareness metrics here are mainly suggested for campaigns that seek to change attitudes but not immediately change behaviours.

3


Each of these campaign types has a set of recommended evaluation metrics. Consistent use of these metrics will assist campaign planners in choosing appropriate objectives and enable our profession to establish benchmarks for success.

Metrics are divided according to the four categories identified by AMEC:

• inputs (what we put in, our planning and content creation)

• outputs (what is produced, such as audience reach)

• outtakes (subject-oriented stakeholder experiences and communicator-oriented learning about communication practice)

• outcomes (stakeholder behaviour, what the impact of communication and engagement activity is, and whether we achieved the desired organisational impact or policy aim)

The most important of these is outcomes: how effective communication activity is in achieving policy aims and delivering organisational impact. Outtakes are also important for measuring how well communication activity has worked, for example, by assessing the penetration of a campaign message.

This Framework also provides a set of metrics that can be used to assess low cost and no cost activity, which can equally be applied to internal communications and stakeholder engagement activity.

4


EVALUATION

INPUTS OUTPUTS OUTTAKES OUTCOMES

INPUTS (Evidence-based insights about issues and stakeholders)

OUTPUTS (Communication and stakeholder engagement activities)

OUTTAKES (Stakeholder experience of communication activities- lessons learned)

OUTCOMES (Stakeholder behaviour and behaviour change)

What? Research, planning, design, cost and effort that were put in. What has been learned from previous communications activities? What is your theory of change? Content creation

E.g. partnerships secured, press coverage, target audience reach

Reception, perception and reaction of stakeholders. Campaign efficiency metrics for communicators

Changes in behaviour, changes in attitude and contribution to policy objectives

Why? This reflects what we have done to enable the activity

Tracking assets and collateral allows us to evaluate message selection as well as asset type

See if we have chosen effective messages and channels for engagement. Consider what we have learned about our communication practices

Consider if we have achieved what we set out to do in terms of fulfilling a policy or organisational impact, and see if our communications been effective

Example 1 (encourage people to apply for a home insulation subsidy)

£200,000 Previous campaigns had provided narratives and numbers. Theory of change suggested that infographics are a powerful way to present the complex information about the benefits of home insulation

85% of (2,000,000) target audience group reached with impression

Awareness of issue moved from 52% to 65% in target audience

Applications for a subsidy increased from 25,000 to 50,000.

Example 2 (e.g. decrease instances of drink driving)

5 FTE daysTheory of change suggested that local messages and messengers are effective for persistent offenders

1,000,000 impressions recorded on social media using geographically targeted messages and messengers

Active engagement rate was 4.6% compared to forecast 2.5%

Reported instances of ‘Y’ decreased from 500 to 450 per X number of tests in targeted areas

The stages of a communication activity should be viewed as a linked process. Each element is created by the previous element, and that in turn will be causally linked to the next stage. The evaluation of outcomes, outtakes and other elements should be used to inform campaign optimisation both in realtime and for future iterations.

5


OASIS for evaluationThe OASIS2 campaign planning guide provides government communicators with a framework for preparing and executing effective communication activity. Within OASIS, Objectives and Scoring are especially important for the purpose of evaluation.

ObjectivesObjectives should be C-SMART: Challenging, Specific, Measurable, Attainable, Relevant and Time-bound. For the purpose of evaluation it also important that objectives contain three elements.

1. Baseline: A numerical prediction of what will be observed if no communication activity takes place. Some people would take out a pension even if the government ran no communication activity. A baseline should be set using the most recent data available, but some subject areas can use data from last year’s campaign, or exceptionally even earlier. In most cases we can assume that the no-campaign activity observation would be same as the last relevant measurement. It is important to consider predictable movements of the baseline in addition (e.g. we can predict that there will be around a 30% increase in wearable technology ownership in 2018). If it is cost prohibitive to establish a baseline specifically for campaign purposes, planners can use pre-existing publicly available data, research commissioned by policy colleagues, or a proxy measure as a substitute.

2. Change: A numerical forecast of the difference that the campaign activity will make. For example, improving a level of registrations from a baseline of 80,000 to 100,000 (an increase of 20,000 or 25%). Changes should be for a defined period of time, typically three to six months after a campaign for many large behaviour change campaigns. Where major changes are targeted over a longer period (e.g. five years) then milestones or intermediary targets should be provided for a period of no longer than one year.

3. Explanation: Campaign planners should use an evidence base as a justification for the level of change that is being targeted. Typically this might be in line with previous observations from the last time the campaign was run or by comparison to other similar campaigns that can offer some guidance on what level of change could reasonably be expected. Making assumptions is acceptable as long as they are clearly identified and justified. It is important to signal if the behaviour targeted is a start, stop or maintain behaviour. It is important to distinguish the effect of the campaign from other influential factors such as seasonality, fashion, public concern and evolving social norms.

There is more specific guidance available on Audience Insight (especially motivations and barriers), Strategy and Implementation in the full OASIS3 guide.

2 https://gcs.civilservice.gov.uk/wp-content/uploads/2015/09/OASIS-Guide-1-amended.pdf3 https://gcs.civilservice.gov.uk/wp-content/uploads/2015/09/OASIS-Guide-1-amended.pdf

https://gcs.civilservice.gov.uk/wp-content/uploads/2015/09/OASIS-Guide-1-amended.pdf






6


Scoring (Evaluation)Evaluation, focusing primarily on outcomes and outtakes, should take place throughout a campaign and will inform dynamic optimisation of active campaigns.

It is recommended that approximately 5 to 10% of total campaign expenditure is allocated to evaluation. In addition to operational data, evaluation costs will often include commissioning research to measure awareness and message penetration levels.

A complete evaluation will include the following aspects.

• A comparison of actual outcome data with targets set in objectives. Were the objectives met? If not, what reasons can be offered to explain the variation? If the objectives were surpassed, what has driven that?

• A comparison of outtakes with the targets set in objectives. This will typically include various data sources such as qualitative and/or quantitative research findings.

• Considering the causal link between the subject-oriented outtakes and the outcomes. Some campaigns will be more effective in converting awareness and attitudinal changes into tangible behavioural outcomes. To what extent could this campaign convert awareness to behaviour change?

• Findings for current or future campaign optimisation. Ideally this will include attribution modelling and econometric analysis (scientifically assigning a proportion of ‘cause’ to different elements, messages and channels of a campaign). Even without advanced studies, campaigners can often draw conclusions about which channels have been more or less effective than anticipated. Is there anything that others can learn from your theory of change?

• For active campaigns it is advisable to make small incremental adjustments (a slight up or down weighting of elements, messages or channels) to test theories for improvement. Ideally activity will be tested in a pilot beforehand, but this can even be in a live activity by varying message content and delivery channels.

• A conclusion including whether or not the campaign was successful in achieving its policy aims. This should also include what we would do differently next time or for future similar campaigns.

7


Consistent metricsThe following tables present lists of recommended and potential metrics for campaign measurement and evaluation.

Inputs are presented in blue, outputs presented in orange, outtakes are presented in red, and outcomes are presented in green.

For low cost and no cost campaigns that follow, some metrics are identified as either outtakes or outcomes dependent on the aim of the campaign. These are represented in grey.

Please note:

• This is not an exhaustive list, and campaign planners are encouraged to measure any additional more specialised metrics.

• Typically, all communication activity will want to measure the level of awareness and/or agreement with a campaign message. This might be an ‘end’ or outcome in itself, but it will normally be an intermediary step for campaigns looking to achieve behavioural change.

• Not all metrics will necessarily be applicable for all campaigns. For example, campaigns without websites will not be able to measure dwell time.

• Large budget paid-for campaigns are advised to measure as many of these as possible. Low-cost or no-cost campaigns are advised to choose the one or two properties most suitable to their campaign in each activity phase.

• Thinking through outtakes and outcomes can be used to assist in choosing suitable objectives for campaigns.

8


Behaviour changeMetric Online/

offlineDefinition Measurement

method

Inputs

Total spend to date Both Aggregate total spend so far £

Spend to date Online How much money has been spent on digital media

£

Spend to date Offline Sum of one-off set up costs (manual from PASS) and periodic offline media spend updates

£

What is your theory of change (including evidence base)?

N/A Implementation of behavioural science in planning effective communication

Binary – yes/no is in place? Yes/no – current evidence base

Content creation Both Infographics, videos etc Volume by type

Outputs

Estimated total reach Both Aggregate audience reach Absolute number and proportion of target audience

Reported online reach Online The estimated reach as reported by digital platforms

Absolute number and proportion of target audience

Estimated offline reach Offline Reported audience reach for offline media


9


Outtakes

Cost per outcome N/A The unit cost per behaviour change

£

Engagements/interactions

Online The % of impressions generating an interaction (share/like/comment)

Actions which involve active engagement (e.g. typing, not just 'one-click' endorsements)

Completion/registration rate

Both The proportion of contacts/impressions that go on to complete sign-up/registration

%

Cost per completion/registration

Both Unit cost of registration/completion

£

Unprompted campaign issue awarenesse.g. spontaneous recall metric

Offline The number and proportion of target audience that has unprompted campaign issue awareness


Experience of different messages that relate to aspects of theory of change

Both The extent to which different groups agree/disagree with messages related to theory of change

5 point scale (agreement/ disagreement with aspects of message)

Outcomes

Behaviour change(#, %) e.g. number of licensed anglers vs baseline

Offline The number and proportion of target audience that has changed behaviour


Stated/intended behaviour change

Offline The proportion of target audience that claim they will act in accordance with campaign aim


Advocacy e.g. agreement with value for money statement

Offline The number and proportion of target audience that agree with the campaign message (have positive sentiment)

5 point scale recommended (strongly agree/slightly agree/don't know etc)

Current ROI N/A Unit benefit multiplied by number of behaviour changes

£

10


RecruitmentMetric Online/


method

Inputs



£


£


Outputs


Reported online reach Online The estimated reach as reported by digital platforms




11


Outtakes

Expressions of interest (EOI)

Both The number of people actively expressing interest in applying

Absolute number and % of target

Cost/recruit N/A The unit marketing cost per successful recruit

£

EOI/applicant conversion ratio

Both The proportion of EOIs that go on to be applicants

%

Applicant/recruit conversion

Both The proportion of applicants that go on to be employed

%

Cost/EOI Both Total spend / EOIs £

Cost/applicant Both The unit marketing cost per applicant

£

Engagements/interactions



Sentiment toward profession

Offline The regard in which the profession is held by the target audience or general public

5 point scale recommended (strongly agree / slightly agree / don't know etc)

Outcomes

Recruits Offline The number of people successfully recruited


Intra-profession advocacy

Offline The degree to which current professionals would recommend the job to friends/family


Influencer advocacy Offline The degree to which important influencers (e.g. parents) would support entry to profession


Applications Both Unit cost of registration/completion


12


AwarenessMetric Online/


method

Inputs



£

Content Creation Both Infographics, videos etc Volume by type


£

Outputs


Reported reach Online The estimated reach as reported by digital platforms




13


Outtakes

Cost per outcome N/A The unit cost of raising awareness

£

Active engagements/interactions



CTR Online Click through rate: the proportion of impressions generating a click-through

%

VTR Online View through rate: the proportion of impressions meeting a minimum view-through percentage

%

Dwell time Online The average length of time spend on a campaign site

Minutes and seconds

Bounce rate Online % of site visitors that navigate no further than the landing page

%

Prompted campaign recognisers

Offline The proportion of target audience that recalls seeing the campaign when prompted


Outcomes

Advocacye.g. agreement with value for money statement



Unprompted campaign issue awareness



Agreement (# and %) Offline The number and proportion of target audience that agree with the campaign message (have positive sentiment)


14


All communication activityIncluding low cost, no cost, internal communications and stakeholder engagement activities.

Metric Online/offline

Definition Measurement method

Inputs



Volume of press releases

Offline # of press releases sent out # of press releases sent out

Volume of SM releases Online # of releases to owned social media channels

# of releases to owned social media channels

FTE days Offline Total amount of time invested in campaign preparation in terms of person effort

<1, or whole number

Outputs




Reported/estimated online reach

Online The estimated reach as reported by digital platforms


Direct contacts Both # of direct on/offline contacts (e.g. eDM etc)


Events organised Offline Volume of events # of events

# of attendees

Volume of coverage Both # exposures # press cuts

# broadcasts (break local/national)

Partnerships secured Offline # of partnerships providing amplifying support

Formal sign-up to either actively promote and amplify campaign material or make any form of VIK or financial contribution

15


Outtakes

Passive engagements/interactions

Online The % of impressions generating an interaction (share/like/retweet)

A 'one-click' interaction

Active engagements/interactions

Online The % of impressions generating an interaction (comment/response/quote)

Something that involves proactive engagement (e.g. typing)

CTR Online Click through rate: the proportion of impressions generating a click-through

%

VTR Online View through rate: the proportion of impressions meeting a minimum view-through percentage

%

Dwell time Online The average length of time spend on a campaign site

Minutes and seconds

Bounce rate Online % of site visitors that navigate no further than the landing page

%

Prompted campaign recognisers

Offline The proportion of target audience that recalls seeing the campaign when prompted


Stated/intended behaviour change

Offline The proportion of target audience that claim they will act in accordance with campaign aim


EOI/applicant conversion ratio

Both The proportion of EOIs that go on to be applicants

%

Applicant/recruit conversion

Both The proportion of applicants that go on to be employed

%

Reputation perception

Offline See later guidance See later guidance

Response rate Both % of contacts to respond % of contacts to respond

Cost per outcome

N/A The unit cost of raising awareness £

Cost/applicant Both The unit marketing cost per applicant £

Cost/EOI Both Total spend / EOIs £

Cost per completion/registration

Both Unit cost of registration/completion £

Sentiment Either Degree to which a message has been positively or negatively received

Valid for manual assessment of press coverage or a limited (defined) number of stakeholders. Not currently valid for mass automated approach.

16


Outtakes/outcomes4

Agreement (# and %) Offline The number and proportion of target audience that agree with the campaign message (have positive sentiment)


Completion/registration rate

Both The proportion of contacts/impressions that go on to complete sign-up/registration

%




Unprompted campaign issue awareness (# and %)



Outcomes

Behaviour change (#, %)

Offline The number and proportion of target audience that has changed behaviour


Current ROI N/A Unit benefit multiplied by # behaviour changes

£ and X:Y

Recruits Offline The number of people successfully recruited





Advocacye.g. agreement with value for money statement



Applications Both Unit cost of registration/completion


Attitudinal change Both Degree to which people's attitude has changed in favour of the campaign


4 The input/output/outtake/outcome model is helpful for understanding where metrics fit in the communication activity cycle. Tracking appropriate metrics is more important than defining which category they sit in. Depending on the aims of a campaign, for example, raising awareness can be either an outtake or an outcome

17


18


..

Less

ons

lear

ned

Com

mun

icat

ions

ob

ject

ives

Audi

ence

Key

suc

cess

es

Your

less

ons

lear

nt s

houl

d be

mea

ning

ful –

thes

e sh

ould

incl

ude

item

s th

at w

ould

be

usef

ul fo

r you

to re

mem

ber a

nd y

our c

olle

ague

s to

be

awar

e of

for f

utur

e ac

tivity

. Thi

s co

uld

be a

par

ticul

ar s

ucce

ss a

fter t

ryin

g so

met

hing

new

, or r

efle

ctio

ns o

n so

met

hing

that

cou

ld h

ave

gone

bet

ter.

Rem

embe

r, if

som

ethi

ng w

ent w

rong

, it’

s m

ore

usef

ul to

als

o th

ink

abou

t ste

ps y

ou c

ould

take

nex

t tim

e to

miti

gate

this

, rat

her t

han

just

say

ing

som

ethi

ng d

idn’

t wor

k.

Gol

den

exam

ple :

Hol

ding

a b

usin

ess

stak

ehol

der e

vent

in th

e m

orni

ng ra

ther

than

the

afte

rnoo

n en

able

d m

ore

seni

or re

pres

enta

tives

to a

ttend

.

2. O

utpu

ts(d

eliv

ered

–w

hat y

our a

ctiv

ity d

eliv

ered

and

aud

ienc

e re

ache

d)Th

e ai

m h

ere

is to

refle

ctth

e vo

lum

e of

you

r aud

ienc

e yo

u’ve

reac

hed

via

the

wor

k yo

u di

d in

yo

ur in

puts

sec

tion.

Incl

ude

KP

Is a

long

side

the

outp

uts

belo

w.

Med

ia

Exa

mpl

es in

clud

e:

cove

rage

ach

ieve

d (b

roke

n do

wn

by n

atio

nal,

regi

onal

, tra

de a

nd

broa

dcas

t)

key

mes

sage

pen

etra

tion

%

of c

over

age

incl

udin

g m

inis

teria

l/spo

kesp

erso

n qu

ote

Dig

ital

Exa

mpl

es in

clud

e:

aver

age

impr

essi

ons/

reac

h ac

ross

con

tent

(by

chan

nel)

w

ebsi

te/G

OV

.UK

pag

e/co

nsul

tatio

n vi

sits

(tot

al a

nd u

niqu

e)If

utili

sing

pai

d fo

r act

ivity

, sep

arat

e th

is a

nd o

rgan

ic o

utpu

ts.

Exte

rnal

affa

irsE

xam

ples

incl

ude:

co

mm

unic

atio

ns d

eliv

ered

and

aud

ienc

e re

ache

d (n

umbe

r of

lette

rs/e

mai

ls/p

acks

issu

ed, c

alls

mad

e, s

take

hold

ers

reac

hed)

ev

ents

suc

cess

fully

del

iver

ed/a

ttend

ance

supp

ortiv

e st

atem

ent s

ecur

ed fr

om p

artn

ers/

stak

ehol

ders

Inte

rnal

Exa

mpl

es in

clud

e:

audi

ence

reac

hed

(blo

g/st

ory

view

s)

even

ts s

ucce

ssfu

lly d

eliv

ered

/atte

ndan

ce

Act

ivity

title

Dat

es th

is e

valu

atio

n co

vers

Rel

ated

obj

ectiv

e an

d pr

ogra

mm

e

Your

obj

ectiv

es s

houl

d be

as

SM

AR

T as

pos

sibl

e –

spec

ific,

mea

sura

ble,

ach

ieva

ble,

real

istic

and

timel

y.G

olde

n ex

ampl

es: ‘

Incr

ease

reco

gniti

on a

mon

gst 1

8-34

s of

the

econ

omic

opp

ortu

nitie

s of

***

by

5% b

y D

ec 2

019

(vs

Jan

2019

bas

elin

e of

20%

)’ ‘T

o dr

ive

resp

onse

s to

***

cons

ulta

tion/

gree

n pa

per a

nd g

ener

ate

500

resp

onse

s’.

Som

etim

es it

is h

ard

to m

ake

your

obj

ectiv

es p

erfe

ctly

SM

AR

T if

you

don’

t hav

e su

ppor

ting

track

ing

data

. An

exam

ple

of a

n al

mos

t SM

AR

T ob

ject

ive

mig

ht b

e:

‘Rai

se a

war

enes

s of

***

anno

unce

men

t, ge

nera

ting

***

piec

es o

f nat

iona

l cov

erag

e an

d **

* ad

voca

cy s

tate

men

ts’.

This

sec

tion

shou

ld d

etai

l the

thin

gs y

ou’re

mos

t pro

ud o

f! It

coul

d be

thin

gs li

ke:

•m

eetin

g (o

r eve

n ex

ceed

ing)

you

r KP

Is (k

ey p

erfo

rman

ce in

dica

tors

)•

apa

rticu

larly

impa

ctfu

l pie

ce o

f cov

erag

e•

gain

ing

advo

cacy

from

real

ly k

ey s

take

hold

ers,

or m

ovin

g a

noto

rious

ly n

egat

ive

stak

ehol

der i

nto

a m

ore

posi

tive

posi

tion

•po

sitiv

e fe

edba

ck o

n an

inte

rnal

sta

ff ev

ent

4. O

utco

mes

(wha

t was

the

resu

lt)Th

e ai

m h

ere

is to

refle

ctth

e ov

eral

l res

ult o

f you

r act

ivity

as

it re

late

s to

a c

hang

e of

atti

tude

or

beha

viou

r e.g

. did

you

r tar

get a

udie

nce

adop

t an

inte

nded

ser

vice

? In

clud

e K

PIs

.

Th

is s

pace

is to

sho

w h

ow y

ou m

ade

prog

ress

and

met

you

r obj

ectiv

es. T

his

coul

d be

sho

wn

thro

ugh

advo

cacy

, atti

tude

cha

nge

thro

ugh

pre

and

post

ac

tivity

pol

ling,

sta

ff su

rvey

resu

lts, o

nlin

e re

gist

ratio

ns o

r for

m

com

plet

ions

.

Rem

embe

r, as

wel

l as

deta

iling

advo

cacy

, mov

ing

stak

ehol

ders

from

a

nega

tive

to n

eutra

l pos

ition

is s

till a

suc

cess

.

3. O

utta

kes

(res

pons

es to

con

tent

–w

hat y

our t

arge

t aud

ienc

e th

inks

, fe

els

and

does

)Th

e ai

m h

ere

is to

enc

apsu

late

you

r tar

get a

udie

nce’

s re

spon

se to

you

r act

ivity

–w

ere

they

po

sitiv

e ab

out i

t, di

d th

ey ta

ke a

ctio

n as

a re

sult

of y

our c

omm

unic

atio

ns?

Incl

ude

KP

Is.

Med

ia

Exa

mpl

es in

clud

e:

% s

entim

ent o

f cov

erag

e (p

ositi

ve a

nd/o

r neu

tral)

re

spon

ses

(and

any

cov

erag

e th

at in

clud

ed th

ird p

arty

reac

tion)

Dig

ital

Exa

mpl

es in

clud

e:

soci

al m

edia

eng

agem

ent (

follo

ws,

like

s, s

hare

s, c

omm

ent/r

eplie

s,

retw

eets

by

chan

nel)

av

erag

e en

gage

men

t rat

e (b

y ch

anne

l)

shar

es o

f con

tent

from

key

sta

keho

lder

s

use

of o

ffici

al #

men

tions

ac

tions

(clic

k th

roug

hs, c

ost p

er c

licks

, dow

nloa

ds, s

ubsc

riptio

ns)

se

ntim

ent (

via

man

ual d

ippe

d sa

mpl

e ch

eck)

av

erag

e tim

e on

site

Exte

rnal

affa

irsE

xam

ples

incl

ude:

st

akeh

olde

r sen

timen

t (po

sitiv

e en

gage

men

t, nu

mbe

rs o

f pos

itive

st

atem

ents

, adv

ocac

y st

atem

ent –

refe

r to

the

advo

cacy

gui

de fo

r gu

idan

ce o

n th

is)

st

akeh

olde

r fee

dbac

k (s

urve

y fe

edba

ck a

bout

eve

nt)

Inte

rnal

Exa

mpl

es in

clud

e:

resp

onse

s/fe

edba

ck (l

ikes

and

com

men

ts o

n bl

ogs/

new

s, s

urve

y fe

edba

ck a

bout

eve

nt)

en

gage

men

t (cl

ick

thro

ughs

)

surv

ey re

sults

(aw

aren

ess,

reca

ll)

1. In

puts

(pre

para

tion

–th

e w

ork

you

do b

efor

e or

dur

ing

the

activ

ity)

The

aim

her

e is

to re

flect

on th

e w

ork

you’

ve p

ut in

to a

n ac

tivity

. You

sho

uld

also

incl

ude

wor

k th

at m

ay n

ot h

ave

resu

lted

in a

n ou

tput

/out

take

/out

com

e e.

g. a

n op

-ed

that

was

n’t e

vent

ually

in

clud

ed in

a p

ublic

atio

n. U

se th

e le

sson

s le

arnt

to e

xpla

in w

hy s

omet

hing

did

n’t w

ork.

Med

iaE

xam

ples

incl

ude:

co

nten

t cre

atio

n (p

ress

not

ices

, min

iste

rial/s

poke

sper

son/

stak

ehol

der

quot

es, s

peec

hes,

op-

eds,

Q&

As,

med

ia a

nd tr

ail s

crip

ts)

pr

e-en

gage

men

t act

ivity

(brie

fing

jour

nalis

ts a

nd/o

r med

ia o

utle

ts,

brie

fing

OG

Ds

med

ia te

ams,

org

anis

ing

inte

rvie

ws/

med

ia ro

unds

)

Dig

ital

Exa

mpl

es in

clud

e:

pl

anni

ng a

nd s

ched

ulin

g (c

onte

nt c

alen

dar d

evel

oped

, eng

agem

ent p

lan

prod

uced

, cha

nnel

s ch

osen

)

cont

ent d

evel

opm

ent (

web

pag

e de

velo

pmen

t, m

essa

ge te

stin

g,

audi

ence

seg

men

tatio

n fo

r pai

d-fo

r act

ivity

)

cont

ent c

reat

ion

(soc

ial m

edia

pos

ts d

rafte

d,

info

grap

hics

/vid

eos/

quot

epic

scr

eate

d, G

OV

.UK

pag

es c

reat

ed,

cons

ulta

tion

page

cre

ated

, ad

copy

pro

vide

d fo

r pai

d fo

r act

ivity

, tra

cked

lin

ks c

reat

ed w

hich

sho

uld

be s

epar

ated

by

chan

nel e

.g. ‘

4 Tw

itter

in

fogr

aphi

cs c

reat

ed’ i

f rel

evan

t)

pre-

enga

gem

ent (

influ

ence

r/par

tner

s id

entif

ied

and

enga

ged,

dig

ital

tool

kits

sha

red,

OG

Ds

digi

tal t

eam

s br

iefe

d)If

utili

sing

pai

d fo

r act

ivity

, sep

arat

e it

from

org

anic

inpu

t met

rics.

Exte

rnal

affa

irsE

xam

ples

incl

ude:

pl

anni

ng (s

take

hold

er m

appi

ng, s

take

hold

er m

anag

emen

t pla

n, c

all l

ist

prep

ared

)

cont

ent c

reat

ion

(tool

kits

, pac

ks, c

ase

stud

ies

prod

uced

, dra

fted

emai

ls,

call

scrip

t dra

fted,

lette

r dra

fted)

even

ts o

rgan

ised

(ven

ue s

ecur

ed, a

ttend

ees

invi

ted,

roun

dtab

les)

pr

e-en

gage

men

t act

ivity

(sta

keho

lder

s id

entif

ied,

OG

D e

xter

nal a

ffairs

te

am b

riefe

d, s

take

hold

ers

brie

fed)

Inte

rnal

Exa

mpl

es in

clud

e:

plan

ning

(stra

tegi

c na

rrat

ive

draf

ted)

ev

ents

org

anis

ed (s

taff

brie

fing,

dro

p in

s, s

tand

ups

, web

chat

)

cont

ent c

reat

ed (b

log/

new

s st

ory

draf

ted,

scr

een

info

grap

hics

de

velo

ped,

vid

eo c

reat

ed, c

ase

stud

y dr

afte

d, in

trane

t ban

ner c

reat

ed)

pr

e-en

gage

men

t (st

akeh

olde

rs b

riefe

d, s

urve

y is

sued

)

Com

mun

icat

ions

, par

tner

ship

s an

d go

vern

ance

Like

you

r obj

ectiv

es, t

ry to

be

as d

etai

led

as p

ossi

ble

whe

n de

scrib

ing

you

audi

ence

. Thi

s co

uld

be in

term

s of

age

, sec

tor,

geog

raph

ic lo

catio

n et

c. T

his

will

mak

e it

easi

er

whe

n ev

alua

ting

whe

ther

you

reac

hed

this

aud

ienc

e th

roug

h yo

ur a

ctiv

ity.

Gol

den

exam

ple:

Ins

tead

of s

ayin

g ‘g

ener

al p

ublic

’, ar

e yo

u ac

tual

ly a

imin

g fo

r ‘18

-24

year

old

s’ o

r ‘18

-24

year

old

s w

orki

ng in

a p

artic

ular

sec

tor’?

This

gui

de w

ill ta

ke y

ou th

roug

h w

hat y

ou

coul

d in

clud

e fo

r eac

h se

ctio

n in

an

eval

uatio

n re

port

for a

low

cos

t/no

cost

act

ivity

. Not

all

met

rics

will

be

rele

vant

to e

very

type

of a

ctiv

ity

–ju

st u

se th

e on

es a

pplic

able

to y

our w

ork.

Fo

r fur

ther

gui

danc

e co

ntac

t the

insi

ght a

nd

eval

uatio

n te

am.

Example dashboard5

5 The full example of this template, and a blank version for you to populate can be found at:

19


..

Less

ons

lear

ned

Com

mun

icat

ions

ob

ject

ives

Audi

ence

Key

suc

cess

es

Your

less

ons

lear

nt s

houl

d be

mea

ning

ful –

thes

e sh

ould

incl

ude

item

s th

at w

ould

be

usef

ul fo

r you

to re

mem

ber a

nd y

our c

olle

ague

s to

be

awar

e of

for f

utur

e ac

tivity

. Thi

s co

uld

be a

par

ticul

ar s

ucce

ss a

fter t

ryin

g so

met

hing

new

, or r

efle

ctio

ns o

n so

met

hing

that

cou

ld h

ave

gone

bet

ter.

Rem

embe

r, if

som

ethi

ng w

ent w

rong

, it’

s m

ore

usef

ul to

als

o th

ink

abou

t ste

ps y

ou c

ould

take

nex

t tim

e to

miti

gate

this

, rat

her t

han

just

say

ing

som

ethi

ng d

idn’

t wor

k.

Gol

den

exam

ple :

Hol

ding

a b

usin

ess

stak

ehol

der e

vent

in th

e m

orni

ng ra

ther

than

the

afte

rnoo

n en

able

d m

ore

seni

or re

pres

enta

tives

to a

ttend

.

2. O

utpu

ts(d

eliv

ered

–w

hat y

our a

ctiv

ity d

eliv

ered

and

aud

ienc

e re

ache

d)Th

e ai

m h

ere

is to

refle

ctth

e vo

lum

e of

you

r aud

ienc

e yo

u’ve

reac

hed

via

the

wor

k yo

u di

d in

yo

ur in

puts

sec

tion.

Incl

ude

KP

Is a

long

side

the

outp

uts

belo

w.

Med

ia

Exa

mpl

es in

clud

e:

cove

rage

ach

ieve

d (b

roke

n do

wn

by n

atio

nal,

regi

onal

, tra

de a

nd

broa

dcas

t)

key

mes

sage

pen

etra

tion

%

of c

over

age

incl

udin

g m

inis

teria

l/spo

kesp

erso

n qu

ote

Dig

ital

Exa

mpl

es in

clud

e:

aver

age

impr

essi

ons/

reac

h ac

ross

con

tent

(by

chan

nel)

w

ebsi

te/G

OV

.UK

pag

e/co

nsul

tatio

n vi

sits

(tot

al a

nd u

niqu

e)If

utili

sing

pai

d fo

r act

ivity

, sep

arat

e th

is a

nd o

rgan

ic o

utpu

ts.

Exte

rnal

affa

irsE

xam

ples

incl

ude:

co

mm

unic

atio

ns d

eliv

ered

and

aud

ienc

e re

ache

d (n

umbe

r of

lette

rs/e

mai

ls/p

acks

issu

ed, c

alls

mad

e, s

take

hold

ers

reac

hed)

ev

ents

suc

cess

fully

del

iver

ed/a

ttend

ance

supp

ortiv

e st

atem

ent s

ecur

ed fr

om p

artn

ers/

stak

ehol

ders

Inte

rnal

Exa

mpl

es in

clud

e:

audi

ence

reac

hed

(blo

g/st

ory

view

s)

even

ts s

ucce

ssfu

lly d

eliv

ered

/atte

ndan

ce

Act

ivity

title

Dat

es th

is e

valu

atio

n co

vers

Rel

ated

obj

ectiv

e an

d pr

ogra

mm

e

Your

obj

ectiv

es s

houl

d be

as

SM

AR

T as

pos

sibl

e –

spec

ific,

mea

sura

ble,

ach

ieva

ble,

real

istic

and

timel

y.G

olde

n ex

ampl

es: ‘

Incr

ease

reco

gniti

on a

mon

gst 1

8-34

s of

the

econ

omic

opp

ortu

nitie

s of

***

by

5% b

y D

ec 2

019

(vs

Jan

2019

bas

elin

e of

20%

)’ ‘T

o dr

ive

resp

onse

s to

***

cons

ulta

tion/

gree

n pa

per a

nd g

ener

ate

500

resp

onse

s’.

Som

etim

es it

is h

ard

to m

ake

your

obj

ectiv

es p

erfe

ctly

SM

AR

T if

you

don’

t hav

e su

ppor

ting

track

ing

data

. An

exam

ple

of a

n al

mos

t SM

AR

T ob

ject

ive

mig

ht b

e:

‘Rai

se a

war

enes

s of

***

anno

unce

men

t, ge

nera

ting

***

piec

es o

f nat

iona

l cov

erag

e an

d **

* ad

voca

cy s

tate

men

ts’.

This

sec

tion

shou

ld d

etai

l the

thin

gs y

ou’re

mos

t pro

ud o

f! It

coul

d be

thin

gs li

ke:

•m

eetin

g (o

r eve

n ex

ceed

ing)

you

r KP

Is (k

ey p

erfo

rman

ce in

dica

tors

)•

apa

rticu

larly

impa

ctfu

l pie

ce o

f cov

erag

e•

gain

ing

advo

cacy

from

real

ly k

ey s

take

hold

ers,

or m

ovin

g a

noto

rious

ly n

egat

ive

stak

ehol

der i

nto

a m

ore

posi

tive

posi

tion

•po

sitiv

e fe

edba

ck o

n an

inte

rnal

sta

ff ev

ent

4. O

utco

mes

(wha

t was

the

resu

lt)Th

e ai

m h

ere

is to

refle

ctth

e ov

eral

l res

ult o

f you

r act

ivity

as

it re

late

s to

a c

hang

e of

atti

tude

or

beha

viou

r e.g

. did

you

r tar

get a

udie

nce

adop

t an

inte

nded

ser

vice

? In

clud

e K

PIs

.

Th

is s

pace

is to

sho

w h

ow y

ou m

ade

prog

ress

and

met

you

r obj

ectiv

es. T

his

coul

d be

sho

wn

thro

ugh

advo

cacy

, atti

tude

cha

nge

thro

ugh

pre

and

post

ac

tivity

pol

ling,

sta

ff su

rvey

resu

lts, o

nlin

e re

gist

ratio

ns o

r for

m

com

plet

ions

.

Rem

embe

r, as

wel

l as

deta

iling

advo

cacy

, mov

ing

stak

ehol

ders

from

a

nega

tive

to n

eutra

l pos

ition

is s

till a

suc

cess

.

3. O

utta

kes

(res

pons

es to

con

tent

–w

hat y

our t

arge

t aud

ienc

e th

inks

, fe

els

and

does

)Th

e ai

m h

ere

is to

enc

apsu

late

you

r tar

get a

udie

nce’

s re

spon

se to

you

r act

ivity

–w

ere

they

po

sitiv

e ab

out i

t, di

d th

ey ta

ke a

ctio

n as

a re

sult

of y

our c

omm

unic

atio

ns?

Incl

ude

KP

Is.

Med

ia

Exa

mpl

es in

clud

e:

% s

entim

ent o

f cov

erag

e (p

ositi

ve a

nd/o

r neu

tral)

re

spon

ses

(and

any

cov

erag

e th

at in

clud

ed th

ird p

arty

reac

tion)

Dig

ital

Exa

mpl

es in

clud

e:

soci

al m

edia

eng

agem

ent (

follo

ws,

like

s, s

hare

s, c

omm

ent/r

eplie

s,

retw

eets

by

chan

nel)

av

erag

e en

gage

men

t rat

e (b

y ch

anne

l)

shar

es o

f con

tent

from

key

sta

keho

lder

s

use

of o

ffici

al #

men

tions

ac

tions

(clic

k th

roug

hs, c

ost p

er c

licks

, dow

nloa

ds, s

ubsc

riptio

ns)

se

ntim

ent (

via

man

ual d

ippe

d sa

mpl

e ch

eck)

av

erag

e tim

e on

site

Exte

rnal

affa

irsE

xam

ples

incl

ude:

st

akeh

olde

r sen

timen

t (po

sitiv

e en

gage

men

t, nu

mbe

rs o

f pos

itive

st

atem

ents

, adv

ocac

y st

atem

ent –

refe

r to

the

advo

cacy

gui

de fo

r gu

idan

ce o

n th

is)

st

akeh

olde

r fee

dbac

k (s

urve

y fe

edba

ck a

bout

eve

nt)

Inte

rnal

Exa

mpl

es in

clud

e:

resp

onse

s/fe

edba

ck (l

ikes

and

com

men

ts o

n bl

ogs/

new

s, s

urve

y fe

edba

ck a

bout

eve

nt)

en

gage

men

t (cl

ick

thro

ughs

)

surv

ey re

sults

(aw

aren

ess,

reca

ll)

1. In

puts

(pre

para

tion

–th

e w

ork

you

do b

efor

e or

dur

ing

the

activ

ity)

The

aim

her

e is

to re

flect

on th

e w

ork

you’

ve p

ut in

to a

n ac

tivity

. You

sho

uld

also

incl

ude

wor

k th

at m

ay n

ot h

ave

resu

lted

in a

n ou

tput

/out

take

/out

com

e e.

g. a

n op

-ed

that

was

n’t e

vent

ually

in

clud

ed in

a p

ublic

atio

n. U

se th

e le

sson

s le

arnt

to e

xpla

in w

hy s

omet

hing

did

n’t w

ork.

Med

iaE

xam

ples

incl

ude:

co

nten

t cre

atio

n (p

ress

not

ices

, min

iste

rial/s

poke

sper

son/

stak

ehol

der

quot

es, s

peec

hes,

op-

eds,

Q&

As,

med

ia a

nd tr

ail s

crip

ts)

pr

e-en

gage

men

t act

ivity

(brie

fing

jour

nalis

ts a

nd/o

r med

ia o

utle

ts,

brie

fing

OG

Ds

med

ia te

ams,

org

anis

ing

inte

rvie

ws/

med

ia ro

unds

)

Dig

ital

Exa

mpl

es in

clud

e:

pl

anni

ng a

nd s

ched

ulin

g (c

onte

nt c

alen

dar d

evel

oped

, eng

agem

ent p

lan

prod

uced

, cha

nnel

s ch

osen

)

cont

ent d

evel

opm

ent (

web

pag

e de

velo

pmen

t, m

essa

ge te

stin

g,

audi

ence

seg

men

tatio

n fo

r pai

d-fo

r act

ivity

)

cont

ent c

reat

ion

(soc

ial m

edia

pos

ts d

rafte

d,

info

grap

hics

/vid

eos/

quot

epic

scr

eate

d, G

OV

.UK

pag

es c

reat

ed,

cons

ulta

tion

page

cre

ated

, ad

cop y

pro

vide

d fo

r pai

d fo

r act

ivity

, tra

cked

lin

ks c

reat

ed w

hich

sho

uld

be s

epar

ated

by

chan

nel e

.g. ‘

4 Tw

itter

in

fogr

aphi

cs c

reat

ed’ i

f rel

evan

t)

pre-

enga

gem

ent (

influ

ence

r/par

tner

s id

entif

ied

and

enga

ged,

dig

ital

tool

kits

sha

red,

OG

Ds

digi

tal t

eam

s br

iefe

d)If

utili

sing

pai

d fo

r act

ivity

, sep

arat

e it

from

org

anic

inpu

t met

rics.

Exte

rnal

affa

irsE

xam

ples

incl

ude:

pl

anni

ng (s

take

hold

er m

appi

ng, s

take

hold

er m

anag

emen

t pla

n, c

all l

ist

prep

ared

)

cont

ent c

reat

ion

(tool

kits

, pac

ks, c

ase

stud

ies

prod

uced

, dra

fted

emai

ls,

call

scrip

t dra

fted,

lette

r dra

fted)

even

ts o

rgan

ised

(ven

ue s

ecur

ed, a

ttend

ees

invi

ted,

roun

dtab

les)

pr

e-en

gage

men

t act

ivity

(sta

keho

lder

s id

entif

ied,

OG

D e

xter

nal a

ffairs

te

am b

riefe

d, s

take

hold

ers

brie

fed)

Inte

rnal

Exa

mpl

es in

clud

e:

plan

ning

(stra

tegi

c na

rrat

ive

draf

ted)

ev

ents

org

anis

ed (s

taff

brie

fing,

dro

p in

s, s

tand

ups

, web

chat

)

cont

ent c

reat

ed (b

log/

new

s st

ory

draf

ted,

scr

een

info

grap

hics

de

velo

ped,

vid

eo c

reat

ed, c

ase

stud

y dr

afte

d, in

trane

t ban

ner c

reat

ed)

pr

e-en

gage

men

t (st

akeh

olde

rs b

riefe

d, s

urve

y is

sued

)

Com

mun

icat

ions

, par

tner

ship

s an

d go

vern

ance

Like

you

r obj

ectiv

es, t

ry to

be

as d

etai

led

as p

ossi

ble

whe

n de

scrib

ing

you

audi

ence

. Thi

s co

uld

be in

term

s of

age

, sec

tor,

geog

raph

ic lo

catio

n et

c. T

his

will

mak

e it

easi

er

whe

n ev

alua

ting

whe

ther

you

reac

hed

this

aud

ienc

e th

roug

h yo

ur a

ctiv

ity.

Gol

den

exam

ple:

Ins

tead

of s

ayin

g ‘g

ener

al p

ublic

’, ar

e yo

u ac

tual

ly a

imin

g fo

r ‘18

-24

year

old

s’ o

r ‘18

-24

year

old

s w

orki

ng in

a p

artic

ular

sec

tor’?

This

gui

de w

ill ta

ke y

ou th

roug

h w

hat y

ou

coul

d in

clud

e fo

r eac

h se

ctio

n in

an

eval

uatio

n re

port

for a

low

cos

t/no

cost

act

ivity

. Not

all

met

rics

will

be

rele

vant

to e

very

type

of a

ctiv

ity

–ju

st u

se th

e on

es a

pplic

able

to y

our w

ork.

Fo

r fur

ther

gui

danc

e co

ntac

t the

insi

ght a

nd

eval

uatio

n te

am.

Example dashboard5

5 The full example of this template, and a blank version for you to populate can be found at: www.gcs.civilservice.gov.uk//guidance/evaluation/

http://www.gcs.civilservice.gov.uk//guidance/evaluation/

20


Calculating return on investment (ROI)GCS recommends using the following five step process:

1. Objectives

These should be focused on quantifiable behavioural outcomes (such as the number of direct foreign investments generated or the number of teachers recruited).

2. Baseline

Establish the status quo or expectation for the metrics in question if we do nothing.

3. Trend

A forecast of how the baseline will naturally move over the period of measurement. There has been an 8% reduction in adult smoking rate over the last 15 years, so we expect that the next year would see a 0.5% reduction, all other things being equal.

4. Isolation

Exclude or disaggregate other factors that will affect the outcome you are measuring to make sure that the change observed has been caused by the campaign. Recruitment campaigns will normally want to allow for the underlying rate of employment. Communication activity that is accompanied by a tax or legislative change should try to apportion the total observed affect between these different methods of government policy implementation.

5. Externalities

Account for any consequential or collateral effects of your campaign, positive and negative. For example, reducing theft offences may have positive consequences for the insurance industry (reducing their costs and therefore premium levels generally) and negative effects on other types of crime (which some criminals may take up as a substitute).

Assumptions: please note that assumptions are not just acceptable in calculating ROI, but in many cases will be necessary. Assumptions should be clearly identified, reasonable and, where possible, justified. Part of post-campaign evaluation will involve refining assumptions and considering their validity.

21


Return on investment: a worked exampleThe Department for Health and Social Care are running a campaign costing £3 million to reduce the volume of inappropriate A&E attendances for low- urgency cases. The campaign aims to divert people to a GP surgery where they can be better and more efficiently handled. (Please note this is a fictitious example to demonstrate the process and level of detail expected only).

1. Objectives. The campaign will reduce the number of inappropriate NHS England A&E attendances by 3% or 85,000 in 2018/19 compared to the 2017/18 baseline of 2,865,377.

2. Baseline. The appropriate baseline for comparison here is the previous 12 months of operational data or observations. In 2017/18 there were 23,878,145 A&E attendances in England. 12% of these were found to be ‘inappropriate’ (did not require A&E attendance as could have been handled by a GP or pharmacist). The baseline for inappropriate attendances is 2,865,377 (23,878,145 × 0.12).

3. Trend. Over the past three years we have seen a steady increase in A&E attendances of 2% year-on-year, driven by population growth and other factors. We can forecast 24,355,708 attendances (23,878,145 x 1.02) in 2018/19. The rate of inappropriate attendance has remained broadly constant at 12%. The trend-adjusted baseline for inappropriate attendances is therefore 2,922,685 (24,355,708 × 0.12).

4. Isolation. The NHS is also starting to provide and promote out-of-hours GP surgery appointments. The rate of inappropriate A&E attendance is 4.5 percentage points higher than average at times when GP surgeries are not currently open. We assume that the new offer of out-of-hours service by GPs will reduce the total number of inappropriate attendances by 2.25 percentage points (half of the total observed affect because this only affects half the hours in day). We expect this to independently reduce the number of inappropriate attendances observed by 65,760 (2,922,685 × 0.0225) to 2,856,925.

5. Externalities. Aside from the direct cost benefit of optimising points of treatment across NHS frontline services (in the conclusion), there are indirect benefits or positive externalities that should be considered in this case.

Inappropriate attendances rarely have to be treated, so there will be a negligible cost for this, and operational overheads will remain as a fixed cost. A reduction in inappropriate A&E attendance of 3% is approximately equivalent to 3% uplift in staffing resource (which can be redeployed to urgent cases). The total annual cost of A&E operation is £2.7 billion and staffing makes up 30% of this, which is equal to £810 million. 3% of £810 million is equal to £24.3 million. The indirect benefit of optimising A&E attendance, or the effective ‘opportunity cost’ of not optimising staff resource, is £24.3 million.

22


Conclusion:The average cost of an A&E attendance is £148. The average cost of a GP appointment is £46. Therefore every potential A&E attendance that is redirected to a GP reflects a saving to the NHS of £102 (£148 - £46).

If 85,000 cases are redirected in this way the health service overall will be £8,670,000 (£102 × 85,000) better off.

The positive externalities generated also create £24.3 million of value for the public sector and society.

The total benefit of this campaign, or return on investment, will be £32.97 million. For every £1 spent on this campaign, society will be £11 better off. This is commonly expressed as a ratio, in this case 11:1.

These results can be validated after the campaign has run by comparing the actual number of inappropriate A&E attendances with the isolated trend-adjusted baseline of 2,856,925.

23


Measuring and managing reputationThe reputation of an organisation is now well established as an indicator of organisational success. Positive reputations are associated with supportive behaviours from stakeholders, while negative reputations are associated with less support or even hostile responses from stakeholders. Despite good evidence on this link, there is still much confusion about how to best measure and manage reputation, with many seemingly competing models and approaches.

In this guide we build upon more than 20 years of research of the John Madejski Centre for Reputation to provide a simple guide to measuring and managing reputation.5 Measuring reputation is a good form of organisational listening and can help to demonstrate performance and guide strategic action. The guide will conclude by providing an integrated reputation management framework to use when considering reputation management issues.

We suggest that practitioners should ask three questions when considering current and future reputation management projects:

• Reputation with whom?

• Reputation for what?

• Reputation for what purpose?

Reputation with whom?Organisations have different reputations with different groups and individuals. Although all people’s perceptions may be informative, it is important to consider which stakeholders are most critical.

To choose meaningfully it is vital to consider what the purpose of your organisation is and which stakeholders it serves. When considering stakeholders it is important to consider if they are vulnerable or powerful, have demands that are legitimate (socially accepted and expected) or are time critical. A common issue facing many organisations is that urgent issues often take priority over important ones, and attention and resource is given to stakeholders that are making the most noise (often through social media).

5 This draws on a number of key publications and adapt the models and approaches presented within them to the context of reputation management in the public sector. Key publications include: Money, K., Saraeva, A., Garnelo-Gomez, I., Pain, S. and Hillenbrand, C. (2017) ‘Corporate reputation past and future: a review and integration of existing literature and a framework for future research’, Corporate Reputation Review, 20 (3-4)Ghobadian, A., Money, K. and Hillenbrand, C. (2015) ‘Corporate responsibility research: past – present – future’, Group and Organisation Management, 40 (3), pages 271-294Money, K., Hillenbrand, C., Henseler, J. and da Camara, N. (2013) ‘Exploring unanticipated consequences of strategy amongst stakeholder segments: the case of a European Revenue Service’, Long Range Planning, 45 (5-6)Money, K., Hillenbrand, C., Hunter, I. and Money, A.G. (2012) ‘Modelling bi-directional research: a fresh approach to stakeholder theory’, Journal of Strategy and Management, 5 (1)Money, K. and Hillenbrand, C. (2006) ‘Using reputation measurement to create value: an analysis and integration of existing measures’, Journal of General Management, 32 (1)MacMillan, K., Money, K., Downing, S. and Hillenbrand, C. (2004) ‘Giving your organisation SPIRIT: an overview and call to action for directors on issues of corporate governance, corporate reputation and corporate responsibility’, Journal of General Management, 30 (2)

24


It is therefore important to consider aspects of organisational purpose, stakeholder need, and legitimacy before responding to stakeholders and in managing and measuring reputation. At its best, choosing to measure reputation with a particular group can help to give them a voice, allow your organisation to listen, and to guide and justify the actions of your organisation.

Reputation for what?Reputation is defined as a perception of character. But in practice reputation is often measured as an aggregate of stakeholders’ trust, admiration and respect for an organisation. This is sometimes referred to as emotional appeal.

This aggregate measure of reputation is appealing because it allows organisations that are different (e.g. the armed services and Amazon) to be compared in the extent that they are trusted. There are many other measures of reputation that link to factors such as:

• organisational characteristics (e.g. products, leadership, financial performance)

• relationships (e.g. customer service, listening, the appropriate use of power)

• third party influence (e.g. if important third parties recommend your organisation)

However, they can more usefully be defined and measured as the causes of reputation that make stakeholders trust, admire or respect your organisation (see Figure 1).

Measuring such factors and causally linking them to reputation through multivariate statistical analysis is important in helping to

develop a theory of change. For example, if good service experience is found to be a key factor driving trust in the HMRC, more so than other factors, it would seem reasonable to focus activities on service if it was cost effective.

Reputation for what purpose?In reputational terms it is useful to answer this question in terms of what stakeholder behaviour or attitude you are seeking to maintain or change through the activities of your organisation.

These behaviours can be usefully considered to be consequences of reputation. Essentially, if your organisation is trying to influence a particular behaviour. what is it trying to cause? This could include engaging in activities such as volunteering, paying taxes or engaging in a healthier lifestyle.

You may ask yourself the question “What is my organisation trying to cause?” It is likely that you will identify different stakeholder behaviours in different contexts. The more specifically you can define a behaviour (time, place, duration), the more reliably you will be able to predict it. In this way it is important to consider the three types of behaviour outlined in the guide: start (e.g. recruitment, volunteering), stop (e.g. smoking, speeding, drinking) and maintain (current positive behaviours such as sharing information or paying tax).

As a general rule maintaining behaviours is easier than starting or stopping behaviours, which can be more difficult if it involves breaking long established habits.

MeasurementIt is important to measure not only reputation, but also its causes and

25


consequences in a way that allows for robust statistical linkages between these factors, e.g. through multivariate techniques such as regression.

In practical terms this often means using established measurement scales that are on five or seven points (see footnote five for sample scale items that can be used). In this way you will be able to identify which stakeholder experiences link to reputation and its associated consequences.

It is also important to include measures that benchmark your organisation against other organisations, especially in terms of aspects of trust, admiration and respect. Historical comparisons to other organisations in terms of the start, stop, and maintain criteria may also be useful.

26


Cau

ses

Str

ateg

ic a

ctio

ns(O

utpu

ts)

Sta

keho

ld e

xper

ienc

es/

ob

serv

atio

nsS

take

hold

fee

ling

s/b

elie

fsS

take

hold

beh

avio

urs/

inte

ntio

ns/e

nd s

tate

s

Go

od

will

/int

ang

ible

ass

ets

(Out

take

s)K

ey p

erfo

rman

ce in

dic

ato

rs(O

utco

mes

)

Rep

utat

ion

Co

nseq

uenc

es

Mai

ntai

ning

: beh

avio

urs

/in

tent

ions

/ e

nd s

tate

s(e

.g. c

omm

itmen

t, co

mpl

ianc

e,

turn

over

, eng

agem

ent,

wel

lbei

ng,

satis

fact

ion)

Func

tio

nal d

rive

rse.

g. p

rodu

ct q

ualit

y, w

orkp

lace

en

viro

nmen

t, fin

anci

al s

ound

ness

, co

rpor

ate

soci

al re

spon

sibi

lity)

See

Fo

mbr

un a

nd o

ther

s, 2

015;

Wal

sh a

nd

othe

rs, 2

009

Rep

utat

ion

as c

og

niti

veJu

dgem

ent a

nd d

iffer

entia

tor(e

.g.

corp

orat

e pe

rson

ality

dim

ensi

ons

such

as

mod

ern,

trad

ition

al, c

hic)

See

Dav

is a

nd

othe

rs, 2

001;

Mon

ey a

nd o

ther

s, 2

012

Rep

utat

ion

as e

mo

tio

nal

judg

emen

t and

com

para

tive

conc

ept(e

.g.

emot

iona

l app

eal,

trus

t, ad

mira

tion,

re

spec

t) S

ee F

ombr

un a

nd o

ther

s, 2

015

Feed

bac

k lo

op

of

lear

ning

(inp

uts)

Res

earc

h in

form

atio

n, c

alcu

late

d re

turn

on

inve

stm

ent,

linki

ng

to fi

nanc

ial i

ndic

ator

s, s

take

hold

er e

xpec

tatio

ns

Rel

atio

nal d

rive

rs(e

.g. b

enefi

ts, c

omm

unic

atio

ns

(incl

udin

g lis

teni

ng, u

se o

r ab

use

of

pow

er, s

hare

d va

lues

) See

Mac

mil

lan

and

othe

rs, 2

004;

Mon

ey a

nd

othe

rs, 2

012

Mo

tiva

tio

nal d

rive

rs(e

.g. a

cqui

ring

mat

eria

l wea

lth a

nd s

tatu

s, d

efen

ding

so

met

hing

of v

alue

, bon

ding

with

ot

hers

, lea

rnin

g an

d ha

ving

pur

pose

)S

ee L

awre

nce

and

Noh

ria, 2

004;

M

oney

and

Pai

n, 2

016)

Thi

rd p

arty

infl

uenc

e d

rive

rs(e

.g. a

dver

tisin

g, p

ublic

rela

tions

, pee

r-to

-pee

r in

fluen

ce, w

ord

of m

outh

) See

Ebe

rle a

nd

othe

rs, 2

013;

Li a

nd o

ther

s, 2

013;

Dys

on a

nd

Mon

ey, 2

017)

Sto

pp

ing

: beh

avio

urs

/int

enti

ons

/

end

sta

tes

(e.g

. ove

r-co

nsum

ptio

n, a

ntis

ocia

l be

havi

our,

self

harm

ing

beha

viou

r)

Sta

rtin

g: b

ehav

iour

s /i

nten

tio

ns /

en

d s

tate

s(e

.g. c

o-op

erat

ing,

con

sum

ing,

en

gagi

ng, p

ro-s

ocia

l beh

avio

urs,

re

silie

nce,

wel

lbei

ng, n

ew re

latio

nshi

ps)

27


Introduction to the government Data Ethics FrameworkThe Data Ethics Framework7 guides the design of appropriate data use in government and the wider public sector. This guidance is aimed at anyone working directly or indirectly with data in government, including data practitioners (statisticians, analysts and data scientists), policymakers, operational staff and those helping produce data-informed insight.

1. Start with a clear user need. Using data in more innovative ways has the potential to transform how government works. We must always be clear about what we are trying to achieve for users, both citizens and civil servants.

2. Be aware of relevant legislation and codes of practice. You must have an understanding of the relevant laws and codes of practice that relate to the use of data. When in doubt, you should consult relevant experts.

3. Use data that is proportionate to the user need. The use of data must be proportionate to the user need. You must use the minimum data necessary to achieve the desired outcome.

4. Understand the limitations of the data. Data used to inform policy and service design in government must be well understood. It is essential to consider the limitations of data when assessing if it is appropriate to use it for a user need.

5. Use robust practices and work within your skillset. Insights from new technology are only as good as the data and practices used to create them. You must work within your skillset recognising where you do not have the skills or experience to use a particular approach or tool to a high standard.

6. Make your work transparent and be accountable. You should be as open as possible about the tools, data and algorithms you used to conduct your work. This allows other researchers to scrutinise your findings and citizens to understand the new types of work we are doing.

7. Embed data use responsibly. It is essential that there is a plan to make sure insights from data are used responsibly. This means those teams understand how findings and data models should be used and monitored with a robust evaluation plan.

7 Written by Dr Sarah Gates, Data Policy, DCMS. Sarah can be contacted at [email protected] for further information.

mailto:[email protected]

28


DELIVERING WORLD-CLASS COMMUNICATIONS

28


Library of further resources

The Government Communication Service (GCS) provides a series of guides, frameworks and tools to support government communicators in their work.

All these are available on gcs.civilservice.gov.uk

Handbook

Campaign highlights

Campaign Solutions Framework

Communications Services Framework

Collaboration Toolkit

Communications and behaviour change

Competency framework

Delivering excellence in partnership

Design102

Diversity and inclusion strategy

Modern media operations

Modern Communications Operating Model (MCOM)

OASIS campaign guide

Professional standards

Propriety guidance

Recruitment guidance

Recommended reading list

7 trends in leading-edge communications

GDS Social Media Playbook

Style guide

The IC Space

Previous Government Communication Plans

http://bit.ly/2pn3YFu

http://bit.ly/2pn3YFu

http://bit.ly/2G6BcUg

http://bit.ly/2GH7Sl5

http://bit.ly/2GH7Sl5

http://bit.ly/commssolframework

http://bit.ly/partnershipmarketing

http://bit.ly/2HLW6Fo

http://bit.ly/2u0BarX

http://bit.ly/2FKCK6W

http://bit.ly/2HNqoHP

http://bit.ly/gcsdandi

http://bit.ly/2GJ7bry

http://bit.ly/GCSmcom

http://bit.ly/GCSoasis

https://bit.ly/2I30hwu

http://bit.ly/proprietyguidance

http://bit.ly/2DEGNf0

http://bit.ly/2G5KNKI

http://bit.ly/7trendscomms

http://bit.ly/2GHJWOe

http://bit.ly/CivilServiceCode

https://communication.cabinetoffice.gov.uk/ic-space/

http://bit.ly/2IBkFWP

GC

S M

od

elO

rgan

isat

ion/

po

licy

ob

ject

ives

Co

mm

unic

atio

n O

bje

ctiv

es

Cam

pai

gn e

valu

atio

n an

d fu

rthe

r in

sigh

t to

info

rm fu

ture

pla

nnin

gg

On-

goin

g in

sigh

t to

info

rm fu

ture

pla

nnin

gg

Sta

ges

:In

put

sO

utp

uts

Out

take

sO

utco

mes

Org

anis

atio

nal

Out

com

es

Thin

gs y

ou

need

to

do

trac

k an

d/o

r ac

hiev

e

Wha

t d

o yo

u d

o b

efor

e an

d d

urin

g th

e ac

tivity

(e.g

):

·p

lann

ing

·p

rep

arat

ion

·p

re-t

estin

g ·

pro

duc

tion

Wha

t is

del

iver

ed/t

arge

t au

die

nce

reac

hed

(i.e

..):

·d

istr

ibut

ion

·ex

po

sure

·re

cep

tion

·re

ach

Wha

t th

e ta

rget

aud

ienc

e th

ink,

feel

or

cons

ider

doi

ng

(i.e.

):

·aw

aren

ess

·un

der

stan

din

g ·

inte

rest

·en

gag

emen

t ·

pre

fere

nce

·su

pp

ort

The

resu

lt of

you

r act

ivity

on

the

targ

et a

udie

nce

(e.g

.):

·im

pac

t ·

influ

ence

·

effe

cts:

-at

titud

e -

beha

viou

r

The

quan

tifiab

le im

pact

on

the

orga

nisa

tion

goal

s/ K

PIs

(e

.g.):

·re

venu

e ·

cost

s re

duc

tion

·co

mp

lyin

g a

ctio

ns ·

rete

ntio

n, ·

rep

utat

ion.

Met

rics

&

Mile

sto

nes

Sel

ect

the

right

met

rics

from

the

fram

ewor

k to

hel

p y

ou m

easu

re a

nd e

valu

ate

the

per

form

ance

of

your

in

tegr

ated

com

mun

icat

ion

activ

ities

.S

elec

t the

righ

t bus

ines

s K

PIs

to

trac

k pe

rform

ance

of

your

In

tegr

ated

com

mun

icat

ion

activ

ities

ag

ains

t you

r org

anis

atio

ns g

oals

.M

etho

ds

Use

a m

ix o

f qua

litat

ive

& q

ualit

ativ

e m

etho

ds (e

.g. s

urve

ys, i

nter

view

feed

back

, foc

us g

roup

s, s

ocia

l med

ia a

naly

tics,

trac

king

etc

).

Evaluation Framework 2€¦ · Evaluation Framework 2.0. This updated Evaluation Framework is ....

Documents

Transcript of Evaluation Framework 2€¦ · Evaluation Framework 2.0. This updated Evaluation Framework is ....