Training on Setting-up Awareness Campaigns Prof. dr. Tom Brijs Transportation Research Institute...
-
Upload
eleanore-walton -
Category
Documents
-
view
218 -
download
1
Transcript of Training on Setting-up Awareness Campaigns Prof. dr. Tom Brijs Transportation Research Institute...
Training on Setting-up Awareness Campaigns
Prof. dr. Tom BrijsTransportation Research
InstituteHasselt University (BE)
Disclaimer: Significant parts of the presentation were taken from the European Manual for Designing, Implementing, and Evaluating Road Safety Communication Campaigns (FP6 project CAST)
Outline
1.1. Intervention mapping as a stepwise Intervention mapping as a stepwise approach for creating awareness approach for creating awareness campaignscampaigns
2.2. Practical implementation of the Practical implementation of the stepwise approachstepwise approach
Part I:Part I:Intervention mapping as a Intervention mapping as a
stepwise approach for stepwise approach for creating awareness creating awareness
campaignscampaigns
Introduction
• Campaigns are all about changing behavior. So a crucial point is to understand the determinants of behavior– Predisposing factors– Reinforcing factors– Enabling factors
Predisposing factors:Predisposing factors:
““learninglearning-related factors -related factors within the within the persons of the persons of the target population that target population that facilitate or hinder facilitate or hinder motivationmotivation for for change” change”
e.g., e.g., awareness of the awareness of the problem, beliefs about the problem, beliefs about the problem, attitudes toward the problem, attitudes toward the problem, perceptions about problem, perceptions about wearing helmetswearing helmets
Reinforcing factors:Reinforcing factors:
““factors that encourage or factors that encourage or discourage discourage continuationcontinuation of the of the behaviour” behaviour”
e.g., e.g., social support, peer social support, peer pressure, incentives, pressure, incentives, punishment (fines, demerit punishment (fines, demerit points)points)
Enabling factors:Enabling factors:
““factors that can factors that can help or help or hinderhinder the the desired behaviours desired behaviours as well as as well as environmental environmental changeschanges””
e.g. e.g. Skills: how to correctly wear Skills: how to correctly wear the helmetthe helmet Resources: money to buy a Resources: money to buy a helmet helmet Barriers: hot weatherBarriers: hot weather
Explaining Human Behaviour
Theory of planned behaviourTheory of planned behaviour
CampaignsCampaigns
Barriers
Skills
Changing needs understanding
but
Understanding is not sufficient for changing
Changing behaviour is a science
It involves applying theories, learning from earlier empirical studies done on the topic or related topic, and collecting data from your target population and the facilitators of the intervention. Foremost, changing behaviour needs a systematic and planned approach.
Changing behaviour is a science
Evidence-based public health programmes are based on empirical data and theory.
Theory and Practice
Applying theory and researchKnowledge of Theory
especially knowledge of the process and conditions
Example: Fear Appeals
Theory and Research
High Personal Relevance Difficult Behavior Change
Fear
= Defensive reaction
+
Theory and Research
High Personal Relevance Difficult Behavior Change
Skills
= Constructive reaction
+
Part II:Part II:Practical implementation Practical implementation of the stepwise approachof the stepwise approach
Intervention Mapping
Step 1: Needs AssessmentStep 2: Defining Campaign ObjectivesStep 3: Selecting Methods and StrategiesStep 4: Designing the CampaignStep 5: Planning for Campaign
ImplementationStep 6: Planning for Campaign Evaluation
Intervention Mapping
Step 1: Needs AssessmentStep 2: Defining Campaign ObjectivesStep 3: Selecting Methods and StrategiesStep 4: Designing the CampaignStep 5: Planning for Campaign
ImplementationStep 6: Planning for Campaign Evaluation
Step 1: Needs Assessment
Task 1: Establish a planning group that includes potential program participants and plan the needs assessment
Task 2: Conduct the needs assessment to analyze traffic and mobility problems and behavioral and environmental causes
Task 3: Define the target population
Task 4: Link the needs assessment to evaluation planning by establishing desired program outcomes
Task 1: Putting together a work group
Representatives of the resource group – the program developers and possibly fundersRepresentatives of the potential program adopters and implementers Representatives of the priority population – ultimate end users and managers or leaders from community organizations
Step 1: Needs Assessment
Representatives of the resource group
• Public authorities– Ministries– National Road Safety Council– World Bank, ADB, …
• Road authorities• Ngo’s: HIB, CRY, GRSP, …• Insurance companies• Vehicle and safety equipment manufacturers• Charities
Step 1: Needs Assessment
Representatives of the potential program adopters and
implementers • Enforcement authorities
– Cambodian National Traffic Police– Local police
• Schools• Driver education training centers• Private organizatons, e.g.
– Petrol stations– Automobile associations, car manufacturers– Festival and event organizers– Restaurant owners– Supermarkets, …
• Volunteer organizations• Hospitals, doctors• Drink producers
Step 1: Needs Assessment
Representatives of the priority population
• Action groups, e.g.– Motorcycle action groups– Road victims association– Local road safety ambassadors
• Private organizations– Sport clubs– Youth organizations– Taxi drivers– …
• A selection of individuals from the target audience
Step 1: Needs Assessment
• What we know from the international literature– A strong coalition increases the chances of
success of the campaign: public-private partnership involving state organizations (from top level political commitment to local commitment from police departments, local road safety ambassadors), legal departments, insurance companies, …
– Early involvement of the different stakeholders is needed
– Strong police leadership needed to ensure that strong enforcement is accepted by police officers
Step 1: Needs Assessment
1. What is the problem? Why? Is it a serious problem? What people are involved?
2. What behaviors and environmental conditions are involved?
3. Brainstorm key determinants of behaviors and environmental conditions
4. Build a theory- and evidence-based explanation
Task 2: Conduct a needs assessment
Step 1: Needs Assessment
Where do we get the necessary information?- crash statistics (RTAVIS)- hospital data- field observations- field surveys- past campaigns- international literature- interviews with experts- focus group discussions
Step 1: Needs Assessment
• Selection of a theoretical model underlying the target behavior in the campaign
– Theory of planned behavior– Health-belief model– Protection motivation theory– Deterrence theory
Step 1: Needs Assessment
Behaviouralbeliefs
Normative beliefs
Controlbeliefs
Attitude
Subjective norm
Perceivedbehavioural
control
Behaviouralintentions
Behaviour
Theory of Planned Behaviour
Step 1: Needs Assessment
skills
barriers
• Examples– subjective norm, perceived behavioral
control (drink driving campaign) http://www.youtube.com/watch?v=qsY_Co-p8Bw
– Subjective norm, perceived behavioral control (drink driving)
http://www.youtube.com/watch?v=23xr3JF-tz0
Step 1: Needs Assessment
Intrinsic benefitsExtrinsic benefits
Self-efficacy
SeverityVulnerability
Response cost
Threat appraisal
Coping appraisal
Protectionmotivation
Protection Motivation Theory
Step 1: Needs Assessment
+
+
=
=
• Examples– Severity, vulnerability, response cost
(helmet wearing campaign Vietnam)
http://www.youtube.com/watch?v=3X46UoXpAME
– Self-efficacy (child restraint)
http://www.youtube.com/watch?v=j4-zRcm38oM
Step 1: Needs Assessment
PerceivedSeverity
PerceivedSusceptibility
PerceivedBenefits
PerceivedBarriers
Perceived Threat
Cost-benefitAnalysis
Cues to action
Likelihood ofAction
Health Belief Model
Step 1: Needs Assessment
• Examples– Benefits (seatbelt wearing campaign)
http://www.youtube.com/watch?v=23xr3JF-tz0
– Action cues (campaign on fatigue) http://www.youtube.com/watch?v=LR4NK_xz7y8
– Action cues (call a “responsible young driver”)
http://www.youtube.com/watch?v=Vq93NCBp9Ik
Step 1: Needs Assessment
– People tend to change their behavior more easily when campaigns are associated with stronger enforcement.
– Communication about enforcement must focus on:• Immediacy: you will be fined immediately• Severity: you will pay a high fine• Likelihood: you have a high chance of being caught
– Example drink drivinghttp://www.youtube.com/watch?v=iwKCjIaT_Xs
Step 1: Needs Assessment
Deterrence Theory
Task 3: Describing the individuals who are the
potential program recipients (target audience)The recipients of campaign benefits may not
necessarily be the population at-risk– E.g., car drivers and pedestrians (pedestrian
accidents)– E.g. parents and children (child protection)
There are often multiple groups targeted by an intervention, some of whom are populations at risk and others, those who influence the environment (e.g. police, school teachers)
Program participants could be environmental agents, such as the road safety managers, police, …
Step 1: Needs Assessment
• Need for segmentation = separating the audience in distinct, relative homogeneous subgroups, called segments.
• Why? – The idea is that since a segment shares key
characteristics, individuals within a segment will react similarly to the same stimuli (e.g. campaign)
– Carefully selecting the target group increases significantly the effect of a campaign
Step 1: Needs Assessment
• Criteria for target segment selection– Severity: how much is the target group at
risk, what is the potential for improvement?
– Accessibility: how can they be identified and reached?
– Responsiveness: are they willing to listen and are they responsive to our message?
– Segment size: is the segment large enough to be cost-effective?
Step 1: Needs Assessment
• Segmentation variables– Demographic: age, gender, education,
occupation, family size, social class, …– Geographic: rural versus urban, province, …– Psychographic: personality, beliefs, values,
interests, lifestyle– Behavioral: user rate (high offenders versus
low offenders)– State of change
Step 1: Needs Assessment
• Segmentation variables– State of change:
• pre-contemplation: individual has no intention to change (believers / non-believers)
• contemplation: individual starts to become aware of the problem. Highlight discrepancy between individual needs and general public needs (traffic safety)
• preparation: individual has begun preparations to change. This can be supported by facilitating change.
• action: change has occurred but risk is still high for fallback. The first actions must be rewarding in order to prevent fallback. Peer pressure, subjective norms play a very important role here. Giving gifts, demerit point systems etc may help.
• maintenance: new behavior has started to become a habit
• termination: new behavior is established and return to old behavior is not likely
Step 1: Needs Assessment
Task 4: Objectives from the needs assessment
Type of objective Safety outcomes
Safety-related behavior outcomes
Safety-related environmental outcomes
Definition What will change in terms of the safety problem? By how much? Among whom? By when?
What safety-related behavior will change? (e.g. helmet wearing)
What environmental conditions will change? (e.g. helmet lockers)
Step 1: Needs Assessment
Task 4: Objectives from the needs assessment
• What will be the target variable of our campaign?– Knowledge about risks associated with not wearing
the helmet– Attitudes with respect to helmet wearing– Behavioral intentions with respect to helmet
wearing– Changed behavior (helmet wearing)
• Observed• Self-reported
-> the closer to behavior, usually the harder it is to prove the effectiveness of the campaign)
Step 1: Needs Assessment
Intervention MappingStep 1: Needs AssessmentStep 2: Defining Campaign ObjectivesStep 3: Selecting Methods and
StrategiesStep 4: Designing the CampaignStep 5: Planning for Campaign
ImplementationStep 6: Planning for Campaign
Evaluation
Step 2: Campaign Objectives
• At this stage, the general aim of the campaign is translated into specific objectives
• What are the expected effects of the campaign in terms of the target variable for the specific target group?– E.g. double the helmet use during peak hours for young people
between 16-25 years old• Other possibilities
– The number or severity of accidents involving non-helmet use– The number of offences
• Alongside the primary objectives, there can also be secondary objectives– E.g. to influence the attitudes towards helmet use– E.g. to influence perceive behavioral control (PBC) by reminding
drivers that they are in charge of their own decision to wear the helmet
Intervention Mapping
Step 1: Needs AssessmentStep 2: Defining Campaign ObjectivesStep 3: Selecting Methods and StrategiesStep 4: Designing the CampaignStep 5: Planning for Campaign
ImplementationStep 6: Planning for Campaign Evaluation
Step 3: Methods and Strategies
Theoretical methods– general technique for influencing change– E.g. use of testimonials
Practical strategies– Specific technique for the application of the
method– E.g. accident victim goes to schools to talk about
his/her experiences and makes young people aware of the dangers
• Methods to influence knowledge– Discussion (e.g. in class rooms, on TV, radio, …)– Images– Active learning– …
• Methods to increase awareness/risk perception– Information about personal risk– Persuasive communication (e.g. role model stories)
• E.g. 6-12 years old: http://www.vtm.be/zeppezikki/#movie07
– Fear arousal– …
Step 3: Methods and Strategies
• Methods to increase self-efficacy– Skill training (e.g. short courses)– Guided practice with feedback (e.g. additional
road education courses)– Reinforcement (
http://www.youtube.com/watch?gl=NL&hl=nl&v=CBZSF5Q4B5c)
• Methods to change habits– Presenting substitute behaviors– Presenting cues (I buckle up before I start the
engine)
Step 3: Methods and Strategies
• To change attitudes– Self-reevaluation (discussing about one’s own
behavior)– Increasing direct experience (e.g. testimonials)– Anticipated regret (e.g. imagine you are
disabled for the rest of your life, how would you feel about that?)
• To change social influence– Building skills for resistance to social pressure (
http://www.youtube.com/watch?gl=NL&hl=nl&v=NTHN8JguKAU)
– Showing positive visible expectations
Step 3: Methods and Strategies
Intervention Mapping
Step 1: Needs AssessmentStep 2: Defining Campaign ObjectivesStep 3: Selecting Methods and StrategiesStep 4: Designing the CampaignStep 5: Planning for Campaign
ImplementationStep 6: Planning for Campaign Evaluation
Step 4: Designing the campaign
• Develop the campaign strategy– Task 1: Define the strategy– Task 2: Develop the message– Task 3: Choose the media and media
plan
Task 1: Define the strategy• Isolated campaign: a single campaign• Combined campaign: accompanied by increased
enforcement, changed legislation and/or educational activities
• Integrated campaign:– Horizontal integration: across sectors (involvement of
different stakeholders)– Vertical integration: across levels (from National to
municipal)– E.g. THINK! (UK) & STEP (USA)
• Multi-year campaign: several waves of the same campaign carried out over a longer period of time
Step 4: Designing the campaign
Task 1: Define the strategy• Mass-media communication (non-
personal)– TV, radio, newspapers, …
• Selective communication (reach target groups simultaneously)– Cinema, direct mail, flyers, brochures
• Interpersonal communication (two or more people communicate directly)– Focus groups– Telephone– Face-to-face
Step 4: Designing the campaign
• What we know from international research– Combined and integrated campaigns are more
effective than isolated campaigns– Increased enforcement in combination with
enforcement awareness campaigns are very successful
• 2 weeks of advertizing about increased enforcement: 1 week before and 1 week during the increased enforcement period
– Campaigns should be accompanied by measures to assist the change process
– The use of campaign waves is very successful: 2-3 times per year
– Feedback of the results at the end of the campaign towards the general public increases the acceptability of campaigns
– Inform road users about intermediate results: information panels indicating the number of offences or the number of correct users
Step 4: Designing the campaign
Task 2: Develop the message• General principles about the campaign message
– Clear– Consistent– Unambiguous– Well understood by the target audience
• Content of the message versus style of the message
• You might want to involve an advertising agency to propose different communication concepts
• Campaign message ≠ campaign slogan
Step 4: Designing the campaign
• The message content: what will be said– Structure of the message
• One sided message: only arguments in favour of the theme of the campaign are presented
– E.g. wear a helmet because it reduces your probability of serious injuries or death in an accident
• Two-sided message: arguments in favour and against the theme are presented, then arguments to counter the opposing view are included
– E.g. wearing a helmet reduces your probability of serious injuries. We know that it is bad for your looks, but do you really think that it is worth driving without a helmet knowing that after having an accident your good looks will be gone forever.
Step 4: Designing the campaign
• When to adopt one-sided messages– When the target audience is sympathetic to the message– When the aim is an immediate/short term opinion
change
• When to adopt two-sided messages– When the target audience is unsympathetic to or initially
disagrees with the campaign message. I.e. when the negative aspects of the wanted behavior are well known among the target audience
– When it is likely that the target audience will be exposed to contradictory messages
Step 4: Designing the campaign
• The message style: how it will be said– Emotional and non-factual versus cognitive and
rational approach– Emotional messages can be either positive (love,
hapiness, pleasure) or negative (fear, anger, sadness)
– One must be cautious using fear appeals. Two types of reactions can take place:
• Cognitive reaction: the respondent wants to reduce the danger by adapting his/her behavior ( = control the danger)
• Emotional reaction: rejection of the message, i.e. activation of various mechanisms to protect the person from the threat (deny, defensive avoidance or reactance) (= control the fear)
Step 4: Designing the campaign
• Examples– Informational approach
• Drink driving (testimonial, social cost) http://www.youtube.com/watch?v=9kplg8O6GPQ
• Helmet wearing (celebrity endorsement)http://www.youtube.com/watch?v=BmZBDBjCh2o
• Speed
http://www.transport.qld.gov.au/resources/file/eb6d2f00d77fddb/stopping_distances_tvc.mpg
– Emotional approach• Drink driving (social cost)
http://www.youtube.com/watch?v=JdCTxTM_GtU
Step 4: Designing the campaign
• Examples– Fear appeal approach
Step 4: Designing the campaign
• Message framing– Problem avoidance: the message is
directed at avoiding getting into trouble by adopting the correct behaviorE.g. (seatbelts) http://www.youtube.com/watch?v=Xx6v9CNcQ04
– Promotion oriented: the message is directed at showing the benefits of adopting the correct behavior
Step 4: Designing the campaign
• Message execution: who’s saying it (crucial for credibility of the message)– Organizations instead of an individual– The use of testimonials– Celebrities
Step 4: Designing the campaign
• Campaign identifiers– Mascots– Branding– Logo– Celebrities-> they increase acceptance and/or credibility of
the message-> they are useful as a cornerstone of a complete
campaign package
Step 4: Designing the campaign
• What we know from international research– Persuasive and emotional messages work better than
rational and informative (educational) messages– Message must be relevant to the target group in order to
increase systematic processing by the target audience. E.g. playing on family values, family e
– Systematic processing increases by exposing the audience to the message several times
– Fear appeals can work in specific circumstances and when they are not too aggressive. Otherwise, a defensive reaction will take place. Therefore, it is very important to provide efficient mechanisms to avoid the threat and that the target audience finds itself capable of carrying out the recommended actions.
Step 4: Designing the campaign
Intervention Mapping
Step 1: Needs AssessmentStep 2: Defining Campaign ObjectivesStep 3: Selecting Methods and StrategiesStep 4: Designing the CampaignStep 5: Planning for Campaign
ImplementationStep 6: Planning for Campaign Evaluation
Step 5: Planning for campaign implementation
• Choice of media• Media planning• Pre-testing (if possible)
Step 5: Planning for campaign implementation
• Choice of media and mediators– Depends on campaign budget– Depends on the target audience’s media habits
• Time most likely to be exposed to the message (e.g. young males watch football matches)
• Time most likely to adopt the problem behavior (e.g. in bars ~ drink driving)
– Depends on geographic coverage of the campaign
– Depends on advantages and disadvantages of different media types
• Strengths/weaknesses different media types– TV
• S: large reach, high frequency, accessible to everyone• W: high production cost, short lived messages
(seconds)
– Radio• S: large reach, also local reach, dynamic, low
production cost• W: low attention, short lived messages, not suitable for
complicated messages
– Cinema• S: selective, high attention, allows for more complexity• W: high production cost
Step 5: Planning for campaign implementation
• Strengths/weaknesses different media types– Newspapers
• S: large reach, geographical selectivity, high credibility, high information capacity, short lead time
• W: short lifespan, poor demographic selectivity, poor reproduction quality, high noise ration
– Magazines• S: excellent for segmented audience, relatively long
lifespan, good reproduction quality• W: high insertion cost, low frequency
– Flyers, leaflets, brochures• S: high selectiveness, allows for complex messages,
low cost• W: low attention
Step 5: Planning for campaign implementation
• Strengths/weaknesses different media types– Direct mailings
• S: selective communication, for small target groups, high information capacity
• W: ‘junk mail’, need for address lists
– Billboards, small-size posters/banners• S: high exposure, on-the-spot presence,
geographically selective, long life span• W: low attention, low information capacity
– Face-to-face contacts• S: effective, interaction with target group• W: low exposure, high cost per contact
Step 5: Planning for campaign implementation
Step 5: Planning for campaign implementation
• Media planning= Scheduling the distribution of campaign
materials (spots, billboards, website, …) and supportive activities (e.g. education, enforcement, …)
– Duration and frequency• single-phase versus multi-phase
– Time of campaign implementation• Seasonal issues (e.g. fatigue in summer period)• Event related (e.g. drink driving around Khmer
New Year)
Step 5: Planning for campaign implementation
• If possible: Pre-testing !!– Is the message appropriate for the target audience?– Is the message understood as intended?– Is the message clearly stated?– Is the message perceived as useful to the target
group?– Is the message well recalled or remembered?– Is the message provoking unexpected feelings or
reactions in the target group?
Step 5: Planning for campaign implementation
• How to pre-test a campaign?– Interviews– Questionnaires– Focus groups– Thought listing task: record immediate
verbalised responses
Intervention Mapping
Step 1: Needs AssessmentStep 2: Defining Campaign ObjectivesStep 3: Selecting Methods and StrategiesStep 4: Designing the CampaignStep 5: Planning for Campaign
ImplementationStep 6: Planning for Campaign Evaluation
• Types of evaluation– Process evaluation– Outcome evaluation– Economic evaluation
• Evaluation designs– Non-experimental– Quasi-experimental– Experimental (randomized control trials)
• Statistical issues– Sample size– Data analysis
Step 6: Planning for program evaluation
• Types of evaluation– Process evaluation:
• Did the implementation of the campaign take place as planned?
• Where there any external factors disrupting the campaign process? If yes, what and how to prevent next time?
• How many paid and unpaid stories came out in the media?
• Objective exposure– Number of TV/radio spots, audience figures, number of
billboards, …– TARP = Target Audience Rating Points (TARP>100)
• Subjective exposure– Was the target audience aware that the campaign took place
(campaign awareness and message recall)– Did the target audience understand the campaign message?
Step 6: Planning for program evaluation
• Types of evaluation– Outcome evaluation:
• Road crashes, injuries, fatalities (e.g. from RTAVIS). However, beware that these may not be the direct result of your campaign!
• Behavior as outcome measure– V85 speed, median or average speed– BAC levels– Helmet wearing
How to measure: Observed behavior, self-reported behavior, number of offences. Beware of confounding factors: use of control group!
Step 6: Planning for program evaluation
• Types of evaluation– Economic evaluation:
• Cost of the campaign– Development of the campaign– Cost of media placement (e.g. TV advertising)– Beware: direct vs. indirect costs
• Benefits of the campaign– Benefits to the society due to reduced number of
fatalities, serious injuries and slight injuries» E.g. willingness-to-pay method» E.g. value of a victim (e.g. in Europe: 2.5
million dollar/fatality)
Step 6: Planning for program evaluation
• Types of evaluation– Economic evaluation (continued):
• Method of evaluation– Cost Effectiveness Analysis (CEA) = when
monetary evaluation of the benefits is not possible.
– E.g. Campaign or activity ‘X’ is most cost-effective in reducing the speed per km/h
– Cost Benefit Analysis (CBA) = when it is possible to value the benefits in a monetary way.
– E.g. For each dollar spent in campaign or activity ‘X’ society’s benefits are ‘$’ (cost-benefit ratio)
Step 6: Planning for program evaluation
• Evaluation designs– Non-experimental = simple before-after
comparison for a specific outcome measure in the treatment group without use of comparison group
– Limitations:• Several confounding factors: you are not sure that
what you measure is really the effect of your campaign
Step 6: Planning for program evaluation
Target audience
Before
AfterCampaign
• Evaluation designs– Quasi-experimental = a before-after analysis
using treatment (exposed to treatment, e.g. 16-25 years old) and comparison group (not exposed to treatment, e.g. 40-45 years old)
– Limitations:• Individuals are not randomly allocated to either group.
Ideally, the comparison group should contain 16-25 years old who were not exposed to the campaign, but sometimes this is practically not possible.
Step 6: Planning for program evaluation
Target audience
Before
AfterCampaign
Comparison group
• Evaluation designs– Experimental (=randomized control design) =
before-after study where the subjects are randomly allocated to either the treatment group or control group
– Limitations:• Theoretically the most powerful evaluation method,
however sometimes hard to implement in practice. For instance, when the target audience is the entire population of a particular age group.
Step 6: Planning for program evaluation
Target audience
Before
AfterCampaign
Control group
• Some more comments about evaluation designs– When to evaluate?
• Immediate after-period analysis• Longer-term after-period analysis (e.g. a number of
months after the campaign has finished) so evaluate the long-term effect of the campaign
– How to isolate effects of several campaign activities?
Step 6: Planning for program evaluation
Pre/post measurement
Media Enforcement
No program
Group 1 X
Group 2 X
Group 3 X X
Group 4 X
• Statistical issues– Sample size depends on a number of issues
• Statistical issues– Power of the test (we want to maximize this
probability) » Null hypothesis states there there is no
observable difference between the outcome in the treatment group versus the control group
» Power = the probability, assuming that the null hypothesis is false (i.e. effect of campaign is significant) of obtaining a result that will allow the rejection of the null hypothesis. So, if there is an effect, we want to maximize the probability of observing it in the data
» Typically, power >= 0.8
Step 6: Planning for program evaluation
• Statistical issues– Sample size depends on a number of issues
• Statistical issues (continued)– Significance of the test (we want to minimize this
probability)» Probability to measure an effect if it is not there (false
positive)» Typically, significance <=0.05
– Difference: what difference between the treatment group and comparison/control group does the test need to detect?
» E.g. we want the test to find a difference of 10% between both groups
» Decreasing this number will increase the number of respondents that you need in both groups
• Practical issues– Budget available for carrying out the evaluation study
Step 6: Planning for program evaluation
• Example of sample sizes
Step 6: Planning for program evaluation
Power Significance
Difference
Sample size each
group0.8 0.05 5% 1,237
0.8 0.05 10% 310
0.8 0.05 20% 78
• Statistical issues– Data analysis
• Descriptive statistics– Comparing sample mean, standard deviation– Simple graphs: histograms, box plots, scatter diagrams,
line charts
• Inferential statistics– Hypothesis testing: testing whether the observed
difference in the outcome measure between the treatment group and the control group is statistically significant
» E.g. T-test– Advanced methodologies
» Time series analysis» Structural equation modelling
Step 6: Planning for program evaluation
Step 6: Planning for program evaluation
Example structural equation model