Unique test database reveals gold nuggets

Fiona McPhee

Why do we test?

Huge database of test results

No guarantees – nothing stays the same

Short-term vs. Long-term

World of myths to bust

Testing for over 12 years

Keep In Mind

Have a test strategy & hypothesis for each

Does it have a chance of being valid?

Can you do anything with the potential results?

Have a control group & test group(s)

Have a test budget

A / B??

What is affecting response

Digital: What has and hasn’t worked

Channels: Direct Mail, Digital, Phone

What is affecting income

Testing Propositions & Language

What is on the horizon?

% $

The biggest things that influence response rates

Asking

Audience

Contact Rates / Open rates

Ability to contact in future

Should we ask?In a direct mail survey test the

inclusion of an ask at the end of the survey significantly

increased response rate, return on investment and net income.

Test Segment Mailed # Resp Resp Rate Avg Gift Gross Income Net Income ROI

Ask 4847 285 5.9% $49.55 $14,123 -$2,308 0.86No Ask 4855 198 4.1% $57.23 $11,331 -$5,127 0.69

Should we make specific asks?

In a direct mail appeal test we asked in the copy and on the response mechanism against just on the response mechanism. The ask in copy and on the response mechanism significantly increased response rate

Test Segment Mailed # Resp Income Response Rate AvgGift

Ask in Copy & RM 5248 354 $18,818.00 6.75% $ 53.16

Ask on RM Only 5248 299 $17,765.20 5.70% $ 59.42

This is a sample text.Insert your desired text here.

Sample text

The low multiplier significantly increased response rate for a direct mail regular giving conversion ask

Test Segment Mailed # RG Resp RG Resp Rate

Avg Annual RG Value Annual Value

High Ask 2,241 59 2.6% $246.10 $14,520Low Ask 2,242 95 4.2% $171.66 $16,308

How high should we ask?

Audience Testing

Mail: Response highest from most recentPhone: Variables such as channel and pay type also have significant impact

Warm

Strong respondersThere is a cost to swapping

Charity Givers

Mail: Quality of data criticalPhone: Your own leads can increase response

Tepid

Cheap = cheapKnow what you are testing

Cold

WarmWarm

TepidTepid

ColdCold

Charity GiversCharity Givers

Response Rates

Direct Mail Phone Online

Face to Face Media Other0%

25%

50%

75%

100%

0%

25%

50%

75%

100%

2006

2007

2008

2009

2010

2011

2012

2013

2014

2015

2006

2007

2008

2009

2010

2011

2012

2013

2014

2015

2006

2007

2008

2009

2010

2011

2012

2013

2014

2015

Mai

labi

lity

Rate

Contact Rates

Contact RatesCharity A: ~40k RGs Year 1 Year 2 Year 3

# Campaigns / Batches

1 2 5

Conversations 6,960 12,828 19,685Upgrades 2,288 4,840 5,827Yr 5 Net Income $1,086,000 $2,719,000 $2,907,000

Charity B: ~16k RGs Year 1 Year 2

# Campaigns / Batches 1 2

Conversations 5,174 10,901Upgrades 2,070 4,197Yr 5 Net Income $1,141,000 $1,977,000

Contact detail capture: Digital Lead Generation

Mandatory =

Less sign upsHigher quality

90% to call

Not Mandatory =

More volume25% phone opt

out

0

500

1,000

1,500

2,000

2,500

3,000

0%

10%

20%

30%

40%

50%

60%

70%

Addr1 Addr1Email

Addr1Email

Landline

Addr1Email

LandlineMobile

Addr1Email

Mobile

Addr1Landline

Addr1Mobile

Email EmailLandline

EmailMobile

2nd Gift Rate (Year 1) RecruitsBase: 2009 - 2014 Recruits

Multi-contact opportunity donors have the highest

second gift rates. No address sees much lower

conversion.

Address & Mobile potentially an age thing

Information provided by

donors can be increased over

time

Information provided by

donors can be increased over

time

The biggest things that influence income

Asking

Longer is better

Tangible

Asking

Test Segment Qty Mailed Resp Resp Rate Ave Gift Net

Income ROI

Ask 9,831 1,858 19.9% $57.02 $93,610 8.59

No Ask in Copy nor

RM4,240 648 15.3% $47.81 $25,641 5.80

Is longer better?

That longer appeal letters will generate a lower response

8 campaigns – 6 charities2 significant results for response

Only resulted in significantly higher income on one occasion

Test Mailed Resp# RR% AV G Net Return

4pp 9,415 2,554 27.1% $59.03 $124,769 5.80

2pp 9,415 2,287 24.3% $57.44 $106,129 5.21

For an individual test thoughLikely to be marginal difference

Any cost of 4pp vs. 2ppWould have to be less than that.

4pp Generates more large gifts & consistently

more income

Gross Income per ‘000 mailedAn average of an extra $1 per

donor mailed

Is longer better?

Is Tied better?

That asking for Tied funds (vs. General funds) will not increase RR% or Av Gift

Significant result on RR%

Average gift higher for Tied funds but not enough to outweigh lower RR% across the campaign

Test Mailed Response RR% AV G

General 17,324 2,826 16.31% $50.06

Tied 17,325 2,661 15.36% $45.62

“If I can not raise $100,000, I will have to cut X service.”

Vs.“If I can not raise $100,000, we will

have to consider cutting the X program and others like it.”

More Gold?World largest single mass of Gold, 630 pounds

Testing Propositions

Three propositions vying for supremacy

Two taken to the phones with warm donors

One significantly out pulled the other half way through calling

01

02

03

Welcome to Stroke Foundation Life Savers.

Thank you for taking the time to speak with one of the team recently and thank you so much for making the wonderful decision to join Life Savers.- helping us to run a powerful and sustained national campaign to stop so many people dying needlessly. Together we can save a life every hour.

Thank you for taking the time to speak with one of the team recently and thank you so much for making the wonderful decision to join us in spreading the message to save lives - helping us to fill in the gaps in people’s knowledge about stroke.

By choosing to make a monthly donation you are continuing to change lives and will be helping to make sure everyone in New Zealand knows how to reduce the chance of stroke, how to recognise stroke and what to do if a stroke happens.

Testing Language

Does the use of colloquial language increase response rate?

The use of Kiwi (vs. New Zealand) language in variable copy increased response rates. Average gifts were down for the colloquial test group – did we put off posher donors?

Less formal language positively impacting response. Finding the right tone in relation to each appeal / story important and we are looking at testing/adaptation for the Mid-Value Donor program.

01

02

03

Test Segment

Mailed Cash Avg Gift RG Avg Gift Comb

ROIComb RR%

Kiwi 26,534 $46 $13 3.04 4.21%

New Zealand 26,454 $57 $14 3.04 3.83%

Digital: Direct to Donation

10 years ago it worked

Early adopters getting results

Integrating warm was getting traction

Asking to get married before the first date?

Digital: Direct to Donation

Too high a barrier to

donating as first interaction

10 years ago it worked Now: Wall to Wall

advertising

Lifeline

STEP 1 – Drive to Petition

Facebook – best performing creative, Step 1 – Petition

1 2 3

Lead AcquisitionSignatures: 65,027

Email opt-ins: 31,609 Phone opt-ins: 8,838

Some of the things we are testing in the digi space now

• Digital to phone• Digital to digitalConversion Step• Petition• Survey• Pledge

Lead Sources

• Targeting within mediaMedia• Link Ad• Canvas• Carousels• Video Slideshows

Format• Stories• Images• Subject lines

Creative

Channel Integration

Does the provision of online options increase survey response?

Including both the launch email and the post email had the most significant impact on Survey response. The emails had no significant impact on financial donations to the Survey.

Email integration had a positive impact on the key objective of the appeal. Continuing to increase email penetration rates and testing email execution needed.

01

02

03

Ask Tests

Level change against each

other

Fixed string against

calculated

Number of asks

Type of asksCombining NL/Survey with an ask

Highlight asks

Reversing sequence

Other than ask1 in copy,

no ask in copy

My choice on RM

Having no calculated asks on RM and copy suppresses the response and results in lower average gift

Numbers of asks in copy – insignificant outcomes

Adding ask at end of survey increases

response rate

5 Charities, 6 tests, 1 significant

Number of Asks

Ask Testing

Sample text

Highlight asks

Insignificant

Reverse Sequence Ask 2 As Main

Lowers Response but appears to increase

average gift

Insignificant Significant

Direct Mail Pack Additions Testing

Does inclusion of a premium (Note pad) vs no premium uplift response rate?

Inclusion of a Notepad Premium uplifted response rate to the appeal – however due to the higher cost of this pack, ROI is lower

Premiums continue to uplift response, however cost and audience needs to be carefully balanced to make it a valid option going forward

01

02

03

Pack Additions Testing

Significant higher response rate and net income

Include a Calendar

Significantly lower average gift

More responded

Significantly higher response rate and average gift

Include an Involvement Device

Significantly lower net income

Sample text

Significantly higher response rate

Post It Notes

Increased net income

Significantly lower response rate & net income

Adding Newsletter

Higher average gift with newsletter

More responded More responded Less responded

What is on the horizon?

1

2 Splitting incorrectly & not sticking with it

What is it? How do you provide it? How can you test its impact/value?

Measure the income & retention vs. total cost over an extended period of time

3

Stewardship / Donor Care

Wrong

Right

What is on the horizon?

1

2 Are we under emailing? Under mailing? Under communicating

What is appropriate? What is effective?

One channel / more channels? Or is it all too much?3

Frequency of Contact

Too Little?

Too Much?

fiona.mcphee@paretofundraising.com