DM 201- Avalon Consulting + Donor Services Group - Multi-Channel Marketing
DM 201 - Avalon Consulting - Testing Creative
-
Upload
avalon-consulting -
Category
Marketing
-
view
133 -
download
0
description
Transcript of DM 201 - Avalon Consulting - Testing Creative
![Page 1: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/1.jpg)
1Avalon Consulting Group, Inc.
All rights reserved, 2014
DIRECT MARKETING 201The Strategy and Metrics of Creative Testing
Avalon Consulting Group
May 14, 2014
www.avalonconsulting.net
![Page 2: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/2.jpg)
2Avalon Consulting Group, Inc.
All rights reserved, 2014
Testing QUIZ!Which Test Won?
![Page 3: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/3.jpg)
3Avalon Consulting Group, Inc.
All rights reserved, 2014
A
B
![Page 4: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/4.jpg)
4Avalon Consulting Group, Inc.
All rights reserved, 2014
![Page 5: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/5.jpg)
5Avalon Consulting Group, Inc.
All rights reserved, 2014
A
B
![Page 6: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/6.jpg)
6Avalon Consulting Group, Inc.
All rights reserved, 2014
A
B
![Page 7: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/7.jpg)
7Avalon Consulting Group, Inc.
All rights reserved, 2014
A B
![Page 8: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/8.jpg)
8Avalon Consulting Group, Inc.
All rights reserved, 2014
B
A
A
A
And the Winners Are…
![Page 9: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/9.jpg)
9Avalon Consulting Group, Inc.
All rights reserved, 2014
In case you’ve forgotten, your opinion doesn’t matter…
![Page 10: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/10.jpg)
10Avalon Consulting Group, Inc.
All rights reserved, 2014
Creative Strategy & Testing
![Page 11: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/11.jpg)
11Avalon Consulting Group, Inc.
All rights reserved, 2014
TESTING STRATEGY
Testing is essential to have a healthy, productive direct marketing program.
1. Each organization is different and not all best practices are universal.
2. Used to determine what works for your organization.
3. Test making incremental or large changes, seeing their impact, while minimizing the overall risk.
4. Help find ways to improve performance, can save money and improve the bottom-line.
Things to remember about testing:
Test one element at a time
Test to save cost (formats, components) or boost response rate or average gift (offer, teasers, treatments)
Ensure the statistical significance of your test panel.
![Page 12: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/12.jpg)
12Avalon Consulting Group, Inc.
All rights reserved, 2014
CASE STUDY: A Snapshot of the NPCA Acquisition Testing History
![Page 13: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/13.jpg)
13Avalon Consulting Group, Inc.
All rights reserved, 2014
NPCA Acquisition Control Packages
![Page 14: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/14.jpg)
14Avalon Consulting Group, Inc.
All rights reserved, 2014
DM ACQUISITION CONTROL PERFORMANCEJu
ly 2
011
Aug
2011
Sept
201
1
Oct
201
1
Nov
201
1
Dec 2
011
Jan
2012
Feb
2012
Mar
ch 2
012
April
201
2
May
201
2($30.00)
($25.00)
($20.00)
($15.00)
($10.00)
($5.00)
$0.00
Survey net/donor
Member card net/donor
• The FY12 overall results of
the acquisition controls have
shown a weakening of the
member card, with the
survey more cost effectively
acquiring donors.
• However, Avalon analysis of
the member card showed a
stronger ROI, so both
packages continue to be
mailed as co-controls in
FY13.
![Page 15: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/15.jpg)
15Avalon Consulting Group, Inc.
All rights reserved, 2014
Test Within Your Acquisition Control Packages
Premium (cost savings)CONTROL: Calendar and bucket hat TEST: Calendar only WINNER: Control - Calendar bucket hat
Ask String (boost avg. gift)CONTROL: $15 minimum gift to receive premiumTEST: $25 upgrade ask testWINNER: There was no statistical winner in the original test, but we mailed a panel of this package every month. Backend analysis shows ROI of $25 ask is stronger.
Combine elements of co-controlsCONTROL: Member card package or Survey packageTEST: Member card in Survey package WINNER: Test – Member card in the Survey package. Re-test confirmed results.
Combine tone of Survey and Member Card CONTROL: Member card package or Survey packageTEST: Combine the urgency of the member card with the emotion of the survey letter. WINNER: There was no statistical winner. Re-testing to get significant results.
Phrase-based formatting (boost response rate)CONTROL: Control acquisition packageTEST: Use ReadSmart to make subtle changes in the arrangement of the text on the page WINNER: There is no statistical winner.
Removal of insert (cost savings)CONTROL: Control acquisition packageTEST: Remove glossy color insert WINNER: Control with the insert.
![Page 16: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/16.jpg)
16Avalon Consulting Group, Inc.
All rights reserved, 2014
Test New Acquisition Packages
Map package Included a foldout wall map and personalized teaser with a national park in the prospect’s state. The test has a 0.91% response rate – comparable to the control’s 0.92%, and a higher average gift than the control ($23.13 vs. $22.73), but the costs were too high.
Patriotic package Used a stirring, patriotic tone to highlight the bond Americans feel with their national parks, and included a lift note with a quote from Ken Burns. Avg. gifts are only $0.30 apart, the strong control had the higher response rate (0.99% vs. the test’s 0.83%) and revenue/M - $234 vs. the test’s $196.
![Page 17: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/17.jpg)
17Avalon Consulting Group, Inc.
All rights reserved, 2014
ACQUISITION TESTS – NEW PACKAGES
Postcard Package (Merge K) This package was developed to try and incorporate more park images into the acquisition with the theory that if someone was reminded of the beauty they were helping to protect, they would be more responsive. The “Wish you were here!” package included a 4-color insert with four postcards showcasing the grandeur of the parks. The letter invited members to enjoy our parks now, and support them for generations to come, by joining NPCA.
The postcard test had a significantly weaker revenue/M, compared to the member card and the survey.
![Page 18: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/18.jpg)
18Avalon Consulting Group, Inc.
All rights reserved, 2014
ACQUISITION TESTS – NEW PACKAGES
Magazine Package The magazine subscription package included a 4-color sample of the National Parks magazine, and emphasized the magazine as NPCA’s premier membership benefit.
In addition to a weaker average gift, the package currently has a 10% weaker RR than the survey package and a 30% weaker RR than the member card, as well as a weaker revenue/M.
![Page 19: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/19.jpg)
19Avalon Consulting Group, Inc.
All rights reserved, 2014
Review Acquisition Results by Premium
Anorak Bison Blanket Bucket hat
Tote bag$0.00
$5.00
$10.00
$15.00
$20.00
$25.00 Survey
Survey
Review back-end premium costs to acquire a donor. Include front-end as well as back-end costs.
The anorak, blanket and bucket hat all have comparable results in the member card package and can be mailed interchangeably, but in the survey, the blanket, tote bag and bucket hat are strongest because their costs to acquire a donor are lower.
Anorak Bison Blanket Bucket hat Tote bag$10.20
$10.40
$10.60
$10.80
$11.00
$11.20
$11.40
$11.60
$11.80 Member card
Member card
![Page 20: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/20.jpg)
20Avalon Consulting Group, Inc.
All rights reserved, 2014
Online Activist Conversion Results
The Sept pre-email was the first contact in the series. Both Group A and Group C had a 0.05% response rate.
The Oct DM acquisition was sent to groups A and B – group A had received the pre-email and group B did not. Group A had a 1.03% response rate, while group B had a 0.98% response rate. By comparison, the overall mailing had a 0.86% response rate.
The Nov TM acquisition made 4,723 contacts and 286 members joined with an average gift of $27.83. Groups A and B had nearly identical results on the phone.
The Dec post-email was the last contact in the series. Group B had a 0.10% response rate, while Group C had 0.08%. The modeled groups had an average gift of $31.30. In addition, the non-modeled group had a $26.75 average gift.
![Page 21: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/21.jpg)
21Avalon Consulting Group, Inc.
All rights reserved, 2014
Testing for Impact
![Page 22: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/22.jpg)
22Avalon Consulting Group, Inc.
All rights reserved, 2014
Testing Carriers: Getting them into the envelope
![Page 23: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/23.jpg)
23Avalon Consulting Group, Inc.
All rights reserved, 2014
Image vs. No Image
![Page 24: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/24.jpg)
24Avalon Consulting Group, Inc.
All rights reserved, 2014
Plain Carrier Test
Tested against 4-color control
carrier
10
Plain vs. 4-Color Carrier
![Page 25: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/25.jpg)
25Avalon Consulting Group, Inc.
All rights reserved, 20149
Teaser and Photo Testing
![Page 26: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/26.jpg)
26Avalon Consulting Group, Inc.
All rights reserved, 2014
Testing Offers: Compelling them to Respond
![Page 27: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/27.jpg)
27Avalon Consulting Group, Inc.
All rights reserved, 2014
Discount Offer Test
Added discount to carrier teaser/
changed PS discount
language from $3 to 10% discount
9
Discount offer
![Page 28: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/28.jpg)
28Avalon Consulting Group, Inc.
All rights reserved, 2014
Ask string testing
JUNE: Control asking string was tested against descending asking string and askingstring with only three asks. In both cases, control is maintained with higher response rate and higher
gross/thousand.
Control
Test 1
Test 2
![Page 29: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/29.jpg)
29Avalon Consulting Group, Inc.
All rights reserved, 2014
C3 vs. C4 DM & EM SURVEY APPEAL
JANUARY mailed annual survey in official-looking carrier with companion e-appeal. Messaging focused on state of our democracy.
TESTS: 501c3 version of appeal was tested.
DM: C3 test version was winner at 85% confidence with higher response rate (4.85% vs. 4.75%) and $1.46 higher average gift.
EM: C4 control had higher response rate (0.28% vs. ).24%), but $17.91 lower average gift. There was no statistical winner.
![Page 30: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/30.jpg)
30Avalon Consulting Group, Inc.
All rights reserved, 2014
Testing Engagement Devices: Engaging Beyond Giving
![Page 31: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/31.jpg)
31Avalon Consulting Group, Inc.
All rights reserved, 20149
A Petition as a Call to Action
![Page 32: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/32.jpg)
32Avalon Consulting Group, Inc.
All rights reserved, 2014
The NPCA Mining Appeal exceeded budgeted revenue goal by 17% and net by 33%.
The drive to web test, with the URL listed on the reply and in the P.S of the letter earned more revenue due to a higher response rate and beat the control at 84% confidence.
Drive to Web Test
![Page 33: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/33.jpg)
33Avalon Consulting Group, Inc.
All rights reserved, 2014
Testing Messaging
![Page 34: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/34.jpg)
34Avalon Consulting Group, Inc.
All rights reserved, 2014
PILOT CAMPAIGN Package #1: From the Field/Mike Fay
![Page 35: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/35.jpg)
35Avalon Consulting Group, Inc.
All rights reserved, 2014
PILOT CAMPAIGNPackage #2: RSVP/Invitation
![Page 36: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/36.jpg)
36Avalon Consulting Group, Inc.
All rights reserved, 2014
Testing Tone and Language: Aggressive
![Page 37: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/37.jpg)
37Avalon Consulting Group, Inc.
All rights reserved, 2014
Testing Tone and Language: Cooperative
![Page 38: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/38.jpg)
38Avalon Consulting Group, Inc.
All rights reserved, 2014
Testing to Settle an Argument
![Page 39: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/39.jpg)
39Avalon Consulting Group, Inc.
All rights reserved, 2014
Testing a Long-Standing Control
![Page 40: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/40.jpg)
40Avalon Consulting Group, Inc.
All rights reserved, 2014
Matching Gift Appeal Test
![Page 41: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/41.jpg)
41Avalon Consulting Group, Inc.
All rights reserved, 2014
Matching Gift Appeal Test
![Page 42: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/42.jpg)
42Avalon Consulting Group, Inc.
All rights reserved, 2014
A: Control
B: New Logo Test
C: Tagline Test
Logo Testing
![Page 43: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/43.jpg)
43Avalon Consulting Group, Inc.
All rights reserved, 2014
2 p vs. 4 p
![Page 44: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/44.jpg)
44Avalon Consulting Group, Inc.
All rights reserved, 2014
And Don’t Forget: Testing to Reduce Costs
![Page 45: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/45.jpg)
45Avalon Consulting Group, Inc.
All rights reserved, 2014
Some of our Favorite Cost-Saving Tests:
Test 1st vs. 3rd Class postage
Test a modified or standard size or format
Test eliminating an insert
Test an E-R0 effort to invite people to renew online and not receive a DM renewal
![Page 46: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/46.jpg)
46Avalon Consulting Group, Inc.
All rights reserved, 2014
The Analytics of Testing
![Page 47: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/47.jpg)
47Avalon Consulting Group, Inc.
All rights reserved, 2014
• There are two phases to validating and analyzing direct marketing
testing. The first is ensuring that the test sample size is valid. • A sample size that is too small will not provide reliable results.
Part 1: Sample Size Calculator
Go to:
http://
www.avalonconsulting.net/
Sample-Size-Calculator.html
Recommendation: Perform this
calculation before the campaign is
conducted to guide the size of the
test and to ensure testing is
worthwhile.
![Page 48: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/48.jpg)
48Avalon Consulting Group, Inc.
All rights reserved, 2014
• Step 1: How much do I need to mail?• Enter proposed volume for control or test
panels.• Step 2: Is there a valid percent return?
• Enter expected response (pre-campaign,
use a conservative estimate.)• Step 3: Is my confidence above 80%?
• If not, remove either quantity or response to
determine what values are needed.
Repeat with actual campaign results
if needed and if fluctuating from
proposed.
Part 1: Sample Size Calculator
![Page 49: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/49.jpg)
49Avalon Consulting Group, Inc.
All rights reserved, 2014
Part 2: Testing Confidence Calculator
• After a valid sample size is achieved, the next step is to determine
if the results of the test are significant.
• Important: Just because there is a difference in performance
does not mean the performance can be replicated. Often, it is
within the margin of error – a flip of the coin.
• Only make decisions on valid results.
• Goal: Determine if the difference in net per thousand mailed
(other metrics can be used) is significant or within the margin
of error. Typically, a confidence level of 80% is a starting point,
with 95% being ideal.
![Page 50: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/50.jpg)
50Avalon Consulting Group, Inc.
All rights reserved, 2014
Testing Confidence Calculator – Process
• Step 1: • Gather control and test panel quantities, gifts, revenue, and roll out costs.• Remove outlier gifts from quantity, gifts, and revenue. A typical rule of thumb is
$1,000 gifts or more.• Step 2:
• Calculate “upper bound” and “lower bound” for the response rate, revenue per
thousand, and/or net per thousand. It is typical to start with the standard normal
deviate of 1.96.• Step 3:
• If there is no overlap between the upper bound of one panel and the lower bound
of another, a confidence level will be found.
Upper LowerNet/M % # Bound Bound
Test Group Net/M Winner Conf. Conf. Net/M Net/M
Control $1,477 - $1,611 $1,343Test $1,826 WINNER $1,982 $1,670
95% 1.96
![Page 51: DM 201 - Avalon Consulting - Testing Creative](https://reader033.fdocuments.net/reader033/viewer/2022061200/54777741b4af9fa5248b47c0/html5/thumbnails/51.jpg)
51Avalon Consulting Group, Inc.
All rights reserved, 2014
Thank you!
Allison PorterPresident
Avalon Consulting Group2030 M Street, NW Suite 700
Washington, DC 20036202-429-6080 x102
Follow Avalon on LinkedIn Today!