Post on 15-Apr-2017
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Executing and A/B Testing Cross-Channel Personalization
Andy Zimmerman, Evergage Shanelle Mullin, ConversionXL
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Agenda• The Cross-Channel Imperative • Challenges • Solution Requirements • Examples • A/B Test Process • Prioritization • Duration • Results and Archiving • Q & A
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
The Cross-Channel Imperative
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Challenge: Cross-Channel is Happening Now
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Challenge: Cross-Channel is Happening Now
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
70% OF WALMART.COM
TRAFFIC LAST HOLIDAY SEASON
WAS MOBILE DEVICES.
Forrester: Your Digital Experience Technology Starts With A Customer Journey Map
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
69%
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
$1.5 Trillion retail sales generated
by cross-channel interactions – Forrester, The Omnichannel Commerce Playbook for 2016
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
47% of bank customers engaged in
at least one cross-channel interaction over 90 day period.– Forrester, Design Better Cross-Channel Banking Journeys
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Top Goal is Customer Experience
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
4 Key Elements of a Great Customer Experience
4
1 Remember me.
2 Understand me.
3 Help me achieve my goals.
Surprise me. Delight me.
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Cross-Channel Personalization
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
77% of consumers have chosen, recommended, or paid more for a
brand that provides a personalized service or experience.
– Forrester’s North American Consumer Technographics® Brand Compass Survey, Q3 2015
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
The Cross-Channel Challenge
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Challenge: Organizational Misalignment
– Forrester, The Omnichannel Commerce Playbook for 2016
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Challenge: Organizational Misalignment
– Forrester, The Omnichannel Commerce Playbook for 2016
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Challenge: Data is SiloedMarketers are using SIX or more tools to collect performance data.
Marketing Analytics Tool
Website Analytics Tool
Marketing Automation Tool
Operational Analytics Tool
Business Intelligence System
Sales Automation Tool
Telemarketing or Online Surveys
Channel Performance/Analytics Tool
CRM System
ERP System
0% 20% 40% 60% 80%
30%
42%
43%
44%
54%
54%
57%
57%
69%
79%
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Requirements for a Cross-Channel Personalization Solution
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Track Behavior Across ChannelsWebsite Email SearchMobile
Web App
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Synch Data with Other Systems
CRM
Marketing Automation
Email Marketing
Data Warehouse
Data Lake
Call Center
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
One Unified Customer Profile
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Core Capabilities of Personalization Platform
Cross-channel Targeting
Algorithmic Recommendations
Real-Time Segmentation
A/B & MV Testing
Analysis & Attribution
Behavioral Tracking
Unified Customer Profile
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Cross-Channel Personalization Maturity Evolution
Beginning Intermediate Part A
Intermediate Part B Advanced
Step 1 Digital Channel 2 or 3 Digital Channels
Synch with other systems
All channels, including agents
Web Mobile App Open-time Email Search
Web Mobile App Open-time Email Search
CRM System Email/MAP Data Warehouse
Call Center In-Store Kiosk Reps/Agents
Benefit One Team Multiple Teams Better data, Targeting x-channel Organization-wide
Challenge Getting Started Cross-Team coordination needed
Coordinate synch if not out-of-box, APIs
Additional Integrations
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Examples of Cross-Channel Personalization
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Ski Resort Example
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Ski Resort Example
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Ski Resort Example
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Rue La La Example
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
How to A/B Test
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Only 32% of small companies and 42% of large companies have a structured, documented process.
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
1. Research 2. Prioritize 3. Experiment 4. Analyze and Learn
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Prioritization Problems
1. Ease and impact are too subjective 2. Too rigid. 3. Doesn’t encourage a data-driven culture
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Prioritization Fixes
1. Binary responses 2. Customizable variables 3. Ask everyone to bring data
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Grab the Template
bit.ly/pxlmodel
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Determining How Long to Test
1. Calculating sample size 2. Running for full weeks, usually two cycles
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Use the Calculator
bit.ly/evansample
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Ensuring a Representative Sample
1. Every day of the week 2. Various traffic sources 3. Ad, email, social, content schedule 4. External factors (e.g. holidays) 5. Comparison shoppers
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
On External Factors...
1. Season 2. Day of the Week 3. Press 4. PPC / SEM / SEO 5. Word of Mouth
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
On Sample Pollution...
1. Length pollution 2. Device pollution 3. Browser pollution 4. Cookie pollution
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Watch Out...
1. Regression to the mean 2. The novelty effect 3. Simultaneous testing
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Creating a Hypothesis
1. Is it designed to solve a conversion problem? 2. Is it measurable / testable? 3. Will you gain insight, win or lose?
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Skills That Help with Testing
1. HTML, CSS, JavaScript, jQuery 2. Design 3. Copywriting
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Get the Details
bit.ly/gaabresults
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
No Difference? Not So Fast...
1. Maybe the implementation sucked. 2. The variation might’ve won in a segment.
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Segmentation 101
1. Desktop vs. Mobile 2. New vs. Returning 3. Traffic Direct on the Page vs. Traffic via Internal Link
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Insights > Wins
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Why Archive?
1. To avoid repeating tests 2. To better communicate wins and insights 3. To improve the quality of future tests 4. To create an insights > wins culture
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Archive Methods
1. Iridion 2. Effective Experiments 3. GrowthHackers Projects 4. Experiment Engine 5. Testing Tool 6. Excel
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Conversion optimization training program.
evergage-webinar
WEBINAR: EXECUTING AND A/B TESTING CROSS-CHANNEL PERSONALIZATION
Questions?