Amazon AB Testing - Modifying & Measuring App Behaviors Without Republishing

#mceconf #mce-addingknobs 11 Jan 2014 © 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

Adding Knobs to your App – Modifying & Measuring Behavior in the Wild (without Republishing) Cheng Luo, Amazon Appstore 11th January, 2014 @chengluo

#mceconf #mce-addingknobs 11 Jan 2014

Agenda •  Why Amazon Mobile SDK •  How to use A|B testing •  Add your A|B testing in 10 minutes •  Demo

30% 25%

<$50 $50-‐$200 $200+

Monthly Unique Visitors (US):

6.6 Active Global Customer

Accounts:

200M Avg. Daily Unique Visitors

Note: Active Customer Accounts are unique email addresses that have placed an order in the last 12 months Sources: 1. Amazon Investor Relations; 2. ComScore PlanMetrix January, 2013.

<$25K $25K-‐$75K $75K +

1 2 2 2

0-‐18 18-‐34 35-‐44 45+

Household Income of Amazon Unique Visitors2

Age Composition of Amazon Unique Visitors2

% of Amazon Shoppers Who have Made an Online Purchase2

Amount Spent

Avg. Visits per Visitor per

Month (US):

Active Users

•  Amazon Appstore runs Android apps

•  75% of Android apps work without any need for customisaTon •  Simple APIs to publish free-‐to-‐user, freemium and premium apps

A/B TesTng Service API

GameCircle API

Maps API Device Messaging API

In-‐App Purchasing API

Simplicity

The latest Kindle Fire HDX

Kindle Fire HDX 7" Tablet $229, with 24x7 May Day support

What A|B Testing can do for you

Precisely control who is affected

Change in the wild; No new APK needed

Safely test back-‐end load

Case Study - Air Patriots Used A/B Testing to decide: •  Include in-game advertising •  Provide purchasing short-cuts •  Notification frequency •  Level difficulty •  Promotional imagery

Case 1 – Image

Case 2 – Ads placing

Case 3 - Notification Notification appears •  x1 day •  x 3 days → Sweet spot •  x 7 days

Case 4 - User Retention

Level: 1 Difficulty: Easy

Level: 5 Difficulty: Medium

Air Patriots Sample Results

Add A/B Testing to Your App

Initialize A/B Testing in Your App private static final String APP_KEY = "876c809ba7084b9fbe2c08a22e5acda1"; private static final String PRIVATE_KEY = "9nxxk4//qL89ymXl/P5aL7w8uhs3DVDrLcyYViG6yn0="; AmazonInsights insightsInstance; @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.activity_main); . . . InsightsCredentials credentials = AmazonInsights.newCredentials(APP_KEY, PRIVATE_KEY); insightsInstance = AmazonInsights.newInstance(credentials, getApplicationContext()); }

Create an A/B Testing Project

Add a New Launch

Multiple Projects Are Allowed

private static final String AB_PRJ_ACC = "Acceleration"; //Project name private static final String AB_VAR_MAXACC = "maxAcceleration"; //Variable name private static final String AB_PRJ_VEL = "Velocity"; //Project name private static final String AB_VAR_MAXVEL = "maxVelocity"; //variable name

Load Variation Variables private static final int DEF_MAXACC = 5;

private static final int DEF_MAXVEL = 50;

private int maxAcc;

private int maxVel; private void loadTreatment() { ABTestClient abClient = insightsInstance.getABTestClient(); abClient.getVariations(AB_PRJ_VEL, AB_PRJ_ACC).setCallback(new InsightsCallback<VariationSet>() { @Override public void onComplete(VariationSet variations) { Variation variation = variations.getVariation(AB_PRJ_ACC); maxAcc = variation.getVariableAsInt(AB_VAR_MAXACC, DEF_MAXACC); variation = variations.getVariation(AB_PRJ_VEL); maxVel = variation.getVariableAsInt(AB_VAR_MAXVEL, DEF_MAXVEL); } }); }

Changing Values on the Fly

Add a Segment

Use the Segment

//get user age from the app settings SharedPreferences myPrefs; myPrefs = getSharedPreferences("com.example.testingapp", Context.MODE_PRIVATE); int userAge = myPrefs.getInt("com.example.testingapp.USER_AGE", ""); //Initial the segments here

UserProfile userProfile = insightsInstance.getUserProfile(); userProfile.addDimensionAsNumber("age", userAge);

Use App Segngs to idenTfy your user segments

Wrap up •  A/B Testing

– Free cross platform services – Precise measuring behaviours using custom

segments – Modifying app behaviors without

republishing

Thank you@AmazonAppDev

Amazon AB Testing - Modifying & Measuring App Behaviors Without Republishing

Technology

Transcript of Amazon AB Testing - Modifying & Measuring App Behaviors Without Republishing

Self Modifying Code

Modifying Irrevocable Trusts Using Nonjudicial Settlement ...media.straffordpub.com/products/modifying... · 9/19/2017 · Modifying Irrevocable Trusts Using Nonjudicial Settlement

Multidrug-resistant genes of aminoglycoside- modifying ... · Multidrug-resistant genes of aminoglycoside-modifying ... We examined the distribution of genes of aminoglycoside-modifying

Modifying Jython Examples

Final Ph.D synopsisshodh.inflibnet.ac.in/.../123456789/1228/1/02_synopsis.pdf1 1.0.0 INTRODUCTION Learning is acquiring new or modifying existing, knowledge, behaviors, skills, values,

Behavior Modifying Chemicals

Modifying Interface 10en

Modifying padlets

Learning is acquiring new, or modifying and reinforcing existing knowlege, behaviors, skills values, or preferences and may involve synthesizing different.

Modifying Styles

Modifying Wordpress Themes

Certain Footwear: Recommendations for Modifying … · Certain Footwear: Recommendations for Modifying the Harmonized Tariff ... Recommendations for Modifying the Harmonized Tariff

Modifying Tables

Modifying Objects

beam modifying devises

Replacement Behaviors Teaching & Changing Unexpected Behaviors to Expected Behaviors.

Modifying Vertical Geometry

Fitting Modifying

Modifying a Presentation

Micronesian Educatop Vol. 1, No. 1 - Modifying student's behaviors to encourage oral participation