Optimizely Workshop 1: Prioritize your roadmap

53
Prioritize Your Roadmap Strategy Consultant: Ryan Lillis August 12th, 2015 An Optimizely Online Workshop

Transcript of Optimizely Workshop 1: Prioritize your roadmap

Prioritize Your Roadmap

Strategy Consultant: Ryan Lillis

August 12th, 2015

An Optimizely Online Workshop

Introduction

Ryan Lillis Strategy Consultant

[email protected]

Today’s Takeaways

Today’s Takeaways

#1 Why Prioritize & Key Components

#2 Tools to Prioritize & Effective Usage

Today’s Tools

Today’s Tools

#1 Your List of Test

Ideas

Today’s Tools

#1 Your List of Test

Ideas

#2 A Blank

Spreadsheet

1. Prioritization Components 2. Components in Action 3. Tools for Organization & Prioritization

AgendaKey Focus Areas

Three Questions

#1 - Poll: Do you have a list of test ideas?

#2 - Poll: How many tests are in your list?

#3 - Poll: Do you have a prioritization framework and

is it followed?

Prioritization Components

Effort Impact

Effort

Activity: Write down what takes effort to set up a test.

Activity: Write down what takes effort to set up a test.

Effort

Teams (or People)•Power user(s) •Designer(s) •Developer(s) •Graphic designer(s) •Photography team •SEO and/or SEM team •IT department •QA department •Executive sponsor

•HTML •CSS •Javascript •Backend changes (reordering pages in a checkout flow)

•Advanced audience targeting conditions (cookies, ad campaigns, custom tags etc.)

Technical

Impact

Poll: What is the primary way a test’s impact is measured?

Impact

•Higher Conversion Rate • Purchases • Sign ups • Engagement

•More of Something • Revenue • Pageviews

•Less of Something • Internal cost reductions

Hard Impact - QuantifiablePoll: What is the primary way a test’s impact is measured?

Impact

Soft Impact - Meaningful•Internal Buy-in

• Executive sponsorship • Excitement (test ideas from more people)

• Resource allocation (dedicated developer)

• Access (places testing is allowed to happen)

•Implementation (winners actually implemented)

Hard Impact - Quantifiable•Higher Conversion Rate

• Purchases • Sign ups • Engagement

•More of Something • Revenue • Pageviews

•Less of Something • Internal cost reductions

Activity: Write down the primary ways your tests impacts will be measured?

Components In Action

Basic Framework Effort vs. Impact

Effort vs. Impact

Test Name Effort Impact

Test 1 Low High

Test 2 Low Medium

Test 3 Low Low

Test 4 Medium High

Test 5 Medium Medium

Test 6 Medium Low

Test 7 High High

Test 8 High Medium

Test 9 High Low

Test Name Effort Impact

Test 1 Low High

Test 2 Medium High

Test 3 High High

Test 4 Low Medium

Test 5 Medium Medium

Test 6 High Medium

Test 7 Low Low

Test 8 Medium Low

Test 9 High Low

Priority

First Second Third

Test Name Effort Impact

Test 1 Low High

Test 2 Low Medium

Test 3 Low Low

Test 4 Medium High

Test 5 Medium Medium

Test 6 Medium Low

Test 7 High High

Test 8 High Medium

Test 9 High Low

Test Name Effort Impact

Test 1 Low High

Test 2 Medium High

Test 3 High High

Test 4 Low Medium

Test 5 Medium Medium

Test 6 High Medium

Test 7 Low Low

Test 8 Medium Low

Test 9 High Low

Prioritized by EffortPrioritized by Impact

Effort vs. ImpactPriority

First Second Third

Chat: What could be done to this framework to make it

more useful?

Use Numbers Instead of Words & Sum the Scores

Quantify Effort & Impact

Test Name Effort Impact

Test 1 Low High

Test 2 Medium High

Test 3 High High

Test 4 Low Medium

Test 5 Medium Medium

Test 6 High Medium

Test 7 Low Low

Test 8 Medium Low

Test 9 High Low

This impact framework…

Use Numbers Instead of Words & Sum the Scores

Quantify Effort & Impact

…becomes this mixed framework.Priority Test Name Effort Impact Score

1 Test 1 3 3 6

2 Test 2 2 3 5

3 Test 4 3 2 5

4 Test 3 1 3 4

5 Test 5 2 2 4

6 Test 7 3 1 4

7 Test 6 1 2 3

8 Test 8 2 1 3

9 Test 9 1 1 2

Test Name Effort Impact

Test 1 Low High

Test 2 Medium High

Test 3 High High

Test 4 Low Medium

Test 5 Medium Medium

Test 6 High Medium

Test 7 Low Low

Test 8 Medium Low

Test 9 High Low

This impact framework…

Poll: Who’s satisfied with this framework?

Consider Your Key Components & Create a Rubric

Objectively Quantify Effort & Impact

Create a Classic, Detailed Rubric

Rubric Method #1

Create a Classic, Detailed Rubric

Rubric Method #1

Effort Component 3 2 1

HTML No HTML requiredHTML required but can be done within Optimizely’s visual editor

Detailed, custom HTML needs to be created

Javascript / jQuery No Javascript requiredSimple manipulation of jQuery in Optimizely’s “Edit Code” section

Detailed, custom Javascript or jQuery needs to be created

Graphic Design No graphic design required

Quick graphic design element needed

Detailed graphic design work needed

Executive Sponsor Executive sponsor not needed [intentionally blank] Executive sponsor

approval needed

Example Effort Rubric

=

Create a Classic, Detailed Rubric

Rubric Method #1

Impact Component 3 2 1

Conversion RateTest has a high likelihood of directly impacting our conversion rate.

Test may or may not directly impact our conversion rate but should impact a leading indicator metric.

This test will likely only impact a leading indicator metric.

ExcitementThis test will generate a lot of internal excitement for testing and build momentum.

This test will interest and satisfy some team members.

This test will only matter to a few people and get little visibility regardless of the outcome.

ImplementationIf this test has a winning variation it will definitely be implemented immediately.

If this test has a winning variation it will probably be implemented or it will take some time before it is.

Even if this test produces a winning variation, there is only a small chance it will be implemented due to buy-in or technical constraints.

Example Impact Rubric

=

Example Spreadsheet

Let’s Look at an Example of What a Prioritized Spreadsheet May Look

Like

Tools for Organization & Prioritization

Three Tool Categories

Idea Prioritization Workflow ManagementIdea Collection

Three Tool Categories

Idea Prioritization Workflow ManagementIdea Collection

Poll: How can test ideas be collected in a way that

incorporates rubric scoring…automatically?

Poll: How can test ideas be collected in a way that incorporates rubric scoring…automatically?

#1 Impossible! It simply cannot

be done!

Poll: How can test ideas be collected in a way that incorporates rubric scoring…automatically?

#1 Impossible! It simply cannot

be done!

#2 We’ll create an

exploratory committee to discover this

method. We should have an answer when Tesla has a flying car.

Poll: How can test ideas be collected in a way that incorporates rubric scoring…automatically?

#1 Impossible! It simply cannot

be done!

#2 We’ll create an

exploratory committee to discover this

method. We should have an answer when Tesla has a flying car.

#3 Create a test idea submission form that utilizes questions asking what

resources, skill-sets etc. are required for a given idea. The

responses to this form populate a spreadsheet with built in formulas that add or subtract points based on the responses. Therefore as soon as an idea is submitted, a

score is generated and the list can be sorted by scores.

Poll: How can test ideas be collected in a way that incorporates rubric scoring…automatically?

#1 Impossible! It simply cannot

be done!

#2 We’ll create an

exploratory committee to discover this

method. We should have an answer when Tesla has a flying car.

#3 Create a test idea submission form that utilizes questions asking what

resources, skill-sets etc. are required for a given idea. The

responses to this form populate a spreadsheet with built in formulas that add or subtract points based on the responses. Therefore as soon as an idea is submitted, a

score is generated and the list can be sorted by scores.

So the answer is #3. Now what?

1. Create a test idea submission form

2. Include questions about required resources & skill-sets etc. required.

3. Modify the spreadsheet with built in formulas that add or subtract points based on the responses.

4. Sort the spreadsheet based on the final prioritization score.

+

Auto-Prioritization Recipe

= +

Scoring: Common Spreadsheet Formulas

Forms: Google Forms Responses: Google Sheets

Scoring: Common Spreadsheet Formulas

Quick Chat: Name one* tool you can use to help with

prioritization?

Quick Chat: Name one* tool you can use to help with prioritization?

Forms: Google Forms Responses: Google Sheets

* Google Forms uses Google Sheets automatically.

Three Tool Categories

Idea Prioritization Workflow ManagementIdea Collection

Idea Prioritization Tools

Google Sheets

Automatically integrates with Google Forms, always

shareable in the cloud

Microsoft Excel

Tried and true with advanced functionality for

graphing etc., can be cloud-based

Smartsheet

Set up for seamless project management and timelines,

cloud-based

Atlassian’s JIRA

Ticketing system incorporating elements of

form submission tool and a sortable spreadsheet

Three Tool Categories

Idea Prioritization Workflow ManagementIdea Collection

Poll: Why would a “workflow management” tool help with

prioritization?

Poll: Why would a “workflow management” tool help with prioritization?

•Choice #1 - A test can’t be the first priority if it needs the expertise of someone who’s on vacation.

•Choice #2 - A test can’t be the last priority if it’s part of a time-sensitive marketing campaign.

•Choice #3 - Any test idea submitted by my boss is always the top priority.

•Choice #4 - Both #1 and #2 are correct but definitely not #3.

Workflow & Project Management Tools

Smartsheet

Set up for seamless project management and timelines,

cloud-based

Atlassian’s JIRA

Ticketing system incorporating elements of

form submission tool and a sortable spreadsheet, also

can utilize “cards” that track the status of each

step

Trello JIRA

One of the most popular testing workflow

management tools, incorporates “cards”, to-do lists and is one of the more

visual systems out there

Recap

1. Prioritization Components 2. Components in Action 3. Tools for Organization & Prioritization

Original Agenda

Additional ResourcesOptiverse contains content on a variety of topics in a

variety of formats including our online user community, knowledge base, and learning academy!

Community Knowledge Base Learning Academy

Thank You For Attending!