Recommendation Eval Framework V2!09!22 2011

8/3/2019 Recommendation Eval Framework V2!09!22 2011

1/15

Customer Scenario and Customers.com are registered trademarks and Customer Flight Deck and Quality of Customer Experience (QCE) are service marks of the

Patricia Seybold Group Inc. P.O. Box 783, Needham, MA 02494 USA www.customers.com Unauthorized redistribution of this report is a violation of copyright law.

Patricia Seybold Group / Evaluation Framework

Recommendation Evaluation Framework, Version 2Evaluating Solutions for Personalization and Recommendations

By Susan E. Aldrich, Sr. VP and Sr. Consultant, Patricia Seybold Group September 22, 2011

NETTING IT OUT

Recommendations are hitting their stride. A decade ago merchants and publishers saw therecommendations on Amazons site and wantedit. A decade ago they had to spend afortune to get it. Today, recommendation engines can be had for some thousands of dollarsa month and can be implemented in a few weeks. My guess is that recommendations will

be ubiquitous within the next two to three years. Im always optimistic on these guesses, butsince recommendations are widely available as a service, rollout can be very swift.

SaaS offerings greatly simplify the technology aspect of implementation and virtuallyeliminate up-front costs, making it relatively easy for customers to sign up with a vendor.Nevertheless, choosing a vendor wisely is always better than choosing often. To aid in thatchoice, I offer a set of requirements and evaluation criteria set forth in this evaluationframework. This framework updates and replaces the recommendation evaluationframework published in January, 2010. I will be using the framework to evaluate a numberof the leading products using the framework during 2011-12, leading to a detailedcomparison.

WHATS INTERESTING ABOUT RECOMMENDATIONS

What Are Targeting and Personalization

Targeted marketing selects content (including products and offers) for consumers based on traits

such as consumer context or behavior. The consumer is likely anonymous. Personalization requires a

consumer profile that includes traits such as demographics, purchase history, and behavior, and this

profile is used by the algorithms that select the content to be presented. While registering and log-

ging in increases the customer profile data, the consumer is most often identified by web browser

cookies and an associated identification number. Without registration, it is likely the profile does not

contain user information such as name and address, so consumer anonymity is retained.

What Is Recommendation Technology

Web site owners worldwide have yearned for recommendations ever since Amazon started tell-

ing us that people who bought this also bought that and today tell us 52 percent of people who

looked at this bought that; 26 percent bought this other thing. A decade ago marketers had to spend

a great deal to implement recommendations on their sites. Today, recommendation engines can be

had for some hundreds of dollars a month and can be implemented in a few days. Dont you just love

SaaS?

Direct link: http://dx.doi.org/10.1571/fw09-22-11cc
http://www.customers.com/http://dx.doi.org/10.1571/fw09-22-11cchttp://dx.doi.org/10.1571/fw09-22-11cchttp://www.customers.com/http://dx.doi.org/10.1571/fw09-22-11cc


2/15

2 Evaluation Framework

Customers.com Research Service Unauthorized redistribution of this report is a violation of copyright law.

2011 Patricia Seybold Group For Reprints/Redistribution rights, contact: [email protected]

For the user, recommendations on a site can mean pretty darn good search results, much of the

time; emails with interesting offers; banner ads that catch your eye; useful guidance when you are se-

lecting content, whether its a digital camera, a news article, a problem resolution, or a research

document; clicks that take you from Google right to the content you need.

For the content ownerwhether merchant selling products, publisher presenting articles, or

marketer presenting offersrecommendation technology means delivering the most attractive itemin those few seconds before you lose your audiences attention.

A recommendation engine can double or quadruple the click-through rate as compared with the

recommendations selected by the expertsthe merchandisers or researchers or support specialists. It

typically has significant impact on revenue, time on site, employee productivity, and customer satis-

faction.

Where Are Recommendations Used

Because recommendations had their most visible debut in ecommerce, people tend to associate

recommendations with shopping. In the ecommerce arena, recommendations are used most often on

product pages, shopping cart pages, category pages, and order confirmation pages. Recommenda-

tions are also used to tailor the content of those emails that entice you to stop work for a moment andshop. As a marketing team becomes more familiar and more confident with using recommendations,

they expand recommendations to cover more of the interactions across the customer lifecycle.

But I think product recommendations are the tip of the iceberg for recommendations. They be-

long everywhere content must be winnowed for a user, or everywhere that personalization improves

a users productivity or experience. For example:

A news site that knows I love football and dont care about rugby, and always shows methe most interesting world news

A personalized view of the corporate intranet that highlights my department, my division,and my projects

My view of corporate research knowledge bases, weighted to what Im working on

Web sites that deliver coaching, e.g., for runners, dieters, investors, tailored to the mystyle and goals

My personalized support portal to the corporate help desk

My companys portal to a key supplier, e.g., Cisco, personalized by role or person

My dashboard with my KPIs and corporate reports, with the hottest items on the firstpage

A personalized investment site, e.g., Fidelity, tailored to the kind of investor I am

A personalized commerce site tailored to my relationship, e.g., the car I drive, the sports Iplay

Billing inquiry that always shows the disputed bills first

Order history inquiry that always shows the most referenced order first
mailto:[email protected]:[email protected]:[email protected]


3/15

Evaluation Framework for Personalization/Recommendations Solutions 3

Unauthorized redistribution of this report is a violation of copyright law. Customers.com Research Service

For Reprints/Redistribution rights, contact: [email protected] 2011 Patricia Seybold Group

My guess is that recommendations will be ubiquitous within the next two to three years. Im al-

ways optimistic on these guesses, but since recommendations are widely available as a service, roll-

out can be very swift.

The technology effort, and the up-front costs, for deploying recommendations are relatively

small, which makes it reasonably easy for customers to sign up for a recommendation service. The

greatest effort for recommendation customers seems to be in learning how to use them, where to usethem, and how to use them effectivelyskills which will be portable to any vendors solution. Nev-

ertheless, choosing a vendor wisely is always better than choosing often. To aid in that choice, I have

compiled a set of requirements and evaluation criteria set forth in this evaluation framework. And I

will be evaluating a number of the leading products using the framework during 2011 and 2012,

leading to a detailed comparison.

REQUIREMENTS

The requirements for recommendation services are derived from customer requirements. Cus-

tomers for recommendation services cover several roles, including the end-consumers of recom-

mendations, the business people managing recommendations, and technical staff. Their requirements

generate the evaluation criteria which are listed in the Table.

Recommendation Consumer Requirements

People consuming recommendationsthe visitors, shoppers, readers, researchers to whom con-

tent is being recommendedof course, need relevant and enticing recommendations, thats the

whole point. But they also need privacy and may wish they had some control over what personal in-

formation is used and how it is used. As recommendations are increasingly used to personalize ex-

periences, consumers may also want a mechanism to indicate the persona they represent. Dont

recommend for me, recommend for my [boss, niece, colleague].

Recommendation Manager Requirements

The marketers, merchandisers, editors, business analysts and other business people who are us-

ing recommendations as a tool to improve user experience have a broad range of requirements, start-

ing from the initial deployment.

The business people who are responsible for recommendations need help making great ones, in-

cluding:

Guidance on how to deploy recommendations effectively

Advice on how to increase recommendation effectiveness

Training and tools to track and analyze recommendation effectiveness

Business people need tools to deploy, test, analyze, and optimize recommendations:

GUI or wizards for specifying rules for how recommendations are selected and presented

Consistent interfaces designed for their business processes, not for the structure of therecommendation product

Granularity in controlling the content selection process, e.g., using customer history toselect sports content but crowd wisdom in selecting fashion


4/15




Aggregation in controlling the content selection process, e.g., global rules that apply tomultiple sites, countries, and pages

Automation of deployment, analysis, and optimization

Testing and reporting that will compare different recommendation deployments

Integration with other marketing tools

Business people need to manage the recommendation service, and therefore need capabilities

that include:

Reporting on recommendation results, e.g., click through, conversion, revenue

Reporting on recommendation engine service level, e.g., response time and availability ofservice

Security of their content and user information; privacy of consumer information

Reliable, high performance

Controlled, role-based access to recommendation management functionality

IT Requirements

IT personnel are potentially involved in deploying and managing recommendation technology,

either on premise or as part of the SaaS implementation. They need:

APIs or Web Services for requesting recommendations from other applications

Ease of adding and testing recommendation data gathering to Web pages

Ease of providing a content data feed to recommendation service provider or enabling acrawl

Ease of creating a zone on Web pages, emails, phones, or other venues, where therecommendations will be displayed

Capability to incorporate management of the recommendation engine or service intoenterprise management

Security for data provided to the recommendation service provider

Integration of recommendation inputs and outputs with other segmentation, behavioralanalytics, Web statistics, and data modeling applications

Importance, Visibility, and Differentiation

As I prepare my requirements and evaluation criteria, I confess to taking certain shortcuts. If all

products offer a capability, I will drop it from my criteria regardless of its importance in order to

simplify the evaluation process. For example, it is unarguably critical that you be able to start and


5/15




stop the recommendation service. All products offer this capability, so its not a feature cluttering up

my evaluation matrix.

More problematical are the capabilities that are very important but cant be observed or meas-

ured. At the very top of this list is, for a specific web site, do Vendor As algorithms deliver better

recommendations than Vendor Bs? Algorithm quality will be very important to the recommendation

manager, but perhaps not in the first months or even years of deployment, when she is developingher recommendation expertise. During that time, vendor expertise and guidance are far more critical.


6/15




2011 Patricia Seybold Group Inc.

Recommendation Platform Requirements

Category Recommendation Platform Evaluation Criteria

Solution Positioning What solutions are offered by the vendor?

o If the vendor has multiple product l ines, list the major categories of products.

o Briefly describe the solutions offered in the personalization, recommendations,

and targeted marketing arena. Include the unique value proposition for each

solution.

Guidance and Advice What are the vendors target markets, by industry and company size?

o What is the vendors distribution of clients across industries: retail, B2B

ecommerce, media, travel, online services? See Table B.

o What is the vendors distribution of clients across retail categories: apparel +

accessories, books/film/music, computers/electronics, flowers/gifts/jewelry,

food/drug, hardware/home improvement, housewares/home furnishings, mass

merchants/wholesale clubs/department stores, specialty/non-apparel, sporting

goods? See Table B.

o What geographies are supported? What languages do the vendors customers

work in? In what languages are they deploying recommendations? What is the

distribution of the vendors clients, by continent: North America, South America,

Europe, Asia, Africa? See Table C.

o What percentage of clients are multi-site? Multi-country? Multi-currency?

o Does the vendor leverage its client base by offering collaborative or syndicated

recommendations? Syndicated recommendations include product catalogs of

other merchants in a set of recommendations; collaborative recommendationsrecommend products at other retailers that match the customers interest, based

on cross-matching activities on both sites.

How successful are this vendors clients?

o For ecommerce clients, what percent of lift (e.g., revenue increase for

ecommerce sites, ad impressions for media sites) is provided by direct action on

recommendations (shopper clicks on a recommendation and immediately buys

the item), and what percent by delayed action on recommendations (shopper

buys an item previously recommended)?

o What is the average annual growth rate in number of recommendations

consumed per client?

o What is the retention rate of clients?

o What is the average number of recommendations touchpoints deployed per

client?

o What is the distribution of usage of key solution capabilities per client: e.g., web

recommendations, email recommendations, mobile recommendations, ads,

APIs? See Table D.


7/15





Recommendation Platform Requirements (continued)


Guidance and Advice

(continued)

What types of coaching and training are provided and at what points in the

relationship? E.g., Implementation Classes for IT, 1:1 coaching for merchandisers

new to recommendations, self-training for advanced merchandisers, user

conferences, videos, blogs. What services are provided on site?

What type of account management and relationship management services are

provided (business guidance, best practices, integration support)? What is the

frequency of business, planning or strategy review? What is the experience level of

the people performing the reviews?

Recommendation

Structure

What are the sources of content that can be presented in recommendations? For

example, catalog/databases, content management repositories, CRM systems.

What types of recommendations are supported? For example, people who

bought/viewed this bought that; most popular; your-friends-liked; a dynamic

bundle with a dynamic price; questions that collect and analyze answers; search

results, etc. See Table E.

If algorithms can be customized, what factors can be included, and who performs

the customization?

If behavioral recommendations are supported, what behaviors are observed and

used? E.g., incoming site, search terms, views, time spent viewing, navigation

selections, cart adds, cart abandons. What behavior-based recommendation types

are provided?

Personalization: If customer profiles are supported, what data is collected? How are

customers recognized across sessions and touchpoints? What customer profile-based recommendations are provided?

How does the solution use social media? For Facebook and for LinkedIn, what data

is used, and what algorithms (strategies) are enabled? For ratings and reviews,

what data is used, from what sources, for what algorithms?

Describe the scientific principles, algorithms and data models used for each

recommendation pattern. What associations do the vendors algorithms support?

E.g., item-item; item(s)-item(s), person-item, person-person, other.

What recommendation types are supported for each touchpoint: web, email, kiosk,

mobile, business application, iOs app? When are the recommendations generated

for each touchpoint: batch, at send time (for email), at open time? See Table E.

Can the recommendation systems insights/analysis be used by other systems,

such as search, customer segmentation, data warehouse? Describe.


8/15







Managing

Recommendations

Testing

Describe the approach to testing recommendations: Is it A/B or multivariate; what is

the limit for simultaneous recommendation tests?

How does the solution measure success of the recommendations? Can it measure

based on client-specified KPIs?

In what ways can the solution automatically optimize recommendation approaches

to reach client-specified business goals?

Interface for managing recommendations:

Describe the interface. How is it organized? What is the set of symbols, tabs, and

other interface elements that is consistent (in appearance and meaning) from task

to task? Does it include a preview of changes being made, and provide a workflowfor managing review and approval of changes? Is access controlled based on

roles? What controls are in place that enable groups of people to manage

recommendations without getting in each others way?

What tasks can be handled by business people via the interface without calling on

technical staff for help? Or which tasks require technical (IT) staff?

Describe the principles by which business people can control recommendations.

For example, do they create (or use) templates, what does the template control.

How can rules or filters be applied, and to what objects (e.g., pages, templates,

segments, time periods)? What data can be specified in rules and filters, e.g.,

inventory, margin, segment, advertiser, combinations of factors?

What support is provided for managing multi-site, multi-country, and multi-currency

clients? For example, global and local rules; copy and edit templates; copy and edit

configuration.

Reporting:

Does reporting and analysis provide suggestions on how to improve

recommendations to achieve KPIs? Does the vendors staff?

How does the reporting include targets, make forecasts, provide alerts, and analyze

contribution to KPIs?

What is the most recent period that can be viewed in reports? I.e., how near real

time? What is the mechanism for scheduling automatic report creation and distribution?


9/15







Integration and

Ecosystem

How does the solution fit into the marketing ecosystem? What information is made

available to other marketing tools? What information from other marketing tools is

used by the recommendation solution? How would recommendations contribute to

a marketing campaign? How would recommendations that are part of campaign be

monitored and managed? How would the results of a campaign be reflected in

recommendations?

How does the recommendation solution fit into the web site ecosystem, e.g., site

search, content management, SEO, SEM, advertising, and other applications?

Describe what data is required by the recommendation engine, and how it is

collected, including the format of feeds, language of collectors, and testing process.

Does the solution have auto-discovery capability for content and items?

Describe the mechanisms for ensuring data privacy and security, for customer anduser data.

Describe APIs provided. What languages/methods are provided? What services are

available via API?

Describe partner programs. What support is provided partners who seek to provide

more services to clients? What is the metric for partner value to the ecosystem?

Describe the typical implementation cycle, including activities, milestones,

timeframes, and project management tools. What role/skill typically manages the

project (for vendor and for client)?

Operations For on-premise software managed by the customer: What is the mechanism for availability and scalability? If multiple machines are

involved, what is the mechanism for managing multiple machines, both day-to-day

and during software upgrades?

Are SNMP alerts provided for integration with enterprise management?

What are the scalability limits, in terms of content and users?

For Hosted Software as a Service (SaaS):

Describe your SLA, including guarantees, penalties, and limits; include a summary

of SLA performance for past 12 months.

What is the typical response time for the recommendation service? How many recommendations were served in the past three months? Please define

what you mean by recommendation: is it a query, or the number of items returned,

or something else?

How many data centers support the recommendation engine? Where are they

located?


10/15







Table A. The evaluation criteria for recommendation engines or platforms are based on customer requirements.We will use the criteria to analyze leading recommendation solutions.

Vendors Development

and Maintenance

Do you package named releases? If so, describe your release frequency for new

functionality and for bug fixes.

What methodology or standards are followed in support, bug correction, and

testing?

Describe the escalation path for performance or quality issues.

Company and Product

Viability

Product background and release history: month/year of major releases, beginning

with the initial release

Product plans: frequency of major/minor releases; enhancements planned for the

next three months

Partner and OEM strategy for recommendation solution

Number of clients and sites

Pricing for recommendation solutions:

o What is the basis for pricing? If recommendation success is a factor, please

describe how recommendation success is defined.

o What is the average low end and high end?

Company history: date founded, founders, investors, # employees

Financial performance as available


11/15





Table B. Each vendor targets specific industries with its marketing, sales, technology, and customer care.

Industry Focus

Industry Sub category Percent ofclients

Total for all retail

Apparel and accessories

Books/film/music

Computers/electronics

Flowers/gifts/jewelry

Hardware/home improvement

Housewares/home furnishings

Mass merchants/wholesale clubs/department stores

Specialty/non-apparel

Sporting goods

Retail

Other

B2B

Media

Travel

Telecom

Online Services

Advertising


12/15




Geographic Focus

Percent of

Clients

List of Clients Countries Languages Supported for Console

North

America

South

America

Europe

Asia + Pacific

Africa


Table C. Each solution is deployed on multiple continents and on Web sites in many languages. The consoleprovides the interfaces that marketers and merchandisers use to manage and optimize recommendations.

Use of Services

Services provided by this vendor (add or remove services as

appropriate)

Percent of clients using the service

Web recommendations

Email recommendations

Mobile recommendations

Kiosk recommendations

Display Ads

APIs


Table D. Successful clients expand their use of vendors services. This table reflects the level of adoption of keyservices.


13/15

Recommendation Evaluation Framework, Version 2 13



Recommendation Types and Structures

Recommendation Structures Supported

Item-item(s) associations X

Many items to item(s)

Person-person associations

Item(s)-Person(s) associations

Recommendation Types (list here)

Behavior-based

Profile-based

Social-Media

Web Email Mobile Kiosk Business

Appl.

Generated as batch; at send; or at open

Rules driven (e.g., top

sellers, lists such as

staff picks)

X Open Send Batch Open

Rules modified (e.g.,

white and black list)

X

People who viewed

this, viewed that

People like you whoviewed this, viewed

that

People who viewed

this, bought that

People like you who

viewed this, bought

that

People who bought

this, bought that

People like you who

bought this, bought

that



14/15




Recommendation Types and Structures (continued)

Recommendation Types (list here)

Behavior-

based

Profile-

based

Social-

Media

Web Email Mobile Kiosk Business

Appl.

Generated as batch; at send; or at open

Search results

selected and ranked

Landing page items

selected and ranked

..etc

Syndicatedrecommendations

Collaborative

recommendations


Table E. Each solution supports a variety of recommendation types. This is due in part to the recommendationstructures that are supported. Each recommendation type may be available to multiple touchpoints. Sometouchpoints allow for the most dynamic recommendations, that is, recommendations that are selected at themoment the page is accessed (such as opening an email). Other touchpoints call for the least dynamicrecommendations, produced in a batch. A kiosk in offline mode would require batch recommendations.

CONCLUSION

This evaluation framework will be applied to the leading recommendation solutions during the

coming months. After we have evaluated a number of products, we will compare the leaders and

analyze which solutions best suit which applications and markets. You are welcome to use our

framework for your own evaluation. We welcome your comments.


15/15

About Susan E. Aldrich and Patricia Seybold Group

Customers.com

Research Service Unauthorized redistribution of this report is a violation of copyright law.

2011 P t i i S b ld G F R i t /R di t ib ti i ht t t l @ t

ABOUT THE AUTHOR

SUSAN E. ALDRICH is a Senior Vice President and Senior Consultant at thePatricia Seybold Group.

Aldrich is a senior analyst for the firms Advisory Service. As leading authorityon worldwide technologies, custom er requirements, practices, and

governance for finda bility, she manages the Sear ch, Navigation, and

Discovery Research Practice. Her research foc uses on customer self-service,information management an d technologies and practi ces for mo nitoring,

measuring, and managing the Quality of Customer ExperienceSM (QCE).

Aldrichs experience includes commercial a pplications development, deployment, and

implementation and operating systems deve lopment. She has provided informationmanagement, customer relationship, and distributed systems management consulting

worldwide.

Patricia Seybold GroupTrusted Advisors to Customer-Centric Executives

If you're a visionary customer-focused executive, thePatricia Seybold Group should be your firstchoice for ongoing strategic advice, business and technology guidance, customer experience best

practices, and help with customer-centric initiatives.

Founded in 1978 a nd based in Boston, we provide consulting, research and advisory services,

peer groups, and in teractive workshops. We h elp clients to design an d continuously improve

their customer-focused business strategies and processes using our pro ven consultingmethodology,Customer Scenario Design.

The CEO and founder, Patricia Seybold, is t he New York Times best-selling author of

Customers.com and The Customer Revolution. Patty's latest book, Outside Innovation, isavailable now.

Patricia Seybold Group

P.O. Box 783Needham, MA 02494

Phone: (617) 742-5200

Fax: (617) 742-1028

Email: [email protected]: http://www.customers.com
mailto:[email protected]:[email protected]://www.customers.com/http://www.customers.com/http://www.customers.com/http://www.psgroup.com/consulting_csmcert.aspxhttp://www.psgroup.com/consulting_csmcert.aspxhttp://www.psgroup.com/consulting_csmcert.aspxhttp://www.psgroup.com/consulting_csmcert.aspxhttp://www.psgroup.com/consulting_csmcert.aspxhttp://www.psgroup.com/consulting_csmcert.aspxhttp://www.psgroup.com/books_oi.aspxhttp://www.psgroup.com/books_oi.aspxmailto:[email protected]:[email protected]://www.customers.com/http://www.customers.com/http://www.psgroup.com/signup.aspxhttp://www.psgroup.com/books_oi.aspxhttp://www.psgroup.com/consulting_csmcert.aspxhttp://www.customers.com/mailto:[email protected]://www.customers.com/mailto:[email protected]

Recommendation Eval Framework V2!09!22 2011

Documents

Transcript of Recommendation Eval Framework V2!09!22 2011