Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM...

89
Laurel Poertner Head of Training Intermediate Analytics

Transcript of Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM...

Page 1: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Laurel Poertner

Head of Training

Intermediate Analytics

Page 2: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Introductions

Laurel Poertner

Over 15 years experience Managing technical support and education teams

Delivered technical and business process training

Supported, used and managed Knowledge Management tool, Knova

Built virtual classroom offerings for an Enterprise Resource Planning software package

Implemented several CRM and KM systems internally and as a Professional Services consultant

Certified Trainer of Knowledge Centered Service v6SM Practices

Page 3: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

1. Review why Analytics

2. Basic Analytics - Key concepts

3. Set up your Lab Environment

4. Understand - Dashboards

5. Understand – Solution Adoption

6. Act – Content Gap and Query Performance

7. Act – Common Boosting Scenarios

8. Understand – Workflow and Self-Service

9. Automate - Machine Learning

Plan

Page 4: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Goal: Gain practical experience using all the Usage Analytics Tools to help

▶ Understand your users to gain additional insight into search behaviors and

patterns

▶ Analyze Analytics data using reports and the Visit Browser to identify areas

to optimize the search experience

▶ Understand the importance of Relevance and how to improve it using

Query Pipeline rules

▶ Knowing the most appropriate action to take to improve the search

experience based on performance and utilization metrics

▶ Knowing when and how to set up Machine Learning and manual search

tuning to increase relevancy

About This Course

Page 5: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

A Review

5 © 2017 Coveo Solutions Inc - Proprietary and Confidential

Why Analytics

Page 6: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Change is constant

And so is the need for Relevance

Providing Relevance is a Journey in which you continuously

Understand the Behavior of your users, gain insight

Act on this insight, to provide to improve Relevance

Leverage Machine Learning to Automate the Relevance Improvement

Relevance is a Journey, not a destination

Page 7: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

The Triumvirate of Analytics

Understand Act

Automate

What your user are searching for

What they are clicking on

Are they finding what they are looking for

What is relevant to them

Optimize the search experience

based on what your user are

doing

Tune the experience to provide

ever-improving relevance

Leverage Machine learning to automate the

Relevance tuning process as you go

Free time to do further research

1 2

3

Page 8: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Co

pyr

igh

t ©

20

16 C

ove

o S

olu

tion

s In

c.

All

rig

hts

re

serv

ed

. Pro

pri

eta

ry a

nd

CO

NF

IDE

NT

IAL

The Tools of Analytics

User Interface

Query Pipelines

Machine LearningUsage Analytics

Connectors

Salesforce Sitecore Web Exchange

Index

Source

Items

Source

Items

Source

Items

Source

Items

Understand 1

Act 2

Automate3

Page 9: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Coveo Relevance Maturity Model

manual/hard automated/easy

content intelligence

Positive

business

impact &

ROI

Negative

economic

ImpactBasic search captures user’s intent, and saves time retrieving information.

Entry-level search is resource intensive, relies heavily on taxonomies for relevance.

Federated search fails at ranking combined result sets relevantly.

Poor relevance affects conversions, clickthroughs, satisfaction, deflection, escalations, productivity, innovation, work redundancy,

...

Net net, entry level search [cheaper] is the most expensive strategy.

Siloed

search

Federated

searches

0Efficiency gains Proficiency gains

LEADERSLAGGARDS

6 Discovers user’s likely intent,

by analyzing behavioral data.

Machine learning auto-tunes recommendations and

ranking to maximize business outcome, enabling true

one-to-one user engagement and upskilling.

Self learning predictive

recommendations

PREDICTIVE[relevance is predictable]

5

…related content, experts,

products or services pushed to

expand user’s knowledge and

abilities to do more, buy more,

learn more, engage more, etc.

Contextual

suggestions

3

4

…adapt relevance through

weighting of ranking factors,

query ranking expressions,

based on known content

characteristics.

…in-product, in-task, and other

contextual signals factor the

uniqueness of the user and

query context and ranks results

with higher relevance.

Tunable

relevance

Contextual

relevance

PROACTIVE[relevance is contextual]

1

2

Unifies and ranks

information from

multiple sources.

Configurable rich facets, search

tabs, folding and security

trimming provide the very first

step of personalization for users.

Content

navigationSecured

unified

ranking

RESPONSIVE[relevance is personal]

CRMM™

stages

Page 10: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Reactive

Low search success

Visits w/ no results

Searches w/ no results

Low click-through/rank

Responsive

Visits w/ searches

Searches w/ clicks

Click-through increases

Avg click rank decreases

Proactive

>50% customers visit

Self-service portal

Cases submitted

decreases

Predictive

>85% customers visit

first

Click-through >50%

Avg Click rank <3

10

Self-Service maturity model

Page 11: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Search Clicks Search Success Relevance

11

Top Queries

Top Keywords

Visit Click-Through %

Top searches w/clicks

Top documents

clicked

Search Click-Through

%

Visits w/Searches and

a click

Visits w/results

Click-through %

Avg Click rank

Queries w/ high

relevance

Top documents by

average click rank

Identify what is successful for your users (ie. Did they find what they were looking for?)

Understand

Page 12: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Search Clicks Search Success Relevance

12

Top Queries w/o

results

# Queries w/o results

Queries w/o clicks

Lowest clicked

documents

Top keyword for case

creation subject

Top documents

clicked with case

creation

Queries w/ low

relevance scores

Lowest average click

rank queries

Identify what is NOT successful for your users

Understand

Page 13: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Key Concepts

13 © 2017 Coveo Solutions Inc - Proprietary and Confidential

Basic Analytics

Page 14: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

1. Review why Analytics

2. Basic Analytics - Key concepts

3. Set up your Lab Environment

4. Understand - Dashboards

5. Understand – Solution Adoption

6. Act – Content Gap and Query Performance

7. Act – Common Boosting Scenarios

8. Understand – Workflow and Self-Service

9. Automate - Machine Learning

Plan

Page 15: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Review the important key concepts of basic analytics to delve further into more technical

areas.

Learning Objectives

1. Understand the big picture of where Usage Analytics data fits in with the Coveo

Organization

2. Review the concepts of Visits, Events, Dimensions and Filters

3. Understand the different dashboard templates and metrics

In This Module

Page 16: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Where are We

User Interface

Query Pipelines

Machine LearningUsage Analytics

Connectors

Salesforce Sitecore Web Exchange

Index

Source

Items

Source

Items

Source

Items

Source

Items

6Analyzing

behavioral data

Page 17: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Big Pictures

[Facet Component]

Usage Analytics

[Recommendations Component]

[Omnibox Component]query suggestionssuggestion engine

Index

[Search Result List]

Query Pipeline

Coveo Search Interface

Coveo Cloud Platform

queyr suggestions

Search Result(Coveo items)

Query1

2

Usage AnalyticsDatabase

Event

Coveo Analytics Admin

Explorer Dashboard

Dimensions

Metrics

Dimensions

Metrics

Send a Query

Type a Query

Click on a Query Suggest

Click on a Facet

Click on a Recommendation

Click on a Search Result

Page 18: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Key Concepts: Events, Dimensions and Metrics

1 An Event is composed of Dimensions, each populated with their unique Value

2 3Something happens on the page and triggersan Analytic Event

Coveo Platform compiles Metricsabout each Dimensions and their Values (Numerical Value)

Dimensions Value

User Name Laurel Poertner

User Query Adventure

Event Type Search

Date 5:13:53PM 03/21/2017

Event Cause searchboxSubmit

Origin 1 (Page/Hub) Full Learning Site

Origin 2 (Tab/Interface) All

Page 19: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Comprised of Dashboards to quickly monitor the search usage status and

Explorers which are subsets of dashboards to drill down into the usage analytics data

Permission Filters

Filter data in:

▶ Reports

▶ Exports

▶ Visit Browser

Exports

Access from:

▶ Visit Browser

▶ Explore Data/ Explorer

▶ Export menu

Report Access

Define who can access:

▶ Only Me

▶ Analytics Privileges

▶ Custom

Reporting

Page 20: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

App Store Site

Lab Environment

Page 21: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

1. Review why Analytics

2. Basic Analytics - Key concepts

3. Set up your Lab Environment

4. Understand - Dashboards

5. Understand – Solution Adoption

6. Act – Content Gap and Query Performance

7. Act – Common Boosting Scenarios

8. Understand – Workflow and Self-Service

9. Automate - Machine Learning

Plan

Page 22: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Understand the Coveo Learning site and environment to use throughout the training.

Learning Objectives

1. Become familiar with the App Store search page

2. Review Query Pipelines

3. Understand how the App Store context will be used as our learning environment

4. Use the App Store site to create your own test environment and pipeline

In This Module

Page 23: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Site Dimensions

23

https://platform.cloud.coveo.com/pages/coveolearningintermediateanalytics/Demo#t=All&sort=relevancy

Page 24: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Where are We

User Interface

Query Pipelines

Machine LearningUsage Analytics

Connectors

Salesforce Sitecore Web Exchange

Index

Source

Items

Source

Items

Source

Items

Source

Items

3 4Tunable

relevance

Contextual

relevance

Page 25: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Vocabulary Refresher

Integration

▶ System that hosts a Coveo Search Experience

Hub

▶ Name given to a hosted Search Page or Search Panel

Query Pipeline

▶ Conduit by which an interface transmits a Query to the Index for processing

▶ Alternate set of rules that can be defined to modify queries

▶ Take advantage of query pipelines when you have more than one search interfaces / audiences

▶ Distinct users

▶ Purpose

▶ Different rulesAgents

Customers

Partners

Page 26: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Some Vocabulary

Hub – Full Learning Site Hub – My pipeline

default

Pipeline

Page 27: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

What we Want

Customer

Pipeline

Trainee

Pipeline

Hub – Full Learning Site Hub – My pipeline

Page 28: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Exercise – Create your Own Pipeline

28

▶ Add a new Query pipeline (call it your name or something unique and easy)

▶ Open the search page and add your pipeline in the link

▶ Before the # you add this: ?pipeline=MyOwnPipelineIjustCreated

▶ Example:

https://platform.cloud.coveo.com/pages/coveolearningintermediateanalytics/Demo?pip

eline=Laurel#t=All&sort=relevancy

▶ Test and Validate your pipeline

▶ Search an expression

▶ Press Alt + Double Click beside title of first result

Page 29: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

FAQ

Q: Will I affect my production environment if I add a pipeline?

No. Traffic will first look at condition on the pipeline. If no conditions match my traffic, it will

always use the default pipeline.

Q: What happen if I have two pipelines with the same condition?

Traffic will go through the first pipeline from the top of the list that match my traffic. The other one

won’t be use. You can use AB testing to split traffic with different rules.

Page 30: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Dashboards

Understand

Page 31: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

1. Review why Analytics

2. Basic Analytics - Key concepts

3. Set up your Lab Environment

4. Understand - Dashboards

5. Understand – Solution Adoption

6. Act – Content Gap and Query Performance

7. Act – Common Boosting Scenarios

8. Understand – Workflow and Self-Service

9. Automate - Machine Learning

Plan

Page 32: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Get practical experience in creating analytics dashboards and explorers to analyze the data.

Learning Objectives

1. Create dashboards in the following ways

a) From template

b) From blank dashboard and explorer

c) From configuration file

2. Share dashboard with Students group

In This Module

Page 33: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Self-Service

▶ Visits

▶ Case Creation

▶ Documents

▶ Content Gap

Customer Success

Monthly Overview

▶ Hub summary

▶ Performance

▶ QPM

▶ Custom events

▶ Agents

Community Search

▶ Utilization

▶ Performance

▶ Documents

▶ Content gap

▶ Visitors

▶ Facets

▶ Search cause

Agent

▶ Participation

▶ Attach to case

▶ Performance

▶ Content gap

Recommended Dashboards

Page 34: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Create a dashboard from template

Duplicate a report

Create a blank dashboard

34

Report creation

Page 35: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Exercise – Create/Edit Reports

35

Add a new Dashboard from template

▶ Select Summary template

▶ Add name filter: App Store Site

Add a new blank Dashboard

▶ Edit Configuration

▶ Rename Dashboard with your initial or name

▶ Add name filter: App Store Site

▶ Set the permission to custom and add the instructor member

▶ Save your dashboard

Duplicate the Activity Explorer

▶ Add name filter: App Store Site

Page 36: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

FAQ

Q: Where can I find Custom template?

You can find custom template in our knowledge base. Search for : Dashboard template

Q: Can I share my dashboard with others?

You can copy the URL of your dashboard including a specific horizon or filter that is not necessarily

saved and send it to someone else. It will open with the same settings. (The person needs to have

access to the Cloud analytics with permissions)

Page 37: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Solution Adoption

37

Understand

Page 38: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

1. Review why Analytics

2. Basic Analytics - Key concepts

3. Set up your Lab Environment

4. Understand - Dashboards

5. Understand – Solution Adoption

6. Act – Content Gap and Query Performance

7. Act – Common Boosting Scenarios

8. Understand – Workflow and Self-Service

9. Automate - Machine Learning

Plan

Page 39: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Analyze the data in the dashboards and explorers you created to determine what actions

to make.

Learning Objectives

1. Find out if visitors are utilizing the solution as expected

a) Tab performance

b) Facet performance

c) Custom event performance

2. Add metrics to a dashboard to analyze tab and facet performance

In This Module

Page 40: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Event Value

• submitButton• cancelButton• unloadPage

Event Type

• caseCreation• caseAttach/caseDetach• expandToFullUI• preferencesChange

Event Cause (Salesforce)

• searchBoxSubmit• documentOpen• documentQuickView• inputChange• interfaceChange• facetSelect

40 © 2017 Coveo Solutions Inc - Proprietary and Confidential

Custom Event Dimensions

Page 41: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Performance indicators

Solution Adoption

Total users using the solution

components (facets, tab, quick

view, custom objects etc..)

Facet Performance

Which facets is popular or not

used. Which facets gave

content gap or no click

Page 42: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Drive changes to the UI

Tabs

If your tabs performance is

low, review their location,

size, color and name

Facets

If your facets performance

is low, review their

location, color and name

Page 43: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

▶ BEFORE: Few tabs where only

visible and available in a drop list.

Traffic was very low and content

from those tabs hardly used

▶ AFTER: Changing the color and the

display of the tabs brought an

important increase of traffic

▶ IMPACT: More traffic on hidden

content brought more training

registration and more deflection

with the know issues section

Tab PerformanceBEFORE

AFTER

IMPACT

Page 44: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Exercise – Create/Edit Reports

44

▶ Tabs performance in Self-Service TAB

▶ Open Self-Service tab in your dashboard

▶ In solution adoption, create a metrics box to show total unique users using tabs (search cause =

interface change)

▶ Tabs performance in a new TAB

▶ Add a new TAB called: Tabs

▶ Add time series card with total query, click-through and average click-rank

▶ Filter by tab = All

▶ Add a title: All

▶ Resize the card half page

▶ Duplicate the new section and do same exercise for the other tabs

*** Each tab is affecting the overall performance metrics. Looking at each of them gives you an idea of

which one is the most or less performant

Page 45: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

FAQ

Q: Where can I remove or change my facets

Facet are managed by your site editor. It can also be done in the interface editor depending of

your deployment. Training is available as well.

Q: Where can I remove or change my tabs

Facet are managed by your site editor. It can also be done in the interface editor depending of

your deployment. Training is available as well.

Page 46: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Content Gap and Query Performance

46

Act

Page 47: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

1. Review why Analytics

2. Basic Analytics - Key concepts

3. Set up your Lab Environment

4. Understand - Dashboards

5. Understand – Solution Adoption

6. Act – Content Gap and Query Performance

7. Act – Common Boosting Scenarios

8. Understand – Workflow and Self-Service

9. Automate - Machine Learning

Plan

Page 48: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Review Content Gap and Query Performance reports to determine what query pipeline

rules to add to improve relevance.

Learning Objectives

1. Review query pipeline rules

2. Analyze Content Gap and Query Performance reports

3. Evaluate which actions should be taken to solve the query performance issue

a) Enable ML

b) Add Synonyms

c) Add content

d) Boost results

e) Add Top Features

In This Module

Page 49: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

The Tools of Analytics

User Interface

Query Pipelines

Machine LearningUsage Analytics

Connectors

Salesforce Sitecore Web Exchange

Index

Source

Items

Source

Items

Source

Items

Source

Items

Act 2

Page 50: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Rule

Tool that allows you to modify a Query sent from a given Hub or the Ranking of the items that would be returned to the Hub.

Query Pipeline Rules Review

Modifies the Query- Thesaurus- Stop Words

Modifies the Ranking of Results- Machine Learning (ML)- Featured Results- Ranking Expression(QRE)

Modifies the Search Page- Triggers

Page 51: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Co

pyr

igh

t ©

20

16 C

ove

o S

olu

tion

s In

c.

All

rig

hts

re

serv

ed

. Pro

pri

eta

ry a

nd

CO

NF

IDE

NT

IAL

Query Pipeline Rules

Thesaurus Featured Results

Ranking Expressions

Manage equivalent words used to transparently add expressions to the query before it is sent to the index

Promote specific results to appear at the top of the results list when a query matches or contains a specific expression

Influence the document score by adding query ranking expressions (QRE) to boost or lower the score within search results matching an expression

Triggers

Notify

Redirect

Execute

Query

51

Page 52: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Exercise – Query analysis

52

Exercise:

▶ Open Content Gap Tab explorer to analyze content gap

▶ Add synonym / content

▶ Open Query Performance tab in Portal Search dashboard to analyze poor

relevancy query

▶ Use Visit browser for analysis

▶ Add synonyms

Page 53: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

FAQ

Q: I see content gap with long sentence

Use the partial match which will allow you to match fewer words instead of the whole sentence.

Partial match can be setup in your administration tool

Q: I see content gap with special characters (* & ? % \ / ! @ & # @ {} [] )

Use the disable query syntax option which will be soon available from pipeline. It can be setup in

the administration tool

Q: I see a lot of content gap using the same pattern typo

You can use regex to resolve pattern issue

i.e: Query = “KB 123456” expected query = “KB123456” (no space)

Page 54: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Common boosting Scenarios

54

Act

Page 55: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

1. Review why Analytics

2. Basic Analytics - Key concepts

3. Set up your Lab Environment

4. Understand - Dashboards

5. Understand – Solution Adoption

6. Act – Content Gap and Query Performance

7. Act – Common Boosting Scenarios

8. Understand – Workflow and Self-Service

9. Automate - Machine Learning

Plan

Page 56: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Learn scenarios to add boosting rules (Query Ranking Expressions) in your environment.

Learning Objectives

1. Create a Query Ranking Expressions (QRE)

2. Add a Featured Result rule

3. Understand when and how to use conditions with your pipeline rules

4. Follow best practices to add and test new pipeline rules in a production

environment

In This Module

Page 57: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

A condition is a rule that sets predefined requirements on something before it can be executed

Can be used on • Entire pipeline• Specific pipeline rule (thesaurus, ML

etc.)

57 © 2017 Coveo Solutions Inc - Proprietary and Confidential

Conditions

Page 58: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Exercise - Boosting Rules and Tops Features

Scenario

This month, it is the FIFA Word Cup and people from all over the word will watch the games. You

would like to boost a sport channel and some soccer content during that month.

▶ Add a feature: “Fox Soccer 2Go” when query contains soccer in Sport Tabs

▶ Boost the game categories in Game Tabs

58

Page 59: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Exercise - Boosting Rules (QRE)

Add Conditions:

▶ Steps to do in condition menu:

▶ Add condition 1: Tab is Sports OR Tab is “Myname” (cannot have duplicate condition)

▶ Add condition 2: Tab is Games OR Tab is “Myname” (cannot have duplicate condition)

Add Feature Request:

▶ Steps to add Feature requests

▶ Add Feature request using select document option and search for Fox Soccer

▶ Select the app and Save (do not add a condition yet)

▶ Select add a condition in More menu and choose the Sports Tab condition

Add Ranking Expression:

▶ Steps to do in ranking expression rule

▶ Add a QRE using the meta data @sysconcept = fifa or @sysconcept = soccer

▶ Boost to 300 points (little higher than ML boost)

▶ Select add condition in More menu and choose the Games Tab condition

59

Page 60: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Workflow and Self-Service

60

Understand

Page 61: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

1. Review why Analytics

2. Basic Analytics - Key concepts

3. Set up your Lab Environment

4. Understand - Dashboards

5. Understand – Solution Adoption

6. Act – Content Gap and Query Performance

7. Act – Common Boosting Scenarios

8. Understand – Workflow and Self-Service

9. Automate - Machine Learning

Plan

Page 62: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Learn the difference between Self-Service Success and Case Deflection measurements.

Learning Objectives

1. Understand the metrics of Self-Service and Case Deflection

2. Understand the difference between Assumed case deflection and confirmed case

deflection

3. Gain more experience with the visit car metrics and why we use it in case deflection

In This Module

Page 63: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

• Query Click-Through Rate (CTR) –Best in class 50%

• Average Click Rank (ACR) – Best in class <= 3

• Visit Click-Through Rate – Best in class 50%

[Search] Click-Through (CTR): percentage of searches with at least one opened document from the Coveo search results

Average Click Rank (ACR): measures the average position of opened documents in the Coveo search results list.

Visit Click-Through: percentage of visits with at least one search expression and one opened document from the Coveo search results

Benchmarks

Metrics

Page 64: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

• Unique Visitor ID

• Unique Visit ID

Unique VisitID: a random and unique value generated every time a user visits the search site. A visit ends after 30 min of inactivity.All user interactions of the same visit are recorded using the same Visit ID.

Unique Visitor ID: similar to a Visit ID but never expires. Allows you to know the number of distinct users who submitted a query or clicked a document, counting them only once even if they performed the same events in multiple visits.

Visit Count: this is the old metric that can be seen in reports that were created before the above dimensions were created. Remove this and add one of the Unique dimensions to get a more accurate count.

Used to track how many distinct users did a query or click

Visit Dimensions

Page 65: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Self-Service Success Case Deflection

How do we measure success?

The rate that self-service resources eliminate a customer’s need for live assistance

The rate at which customers find the information they need on your self-service portal, which may or may not have required live assistance.

Page 66: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Self-Service

• Visit Click-Through

Case Deflection

• Confirmed Case Deflection• Assumed Case Deflection

Traffic Workflow

• Portal traffic• Case Deflection traffic

66 © 2017 Coveo Solutions Inc - Proprietary and Confidential

Case Creation

Page 67: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Highlights

• Visit Click-through is 70.8% which is

well over the best in class benchmark of

50%

67 © 2017 Coveo Solutions Inc - Proprietary and Confidential

Self-Service MetricsMonth of March

Page 68: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Highlights

• Traffic Workflow 39% of customers with

the intention of submitting a case will

search or navigate prior to submit the

case.

68 © 2017 Coveo Solutions Inc - Proprietary and Confidential

Site Workflow MetricsMonth of March

Page 69: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Highlights

• Case Deflection 29 cases were deflected

combining assumed and confirmed case

deflection which is 12% deflection.

69 © 2017 Coveo Solutions Inc - Proprietary and Confidential

Case Deflection MetricsMonth of March

Page 70: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Exercise – Visit Card

Open Self-Service Tab in your dashboard:

▶ Self-Service Section

▶ Edit dashboard

▶ Add Visit Card in Self-Service Section

▶ Show Visit where customer select a facet (event cause = facet select)

▶ Title of the card: Total Visits with Facet Selection

▶ Unsuccessful Visits

▶ Click the crayon on the Visit without results and analyze how the condition is built

▶ Click the crayon on the Visit Without Clicks and analyze how the condition is built

▶ Save your dashboard and click on the title: Total visit without Results. You can see which visits that meet this

condition.

70

Page 71: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Machine Learning

71

Automate

Page 72: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

1. Review why Analytics

2. Basic Analytics - Key concepts

3. Set up your Lab Environment

4. Understand - Dashboards

5. Understand – Solution Adoption

6. Act – Content Gap and Query Performance

7. Act – Common Boosting Scenarios

8. Understand – Workflow and Self-Service

9. Automate - Machine Learning

Plan

Page 73: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Learn the how machine learning can help automate the manual actions we just learned

about.

Learning Objectives

1. Understand how machine learning works and the different models

2. Understand the different business cases to use machine learning

3. Know the requirements to set up and activate the different machine learning

models

4. See the difference in search results with machine learning activated and without

In This Module

Page 74: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

The Tools of Analytics

User Interface

Query Pipelines

Machine LearningUsage Analytics

Connectors

Salesforce Sitecore Web Exchange

Index

Source

Items

Source

Items

Source

Items

Source

Items

Automate3

Page 75: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

How do I fix my printer?

Thanks, that last one worked!

How do we fix our printers?

Thanks, the first one worked!

IntelligenceEngine

Use machine learning to learn from your visitors, continually improvesearch results & predict relevant content

Make self-service smart

Page 76: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Coveo ML Information Flow

Usage Analytics

UA

Database

ML Service

Previous Users

UA Events

Type: Page View, Query, Clicks, Custom Events

Build a model

Query Pipeline

Index

Current User

User Request

User context (e.g. user role)

User actions (e.g. page view, query) Request

Query with

ML boosts

Page 77: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Machine Learning Models

Tune Relevance ModelAutomatically optimize the Ranking of Coveo Items returned by a Query

1

2

Query: IPA

3

Sleeman

4

Alexander Keith’s

Goose Island

Inukshuk Island

1

2

3

Goose Island

4

Alexander Keith’s

Sleeman

Inukshuk Island

Query Suggest ModelProvides users with Query Suggestions as they are typing in the Coveo Search Box

RecommendationRecommends Coveo Items that the user might be interested on viewing next.

ipa

Goose Island IPAMolson Canadian IPA

Sleeman IPAMuskoka IPA

Recommended Topic

Stone Delicious IPA

Mill st west coast ipa

Hops & Robbers ipa

Page 78: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Model TypeThe type of Machine Learning relevance optimization

Training SetThe Usage Analytic data that a Model uses to self-optimize.

Update FrequencyThe frequency at which a Training set is updated with new data.

Data PeriodThe period of Usage Analytic data to be included in the Training Set with each update

Condition [optional]

The Condition(s) to trigger the Machine Learning rule

Some Vocabulary

Page 79: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

ML Requirements

1 - Machine Learning Models are applied on a Query Pipeline

2 - Models learns from Users past Behaviors to optimize the search experience ( using data

such as Context, Query, Success rate, Click, etc.)

3 – You need approximately 10 000 Query Events in order to compile a Relevant model

4 – Depending the number of Events, compiling or updating a Model takes between 30 and

60 minutes.

Page 80: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Requirements

[Facet Component]

[Recommendations Component]

[Omnibox Component]query suggestionssuggestion engine

[Search Result List]

Coveo Search Interface

queyr suggestions

Tune Relevance Model

Query SuggestModel

Recommendations Model

Page 81: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

To Remember

Model Requirements What it Does

Tune Relevance None Automatically optimize the Ranking of CoveoItems returned by a Query

Suggest Query Search Interface has an Omnibox component

Provides users with Query Suggestions as they are typing in the Coveo Search Box

Recommendation Search Interface has a Recommendations components

Recommends Coveo Items that the user might be interested on viewing next

Page 82: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Steps:

1. Search using an acronym or term

2. Click an appropriate result

3. Enter the words behind the acronym or a synonym of the term entered above and click the same result

4. Repeat

5. Once model is trained, in the Content Browser, ensure the model returns the expected results

Help Train Machine Learning with Thesaurus Rules

Accelerate the Machine Learning training process by pointing to synonyms as well as words behind acronyms

Page 83: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Highlights

• 42.8% of the clicks have been suggested

by Machine Learning

• Top Queries are about analytics and

machine learning

83 © 2017 Coveo Solutions Inc - Proprietary and Confidential

Search Activities driven by MLTop Queries with Clicks

Page 84: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Highlights

Based on a real scenario,

this is the difference of

results without ML and

With ML.

Without ML we show

topics around list, listed

and with ML we show

topics around thesaurus

84 © 2017 Coveo Solutions Inc - Proprietary and Confidential

ML Impact

Page 85: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

DEMO – Adding a Tune Relevance Model to a Pipeline

Page 86: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Exercise – A/B Testing

Scenario

After reviewing the new Machine Learning model, we need to understand if it is having a positive

impact on query performance.

Exercise:

Use the A/B Test “FLS + ML” to create an A/B Test report dashboard using the A/B Test template

Compare Pipeline A = Full Learning Site to Pipeline B = MachineLearning

86

Page 87: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

• Machine Learning Models are compiled from Usage Analytics Events and requires

around 10 000 Query Events to be Optimized

• Machine Learning Models are added to Query Pipelines

• There are 3 Machine Learning Models: Tune Relevance, Query Suggest and

Recommendations

• Query Suggest Model requires your Search Interface to have an Omnibox Component

• Recommendations Model requires your Search Interface to have a Recommendation

Component

In Summary

Page 88: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

In Summary

Why Analytics

The need for relevance is a constant journey in which you

Understand

Analyze the data to gain insight into the behavior of your users

Act

Know the most appropriate action to take from this insight to provide quality

improvements to relevance and improve search success

Automate

Leverage Machine Learning to automate the relevance improvement

Page 89: Coveo Intermediate Analytics · 2019-12-04 · Certified Trainer of Knowledge Centered Service v6SM Practices. 1. Review why Analytics 2. Basic Analytics - Key concepts 3. ... Salesforce

Laurel Poertner

Training Services Lead

[email protected]

Thank you!

@laurelpoertner

Questions?