SharePoint Saturday New York City 2015#SPSNYC

Fotopedia.com

An Independent Evaluation of Third-Party SharePoint Analytics Offerings

Jonathan RaltonAll trademarks and registered

trademarks are the property of their respective owners.

Jonathan RaltonBlueMetal

An Independent Evaluation of Third-Party SharePoint Analytics Offerings

AgendaSetting the StageClient StoryMeasuring and Trending

DefinitionsFrameworkExamples

SharePoint Analytics

Third-Party Tool CandidatesWhich would you choose?

EpilogueWhich did I choose?How does the story end?

Wrapping UpQuestions

Setting the StageAn Independent Evaluation of Third-Party SharePoint Analytics Offerings

YOU

Audience

What roles are you in?

What SharePoint phase are you in?

What are you hoping to learn?

ME

Presenter

Jonathan Ralton• Senior Information Architect• SharePoint IT Pro since 2005

(WSS/SPS)• No coding!• Document Management,

Content Management, Knowledge Management…

@jonraltonjonathanr@bluemetal.com

blog.jonralton.net

Client StoryAn Independent Evaluation of Third-Party SharePoint Analytics Offerings

Overview• Food and Beverage Manufacturing Company

• One of the top industry leaders in the country• One of the top industry leaders in the world

• Employees in dozens of countries• Hundreds of household brand names

• Some with $1b market shares

Intranet• Redesigned by BlueMetal and relaunched in late 2014• Heavy on video content• Social focus• Personalization

Challenges

Google

Challenges

Communication Justification• Need to prepare for new

management oversight• Need to publish appropriate

news at appropriate times• Want to maximize investment in

the new facility of the intranet

Objectives

Content and Visitors• Learn what content is being

consumed and by whom• Learn which types of content

are most engaging• Gain better insight into what

kinds of content employees will respond well to, and in turn continue to return to myCorptown for

Employee Communications• Help to justify team’s value in

managing myCorptown• Demonstrate higher employee

satisfaction in being informed• Demonstrate higher employee

engagement

Objectives

AdoptionEngagement

Visitors/Visits/Constituencies

New Employees

New Users

Lapsed Users

Executive Employees

Mobile Users

Tablet Users

Laptop Users

Desktop Users

Kiosk Users

Current Tool

Measuring and TrendingMust Love Term Sets

WEB ANALYTICS

def·i·ni·tion [dèffə nísh'n]

Web Analytics

Web analytics is the measurement, collection, analysis, and reporting

of web data for purposes of understanding and optimizing web usage.

METRIC

def·i·ni·tion [dèffə nísh'n]

Metric

A metric is a something that can be measured based on quantifiable data,

or at the very least, estimated based on anecdotal data.

Metrics w/Tools

Web analytics tools mainly provide metrics for consumption via reports and APIs.

KEY PERFORMANCE INDICATOR(KPI)

def·i·ni·tion [dèffə nísh'n]

Key Performance Indicator

A key performance indicator (KPI) is a visual indicator that tells a story

about one or more measurements, in context.

Key Performance Indicators w/Tools

Web analytics tools may already provide some KPIs in dashboards.

Metrics and KPI Framework

Goals

Signals

Metrics KPIs

Goals

Adoption

Engagement

RetentionTask Success

Happiness

Sample Goals – Adoption

Ensure new employees and new users are

engaging with myCorptown content for the first time, including

the email newsletter

Build employee buy-in by telling

our story through our vehicles

Reach our employees ‘where they are’ including

via mobile and kiosks

‘Give them what they want’ by being able to respond to

popularity of content

Sample Goals – Engagement

Get our employees to give feedback on content

including likes and comments, submitting

questions, etc.

Have our employees be conversant in an informed manner

about company news with fellow colleagues

Get our executive team to choose

myCorptown as their communication

vehicle

Get our employees to help each other

(Employees as Communicators)

Sample Goals – Retention

Get our users to return to

myCorptown more often (more than 1x

per week)

Encourage our lapsed users to

return to myCorptown

Reduce stale content in areas on

myCorptown where dynamic content is

expected by our users

Sample Goals – Task Success

‘Give them what they need’ in

addition to what our users want, including

via search

Solve our users’ problems via self-

service where possible (without a

help desk call)

Sample Goals – Happiness

Ensure our employees are able to articulate the company story•What we stand for•How does my work fit into the story•How does my work touch consumers

Support a more transparent

culture at the company

Decrease frustration during

activities on myCorptown

Support our employees

through times of change

Sample Signals

GoalGet our users to return to myCorptown more often (more than 1x per week)

Signals• Providing content relevant to

users• Identify users who return to visit

the site within a week period

Sample Signals

GoalGet our employees to give feedback on content including likes and comments, submitting questions, etc.

Signals• Identifying posts that have

higher numbers of views• Identifying posts that have high

numbers of comments• Monitoring submitted questions

Sample Signals

Goal‘Give them what they need’ in addition to what our users want, including via search

Signals• Identifying usage of tools and

links that have typically been difficult to find• Monitoring search results for

successful and failed queries

Sample KPI w/Trending #1• Number of retained user home

page views minus bounces as percentage of all views• Over time period of 7 days• Indication of current metric

against target w/in a certain window• Trend

57%

StickinessHome Page

Target

70%Window

7 Days

Sample KPI w/Trending #2• Number of views during off-

work hour segments• Over time period of 5 days• Indication of current metric

against target w/in a certain window• Trend

1.6k

Time-Based VisitsOff-Work

Target

2kWindow

5 Days

Sample KPI w/Trending and History #1

• Number of posts related to influencers• Over time period of 30 days• Indication of current metric

against target w/in a certain window• Performance against last

window and previous year’s window• Example of where higher metric

is better

38

InfluencersPosts

Target

50Window

30 Days

YoY

+70.1%Last Month

-23.9%

Sample KPI w/Trending and History #2

• Number of questions submitted to portal team• Over time period of 30 days• Indication of current metric

against target w/in a certain window• Performance against last

window and previous year’s window• Example of where lower metric

is better

12

PortalQuestions Submitted

Target

15Window

30 Days

YoY

-156.5%Last Month

-10.4%

Sample KPI w/Snapshot #1

Average Length of Visit

> 5 m > 4 m > 3 m > 2 m > 1 m

• Average visit length• Segmented into groups• Current metric for one window• Shows spread/breakdown detail• Does not show trend

Sample KPI w/Snapshot #1• Average visit length in minutes• Over time period of 7 days• Indication of current metric

against target w/in a certain window• Trend

1.8

Length of VisitHome Page

Target

2.0Window

7 Days

Sample KPI w/Snapshot #2• Rating of frustration level on

portal from survey• Current metric for one window• Shows spread/breakdown detail• Does not show trend

Low Medium High0

1000

2000

3000

4000

5000

6000

Frustration Level

Frustration Level

Sample KPI w/Snapshot #2• Most common rating of

frustration level from survey• Over time period of 1 fiscal

quarter• Indication of current metric

against target w/in a certain window• Performance against last window

and previous year’s window• Example of where metric is

abstracted

M

PortalFrustration Level

Target

MWindow

Quarter

YoY

+27.6%Last Quarter

+5.8%

SharePoint AnalyticsAn Independent Evaluation of Third-Party SharePoint Analytics Offerings

Changes ⓮ → ⓯

Changes ⓮ → ⓯• Web Analytics Web Part is not supported anymore• Reports Removed

• Browser Traffic• Top Users• Referring URL

Changes ⓮ → ⓯• Engine finds relevant information based on clicks, views, etc.• There are now visual indicators and usage numbers based on number

of views and number of unique visitors• Clicks and views are counted for each document• Content can be recommended• Search results influenced by the priority of an item• Ability to sort search results by "hit"• Engine is now extensible for third-party solutions

SharePoint Usage ReportingUsage

This report shows historical usage information about the site collection/site, such as the number of views and unique users. Use this report to identify usage trends and to determine times of high and low activity.

SharePoint Search ReportingNumber of Queries

This report shows the number of search queries performed. Use this report to identify search query volume trends and to determine times of high and low search activity.

Top Queries by Day/MonthThis report shows the most popular search queries. Use this report to understand what types of information visitors are seeking.

Abandoned Queries by Day/MonthThis report shows popular search queries that received low click-through. Use this report to identify search queries that might create user dissatisfaction and to improve the discoverability of content. Then, consider using query rules to improve the query's results.

No Result Queries by Day/MonthThis report shows popular search queries that returned no results. Use this report to identify search queries that might create user dissatisfaction and to improve the discoverability of content. Then, consider using query rules to improve the query's results.

Query Rule Usage by Day/MonthThis report shows how often query rules trigger, how many dictionary terms they use, and how often users click their promoted results. Use this report to see how useful your query rules and promoted results are to users.

SharePoint Library ReportingMost ViewsMost Views by Unique UsersMost Recommendation Clicks

QUICK EXPLORATION

Third-Party Tool CandidatesAn Independent Evaluation of Third-Party SharePoint Analytics Offerings

Requirements – Compatibility

A tool for myCorptown analytics will require compatibility with (or extensibility to support) traffic delivered through…

Requirements – Awareness

In order to provide the most visibility into activity on myCorptown, a tool for analytics should be specifically aware of traffic, content, and actions related to… • Social Intranet

• Video Cloud

Requirements – Content Segmentation

Intranet administrators would like to segment analytics data by the types of content consumed.This would require support for…

• Metadata• File Extension

Requirements – Visitor Segmentation

Along with the desire to segment the content consumed within myCorptown, intranet administrators will need to examine various properties of the visits and visitor as well.

These attributes of a visit or visitor should be supported…

• Platform• Mobile Device• Browser• Date and Time• Session• Microsoft SharePoint 2013

• User Profile Property• User Audience Membership

• Microsoft Active Directory• User• User Attribute• User Group Membership

Requirements – Extensibility

Reusable MetricsIn order for a dashboard or other display and recalculation of raw metric data into KPIs and reports, an API or REST interface should ideally be provided by the tool.

Email CampaignsIn addition to native web traffic on myCorptown, the tool should support UTM tags commonly used in conjunction with email campaigns and the hyperlinks that refer visitors to the site.

Requirements – Pricing

For the purposes of evaluation, the following data points were used:

• Page Views~1,000,000/month• Users

~30,000

Google AnalyticsGoogle Analytics helps analyze visitor traffic and paint a picture of your audience and their needs. Mainly designed for e-commerce websites, it is a cloud-only/SaaS offering.

Google Analytics

Trackable Attributes/Actions• Page Views• Clickpaths• Visitors• Browser/Platform/Device• Date and Time

Analysis Method• Tracking occurs via JavaScript-

based page tags

Google Analytics• Data is often based on a subset, or percentage of sessions

• This is due to the practice of reporting trends in the subset to reduce processing time• This can occur in the reports, during the data collection itself, or in both

places

Google Analytics

Key Benefits Key Drawbacks In-Page Analytics Clickpath Visualization Sparkline KPIs Real-time Cloud-based

Google takes ownership of your metrics data and may use it how they see fit

Not ‘SharePoint-aware’ Does not analyze search queries No video metrics No Yammer integration No Sitrion integration Visitor attributes require customization Shared login/Google account required

ActualMetrics AngelfishWith its roots in Urchin, Angelfish analyzes IIS logs as a completely separate application, usually installed on a server apart from SharePoint.

ActualMetrics Angelfish

Trackable Attributes/Actions• Page Views• Clickpaths• File Downloads• Visitors• Browser/Platform/Device• Date and Time

Analysis Method• Angelfish relies on the IIS logs• Although tagging with JavaScript

is supported, it is not required

ActualMetrics Angelfish• This tool appears to be in its infancy at this point in time

• A lot of features are in the works/slated for a future release

• The audience for this tool seems to be more technical users than business users• The interface seems like it would be more difficult to navigate for someone

outside of IT

ActualMetrics Angelfish

Key Benefits Key Drawbacks Spike analysis—second-level time

segmentation Unlimited nested segments Bandwidth analysis Broken link analysis Six different tracking methods for visitors

Not ‘SharePoint-aware’ Does not analyze search queries No video metrics No Yammer integration No Sitrion integration Visitor attributes require customization Reports are only exportable in unformatted

form (CSV, XML) Separate hardware recommended

Adobe AnalyticsFormerly known under the brand Omniture, Adobe purchased this analytics suite and it is now part of their Marketing Cloud offering. Mainly designed for e-commerce websites, it is a cloud-only/SaaS offering.

Adobe Analytics

Trackable Attributes/Actions• Page Views• Clickpaths• Visitors• Browser/Platform/Device• Date and Time

Analysis Method• Tracking occurs real-time via

JavaScript-based page tags

Adobe Analytics• Adobe has put a lot of effort into the user interface to enable

visualization of data

Adobe Analytics

Key Benefits Key Drawbacks Sparkline KPIs Real-time Cloud-based; low overhead

Not ‘SharePoint-aware’ Does not analyze search queries No video metrics No Yammer integration No Sitrion integration Visitor attributes require customization

HarePoint Analytics for SharePointUsing its own analytics engine separate from SharePoint and any server logs, HarePoint Analytics installs as part of SharePoint while using its own SQL databases.

HarePoint Analytics for SharePoint

Trackable Attributes/Actions• Page Views• Clickpaths• File Downloads• Search Queries• Visitors• Browser/Platform/Device• Date and Time

Analysis Method• Not requiring JavaScript tags,

the tool utilizes its own tracking within SharePoint to provide metrics

HarePoint Analytics for SharePoint• HarePoint seems committed to this product, as it has adapted it from

SharePoint 2007 through 2010 and 2013, however they have yet to integrate social metrics

HarePoint Analytics for SharePoint

Key Benefits Key Drawbacks Workflows tracked Web part for analytics display within

SharePoint Aware of content within SharePoint, not

just actions by visitorso Unused content can be identified

Real-time No additional hardware required

Must access reports from Central Administration

Can’t track external links No social tracking

Intlock CardioLog AnalyticsDesigned specifically for SharePoint and covers all aspects of usage including portals, extranets, collaboration sites, and social. It is installed on a separate SQL server.

Intlock CardioLog Analytics

Trackable Attributes/Actions• Page Views• Clickpaths• File Downloads• Search Queries• Visitors• Browser/Platform/Device• Date and Time

Analysis Method• Tracking occurs real-time via

JavaScript-based page tags• IIS logs can be imported to

integrate historic data• It does not rely on SharePoint’s

built-in analytics or logs

Intlock CardioLog Analytics• This tool has great in-page feedback options including:

• Message bar• Survey• A/B multivariate testing

Cardio

Key Benefits Key Drawbacks Custom dashboards Direct integration with Active Directory and

SharePoint user profiles Display within SharePoint Page metadata Automatically generated reports Identify unused content Sitrion integration

Requires separate hardware

Piwik SharePoint AnalyticsWith a highly evolved UI for reports, Piwik offers solid SharePoint analytics by activating a feature per site collection.

Piwik SharePoint Analytics

Trackable Attributes/Actions• Page Views• Clickpaths• File Downloads• Search Queries• Visitors• Browser/Platform/Device• Date and Time

Analysis Method• Tracking occurs via JavaScript-

based page tags

Piwik SharePoint Analytics• The installation of Piwik is very streamlined

• A pre-configured server image is delivered and the tool automatically populates the tracking code into all relevant areas

Piwik SharePoint Analytics

Key Benefits Key Drawbacks Real-time viewing of visitor actions Sparkline KPIs Ease of setup

Lack of social tracking Separate hardware required

WebTrends SharePoint AnalyticsClosely aligned with Microsoft, Webtrends is a marquee vendor for web analytics.

Piwik SharePoint Analytics

Trackable Attributes/Actions• Page Views• Clickpaths• File Downloads• Search Queries• Visitors• Browser/Platform/Device• Date and Time

Analysis Method• Embedded JavaScript tags in the

page send information (encrypted in transit) to the tool

Webtrends SharePoint Analytics• Webtrends seems to have a particular focus on adoption and

engagement.• Their pitch is “consumable and actionable insights for business users”

• It has a well-polished UI

Webtrends SharePoint Analytics

Key Benefits Key Drawbacks Cloud-based; no overhead Targeted for consumption by business users Plug-ins for Brightcove Sitrion integration

Feature-heavy product; only small subset of feature set is required

Which would you choose?An Independent Evaluation of Third-Party SharePoint Analytics Offerings

EpilogueAn Independent Evaluation of Third-Party SharePoint Analytics Offerings

Comparison Matrix

Areas of Comparison• Compatibility• Tracking Method• On-Premise vs. SaaS• Content and Process Awareness• Content Segmentation• Visits/Visitor Segmentation• Reporting• Feedback• Extensibility• Price

Which did I choose?An Independent Evaluation of Third-Party SharePoint Analytics Offerings

Recommended Tool

Why?• This recommendation is based largely on the feature set offered by

CardioLog• How those features align with the needs of the client in tracking activity on

myCorptown

• Content as well as activity is monitored• Lack of action can be tracked on documents and items

• In-page feedback mechanisms were very attractive

Why?• Sitrion analytics are built-in and

will require little to no customization

• Brightcove analytics should be able to be integrated via their API and brought into CardioLog

Why?• Other tools…

• Did not have as extensive a feature set to support the client’s requirements• Were less SharePoint-specific• Did not allow for business users to consume their reports as easily• Cost too much

How does the story end?An Independent Evaluation of Third-Party SharePoint Analytics Offerings

Chosen Tool

…and they lived happily ever after.

• Decided to address some of their near-term goals• Have reports against them based

on certain metrics• Some of the KPIs would be able to

be developed from those metrics (manually)

• In order for this to take place, an augmentation of current information being sent over the Internet to Google Analytics is required• This needs to include sending over

key user attributes as well as some classification of the pages being viewed

• Integration with SharePoint user profile data will be required to track things such as a user’s department

Warnings• Complete user information such as first and last names should not be

sent over to Google Analytics, as the data may be reused by Google in any manner• Steps should be taken so that any confidential information or trade

secrets are not accidentally injected into the Google ecosystem in the form of metatags, page titles, etc.

Wrapping UpAn Independent Evaluation of Third-Party SharePoint Analytics Offerings

Wrapping Up

The views expressed in this presentation are not in any way endorsed by any of the product vendors, and I warrant no accuracy in the data presented.

Wrapping Up

Although I concluded that CardioLog Analytics is the leading candidate for this particular engagement, this does not by any means signify future disqualification by myself or BlueMetal in future evaluations of the other tool candidates.

Wrapping Up

In fact, another tool might be the best we could recommend to a completely different future client depending on their needs.

QuestionsAn Independent Evaluation of Third-Party SharePoint Analytics Offerings

THANK YOUEVENT

SPONSORSWe appreciate you supporting the New York SharePoint Community!

• Diamond, Platinum, Gold, and Silver have tables scattered throughout

• Please visit them and inquire about their products & services

• To be eligible for prizes make sure to get your bingo card stamped by ALL sponsors

We Are Hiringin NYC!

• Mobile Web Software Engineer• Cloud & Services Senior Architect• Senior Java Developer• Senior Web Developer• Senior SharePoint Engineer • Devices & Mobility Associate Software

Engineer• Operations Coordinator (Part Time)Learn more:

https://www.linkedin.com/company/bluemetal-architects/careers

ReferenceAn Independent Evaluation of Third-Party SharePoint Analytics Offerings

ResourcesTool Website Tool Download/Request Tool Demo

ActualMetrics Angelfish Link

Adobe Analytics Managed Metadata Overview Link

Google Analytics

HarePoint Analytics for SharePoint Link Link

Intlock CardioLog Analytics Link

Piwik SharePoint Analytics

Webtrends SharePoint Analytics

Google Analytics for SharePoint 2013/Office 365 (CodePlex)

Resources