Research SpotlightLearning’s Big Data Problem: Measuring & Analyzing Impact

David Wentworth, Principal Analyst,

Brandon Hall Group@DavidMWentworth

Mike Rustici, CEO, Watershed

@mikerustici

Michael Rochelle, Chief Strategy

OfficerBrandon Hall Group

HCM Research and Advisory Firm

© Brandon Hall Group 2016 2

FOUNDED IN 1993

Community of 300,000+

10,000 Global Clients

Leading Independent HCM Research & Analyst Firm

Publishing 2+ Pieces of Research Each Day

Sample of Our Clients

How We Help You

© Brandon Hall Group 2016 3

MEMBERSHIP

STRATEGICCONSULTING

EXCELLENCEAWARDS

PROFESSIONAL DEVELOPMENT

Learning & Development

Talent Management

LeadershipDevelopment

Talent Acquisition

WorkforceManagement

Open Surveys

© Brandon Hall Group 2017 4

Visit www.brandonhall.com click Open Surveys

If you would like to join a panel of survey takers, please contact us at success@brandonhall.com

• Recruitment Marketing Practices 2017

• Extended Enterprise Learning 2017

• State of Talent Acquisition 2017

• HCM Technology 2017

How To Ask Questions

© Brandon Hall Group 2016 5

• Submit your questionsor comments about the discussion to our presenters using the Questions tab on your control panel.

• Presentation slides and giveaways can be found in the Handouts tab on your control panel.

Recording & Slides will also be sent out after the webinar.

Agenda

The State of Learning Measurement

Moving and Measuring Beyond Formal

How Do We Get Better?

The Role of Learning Standards

7

23%

27%

38%

40%

53%

66%

78%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

To more strongly link learning and reduced turnover

To provide input to the performance management process

To determine the ROI of learning programs

To more strongly link learning and employee engagement

To more strongly link learning and individual performance

To more strongly link learning and organizational performance

To improve the effectiveness of learning programs

Drivers of Learning Measurement

14%

30%

37%

19%

Formal

8

51%

30%

17%

2%

Informal

43%

32%

16%

7%

Experiential

Not at all Somewhat

Moderately Highly

Effectiveness of Learning Measurement

• Formal Learning/Catalog

• Certification & Compliance

• Courses

• Groups Associated with Classes

• Impressions• Expertise Location

• Idea Engine• Real-Time

Collaboration• Groups &

Workspaces

9

Very Little Learning is Formal

The Blending Effect

4.9%

29.4%

46.1%

19.6%

Not at all

Small extent

Moderate extent

Great extent

Improved Link to Individual Performance

9.1%

37.4%

37.4%

16.2%

Not at all

Small extent

Moderate extent

Great extent

Improved Link to Organizational Performance

95% 91%

Source: Brandon Hall Group HCM Outlook 2017

How Do We Get Better?

© Brandon Hall Group 2016 11

“L&D is responsible for improving our most

important asset, our people. If we can improve

L&D, we can improve the world”

me, I said that

We can do better.

learning happenseverywhere.

Why connectwith xAPI?

To transfer learning records

Example: when you get a new learning platform

For products with different functions to work together

Example: your LMS and a social learning platform

To back up learning records

Example: in case your LMS breaks down

xAPI is a technical specification that makes it easier and

cheaper to connect learning technologies.

How far has xAPI adoption come?

xAPIadoption

Source: http://tincanapi.com/adopters/

Standards define

and confine an industry.

New modelOld model

eLearningLMS

2000

SCORM

eLearningLMS

MobileSocial

InformalSimulations

GamesMOOCs

ILTAdaptive

PersonalizedLRS

Learning Experience PlatformMany more……

2013

xAPI

xAPIdeployments

pilots

2015 – 16

prototypes

2013 – 14

production

2017 –

Why do I have to go beyond CMI-

5?

SCORM: Not evolving, but not dead

yet.

What’s a Learning Record Store?

Three typesof LRS

An Embedded LRS is typically

part of another system. It often

adds xAPI functionality to an

LMS or a sophisticated learning

experience provider. (Example:

Rustici Software’s Engine)

01

Embedded LRS

The main role of this LRS aligns

with the technical definition from

the xAPI spec; it receives,

stores, and provides access to

learning records via xAPI. These

systems are primarily technical

tools with an interface designed

for technical people. (Example:

Learning Locker)

02

Data Storage LRS

A Learning Analytics Platform

(LAP) is a tool that contains an

LRS, but layers a rich reporting

and analytics toolset on top of it.

It’s designed for learning

professionals and facilitates

robust analysis of all the data

collected by xAPI. (Example:

Watershed)

03

Learning Analytics

Platform

Tips for choosing an LRS

01Be on the same page.Make sure everyone involved in the process

shares a common definition for “learning record

store” and an understanding of the type of LRS

you need.

04Verify it meets expectations.Ask what you can do with the data in your LRS. Many

legacy LMS vendors have implemented an

embedded LRS to say they have xAPI, but, most

don’t have out-of-the-box ways to access that data for

reporting once it’s in the system.

02Ensure its xAPI compliant.ADL hasn’t released an xAPI certification

program, but there is a test suite coming soon

that you can run against any LRS to see how it

stacks up.

05Ensure unfettered access to data. One of the core tenants of xAPI is that data should

be freely portable across systems. Make sure your

LRS vendor will allow you to retrieve all the data you

store in your system.

03Join the xAPI community.The xAPI community is friendly and eager to

help. Get involved in discussions and ask more

about each provider. 06

Speak with client references. Make sure your vendor has deployed its product at a

scale and scope similar to your situation and speak

to an actual client to find out how the process and

deployment went.

Key Takeaways• Learning does not just exist within the confines of the

LMS.

• If we want learning to impact behavior and performance, we have to bring it closer to the point of need.

• Don’t let fear of measurement and tracking prevent you from getting started.

• xAPI is a standard that can help pull all the pieces of the puzzle together.

• AN LRS is the tool that lets you see the whole puzzle once it’s together.

Questions?

Submit your questions to the presenters via the

questions tab on your control panel

27

Type your questions here

Thank you for joining us today!

If you have any additional questions, please email us at success@brandonhall.com.

Learn More About UsProducts and services that we offer

© Brandon Hall Group 2016 29

Membership Offers Tailored SupportRESEARCH ACCESS & EVENTS

• Reports • Case Studies, Frameworks & Tools• DataNow® & TotalTech®• Webinars and Research Spotlights• Annual HCM Conference

ADVISORY SUPPORT• Ask the Expert• 1 on 1 Consultations• Research Briefings• Benchmarking

CLIENT SUCCESS PLAN• Your Priorities• Executive Sponsor• Client Associate• Monthly Meetings

© Brandon Hall Group 2016 30

To learn more about becoming a member or to sign up for a free trial, please email us at success@brandonhall.com

Strategic Consulting Offers Expert Solution DevelopmentBENCHMARKING

• Competitive/Comparative

• Maturity Model

• Custom Research

STRATEGY• Business Case

• Planning

• Organization & Governance

TECHNOLOGY SELECTION• Vendor Selection

• Architecture Design

• Systems Evaluation

DEVELOPMENT & INTEGRATION• Program Design

• Assessment

• Survey

• Process Integration

© Brandon Hall Group 2016 31

To learn more about consulting, please email us at success@brandonhall.com

Why Brandon Hall Group

© Brandon Hall Group 2016 32

1

2

3

4

5

We offer flexible, high-value memberships that offer everything you need at a cost you can afford.

Our research is better – more targeted, easily digestible, focused on business results, predictive, prescriptive.

Our solutions gets better results – research-driven, focused on the business, simple, targeted, flexible approach.

Our case studies are unique for their diversity and focus on business results.

Our professional development is practical, tailoredto your needs, and equips you to take action.

We Do This Through…

Research and Analysis that Define New HCM Standards

Success Stories from Global Organizations that Show those Standards in Action

Professional Development that Assesses Current State and Enables Innovation

Flexible Services to Design and Implement Transformative Solutions

Recognition that Validates the Transformation

© Brandon Hall Group 2016 33

What Our Clients Have to Say

Brandon Hall Group’s Excellence Awards allows you to take a deeper dive and evaluate the programs that you’ve created at a level that you might not have the opportunity to do… They have always been a cornerstone for me.

- Katrina Williams, Director, Sales Capability

CDW

https://youtu.be/OgdA_tpM55U

© Brandon Hall Group 2016 34

What Our Clients Have to Say

I describe the attention and that customer service that I’ve experienced as being excellent. They are very attentive and wherever I need to be connected to whomever, they are there to help me.

- Joanne Veech, Global Talent Leader

PwC

https://youtu.be/YkY-v8gfW5k

© Brandon Hall Group 2016 35

HCMx Radio

© Brandon Hall Group 2016 36

THE ONLY PODCAST IN THE HCM ARENA THAT WEAVES CURRENT MARKET RESEARCH, HR TECHNOLOGY AND INDUSTRY LEADERS INTO EACH EPISODE

To catch up on the latest episodes, visit www.brandonhall.com/hcmxradio or subscribe on iTunes