The Half-Time Strategy: Saving Your Organization Time and Money with Diagnostic Testing

Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

2013 Users Conference Baltimore | March 3rd – 6th

The Half-Time Strategy: Saving Your Organization Time and Money with

Diagnostic Testing John LoBianco Steven Torkel

PwC

2013 Users Conference Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark

is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

John LoBianco Senior Associate, Evaluation & Assessment Team

Steven Torkel Director, Evaluation & Assessment Team

Special thanks to Sean Farrell, Senior Manager on the E&A Team, who has been instrumental in this work.

Your Presenters

Slide 2



A global network of professional services firms 180,000 staff members and partners across the network

People development is a top priority PwC was awarded #1 in Training Magazine’s Top 125 for 3

years in a row, and was placed into the “Hall of Fame”

Highly mobile and virtual workforce

Slide 3



How our approach to diagnostic assessment has evolved over the past three years

Best practices for building defensible assessments for the purpose of testing-out of training

Half-time Strategy

Modeling tool

Session Objectives

Slide 4



Diagnostic Assessment

Slide 5

Measures current level of knowledge and skill in specific topic areas.

Results help drive decisions about future development activities.



Why do diagnostic assessments?

Slide 6



How our approach has evolved

Slide 7

2010

• Tax 1 self-studies • New employees

2011

• Tax 1 self-studies • New employees • Enhancements

2012

• Mandatory Training • Experienced Partners

Future

• Classroom Training • Integrating assessments with on-the-job learning



New tax hires must attend Tax 1 after joining the firm College graduates with varying levels of experience First professional job First exposure to PwC technical training

The Tax 1 Program

Slide 8

Program shortened from two weeks to one week

Four years ago:



Tax 1 Re-design

Slide 9

Several self-studies were created as advance preparation New hires were required to complete these self-studies before

attending Tax 1

Learners could not always find the time to complete the self-studies



Test-outs for three of the self-studies were created: New hires could save time by not having to

complete the self-studies Optional The three test-outs were comprised of 25, 28, and

21 questions “One and done”

How could we ease the burden on our learners while still maintaining quality?

Slide 10



On average, there was a slight increase in training time across the learner population after the test-outs were implemented

Why didn’t we achieve the benefit we were hoping for? Not enough learners passed the test-out Test-outs were too long

How much time did we save our learners?

Slide 11



Combined the three test-outs into one “consolidated” test-out

Identified and removed poorly performing items

Adjusted cut score

Required learners to register for test-out

Implemented “half-time” strategy

How did we improve the test-out process?

Slide 12



Half-Time Strategy

Slide 13



Each section of the test-out had 20 questions Each section of the test-out was split in half

Learners needed to pass the first half of each section to proceed to the second half of that section Learners answered anywhere from

30 to 60 questions Saved learners time by bypassing

questions they would probably answer incorrectly

Tax 1 Test-out Half-time Strategy

Slide 14



Test-out Half-time Strategy

Slide 15

Course A (10 questions in 1st half)

Course A (10 questions in 2nd half)

Course B Course C

Course B Course C

Learner scores 80%

or above

Learner scores less than 80%

Learner scores less than 80%

Learner scores 80%

or above

Learner scores 80%

or above

(10 questions in 1st half)

(10 questions in 2nd half)

(10 questions in 1st half)

(10 questions in 2nd half)

Learner is then taken to Course B

Learner is then taken to Course C

Learner reaches end of test-out



Learners were told their results for each section

End of Test-out Feedback

Slide 16

Course A

Course B

Course C



58 hours of training time (about 5%) were saved 871 total hours taking the

self-studies 164 total hours taking the

test-outs 1093 potential total training

hours

Time savings were modest

Slide 17



Mandatory Training test-out option

Slide 18

Mandatory Training for Partners Yearly training event

Option to take a short online test in lieu of completing a 90 minute self-study learning activity

Testout designed to be more challenging than the end of course assessment

Available only to partners with 2+ years experience



Benefit: Expected time savings for the practice

Slide 19

Difficult to quantify as it depends on the number of partners who take the optional test-out, as well as the percentage who pass

Sample estimates based on: A learner population of 1500 partners 90 minute self study learning activity 18 minute test-out activity

Good Better Best

% Taking Test out 50% 70% 90%

% Passing 40% 60% 75%

Training Hours Saved 225 (10%) 630 (28%) 1114 (50%)



Parallel test development: Developed over 50 questions mapped to course content for both the test-out and self-study end of course assessment

Pilot-tested all questions with 68 partners and received 400+ comments

Used statistical item analysis to isolate best performing questions with appropriate level of difficulty

Limited testing time while still maintaining quality 17 question test-out Half-time strategy

Rigorous test development procedures improved the quality of the test-out

Slide 20



17 question test split in two parts: First half 9 questions and second half 8 questions Split the questions to even out content and difficulty

where possible Passing score of 80% (14 of 17) Need to “pass” first half to proceed to second half First half passing score: 6 of 9 which meant 8 of 8 was needed on the second half First half passing score set to allow total pass if perfect on second half

Test-out Half-time Strategy

Slide 21



Partners were satisfied with the test-out process

Slide 22

“I thought this was a great option. Saved a lot of time and it is still not easy to take the test so it also ensures appropriate focus and knowledge.”

“I thought it was a great improvement, and permitted those of us who watch these areas closely to get through the requirement (which is a good requirement to have) easily and efficiently.”

86%



Success Factors

Slide 23

• 231 Partners took the test-out; 20% of target audience • Reasons for not participating:

- Want to take the self-study - Could not find time to complete

Participation Rate

• 67% Pass Rate • Of those who did not pass, 31% were stopped at the

half-way point, saving them time Pass Rate

• 14 minutes average test time; 9 minutes half-time • Saved 236 hours in total; 11% of training time Time Saved

• 86% satisfied with test-out • Increased flexibility for partners • Favorable feedback

Partner Feedback



Modeling Tool Interactive Demo

Slide 24

Assumptions Data

Learner Population 1000

% Taking Test-Out 50%

% Passing Test-Out 50%

Minutes for Training 90

# Questions in Test-Out 15

Minutes Per Question 1

Additional minutes for test out (e.g. instructions) 3

Total Minutes for Test-Out 18

Total Possible Training Hours 1500

Path # of Learners Test-Out Hours Training Hours Total Hours

Learner passed the test out 250 75 75

Learner did not pass the test out 250 75 375 450

Learner did not attempt the test-out 500 750 750

Total 1000 150 1125 1275

Hours Saved 225 15%

Change assumptions to estimate hours saved by using a test out strategy.



Future Deliverables

Slide 25

Classroom Training Integrating assessments with on-the-job learning Traditional training model Curriculum-based Mandatory courses

On-the-job learning {70 / 20 / 10}

Use diagnostic assessment to better target on-the-job learning as well as formal training

The Half-Time Strategy: Saving Your Organization Time and Money with Diagnostic Testing

Technology

Transcript of The Half-Time Strategy: Saving Your Organization Time and Money with Diagnostic Testing