Advanced Performance Measurement Workshop Develop Measures That Drive Performance By David Wilsey

©1997-2012 Balanced Scorecard Institute, a Strategy Management Group company. All Rights Reserved. Do not copy without permission. 1

Advanced Performance Measures Workshop:

Develop Measures That Drive Performance

Advanced Performance Measures Workshop: Develop

Measures That Drive Performance

David Wilsey, BSMP, SMP Vice President of Education & Technology

Balanced Scorecard Institute March 2012

http://tinyurl.com/8wxywgc


Purpose and Agenda


Agenda:

• Driving Breakthrough Strategic Performance and Strategic Alignment with performance information

• A Practical Step-by-Step Approach to developing meaningful performance measures

– Exercise: Identifying Intended Results

– Exercise: Developing Candidate Measures

• Performance Analysis Steps

• Problem-Solving Q&A (time permitting)

Presentation Purpose: Understand how to develop meaningful performance measures that drive

breakthrough strategic performance.


Why Do We Need Effective Strategic Performance

Management & Measurement Now More Than Ever?



We Live in Uncertain Economic Times


Source: Kauffman Economic Outlook, A Quarterly Survey of Leading Economics Bloggers, Tim Kane, First Quarter, 2012.


We Live in a World of Routine Government Cutbacks


• U.S. Federal debt plus unfunded liabilities is now over $100 Trillion

• Projected 2012 State budget shortfalls is over $153 Billion

• 2012 Municipal Budget shortfalls will exceed $53 Billion

In this environment, programs and services

that cannot be defended will be cut!


We Live in a World of Information Overload



We Work in Siloes


“This is not in my Job Description. I draw lines; I don’t remove trees”

Source: John Phatshwe, BSMP, Botswana National Productivity Centre.



Sabermetrics: the use of statistical analysis to analyze baseball records and make determinations about future player performance based on past performance and correlations. Teams which appear to value the concepts of sabermetrics are often said to be playing "MoneyBall".

We Often Measure the Wrong Things


“Sabermetrics” Transformed Baseball Scouting Into a Science


Traditional Scout Evaluations Based on “Tool Set”

Speed

Arm Strength

Hitting for Average

Hitting for Power

“Sabermetrics” Measures

On-Base Percentage

OPS = On-Base + Slugging

Runs Created = (Hits + Walks) * (Total Bases)

WHIP = Walks + Hits per inning pitched


“MoneyBall” is a Case Study on How to Transform an Organization With

Performance Information

• There is value in understanding the “drivers” of overall (financial and customer) success

• Make “fixing gaps” in driver performance a priority

• Focus energy (and resources) on the things that matter most

• Measure what matters


To accomplish this, you have to think differently about your organization!


Why Integrate Strategic Planning & Performance Management?

A Framework Brings Discipline


• Get everyone using the same language

• 'Connect-the-dots' between activities and strategy & vision

• Systematically align vision, strategy, program, budget & people

• Align resources with strategic objectives



Priority & Focus Improves Efficiency & Effectiveness


• Measure only what matters

• Focus on the right initiatives and projects and choose the right products and services

• Process improvements guided by strategy

• Employees focus on results


Align the Organization to Mission, Vision & Strategy


• Strategic Goals / Themes

• Strategic Results

Strategy

Vision Mission

• Customer Values & Needs

• Political Priorities

•Organizational Values

• Stakeholder Considerations

• Policies & Governance

Strategic Objectives

• Strategic Objectives/Map • Measures & Targets • Strategic Initiatives

Budget & Actions

• Programs, Products, Services

• Projects, Activities & Tasks

•Rewards, Recognition & Incentives

• Schedule, Scope, Resources, Risk

• Strategy Budget • Operational Budget • Capital Budget




Strategic Alignment

Mission: Air Botswana is committed to being the airline of first choice through the delivery of high quality service


Building & Implementing a Balanced Scorecard: Nine Steps To Success™



Topics From the Nine Step Framework to Be Addressed Today

• Understand Strategic Context: – Steps 1-4, culminating in Strategic Objectives

• Develop Performance Measures: – Step 5.1: Describe the Intended Results

– Step 5.2: Describe the Measures

– Step 5.3: Describe the Desired Level of Performance

• Analyze Performance Information – Step 7: Performance

Analysis



The Goal is to Develop Meaningful Performance Measures

• Meaningful means strategic

– Focused on desired results

– Derived from the Objectives from a Strategy Map or Strategic Plan

• Meaningful means relevant to the user

– It tells the user what he or she needs to know to make better decisions

– Accepted as valid (measures what it intends)



Example: Airline Balanced Scorecard

Simplified Strategy Map Objectives Measures Targets Initiatives

Financial

Customer

Internal Process

Learning & Growth


Mission: Dedication to the highest quality of Customer Service delivered with a sense of warmth, friendliness, individual pride, and Company Spirit.

Align Ground

Crew

Improve Turnaround

Time

Increase On-Time Flights

More Customers

Lower Prices

Lower Costs

Increase Profits

Increase Revenue

Vision: Continue building on our unique position -- the only short haul, low-fare, high-frequency, point-to-point carrier in America.

Adapted from work by Harvard University / Balanced Scorecard Collaborative

• Increase Profits • Lower Costs • Increase Revenue

• More Customers • Increase On-time

Flights • Lower Prices

• Improve Turnaround Time

• Align Ground Crew

• Market Value • Plane Lease Cost • Seat Revenue

• # of Customers • FAA On time

Arrival Rating • Customer

Ranking

• On Ground Time • On-Time

Departure

• % Ground Crew Stockholders

• % Ground Crew Trained

• 25% per year • 20% per year • 5% per year

• 5% change • First in the

industry • 98%

Satisfaction

• <25 Minutes • 93%

• 70% (100% by year 6)

• 70%

• Optimize routes • Standardize

planes

• Quality Management

• Customer Loyalty Program

• Cycle time Optimization Program

• Stock Ownership Plan

• Ground Crew Training


Organization Capacity

Identify Strategic Objectives Before Developing Performance Measures


Strategic Result (Outcomes)

Financial

Customer

Internal Processes

Strategic Objectives

Strategic Objective: Key strategy component; continuous improvement

activity that must be performed; the “building blocks” that make up a

strategy.

Strategic Objectives are the “DNA” that break

high level strategy into smaller pieces


Strategic Objectives are “Mid-Altitude”


Vision & Mission type language: •“World – class” •“Provider of choice” •“High-performance”

Project, Program or Service Language: “Train staff” “Build a new system” “Deliver services”

Problem: Too Low (project specific)

Risk: too narrowly defined; strictly operational

Problem: Too High (Vague)

Risk: not actionable; will not give actionable focus and direction

30,000 ft.

15,000 ft.

Ground Level

25,000 ft.

Strategic Altitude

Objective language: •“Improve Customer Service” •“Improve Knowledge & Skills” •“Increase Revenue” •“Improve Service Delivery”

Strategic Objective


Strategic Objectives Describe the Intended Results of Continuous Improvement

• Describe the result that we intend to accomplish

• Should be “verb-object” form, normally using a verb that implies continuous improvement: – Improve

– Increase

– Reduce

– Decrease

– Strengthen

– Possible: Optimize (can be confusing)


Increase Economic

Abundance

Good Examples:

Reduce Waste

Improve Product Quality

Improve Access to Services

Improve Brand Awareness

Improve Customer

Satisfaction

Train All Employees

Write a Strategic Plan

Not-So-Good Examples:

Better: Improve Knowledge & Skills

Better: Improve understanding of our strategic direction


Step 5.1: Describe the Results

Step 5.2: Describe the Measure(s)

Step 5.3: Describe the Desired Performance Level and Interpretation

Performance Measurement Development Model


Describe the Intended

Result of the Objective

Choose & define the appropriate

“direct” measure(s)

Define measureable components that

thoroughly describe the Intended Result

Yes

No Yes

Choose & define the

appropriate “indirect” measures

Develop a “composite

index”

Develop targets & thresholds for

each measure

No

Documentation: Intended Result, Measurement Definition & Desired Performance Levels To Step 7:

Performance Analysis

Should measures be

combined into a single

“index”?

Can the Intended Result

be measured directly?


Step 5.1: Describe the Intended Result of Each Strategic Objective

• For each Objective, describe the objective's intended result explicitly, in a few sentences

– Use action verbs

– Avoid words that create ambiguity

– Identify changes in state that accompany the result


Establish agreement across your team on the

desired result!


Measurement Development: First “De-Weasel” the Intended Results

• Strategy is hard to measure because it is often written using “weasel” words, or vague ideals that are hard to interpret, for example:

• We employ “best practices”

• Performance is “Optimized” or “Maximized” or “World Class”

• Employees are more “Efficient,” “Effective” or “Productive”

• “Intended Outcome/Result” in the Objective Commentary should avoid this practice

– Use plain language: Not “Improve Thought Leadership,” but “We create engaging content,” or “More people read (or rave about) our stuff”

– Use sensory language to visualize the desired end result • If you can see, feel or hear it,

you can count it!


Adapted from the Performance Measure Blueprint Workshop, by Performance Measure Specialist Stacey Barr (www.staceybarr.com)


Remove Ambiguity From Intended Results - Examples

• Intended Result: improve thought leadership

– Too ambiguous: it is not clear exactly what thought leadership means

• Not ambiguous: People rave about the articles we write

• Intended Result: we deliver world class service

– Too ambiguous: it is not clear exactly what world class means

• Not ambiguous: Our service is delivered on time and without defect


Improve Brand Image

Improve Service Delivery


Exercise 1: Which of the Following “Intended Results” Still Contain Weasel

Words? And How Might They Be Improved?

Organization: Leading Tablet/Technology Innovator

Objective: Improve Product Development

Candidate Intended Results:

1. Quality increases

2. Enjoyment of our products is optimized

3. We understand our customers

4. Communications improves, both internally and externally

5. We have a sexy user interface

6. We are on the bleeding edge of technology

7. We humanize technology



Exercise 1: Intended Results Possible Solutions

Organization: Leading Tablet/Technology Innovator

Objective: Improve Product Development

1. Quality increases Products meet specifications x or y at z consistency

2. Enjoyment of our products is optimized Customers tell us they love our products

3. We understand our customers Voice-of-the-customer research is consistently used in product development

4. Communications improves, both internally and externally Employees are aware of key initiatives; our marketing advertisements are a hit on free social media outlets

5. We have a sexy user interface Customers tell us they prefer our interface; celebrities want to be seen using our products

6. We are on the bleeding edge of technology Our technology scanning team identifies all important advances relevant to our products

7. We humanize technology Our customers tell us our products are intuitive; non-technical customers use our products



Step 5.2: Describe the Measures

• Analyze each identified Intended Result using this leading question:

– Can the Intended Result of the Objective be measured “directly”? (i.e. is there a clear, discrete, unambiguous way to capture the entire Intended Result?)

• Yes?

– Choose and define the most appropriate “direct” measures

• No?

– Define measureable components that thoroughly describe the intended results



Direct Measurement of the Intended Result is Ideal - Examples

• Intended Result: Services are delivered on time:

– We know when the service is due; we know when it is delivered; thus, we can directly measure whether it was delivered on time

• Intended Result: Interagency is better integrated into planning:

– We know the regular planning activities that occur; we know whether Interagency representatives are included in those activities; thus, we can directly measure whether there is better integration

• Intended Result: All the products that we produce meet specifications:

– If we can define the specifications with physical measures, and measure the products based on those measures, we can measure directly


Improve Service Delivery

Improve Organizational

Alignment

Improve Product Quality


If the Intended Result is a Complex Concept with Multiple Components,

Measure it “Indirectly”

• If necessary, define components that could thoroughly describe the intended result

• Select and define the appropriate indirect measures


Identify those components by looking for related results or for contributing

factors


Indirect Measurement of the Intended Result May Be Necessary

• If the Intended Result can’t be directly measured, develop indirect measures based on a hypothesis around:

– Correlation with intended result : if desired result improves, the indirect measure does also

– Contribution to intended result : if A & B occur, the result will be C; or A might also be a driving factor in B


A

B

C


Process Flow Analysis

Cause-Effect Analysis or Ishikawa (Fishbone) Diagram

Use a Proven Methodology to Generate Candidate Measures


Process Output Input

Logic Model

Effect

Resources

Systems

Methods

Policies

Environment

Outcome


Consider the Purpose for Measuring Performance When Selecting Measures


Evaluate

Control

Budget Motivate

Promote

Celebrate

Learn Improve

Source: Adapted from Robert D. Behn, Why Measure Performance?, Public Administration Review, Sept./Oct. 2003, Vol. 63, No. 5.


Public Sector Managers Have Eight Purposes for Measuring Performance


Purpose Question Answered Common Measurement

Characteristics

Evaluate How well is the agency performing? Outcomes, combined with inputs

Control Are subordinates doing what is asked? Inputs that can be regulated; compliance

Budget What should we spend money on? Efficiency (outputs/inputs) or effectiveness (outcomes/inputs)

Motivate How can I motivate staff to improve? Real-time outputs compared with production targets

Promote How can I convince stakeholders that my agency is improving? How can I demonstrate transparency?

Easily understood; interesting and meaningful for the intended audience

Celebrate What accomplishments can we celebrate? Periodic and significant milestone targets

Learn What is working and what is not working? Disaggregated data that reveals deviances from the expected

Improve What should we do to improve performance? (the overriding high level purpose for all)

Those that connect the dots between operations and desired outcomes

Source: Adapted from Robert D. Behn, Why Measure Performance?, Public Administration Review, Sept./Oct. 2003, Vol. 63, No. 5.


Define and Document all Selected Measure(s)

• Use a Data Definition Table to capture and document all the essential information about the measure:



Step 5.3: Describe the Desired Performance Level and Interpretation

• Determine if “composite” measures are appropriate using this leading question: – Do the individual indicators from Step 5.2 offer useful

indicators of different dimensions or components of the Objective’s Intended Result?

• If so, a composite index will help capture various important dimensions of the Intended Result and combine them into a single measure

• Develop targets & thresholds for each measure



Targets & Thresholds Help With Interpretation

• Target: the desired level of performance for the reporting period in question

• Threshold: the upper and lower limits of desired performance around a target value (e.g., the exact point that we want an indicator to display green to indicate good performance, yellow to indicate satisfactory performance, or red to indicate poor performance)


Green / Yellow Threshold

Yellow / Red Threshold

Target


Exercise 2: Performance Measurement Development & Selection Quiz

• Answer each of the following questions regarding performance measurement development and selection.



• Organization: Leading Tablet/Technology Innovator

• Objective: Improve Product Development

• Intended Result: Voice-of-the-customer (VOC) research is consistently used in product development

• Q: What is the most appropriate measurement?

a) Revenue related to new products in $

b) % of customers that love our products

c) % of products where VOC research procedure was followed

d) % of employees with project management skills


Why? The other measures, while needed, probably focus on other objectives or intended results. Measure C DIRECTLY measures the intended result.


• Organization: Professional Sports Team

• Objective: Improve Player Skills & Abilities

• Intended Result: Our players are skilled in their respective roles

• Q: What is the most appropriate measurement? a) Sales of shirts with shirts with star player names printed on

back (based on a hypothesis around correlation)

b) Our players lead the league in key player statistics relative to their roles (correlation)

c) We win a lot of games (correlation)

d) We spend more on sabermetrics software (contributing factor)


Why? There are too many other factors for A to be a helpful measure. B focuses specifically on the outcome identified in the Intended Result. C is probably

covered in another objective. D is an input that doesn’t monitor follow-through.


Contribution Vs. Correlate Lemonade Stand Example


Strategy Map / Objectives Measures Targets

Financial

• Profit

• Enough to buy a bicycle by the end of summer

Customer

• Repeat business • 80% of target market (parents)

Internal Business Process

• Quality: Sugar content

• Image: “Awww” Index: % of drivers that smile

• 100% sugar content exceeds that of soda

• 80% smile

Organizational Capacity

• Skills: % Mom is angry due to wasted lemons

• Culture: Average age of lemonade stand workers

• 0% anger rate • < 6 years old

Increase Youthful Work

Culture

Improve Lemonade

Quality

Improve Customer

Satisfaction

Increase Profits

Improve Lemon-Squeezing Skills

Improve Brand Image

Profit is a ____ of Profit? a) direct measure of b) correlate of c) contributor to

Sugar content is thought to be a _____ of Quality? a) direct measure? b) correlate? c) contributor?

Mom’s anger is thought to be a _____ of Lemon Squeezing Skills? a) direct measure? b) correlate? c) contributor?


• Organization: Local Elementary School

• Objective: Improve School Technology Capabilities

• Desired Result: Children have access to the technology they will need to succeed in the world

• Q: Which of these is the most appropriate measure? a) # of computers donated to the school (based on a

hypothesis around contribution)

b) Network up time (contribution)

c) Budget spent on technology and staff training (contribution)

d) % of students with access to technology that is less than two years old (direct measure)


Why? It directly measures the Desired Result. The others are inputs that don’t monitor follow-through.


• Organization: State Literacy Program

• Objective: Improve Program Outcomes

• Desired Result: Literacy improves in the state

• Q: I’ve determined that the measurement purpose is promotion, as I need to communicate progress to the program sponsors and to external stakeholders. What is the most appropriate measurement?

a) # of teachers trained in literacy coaching

b) Change in state literacy rate since program was enacted

c) # of students that attend the program per week

d) % of budget focused on literacy programs


Why? It is outcome focused and easily understood, interesting and meaningful for the intended audience


Step Seven is Now Called “Performance Analysis”


7 Performance Analysis


There Are Many Steps to Turning Data Into Information and Knowledge


Source: Adapted from Performance Measurement Diagnosis PuMP Mind Map, by Stacey Barr and The Performance Measurement Cycle, Transforming Performance Measurement, Dean Spitzer.

Collect

Store

Analyze

Interpret / Present

Share / Create Dialog

Take Action

Continuously Review

Turn Data Into Information &

Knowledge


Take a Step-by-Step Approach to Performance Analysis/Analytics

1. Focus on the highest value opportunities first

– Look to Strategic Themes, Results and Strategy Map linkages for a starting point

2. Start with questions and decisions to make, NOT DATA

3. Drive data-driven insights deep into the organization

– Enable dialog

– Share, visualize & simulate

4. Expand (not replace) your analysis capacities

– Focus on both talent & culture

5. Use analysis to improve ongoing strategic planning


Adapted From: Analytics: The New Path to Value – How the Smartest Organizations Are Embedding Analytics to

Transform Insights Into Action, LaValle, Hopkins, Lesser, Shockley & Kruschwitz, MIT Sloan Management Review and

the IBM Institute for Business Value, Fall 2010.


Performance Data Can Analyzed for Actionable Correlations


5 unit increase

1.3 unit increase

0.5 % increase

15 months

Source: Harvard Business Review

Sears “mined” over 70 performance indicators for an actionable correlations. Twenty-two months after the analysis began, they identified three that were highly significant:

Employee Attitude

(Satisfaction)

Customer Impression

(Satisfaction)

Revenue Growth


Gwinnett County Schools Used Performance Data

to Improve Graduation Rates


Graduation

? Pass Algebra I

? Success at eighth-grade creative writing

In 2002, the Gwinnett Count (GA) public school system decided to use performance analytics to improve the (falling) graduation rates for at-risk kids.

Key Questions:

• What aspect of performance most accurately predicted graduation?

• What is the strongest predictor of passing Algebra I?

In Fall 2010, Gwinnett County won the Broad Prize for student achievement and improvement while narrowing achievement gaps between income and ethnic groups

Source: Big Data, Analytics and the Path from Insights to Value, LaValle, Lesser, Shockley, Hopkins & Kruschwitz, MIT Sloan Management Review, Winter 2011.


Other Examples of Performance Analysis Driving Better Decision Making

• Hilton Hotels found that a 5% improvement in customer retention would result in a 1.1% increase in revenue at a typical property*

• Best Buy discovered that a tenth-of-a point increase in employee engagement lead to an increase in store operating income by $100,000*

• The Baltimore Department of Works used data management to reduce absenteeism to reduce operating costs by $13.2m.**

• Victoria’s Secret found that raising its average conversion rate (entering customer who buy something) by 1% led to more than $35m in sales and more than $15m in profit.*

• Store 24, a convenience store, confidently abandoned its “entertaining service experience” strategy after finding a negative relationship between strategy implementation and profits.*


Sources: *The Rise of Analytical Performance Management, Thomas H. Davenport, Harvard Business Digital / SAS Institute. **The CItiStat Model-How Data-Driven Government Can Increase Efficiency & Effectiveness, Perez & Rushing, the Center for American Progress, April 2007.


Use Performance Analysis to Improve Ongoing Strategic Planning

• Use performance information to “manage to results” and inform decision making

• Look for new leading performance indicators – incorporate learning into strategy and business operations

• Communicate strategy effectively, internally and externally

• Periodically check reward and incentive systems for alignment with desired results

• Use Performance Information during Evaluation (step 9) to revise Strategic Themes, Strategic Results, Strategic Objectives, Strategy Maps, etc.

• Don’t be afraid to “test and learn”



Q&A: What Are the Issues You Are Facing in Performance

Measurement Development & Analysis?


Thank You!


Strategic Performance & Change Management Conference

Actuate - The BIRT Company™

Spider Strategies, Inc. Association for Strategic Planning Process Excellence Network Strategy Management Group

If you would like to receive the free IQPC brochure

directly email [email protected]

November 13-15, 2012 in New Orleans, LA

View the Strategic Performance

& Change Management program

here: http://tinyurl.com/8wxywgc



http://www.linkedin.com/in/tarynpharmaiq

http://www.performancemgmtconference.com/Event.aspx?id=803286&utm_campaign=Linkedin-SB2&utm_medium=announcement&utm_source=linkedin&utm_content=text&utm_term=Elainepodcast&MAC=SPCM_LNKSB2

http://www.processexcellencenetwork.com/&utm_campaign=Linkedin-SB2&utm_medium=announcement&utm_source=linkedin&utm_content=text&utm_term=Elainepodcast&MAC=SPCM_LNKSB2

http://www.strategyplus.org/&utm_campaign=Linkedin-SB2&utm_medium=announcement&utm_source=linkedin&utm_content=text&utm_term=Elainepodcast&MAC=SPCM_LNKSB2

http://www.spiderstrategies.com/tour/balanced-scorecard-software/&utm_campaign=Linkedin-SB2&utm_medium=announcement&utm_source=linkedin&utm_content=text&utm_term=Elainepodcast&MAC=SPCM_LNKSB2

http://www.actuate.com/&utm_campaign=Linkedin-SB2&utm_medium=announcement&utm_source=linkedin&utm_content=text&utm_term=Elainepodcast&MAC=SPCM_LNKSB2

mailto:[email protected]

http://www.performancemgmtconference.com/Event.aspx?id=803286&utm_campaign=Linkedin-SB2&utm_medium=announcement&utm_source=linkedin&utm_content=text&utm_term=Elainepodcast&MAC=SPCM_LNKSB2

http://www.processexcellencenetwork.com/&utm_campaign=Linkedin-SB2&utm_medium=announcement&utm_source=linkedin&utm_content=text&utm_term=Elainepodcast&MAC=SPCM_LNKSB2

http://www.strategyplus.org/&utm_campaign=Linkedin-SB2&utm_medium=announcement&utm_source=linkedin&utm_content=text&utm_term=Elainepodcast&MAC=SPCM_LNKSB2

http://www.spiderstrategies.com/tour/balanced-scorecard-software/&utm_campaign=Linkedin-SB2&utm_medium=announcement&utm_source=linkedin&utm_content=text&utm_term=Elainepodcast&MAC=SPCM_LNKSB2

http://www.actuate.com/&utm_campaign=Linkedin-SB2&utm_medium=announcement&utm_source=linkedin&utm_content=text&utm_term=Elainepodcast&MAC=SPCM_LNKSB2



http://www.linkedin.com/groups/Lean-Six-Sigma-37987


Advanced Performance Measurement Workshop Develop Measures That Drive Performance By David Wilsey

Documents

Transcript of Advanced Performance Measurement Workshop Develop Measures That Drive Performance By David Wilsey