My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed)...
Transcript of My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed)...
![Page 1: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/1.jpg)
Nicole Forsgren, PhDDirector of Organizational Performance & Analytics, Chef
My First Year at Chef:Measuring All the Things
@nicolefv
![Page 2: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/2.jpg)
@nicolefv@nicolefv
Who? Why? Huh?
![Page 3: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/3.jpg)
@nicolefv@nicolefv
I rub science on things
Who? Why? Huh?
Basically,
to make your Dev, ITOps and DevOpsbetter with metrics.
![Page 4: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/4.jpg)
@nicolefv@nicolefv
What Did I Do? And How Can You Do It Too?
aka The Talk Outline
• Where do I start?• What numbers should I think about?• What about benchmarking?• What else should I think about?
![Page 5: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/5.jpg)
@nicolefv@nicolefv
What Did I Do? And How Can You Do It Too?
aka The Talk Outline
• Where do I start?• What numbers should I think about?• What about benchmarking?
• What else should I think about?
![Page 6: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/6.jpg)
@nicolefv@nicolefv
Where Do I Start?
Talk about it!
• Start at the highest level.• Is measurement happening?
• Is metrics a thing?
![Page 7: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/7.jpg)
@nicolefv@nicolefv
Where Do I Start?
Talk about it!
• Start at the highest level. • Is measurement happening?
• Is metrics a thing?
• But all we do is fight! Er, talk in circles!• Metrics provide opportunity for communication
and alignment
![Page 8: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/8.jpg)
@nicolefv@nicolefv
Where Do I Start?
Talk about it!
• Start at the highest level. • Is measurement happening?
• Is metrics a thing?
• But all we do is fight! Er, talk in circles!• Metrics provide opportunity for communication
and alignment
• Uh… still not sure?• Ask around! What is important to people?• This can be done at ANY level of an organization
![Page 9: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/9.jpg)
@nicolefv@nicolefv
At Chef:
Lay some groundwork
• Data Driven, by @dpatil and @hmason• Free, and only 20-ish pages!
![Page 10: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/10.jpg)
@nicolefv@nicolefv
At Chef:
Lay some groundwork
• Data Driven, by @dpatil and @hmason• Free, and only 20-ish pages!
• Established Chef Scorecard 2015• Important milestones for every area of the business
• KEY! It must fit on one page. No cheating on font size.
• This can include things that are difficult or not currently
measured
![Page 11: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/11.jpg)
@nicolefv@nicolefv
Chef Scorecard 2015 (scrubbed)Ecosystem, Training, Community• “When I’ve got a problem, do I know to go to Chef” – awareness of what’s in Chef right
now• Community health index – reach (including international), depth
• Internal: Internal chef engineering• International infrastructure (legal)• Open source footprint: % of market learning and using Chef, % of market with Chef on
resume• How effective is Learn Chef?• # people who have been through Chef training
![Page 12: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/12.jpg)
@nicolefv@nicolefv
Chef Scorecard 2015 (scrubbed)Financial• Exit ARR: absolute, incremental by market (US, EMEA)• Product bookings: new (including enterprise logos), expansion, retention / churn • Gross Margin by product, service
• Services Revenue• Exceed cash plan
• Control expenses
Marketing• Conversions: Leads -> Opportunities, Leads -> Wins
• Clear Category Leader: DevOps, Automation, Compliance at Velocity • Strong and increasing external buzz on Chef - enterprise vs startup
• Consistency on how we present our products and our customer
![Page 13: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/13.jpg)
@nicolefv@nicolefv
Chef Scorecard 2015 (scrubbed)People• Chef is an innovative and industry-leading work environment• Chef attracts and retains top talent and uses our people as a competitive advantage • Diversity
• HR material is localized where appropriate
Sales & Business Development• Progress on solution road map: capability by platform
• Pipeline Growth: Size of opportunity pipeline (# adds by deal size) for next 12m m-o-m• Time to close (velocity) – broken down by stage
• New rep ramp-up time• Opportunities -> Wins
![Page 14: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/14.jpg)
@nicolefv@nicolefv
Chef Scorecard 2015 (scrubbed)Product & Engineering• Increase commercial adoption of premium features • Software developer life cycle - hiring, on-boarding, career progression, including diversity
goals
• Ship every day• Rebuild and reboot Hosted Chef• Customer satisfaction and ease of use aka "ease of getting going”
• Deliver against 2015 road map• Become industry reference for development of enterprise software at velocity
• Low touch conversion (download / OSS -> premium: time to value)
![Page 15: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/15.jpg)
@nicolefv@nicolefv
For each metric:
• Define• Set target• Measure – periodically
• Communicate
Metric Baseline Current Target
![Page 16: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/16.jpg)
@nicolefv@nicolefv
Your metrics aren’t set in stone.
Start with an MVP, then iterate and improve.
Toss what doesn’t work.
![Page 17: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/17.jpg)
@nicolefv@nicolefv
What Did I Do? And How Can You Do It Too?
aka The Talk Outline
• Where do I start?
• What numbers should I think about?• What about benchmarking?
• What else should I think about?
![Page 18: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/18.jpg)
@nicolefv@nicolefv
How to think about your world
What kinds of numbers should we be thinking about?
1. External: outward-facing, customer-focused2. Internal: inward-facing, process improvement
3. Cultural: because people matter
![Page 19: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/19.jpg)
@nicolefv@nicolefv
How to think about your world
What kinds of numbers should we be thinking about?
1. External: outward-facing, customer-focused2. Internal: inward-facing, process improvement
3. Cultural: because people matterEXTRA CREDITAlso think about: • Objective v. Subjective• Leading v. Lagging
![Page 20: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/20.jpg)
@nicolefv@nicolefv
Where should I start?
Start with your IMPORTANT THINGS to my organization / team list.
![Page 21: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/21.jpg)
@nicolefv@nicolefv
Where should I start?
Start with your IMPORTANT THINGS to my organization / team list.
Could include things like:• Maintain excellent customer satisfaction
• Increase speed of software delivery• Hit revenue targets• Increase software quality or usefulness
![Page 22: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/22.jpg)
@nicolefv@nicolefv
Where should I start?
Start with your IMPORTANT THINGS to my organization / team list.
Could include things like:• Maintain excellent customer satisfaction
• Increase speed of software delivery• Hit revenue targets
How does this apply to me?
![Page 23: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/23.jpg)
@nicolefv@nicolefv
Where should I start?
Start with your IMPORTANT THINGS to my organization / team list.
Could include things like:• Maintain excellent customer satisfaction
• Increase speed of software delivery• Hit revenue targets
How does this apply to me?
• Gives insight into metrics important to other areas
• Identifies potential external metrics
• Focuses efforts on value-add metrics
![Page 24: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/24.jpg)
@nicolefv
An Example at Chef:Engineering initiative
![Page 25: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/25.jpg)
@nicolefv@nicolefv
Where should I start?
Identify a goal
• Is there existing data?•Who are the key people?•What things are important to support this goal?
![Page 26: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/26.jpg)
@nicolefv@nicolefv
Chef Scorecard 2015 (scrubbed)Product & Engineering• Increase commercial adoption of premium features • Software developer life cycle - hiring, on-boarding, career progression, including diversity
goals
• Ship every day• Rebuild and reboot Hosted Chef• Customer satisfaction and ease of use aka "ease of getting going”
• Deliver against 2015 road map• Become industry reference for development of enterprise software at velocity• Low touch conversion (download / OSS -> premium: time to value)
![Page 27: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/27.jpg)
@nicolefv@nicolefv
Where should I start?
Identify a goal: SHIP EVERY DAY
• Is there existing data?•Who are the key people?•What things are important to support this goal?
![Page 28: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/28.jpg)
@nicolefv@nicolefv
At Chef: Engineering initiative
Identify a goal: SHIP EVERY DAY
• Test Maturity • Unit tests• Component tests• Integration tests• Upgrade tests• Compatibility tests• Ancillary tests
![Page 29: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/29.jpg)
@nicolefv@nicolefv
At Chef: Engineering initiative
Identify a goal: SHIP EVERY DAY
• Test Maturity • Unit tests• Component tests• Integration tests• Upgrade tests• Compatibility tests• Ancillary tests
The data• After investigation, no consistent sources of
objective data available• Initial steps: Subjective data collected• Interviews conducted monthly with team
leads of each product• Percentage complete for each area
calculated and communicated
![Page 30: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/30.jpg)
@nicolefv@nicolefv
At Chef: Engineering initiative
Identify a goal: SHIP EVERY DAY
• Test Maturity • Unit tests• Component tests• Integration tests• Upgrade tests• Compatibility tests• Ancillary tests
Integration Tests: Software tested as part of the distributed application ecosystem prior to release.• Are acceptance tests reviewed and defined as part of feature
definition? • Is acceptance manually reviewed prior to shipping by stakeholders? • Are acceptance tests automated? • Are acceptance test results reviewed as part of the release
process?• Are tests regularly reviewed for accuracy? • Are non-valuable tests deleted (or reviewed for deletion)?• Is there an explicit limit for how long your entire component test
suite will take? • Are tests reviewed for duration?
![Page 31: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/31.jpg)
@nicolefv@nicolefv
At Chef: Engineering initiative
Charts for Test Maturity within a Product
![Page 32: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/32.jpg)
@nicolefv@nicolefv
At Chef: Engineering initiative
Charts for ALL Test Maturity for each Product or Feature
![Page 33: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/33.jpg)
@nicolefv@nicolefv
What Did I Do? And How Can You Do It Too?
aka The Talk Outline
• Where do I start?• What numbers should I think about?
• What about benchmarking?• What else should I think about?
![Page 34: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/34.jpg)
@nicolefv@nicolefv
A word about Benchmarking:
A Benchmark is ESSENTIAL
MUST HAVES:• Truth! Even a bad baseline is good• At least ONE reference group•Within team•Other: Other team? Whole company? Industry?
•Communicated / visible
![Page 35: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/35.jpg)
@nicolefv@nicolefv
At Chef: Engineering initiative
Charts for ALL Test Maturity for each Product or Feature
Even a bad baseline is good
At least one reference group
✔ Truth!✔ Communicated / visible
![Page 36: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/36.jpg)
@nicolefv@nicolefv
Your metrics aren’t set in stone.
Start with an MVP, then iterate and improve.
Toss what doesn’t work.
![Page 37: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/37.jpg)
@nicolefv@nicolefv
At Chef:
Identify a goal: SHIP EVERY DAY
• Test Maturity • Unit tests• Component tests• Integration tests• Upgrade tests• Compatibility tests• Ancillary tests
1. External 2. Internal3. Cultural
![Page 38: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/38.jpg)
@nicolefv@nicolefv
At Chef:
Identify a goal: SHIP EVERY DAY
• Test Maturity • Unit tests• Component tests• Integration tests• Upgrade tests• Compatibility tests• Ancillary tests
1. External 2. Internal3. Cultural
![Page 39: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/39.jpg)
@nicolefv@nicolefv
Westrum cultural measurePathological Power-oriented
BureaucraticRule-oriented
GenerativePerformance-oriented
Low cooperation Modest cooperation High cooperation
Messengers shot Messengers neglected Messengers trained
Responsibilities shirked Narrow responsibilities Risks are shared
Bridging discouraged Bridging tolerated Bridging encouraged
Failure leads to scapegoating
Failure leads to justice Failure leads to inquiry
Novelty crushed Novelty leads to problems Novelty implemented
![Page 40: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/40.jpg)
@nicolefv@nicolefv
Westrum cultural measurePathological Power-oriented
BureaucraticRule-oriented
GenerativePerformance-oriented
Low cooperation Modest cooperation High cooperation
Messengers shot Messengers neglected Messengers trained
Responsibilities shirked Narrow responsibilities Risks are shared
Bridging discouraged Bridging tolerated Bridging encouraged
Failure leads to scapegoating
Failure leads to justice Failure leads to inquiry
Novelty crushed Novelty leads to problems Novelty implemented
15% 52% 33%
![Page 41: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/41.jpg)
@nicolefv@nicolefv
At Chef:Some teams have started measuring Westrum Culture scores quarterly. It is also included in our yearly employee survey.
Rate how strongly you agree (7) or disagree (1) to the following statements:1. On my team, information is actively sought.2. On my team, failures are learning opportunities, and messengers of them are
not punished.3. On my team, responsibilities are shared.4. On my team, cross-functional collaboration is encouraged and rewarded.5. On my team, failure causes enquiry. 6. On my team, new ideas are welcomed.
![Page 42: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/42.jpg)
@nicolefv@nicolefv
Summary of Engineering Initiative inside of ChefCollect 3 types of metrics:• External• Internal• Cultural
Benchmark metrics
Consider:• Subjective v. Objective
• Leading v. Lagging
Initial metrics efforts:• Ship Every Day (among others)• Test Maturity (and components)• Westrum culture
Benchmarked ✔
Next steps as we iterate:• Tooling for Test Maturity (Subj -> Obj)
• Most metrics are lagging (normal). Westrum is leading & lagging. Consider adding WIP limits (leading)
![Page 43: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/43.jpg)
@nicolefv
Another example at Chef: External Customers
![Page 44: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/44.jpg)
@nicolefv@nicolefv
Where should I start?
Identify a goal
• Is there existing data?•Who are the key people?•What things are important to support this goal?
![Page 45: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/45.jpg)
@nicolefv@nicolefv
Chef Scorecard 2015 (scrubbed)Product & Engineering
• Increase commercial adoption of premium features • Software developer life cycle - hiring, on-boarding, career progression, including diversity
goals• Ship every day
• Rebuild and reboot Hosted Chef• Customer satisfaction and ease of use aka "ease of getting going”
• Deliver against 2015 road map• Become industry reference for development of enterprise software at velocity• Low touch conversion (download / OSS -> premium: time to value)
![Page 46: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/46.jpg)
@nicolefv@nicolefv
Where should I start?
Identify a goal: INCREASE COMMERCIAL ADOPTION OF PREMIUM FEATURES
• Is there existing data?•Who are the key people?•What things are important to support this goal?
![Page 47: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/47.jpg)
@nicolefv@nicolefv
At Chef: External customer initiative
Goal: INCREASE COMMERCIAL ADOPTION OF PREMIUM FEATURES
Success gridbecomesDojo = DevOps JourneyAssessment
The data• After investigation, SalesForce data was not always
consistent• Premium Features were being reported as a count
– and nothing else• Technical Account Managers and Customer
Success Engineers knew that a more holistic view was more meaningful
• New data is collected as part of a structured interview with customers. They see their profile emerge.
![Page 48: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/48.jpg)
@nicolefv@nicolefv
At Chef: External customer initiative
Goal: INCREASE COMMERCIAL ADOPTION OF PREMIUM FEATURES
Dojo = DevOps JourneyAssessment
Key areas for new, holistic approach• Organizational success factors• Local development• Code collaboration• Continuous integration• Chef code deployment• Application deployment• Virtualization as a Service• Full stack automation• Continuous deployment
![Page 49: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/49.jpg)
@nicolefv@nicolefv
At Chef: External customer initiativeCharts for DevOps Journey Assessment
![Page 50: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/50.jpg)
@nicolefv@nicolefv
At Chef: External customer initiativeCharts for DevOps Journey Assessment
✔ Truth!✔ Communicated /
visible to customersAt least one
reference group
Even a bad baseline is good
![Page 51: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/51.jpg)
@nicolefv@nicolefv
Summary of External Customer Initiative at ChefCollect 3 types of metrics:• External• Internal
• Cultural
Benchmark metrics
Consider:• Subjective v. Objective• Leading v. Lagging
Initial metrics efforts:• Organizational success factors • Local dev, CI, CD, Chef deployment, Application
deployment, VaaS, Full stack automation
• Code collaboration, Org success factors
Benchmarked ✔• Next steps: create customer benchmark
Next steps as we iterate:• Data will remain subjective for now• Most metrics are lagging (normal).
• Investigate cultural metrics and use of other leading metrics
![Page 52: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/52.jpg)
@nicolefv@nicolefv
What Did I Do? And How Can You Do It Too?
aka The Talk Outline
• Where do I start?• What numbers should I think about?
• What about benchmarking?
• What else should I think about?
![Page 53: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/53.jpg)
@nicolefv@nicolefv
What else should I think about?
Advanced Topics
• Measurement targets should use distributions of probable outcomes. Because targets should include discussions of likelihood.
• Discuss variance of metrics, especially ones thought to influence things. This speaks a lot to the consistency of your process and work.
• Don’t ignore the rate of change in your data. Change is good, yes, but how is it changing? This yields valuable insights.
![Page 54: My First Year at Chef: Measuring All the Things · @nicolefv Chef Scorecard 2015 (scrubbed) Ecosystem, Training, Community • “When I’ve got a problem, do I know to go to Chef”](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec984b7d46526441e3137cf/html5/thumbnails/54.jpg)
@nicolefv@nicolefv
What else should I think about?
Advanced Topics
• Be careful about the use of normalizations in your reporting and metrics. Normalizations are useful for comparison across groups. Improper use can lead to improper smoothing (e.g., a defect is still a defect and someone will still feel pain).
• Distributions in operations and development are rarely normal, so the use of means and standard distributions are inappropriate. Consider the use of means (or transform the underlying distribution first.)