Defining and Evaluating Success: Metrics and Metric Frameworks for Information Architects

IBM Total Information Experience

Defining and Evaluating Success: Metrics and Metrics Frameworksfor Information Architects

Andrea L. AmesInformation Experience Strategist/Architect/Designer

Alyson RileyTIE Content Strategist

STC Webinar27 June 2013

© IBM Corporation 2013. All Rights Reserved

2


About AndreaTechnical communicator since 1983

Areas of expertiseInformation experience design: Content strategy, information architecture, and interaction design for content display and delivery, within products and interactive information delivery systems

Architecture, design, and development of embedded assistance (content within or near the product user interface)

Information and product usability, from analysis through validation

User-centered process for information development and information experience design

IBM Senior Technical Staff Member on corporate Total Information Experience team in IBM CIO’s office

University of CA Extension certificate coordinator and instructor

STC Fellow, past president (2004-05), former member of Board of Directors (1998-2006), and Intercom columnist (with Alyson Riley) of The Strategic IA

ACM Distinguished Engineer 2

3


3

About Alyson

Technical communicator since 1995

Areas of expertise

Content strategy

Content metrics—the business value of content

Information architecture (my first love!)

Interaction design for content delivery vehicles, and interactive content

Information and product usability, from analysis through validation

User-centered processes for content strategy and scenario-driven information architecture

IBM Senior Content Strategist on corporate Total Information Experience team in the IBM CIO Office

Member of STC, and Intercom columnist (with Andrea Ames) of The Strategic IA

4


Agenda

Telling the right story through metrics

Telling the right story to the right audience

The importance of a closed-loop metrics process

Evaluation frameworks

Building an evaluation framework

Questions

5


Jared Spool says it SO well…

Metric: Something you can measureThe time a user spends on a Web site

Whether a particular Web page has a high “bounce rate” (indicates it’s often the last page users visit)

How much revenue a Web site makes

How many times the letter E is used on a Web site home page

All of these are measurable…BUT…being measurable doesn’t make those measures informative or useful

6


We care, because…

An information architect is (among other things) a story-teller:

Define the right vision

Tell a compelling, true story that inspires people to buy in

What makes a story true? Facts—things you can prove.What makes a story compelling? It speaks to what matters most.

What matters most? Depends on your audience. Duh, right?

Prove the value of information architecture and content with metrics

Value is in the eye of the beholder.

Who’s your “beholder?” Understand who your beholders actually are—that is, the real decision-makers and influencers in your world. (Manage your stakeholders!)

Use metrics that target actual decision-makers.

Your actual decision-makers are probably business people—executives, managers, and others who hold the purse-strings.

Figure out what your audience values—their metrics for success.

7


Problem: Metrics have gotten a bad rap

Numbers can be hard for word people

The right numbers are hard for everyone

Getting metrics to work for you requires a significant shift in thinking

Solution: Rethink metrics Metrics are another form of audience analysis (who cares about what?)

Metrics are another form of usability testing (what works for whom?)

Motivation for change: Metrics are a powerful tool for getting what you want (and making sure you want the right things)

Metrics transform opinion into fact

Metrics remove emotion from analysis

Strategize with metrics: Use metrics at every project phase Beginning: identify opportunity, prove the strategy is right

Middle: show incremental progress, course-correct

End: prove value and earn investment for the future

8


Be sure you’re talking to the right audience

Audience 1: Business peopleUnless you can make a direct connection between your IA metrics and the metrics that drive business, you are telling the wrong story for this audience.

You need this audience! The business community funds us. We have to sell our vision to them, with a metrics story that resonates with them.

We must learn to speak “business”—that is, prove the value of content and the information experience using metrics that matter to business.

Audience 2: Content peopleTypically many kinds of content people will help implement an information architecture—your work may span departments and business units.

Content people tend to reflect the values of their leadership and business unit in which they’re located.

This means that even kindred spirits—other content people—can have widely different goals and metrics.

Your job is to define common ground by speaking to what matters most to this audience, too.

9


Selling information architecture to a business audience

The metrics that we use to build effective content strategies don’t resonate with most executives, managers, and finance people.

Sometimes we “talk to ourselves”—that is, use metrics that resonate with content people, not the actual people we need to support our strategy.

“Page hits” resonate with us. “Sales leads” resonates with business.

You cannot directly connect things like page hits and bounce rates to core business metrics.

You need an informational professional’s intuition to know how content supports business metrics—most business people don’t have that intuition.

The business audience funds us. We have to sell our vision and prove our value to them, with a metrics story that speaks to what they care about most.

Examplebusiness metrics:

Revenue streams

Sales leads

Cost per lead

Customer satisfaction

Customer loyalty

Return on investment (ROI)

Time to value

Market share

Mindshare

10


Selling content strategy to a content audience

Analyze each team that contributes to your information architecture

In what business unit are they located?

Who are their executives, sponsors, and stakeholders?

Who “grades them” on their performance?

Who funds them?

What matters to them?

How do they measure their progress or results?

What are they doing well (both in your analysis and theirs)?

Where can they improve (both in your analysis and theirs)?

Identify areas of similarity and differenceWhere do their goals align with yours? build bridges!

Where do their goals conflict with yours? build business cases!

Use metrics to craft a story that:

Shows problems and opportunities that the content team cares about

Maps in key areas to their goals for content

Diverges from their current goals in ways that would increase their value to sponsors and stakeholders

11


Identify and begin managing stakeholders

Your best political asset—your stakeholders!

A rigorous stakeholder management process will help you take rigorous advantage of this key asset

Think through the ways that your stakeholders can help you—start by identifying and analyzing:

Their status relative to your project—advocate, supporter, neutral, critic, blocker

Their top interests and hot issues

Their key performance indicators (KPIs) and metrics

The level of support you desire from them

The role on your project that you desire for them

The actions that you want them to take (and their priority)

The messages that you need to craft for them to enable the outcome you want

The actions and communication that you need to make happen with each stakeholder to achieve your desired outcome

Keep your stakeholder management plan current

“Stakeholder management is

critical to the success of every project in every

organization … By engaging the right people in the right

way in your project, you can make a big

difference to its success...

and to your career.” —Rachel Thompson

Source and free stakeholder management

worksheet here:Thompson, Rachel.

Stakeholder Management:

Planning Stakeholder

Communication. MindTools. Web.

12 April 2013. http://bit.ly/8UnUdj

12


Metrics for an audience of business people (stakeholders)

Use the research you did during the today-state analysis phase

Target the key decisions-makers—those who hold the purse-strings

Identify what the key business decision-makers care about

Use language that resonates with that business audience

Remember: unless you can tie a particular goal or result to a measurement that the stakeholder cares about, that result ultimately doesn’t matter

Stakeholder

Example metrics

Marketing Executive

ROI Cost per lead Campaign performance Conversion metrics

SalesExecutive

Viable leads Sales growth Product performance

SupportExecutive

Call volume Call length Customer satisfaction Ticket deflection

DevelopmentExecutive

Development costs Market share Lines of code Compliance Quality and test results

13


Metrics for an audience of content people

Map the people who deliver content to the metrics they care about

Remember that each content team has their own decision-makers who:

Approve their goalsDetermine their fundingDetermine their futures

Stakeholder

Example metrics

Example associated content teams

Example content metrics

Marketing Executive

ROI Cost per lead Campaign

performance Conversion

metrics

Web team Social team Event team

Web traffic Click-throughs Likes and shares Conversions Collateral distributed Cost per unit

produced

SalesExecutive

Viable leads Sales growth Product

performance

Sales enablement Education &

training Beta programs

Proofs of Concept (PoCs) to sale

Number of classes Beta program

participants Cost per unit

produced

SupportExecutive

Call volume Call length Customer sat. Ticket

deflection

Web support team Call center team

Amount of web information produced

Number of calls reduced

Time of calls reduced Cost per unit

produced

DevelopmentExecutive

Dev cost Market share Lines of code Compliance Quality and

test

Product documentation team

Developers who publish whitepapers and case studies

Product community forums and wikis

Lines of text, number of pages, etc.

Cost per unit produced

Web traffic Number of forum

participants

14


Metrics-based goals

So what are the goals for your information architecture? Express those goals in the form of business metrics and content metrics. Some examples:Business metrics Sample IA metrics Sample content goals

Purchase decisions

(revenue)

Reach—visits, etc. Engagement—

referrals, etc.

Contribute to revenue stream through referrals from technical content that become sales leads.

Product quality

(customer loyalty)

Reach—visits, etc. Engagement—

referrals, etc.

Contribute to product quality through by simplifying the amount of content in the user experience.

Customer satisfaction

(ROI)

Web traffic Direct feedback Ratings Shares (social)

Create high value content that speeds customer time to success.

Perceptions of company (mindshare)

Sentiment—nature of social dialogue, etc.

Direct feedback

Create high quality, highly usable content delivered in an elegant information experience.

15


Critical: A closed-loop process

Closed loop: end up at the beginning!Start with metrics—use at project outset to:

Identify problems and opportunitiesDefine the visionProve that the vision is right

Continue with metrics—use during implementation to:Measure the success of your progress in small incrementsStay on-target through implementationDetermine when it’s time to course-correct (before change gets expensive)Keep your sponsors and stakeholders engaged throughout the long haulEnsure that you remain connected to the broader goals and metrics of the surrounding businessEnsure that you stay responsive and adapt to change

End with metrics—use at project conclusion to:Prove the business value of information architectureProve the business value of your work—enhance your credibility and careerEncourage future investment in information architecture

16


How do you close the loop?

Measure what you’ve said you’ll measure, of course…

Build a framework, aka, scorecard, to evaluate how you’re doing

Before you begin

Identify what to measure and collect

Identify and begin managing stakeholders!

1. Determine types of metrics to include in framework *

2. Normalize “results” to “scores” *

3. Categorize and weight metrics *

4. Create summaries *5. Validate the framework *

6. Launch! *

* Stakeholder-management point

17


1. Determine quantitative metrics

2. Determine qualitative metrics

3. Determine compliance items1Determine metric types

18


What types of metrics are you measuring?

QuantitativeDescribe the what, or how many of the what

Can be measured with numbers—absolutely, mathematically

Examples: See the last three charts Qualitative

Describe intangibles, like the why

Non-numerical

Examples: Sentiment, how important something is, how much the respondents like something, how likely they are to recommend

Compliance itemsQuantitative—you can count them

But you’re only counting “1”

More important is the perceived value of that “1”

Examples: Models applied, templates used, processes followed

19


A simple example: Running

Quantitative

Time

Distance

Heartrate

Compliance

Doping

Qualitative

How I feel at start of run

How I feel at end of run

How the weather conditions were

20


1. Evaluate qualitative metrics – are scales the same

2. Evaluate quantitative metrics – how good is good

3. Evaluate compliance items – how important is it

4. Select a scale• Numerical

• Good/bad

• High/med/low (e.g., priority)

• Crawl/walk/run

Make room for an “innovation” or “exemplary” level—the “top” of your scale should be attainable by everyone!

Consider:

Do teams have a way to be successful from where they are? (e.g., can they reasonably get to crawl?)

Will you be able to measure incremental improvement

2Normalize results to scores

21


Qualitative metrics: Are the result scales the same?


Horrible OK Good Great!



What the weather conditions were*

Freezing Cold Cool Warm/humid Hot/humid

* More accurate to measure temperature and humidity as quantitative metrics

22


Qualitative metrics: Better!





What the weather conditions were*

Horrible OK Good Great!(Hot/humid or freezing cool/dry)

* More accurate to measure temperature and humidity as quantitative metrics

23


Quantitative metrics: How good is “good?”

Time

Beginner: 15 minute mile

Intermediate: 11 minute mile

Athlete: 8 minute mile

Distance: % of goal

25-50%

50-75%

75-100%

Heartrate

Below fat-burning zone

Fat burning zone

Aerobic zone

24


Quantitative metrics: Normalized

TimeHorrible (implied) More than 15-minute mile

OK Beginner: 15 minute mile

Good Intermediate: 11 minute mile

Great Athlete: 8 minute mile

Distance: % of goalHorrible (implied) Less than 25%

OK 25-50%

Good 50-75%

Great 75-100%

Heart rateHorrible (implied) No heart rate OK Below fat burning zone

Good Fat burning zone

Great Aerobic zone

25


1. Determine what to measure by considering:

– Who are your stakeholders?

– What is important to them?

– What are your goals?

– What will best help you tell your story to get your stakeholders aligned around your goals?

2. Use your IA brain; this is a classic IA problem.

3Categorize and weight scores

26


A possible categorization scheme

Think about how you want to tell your story for most impact

What is your clients’ product-use life cycle?

Other possible schemes

By function (to make specific business cases to them)

By “standard” (to understand specific compliance issues)

A generalized view ofIBM’s product-use lifecycle

27


1. Determine what summaries you need to tell the right story at a glance

• For management

• For the larger organization

• For the team being measured

2. Be careful not to create an unwieldy mess

4Create a summaries

28


This could be an entire session by itself!

29


1. Find as many diverse “products” as possible to measure.

2. Gut check with several “independent” IAs• Individual metrics

• Categories

• The whole

3. Compare results across products for consistency

4. Compare results to uncover gaps: Is something note being measured that should be?

5. If possible, find other frameworks to compare to your approach—not necessarily an IA framework…maybe a user experience or support framework

5Validate framework

30


1. Determine how the framework will be used

• As a self-evaluation tool?

• In an objective, outside evaluation?

• Do users need to be “certified” to use it? Or evaluators to evaluate with it?

2. Communicate!

3. Educate!

4. Consult?

6Launch!

31


Questions?

32


References

Jared Spool: KPIs are metrics, but not all metrics are KPIs from UIE Brainsparks blog, Oct 5, 2012: http://bit.ly/VFYvF2

Bhapkar, Neil. 8 KPIs Your Content Marketing Measurements Should Include. Content Marketing Institute. Web. 12 April 2013. http://bit.ly/Wnb7Cy

Klipfolio. The KPI Dashboard—Evolved. Web. 12 April 2013. http://bit.ly/LhzeL9

Muldoon, Pamela. 4 metrics every content marketer needs to measure: Interview with Jay Baer. Content Marketing Institute. Web. 12 April 2013. http://bit.ly/X8IvMJ

Thompson, Rachel. Stakeholder management: Planning stakeholder communication. MindTools. Web. 12 April 2013. http://bit.ly/8UnUdj

34


Proving the business value of content—IBM example

Shameless ad:

Watch for the May issue of

STC’s Intercom magazine for a new article that

we wrote on proving the

business value of content.

At IBM, we’re learning to tell a better story for a business audience

We conducted a survey from 2010-2012 with clients and prospective clients about the value of content—here’s the hot-off-the-press data:

35


A map of content metrics to stakeholder metrics

Tie your IA metrics to the metrics that matter most to your stakeholders so you can tell a story that inspires the outcomes you want.

This means researching how content influences the metrics that are most important to the specific people you need for success.

Start your research with these hints:How does content speed user success and time-to-value?

direct link to customer value

How does content drive purchase decisions?

direct link to the revenue stream

How does content impact product quality?

direct link to customer loyalty

How does content influence customer satisfaction?

direct link to ROI

How does content shape clients’ perceptions of your company?

direct link to mindshare

36


Evaluation frameworks: Heuristics and heuristic evaluations

HeuristicsRules of thumb

No one single “rule” or set of “rules”

Heuristic evaluationsInspection method

Performed by “experts”

Based on heuristics

Both: Based on principles derived from observing and analyzing user actions and intentions

These principles enable development of your own evaluation framework, based on your own user data

Defining and Evaluating Success: Metrics and Metric Frameworks for Information Architects

Self Improvement

Transcript of Defining and Evaluating Success: Metrics and Metric Frameworks for Information Architects