Measuring Gov 2.0 (But First, Lessons from Gov 1.0!)

Andrew Krzmarzick Senior Project Coordinator

The Graduate Schoolhttp://www.graduateschool.edu

http://generationshift.blogspot.comTwitter: @krazykriz

AGENDA

Gov 1.0 Measurement

Advancing Social Media Survey

Gov 2.0 Measurement Hallmarks

3

4

“In many ways, the Internet more resembles an ancient bazaar…”

5

6

Ordinary people…possess information – serious, expert, fact-based, scientific information.

[They] are ready and willing to share that information across geographic, disciplinary, and institutional boundaries.

- Beth Noveck, Wiki Government

GOV 1.0 MEASUREMENT

1. Brookings

2. Forrester

3. OMB’s E-Gov Initiative

4. ForeSee

GOV 1.0 MEASUREMENT

1. Brookings Institution E-Government Study

Since 2000:

• Reviewed > 1,500 State / Federal Websites

• 0 – 100 Point Scale

GOV 1.0 MEASUREMENT

1. Brookings Institution E-Government Study

4 points per feature (72 total possible points) + 28 more for frequency

4 points

4 points

4 points

4 points +2

+1

GOV 1.0 MEASUREMENT

1. Brookings Institution E-Government Study

You got it…or you don’t.

GOV 1.0 MEASUREMENT

1. Brookings Institution E-Government Study

Applied to Gov 2.0:

Should we count the number of social

media tools an agency deploys?

GOV 1.0 MEASUREMENT

2. Forrester Website Benchmark Survey

User Goals

Value

Navigation

Presentation

Trust

Point Scale2 = Strong Pass (Best Practice)

1 = Pass (No problems)

-1 = Fail (One major or many minor problems)

-2 = Strong Fail (2+ Major…)

GOV 1.0 MEASUREMENT

2. Forrester Website Benchmark Survey

Value

• Is essential content available where needed?

• Is essential function available where needed?

• Are essential content and function given priority in display?

GOV 1.0 MEASUREMENT

2. Forrester Website Benchmark Survey

Navigation

• Are menu categories/sub-categories clear?

• Is wording in hyperlinks clear and informative?

• Are keyword searches comprehensive and precise?

GOV 1.0 MEASUREMENT

2. Forrester Website Benchmark Survey

Presentation

• Does site use language that’s easy to understand?

• Does the site use graphics, icons, and symbols “ “ ?

• Is text legible? Text format/layout support easy scanning?

• Are form fields and interactive elements placed well?

GOV 1.0 MEASUREMENT

2. Forrester Website Benchmark Survey

Trust

• Does the site present privacy and security policies?

• Do location cues orient the user?

• Is contextual help available at key points?

• Does the site help users avoid and recover from errors?

GOV 1.0 MEASUREMENT

2. Forrester Website Benchmark Survey

Trust theexperts?

GOV 1.0 MEASUREMENT

2. Forrester Website Benchmark Survey

Applied to Gov 2.0

Blog Review Tool?

Tool Placement?

GOV 1.0 MEASUREMENT

3. OMB E-Government Initiatives

E-Government Act of 2002

“a broad framework of measures

that require using Internet-based IT

to enhance citizen access

to Government information and services...”

WEB 1.0 MEASUREMENT

3. OMB E-Gov Initiatives

WEB 1.0 MEASUREMENT

3. OMB E-Gov Initiatives

• Business.gov

• Regulations.gov

• USALearning.gov

• Grants.gov

• FedBizOpps.gov

• Recreation.gov

• USAJOBS.gov

As of January 2009: • 28/28 agencies had implementation plans

• 87% of milestones met

WEB 1.0 MEASUREMENT

3. OMB E-Gov Initiatives

d) Improve E-Gov Initiatives Performance Measures• Adoption/Participation – Is the relevant community participating?

• Usage – What’s the level of use by the target community?

• Customer Satisfaction – Is the community satisfied w/ products/services?

• Cost Savings/Avoidance – What’s $ value for government /citizens?

• Efficiency - Any decreases in time and/or increases in productivity?

WEB 1.0 MEASUREMENT

3. OMB E-Gov Initiatives

Peer Pressure?

GOV 2.0 DASHBOARD?

GOV 2.0 DASHBOARD?

GOV 1.0 MEASUREMENT

4. ForeSee: E-Government Satisfaction Index

GOV 1.0 MEASUREMENT

4. ForeSee: E-Government Satisfaction Index

E-Gov – What is it?

• Measures 94 e-gov sites on:

1. Navigation

2. Functionality

3. Search

4. Look and Feel

• Key question:

How satisfied are citizens?

• University of Michigan’s

American Customer

Satisfaction Index (ACSI)

GOV 1.0 MEASUREMENT

4. ForeSee: E-Government Satisfaction Index

E-Gov Outcomes as of Q1 2009

• All-time high: 74.1% satisfaction overall

• Most successful sites:

• Citizens find information quickly and easily

• E-Commerce and Transaction functions

WEB 1.0 MEASUREMENT

3. ForeSee: E-Government Satisfaction Index

E-Gov Top Performers

• http://www.ssa.gov/estimator (Score: 89)

• http://medlineplus.gov (Score: 86)

• http://www.niams.nih.gov/index.htm (Score: 82)

• http://www.cia.gov/employment (Score: 81)

WEB 1.0 MEASUREMENT

4. ForeSee: E-Government Satisfaction Index

The customers’

always right?

WEB 1.0 MEASUREMENT

4. ForeSee: E-Government Satisfaction Index

Applied to Gov 2.o

Add/Alter a series of questions related to

social media?

Real-time polling data (shared right away)?

“ADVANCING SOCIAL MEDIA” SURVEY

• 10 questions

• 4,000 potential respondents• Web Manager’s Forum• GovLoop• GovTwit Directory•International Contacts

• 105 responses

• 7 countries

“ADVANCING SOCIAL MEDIA” SURVEY

Questions1. What enables or hinders you from using social media?

2. What social media tools does your agency use?

3. Rate tools per value/importance in achieving mission.

1. Do you establish metrics prior to implementation?1. If yes, for which tools and what variables do you measure?

2. Is privacy, security and monitoring social tools important?

3. How often do you use social media in your job?

4. Thoughts re: gov standards w/browsers, software, etc.?

5. Is CTO/CIO actively involved in social media initiatives?

6. Where are you from/what agency do you represent?

Advancing Social Media in Government

Do you establish metrics prior to implementing any of the above social media tools at your agency?Answer Options Response

FrequencyResponse

Count

Yes 43.8% 46

No 56.2% 59

answered question 105

skipped question 0

“…we don't have benchmark data, nor do we have measurable objectives

for any of our tools.

We basically just put them out there

and hope they work.

It's kind of annoying.”

“ADVANCING SOCIAL MEDIA” SURVEY

Advancing Social Media in Government    

If you answered YES for #4, what tools are you measuring?

  Answer Options Response Frequency

Response Count

1  Blogs 60.4% 292  E-mail 58.3% 283  Twitter 56.3% 274  RSS 43.8% 215  YouTube 33.3% 166  Facebook 31.3% 157  Podcasts 25.0% 128  Mobile devices 16.7% 89  Wikis 14.6% 710  Delicious 8.3% 411  Z-other social networking (e.g. Flickr, Govloop, LinkedIn, Ning) 8.3% 4

12  LinkedIn 6.3% 313  Z-other video sharing (e.g. Hulu, Vimeo, Viddler) 6.3% 314  Z-other social bookmarking (e.g. Digg, StumbleUpon) 4.2% 215  Gaming 2.1% 116  MySpace 2.1% 117  Second Life 0.0% 018  Z-other virtual worlds/3D Web (e.g. YooWalk) 0.0% 0  answered question 48  skipped question 57

GENERAL • Views

Number of views/page Time on site/page

• Visitors Unique Overall Paths to site Paths on site Geographic distribution Searches Satisfaction Comments

Google Analytics

WIKIS•Users•Adoption rate•Edits

BLOGS• Posts

• Views• Comments (quantitative)

• People• Complaints• Comments (qualitative)• Subscriptions

• Paths• Links elsewhere?

RSS• Subscriptions

PODCASTS• Subscriptions• Downloads • Visits

TWITTER•Followers

•Growth•Link click-throughs (when, what types of content)•Retweets•Rankings (Twinfluence, Twitter Grader)•Comparison to similar organizations

•Friends•Conversations

YOUTUBE/VIMEO• Views• Comments • Downloads• Page placement traffic• Ratings• Click-throughs

“Metrics only tell part of the story...

it is difficult to directly measure

influence…”

“ADVANCING SOCIAL MEDIA” SURVEY

40

TRANSPARENCY

ENGAGEMENT

44

For More Information:

Andrew Krzmarzick

Senior Project Coordinator

(202) 821-6288

Andrew_Krzmarzick@grad.usda.gov

GraduateSchool.edu

GenerationShift.blogspot.com

IAmPublicsService.org

Twitter.com/krazykriz

LinkedIn.com/in/AndrewKrzmarzick

Delicious.com/akrzmarzick

Slideshare.net/akrzmarzick

Thank You!

RESOURCES1. Ari Herzog: http://www.ariwriter.com, @ariherzog

2. Andrew Krzmarzick: http://generationshift.blogspot.com, @krazykriz

3. Brookings Institution E-Government Study: http://snurl.com/crpxn

4. Forrester Website Benchmark Survey: http://www.forrester.com/cxpbenchmark

5. ForeSee: E-Government Satisfaction Index: http://snurl.com/crpyu

6. OMB E-Gov Initiative and Report: http://www.whitehouse.gov/omb/e-gov/

7. Social Media and Government (Jeffrey Levy): http://snurl.com/crq0x

8. Federal Web Manager’s “Putting Citizens First:” http://snurl.com/crndj

9. IBM’s “Leveraging Web 2.0 in Government”: http://snurl.com/crq3e

10. Air Force Blog Assessment: http://is.gd/eAYo