EAB Getting the Measure of IT UWMadison...Robert Burton, The Anatomy of Melancholy(1621); EDUCAUSE,...
Transcript of EAB Getting the Measure of IT UWMadison...Robert Burton, The Anatomy of Melancholy(1621); EDUCAUSE,...
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
Getting the Measure of ITDeploying Metrics to Create and Demonstrate IT Value
IT Forum
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
2
IT Supports Analytics for Many Institutional Purposes…
Robert Burton, The Anatomy of Melancholy (1621); EDUCAUSE, “Top 10 IT Issues, 2018,” EDUCAUSE Review, Jan-Feb 2018; EDUCAUSE, Analytics Landscape in Higher Education 2015; Numerify, Visibility to Drive Digital Transformation (2018).
The Shoemaker Goes Barefoot Himself
Analytics-Driven Top 10 CIO Issues
1. Information Security
2. Student Success
3. IT Strategy4. Data-Enabled
Institutional Culture
5. Student-Centered Institution
6. Higher Education Affordability
7. IT Staffing and Organization
8. Data Management & Governance
9. Digital Integrations10. Change Leadership
…But Its Own Capabilities Fall Short
CIOs have full view of all IT processes
35%CIOs agree a full process view is critical to success
80%
Responsibility for Delivering Analytics Services
IT
Shared IT & IR
IR
Other
60%IT
Involvement
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
3
IT Performance Measurement
IT Brand Communication
Decision Support
Imp
orta
nce
Maturity
Maturity=Importance
IT Functional Diagnostic Finds Performance Measurement Maturity Low
Source: IT Forum Functional Diagnostic benchmark data.
Not Our Strongest Suit
Low High
Low
Hig
h
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
4
Traditional IT Metrics Abound But Don’t Shed Light On Today’s Real Goals
A Brighter Lamp Won’t Solve the Problem
• Cycles and storage
• Network uptime
• Cost of infrastructure• Major systems availability
• Help desk wait times
Campus IT Monopoly, 1960s – 2000s
• Contribution to mission
• Business value of IT projects
• Cost per service delivered• Impact on student experience
• Health of key relationships
Hybridized Service Provider, 2010 - ??
Goal: Optimize IT ValueGoal: Optimize Operations
Source: Quentin Hardy, “How Cloud Computing Is Changing Management,” Harvard Business Review, 8 February 2018; Richard Katz, ed., The Tower and the Cloud (EDUCAUSE, 2008); EAB interviews and analysis.
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
5
Business Analysis and Relationship Health Rising in Significance
Source: EDUCAUSE, IT Service Delivery in Higher Education: Current Methods and Future Directions (2015); EAB interviews and analysis.
Softer Skills, Different Problems
39
59
71
2015 2020 2025
Managing Technical Resources
0
100Managing Vendors, Services, Contracts
50
CIO Management Focus
Sof
ter
Har
der
IT S
kills
Tre
nd
Service Metrics• “Business of IT”
• Vendor performance
• Satisfaction & relationships
Operations Metrics• System uptime
• Network speed & capacity
• Data center efficiency
Driver 1: Cloud and Digital Transformation
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
6
Strategic Contribution of IT Must Be Actively Demonstrated
Source: SIM IT Trends Study 2018; CHECS CIO Survey 2018; EAB interviews and analysis.
From Speeds and Feeds to Deeds
39%
78%
46%
83%
56%
13%
TechnicalKnowledge
Leadership RelationshipBuilding
…but Leadership May Not
CIOs Leaders
Skills CIOs Should Have Top 5 CIO Personal Performance Measures CIOs Recognize Their Role in Strategy…
• Internal customer satisfaction
• Value of IT to the business
• IT contribution to strategy• Availability/Uptime
• Cybersecurity related
Drip, Drip“One of a CIO’s biggest fears is to be seen as a plumber, or to be seen as a ‘good enough’ utility.”
Robert Grillo, Vice President and CIOFlorida International University
Driver 2: Contested Strategic Role of IT
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
7
CIOs Aren’t Alone in Advocating for Resources
Driver 3: Internal Competition for Resources
Source: Campus Computing Survey 2017; EDUCAUSE Top 10 IT Issues 2015; EAB interviews and analysis.
A Long-Term Competition for Limited Revenue
Declining HS demographics
Increased price sensitivity
No flight to safety with new markets
High fixed costs
Headwinds for Higher Ed
Decreasing state funding
Intensified competition
Cost of funding strategic ambitions
Higher costs to serve
63%IT organizations that have not recovered from recession-era cuts
9%Share of IT budget available for transformative investment
EAB Program Topic %As
Advancement Forum
Quantifying Advancement's Value to the Institution 61%
Enrollment Management Forum
Making Enrollment Everyone’s Job: Empowering University Staff to be Effective Enrollment Ambassadors
55%
Student Affairs Forum
Telling Our Story and Making the Case for Resources: Student Affairs Marketing and Communication Strategies
49%
University Research Forum
Speaking Their Language: Demonstrating the Value and Impacts of Research
45%
Results from EAB Topic Polls, 2016 to 2018
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
8Value Is the Coin of the Realm
Value as Strategic Alignment“KPIs should go beyond traditional IT metrics involving operational excellence. They need to answer fundamental questions such as, “How well are the IT projects aligned to the business priorities?”
Value as Business Contribution
KPMG
“My vice president cares at some level that our network is up 99.99% of the time, but she cares more about business value.”
Value as Customer Enablement
“We’re not able to report information that causes people to say ‘I got my thing done because of what IT did.’ I’m constantly struggling to demonstrate the value of what we do.”
Value as Good Stewardship
“We operate under resource constraints. We need to be sure we’re getting the best value of the work we’re doing, the staff we’re hiring, the products we buy.”
Many Dimensions, But Common Theme is Showing Impact
CIO, Private Master’s
CIO, Public Doctoral
CIO, Public Master’s
Source: KPMG, Running the Business of IT: Metrics That Matter; EAB interviews and analysis.
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
9
Taking Charge of the Narrative
In the absence of credible information on IT performance, anecdotal evidence about IT failures is likely to dominate the discussion of IT performance in the public sector. With the availability of IT performance metrics, CIOs can both proactively manage performance and demonstrate mission results.”
Kevin Desouza, Arizona State University
Source: Kevin C. Desouza, Creating a Balanced Portfolio of Information Technology Metrics (IBM Center for the Business of Government, 2015).
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
10
Obstacles to Telling IT’s Story Effectively
Information Rich, Communications Poor
Reaching the Right Audience
Shortage of Skills
• Pull predominates over push
• Limited range of communications channels
• Lack of fluency in customer vernaculars
• Dedicated communications teams are rare
• Jargon often invisible to IT professionals
• No story line connecting discrete events and announcements
Source: EAB interviews and analysis.
Blah, Blah, Blah“We get technical performance data from our IT folks, with best intent, that is just noise to the provost, deans, or students.”
CIO, Public Doctoral
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
11Getting the Measure of IT
Enhancing IT communications skills
• IT communications director
1 2 3 Communicate IT Value Effectively
Choose the Right Metrics for the IT Organization
Establish IT’s Value to the Institution
Low-Effort, High-Impact Communication
• Strategic initiatives roadmap• IT-at-a-glance infographics• Impact-based project
promotion• IT success stories
Quantifying IT investment
• Activity-based costing
Measurement-Friendly Strategic Alignment
• IT balanced scorecard• Customer unit service
agreements
The Metrics-Savvy IT Organization
• IT metrics tsar • IT metrics selection
framework
Capturing Customer Perspectives
• Fatigue-fighting customer survey tactics
Developing Value Heuristics
• High-impact IT value stories• IT value heuristic
development process
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
ROAD MAP12
Choose the Right Metrics for the IT Organization1
2 Establish IT’s Value to the Institution
3 Communicate IT Value Effectively
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
13Finding the Signal Amidst the Noise
# of Technical Tickets
Mobile-Enabled Applications
Availability of Public Computers
# of Software Downloads
Help Desk Overall Satisfaction
# of Unique Visitors to Website
Projects On Budget
Conference Line Support
Research Computing Usage
# of Wireless Routers
Centralized v. Distributed IT Staff
Rate of Ticket First Contact Resolution
Ticket Resolution Time
Student Satisfaction with Wi-Fi
Cost of Device Replacement
# of Accounts Compromised
# of Intercepted Phishing Attempts
Data Storage Capacity
Data Storage Utilization
Systems Update and Maintenance Cost
Network-Connected Devices
Staff Completing Security Training
Project Completion Time
Router Capacity
Campus Internet Coverage
Wireless Download/Upload Speed
# of Password Resets
Source: EAB interviews and analysis.
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
14
Goals Cascade
IT All Starts with Strategy
Institutional Goals Cascade and Environmental Scan Help Identify IT Goals and Objectives
SWOT Analysis
IT Strategic Objective
IT Strategic
Goal
IT activity categories result from institutional goals cascade.
Goal: A broad outcome or desired state that fulfills a strategic IT need.
Objective: An action that contributes to realizing a goal. SMART: Specific, Measurable, Achievable, Relevant, and Time-bound.
IT Strategic Objective
IT Strategic Objective
Picture of IT’s Strengths, Weaknesses, Opportunities, and Threats (SWOT)
Source: IT Forum IT Strategic Planning Center of Excellence; EAB interviews and analysis.
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
15
IT Balanced Scorecard
A Framework for Performance at Drake University
Teaching and Learning• Exceptional learning &
living experience• Experiential learning• Inclusive campus
community
Drake’s University-Wide Continuous Improvement Plan (CIP)In lieu of a traditional institutional strategic plan, Drake uses a modified balanced scorecard.
ScorecardBuilt around four key “commitments,” each with objectives and time-trackable performance measures. Progress is reviewed periodically.
Unit AlignmentUnits develop their own CIPs but must follow the basic institutional structure.
Reflection• Data-driven decision
making• Continuous assessment of
learning outcomes
Execution• Vibrant, effective,
efficient facilities• Service excellence• Robust working, learning,
and living technology infrastructure
Stewardship• Financial performance• Environmental
sustainability• Culture of pride and
engagement
Source: Drake University Information Technology Services; EAB interviews and analysis.
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
16
Institutional Categories Drive IT Objectives & Metrics at Drake
Source: EAB interviews and analysis.
Suit the Metric to the Objective, Not Vice Versa
Commitment: Teaching & Learning
• Overall faculty satisfaction• Faculty satisfaction index• Percent courses in LMS• Classroom quality index
• Ensure stability and performance with key services
• Ensure effective project management
• Ensure all our partners gain tech literacy skills
Commitment: Execution
Measures
• Users of Lynda.com• Satisfaction survey
training question scores• Knowledgebase usage
index
• Robustness index (summary measure of major system availability)
• Project execution index
• Overall project satisfaction
Starting in the Right Place“I’m a strong believer in picking something and starting, but instead of starting with what you can measure, start with the objective you hope to hit.”
Chris Gill, CIO Drake University
IT Objective• Enhance the Drake learning
experience by supporting a limited but configurable set of instructional technologies
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
17Creative IT Performance Measurement
Lessons from Drake’s IT Continuous Improvement Plan
Begin by assessing how IT can support institutional objectives
Follow the Institution’s Lead
Using standard survey instruments and measurement models saves time and opens the potential for external benchmarking
Don’t Reinvent
Customer satisfaction and behavior information is rich with strategic implications and complements operational data such as system usage
Go to the Customer
Build Summary Metrics
An index derived from multiple measurements can be a powerful tool for capturing and conveying complex situations; e.g., index averaging responses to five faculty satisfaction survey items
Source: EAB interviews and analysis.
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
18
Outreach and Management• Publish IT-unit alignment
document
• Monitor progress and fulfillment
• Periodic review with unit heads
• Update as needed
IT Strategic Plan Development
• Review institutional strategic plan
• Environmental scan of IT developments
• Draft IT strategic goals and initiatives
University of Denver Customer Agreements Link Unit Strategy to IT Action
PREPARE ALIGN WITH UNITS
1Discuss Unit NeedsMeet with deans and senior staff to identify unit vision and strategic priorities
2Identify IT Supporting ActionsSpecify IT projects and resources that will support the unit
3Establish MeasuresDefine parameters for successful fulfillment of each project
A Sideways Cascade
MONITOR
Customer Unit Service Agreements
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
19
School of Social Work
College of Education
Strategy Sheets Promote Collaboration Below the Institutional Level
Source: University of Denver Information Technology Strategic Alignment Report; EAB interviews and analysis.
Mutual Strategic Development
“Our approach was simple, to create an overall document which described our priorities with measurable objectives for 2018. We wanted not only to be transparent and accountable, but to encourage discussion on what could be done on a wider scale, or in partnership between units.”
Don Harris, Vice Chancellor and CIO
College of BusinessGoal: Integrate traditional pedagogy with latest technologies to be market-ready and technology-adroit.
IT Projects:
• Provide integration support for Internships Initiative
– Measure: Complete bi-directional integration by end Spring Quarter
• Virtual desktops for academic programs
– Measure: All virtual desktops refreshed no later than July 31, 2018
Appendix maps all IT projects to university, IT, and unit strategic plans
IT strategy sheets document unit goals, IT projects & measures
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
20
IT Metrics Tsar
Source: EAB interviews and analysis.
Metrics Conflicts Threaten Strategic Progress
Infrastructure
Security User Satisfaction
Usage
Network Speed
Traffic Patterns
System Up-Time
# of Network Users
# of ERP Users
Peak v. Average Load
# of Incidents
# of User Trainings
# of Password Resets First Call Resolution Rate
Satisfaction with Service
Duration of Phone Consultations
Latency
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
21
University of Nevada Reno Appoints a Metrics and Measurement Director
Source: University of Nevada, Reno Office of Information Technology; EAB interviews and analysis.
An Organizational Home for Metrics
• Select both operational and value metrics in a unified framework that coordinates action
• Identify and implement data-based improvements to business processes, technology utilization, and work practices (e.g., automation)
• Provide evidence of IT performance and needs for high-profile reports and dashboards (e.g., legislative budget request)
• Lead one-on-one and small-group presentations to foster collaborative interaction between OIT and campus stakeholders
• Former project manager reporting to the CIO
Key Functions Fulfilled
Select metrics to align with overall IT strategy
Apply data analysis to process improvement
Communicate IT metrics to campus stakeholders
Duties of the MMD
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
22
Screening Process Helps Narrow Core Metrics from Long Starting List
IT Metrics Selection Framework
Source: EAB interviews and analysis.
Bringing Metric Selection to Life
Four-Step Metric Selection Filtering Process
100’s of Potential Metrics
15-20 KPIs
Consideration Description
Apply a Collection and ReportingReality Check
Set aside metrics not readily accessible, regularly tracked, supported by reliable data, or easily communicated to others
Map toStrategic Objectives
Identify metrics that most directly measure progress on IT strategic objectives
Ensure Balance of Metric Categories
Avoid over-representation of some areas by sorting metrics by function or strategic perspective
Account for Unit-Specific Imperatives
Add “hot-seat” metrics that shed light on pressing yet temporary areas of concern
1
2
3
4
Metrics are quantifiable measurements collected to track organizational activities and processes
Key performance indicators are metrics that indicate progress toward strategic and operational objectives.
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
23
Four Pragmatic Screens to Determine Metric Viability
Source: EAB interviews and analysis.
Apply a Reality Check
Suggested Screens
Metric Screen Description Rationale
Accessibilityof Data
Information system must possess the capability to generate data on metrics.
Time-consuming to manually pull and analyze data for each metric.
Frequencyof Tracking
Metrics elevated to unit reports or dashboards should be monitored at regular intervals (e.g., monthly or quarterly).
Infrequent (e.g., annual) data updates hamper ability to assess performance at regular intervals.
Reliability of Data
Data available from information system should be accurate, consistently defined, and measured across the institution.
Absence of trustworthy data results in stakeholder suspicion toward performance, often resulting in inaction.
Communicabilityof Data
Definition and rationale for metrics should be easy to communicate and understand.
Lack of understanding about metric drivers and relevance hinders ability to inflect performance.
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
24
Ideal Metrics Prompt “Yes” for Every Question in List
Tool: Reality Check Screening for Metrics
Accessibility of Data1. Is the data for this metric collected via an automated system? 2. If not, can someone collect and report the data within a few hours?
3. Is the system capable of calculating and reporting the results for this metric?
Frequency of Tracking4. Can this metric be tracked more than once a year?
5. Can this metric be tracked frequently enough to inform action?
Reliability of Data 6. Do all departments use the same definition for this metric?
7. Is the metric calculated by an automated system?
8. Can you ensure the accuracy of the reported data?9. Do managers trust the data for decision making?
Communicability of Data10. Is this metric easily explained to and understood by leaders outside your
unit?
11. Do managers typically agree on the definition of this metric?12. Are managers aware of the importance of tracking the metric?
13. Do managers understand how performance on this metric impacts institutional goals?
Source: EAB interviews and analysis.
Metric Quality Score
• 10 – 13: Gold standard
• 7 – 9: Use with caution
• 6 or below: Avoid
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
25
Confirm Metrics Directly Measure Strategic Objectives Rather than Initiatives
Source: EAB interviews and analysis.
Map to IT Strategic Objectives
Strategic Goals Strategic Objectives KPIs Targets Strategic
Initiatives
Des
crip
tion
• Backbone for strategy
• Usually four to eight
• Stem from strategic goals;typically 40 to 60
• Adapted annually to every few years
Indicators that track progress toward objectives
• Indicator goals that motivate performance
• Frequently reset to ensure continuous improvement
Set of actions to raise metrics above target levels
Exam
ple
Foster data-drivendecision making
• Create data management architecture
• Improvereliability of data
• Provide self-service BI
• Percent of institutional data items defined
• Data error rates
• Ad hoc report requests
• 300 new data dictionarydefinitions
• 10% y-to-y decrease in errors
• 20% decrease in staff time for ad hocs
• Start up data governance
• External dataquality review
• Implement self-serve fact book
Framework to Map Metrics to IT Strategic Objectives
Metrics should flow directly from strategic objectives
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
26
Equitably Distribute Metrics Across IT Functions or Strategic Goals
Source: EAB interviews and analysis.
Ensure Representation From Various Categories
The most straightforward categorization scheme is to group metrics based on IT functions or capabilities, ensuring a balance of metrics across all responsibilities.
A second categorization scheme sorts metrics by institutional strategic pillars, which helps illustrate the link between IT initiatives and overall institution success.
Sample IT Functions
› Network Operations
› Enterprise Systems
› Service Desk› Cybersecurity
Sample Strategic Goals
› Student Success
› Enrollment
› Research and Scholarly Excellence› Financial Strength and Stewardship
Option 2: Strategic or Institutional Perspective
Option 1: Function or Capability
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
27
Elevate “Hot-Seat” Metrics in Response to External and Internal Pressures
Source: EAB interviews and analysis.
Account for High-Priority Imperatives
2017 KPIs
Metric Status
Help desk satisfaction
Data dictionary definitions
ERP availability
Integrations completed
Student services wait times
Data center electricity consumption
Metric Status
Help desk satisfaction
Data dictionary definitions
ERP availability
Integrations completed
Extent of PII, PHI scanning
Percent of staff completing awareness training
Metric Status
Help desk satisfaction
Data dictionary definitions
ERP availability
Integrations completed
Operating budget execution
Server consolidation
New president appointed; identifies student retention and sustainability as high priorities
State budget cuts force IT to identify operational efficiencies
Data breach incident highlights cybersecurity issues
2016 2017 2018
Example Pressures that Drive New Metrics
2017 KPIs2016 KPIs
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
ROAD MAP28
Choose the Right Metrics for the IT Organization1
2 Establish IT’s Value to the Institution
3 Communicate IT Value Effectively
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
29
Source: EAB interviews and analysis.
The Value Demonstration Problem
IT Contribution to Strategic Achievements Often Significant But Indirect
Budgets Don’t Tell the Whole Story
Outcomes Data Hard to Come By
Complex Processes Competition for Credit
Implications for IT Credit Claims
Stovepiped IT budgets obscure total investment in areas of stakeholder concern
IT dependent on business and academic units to collect performance data
Hard to separate IT contribution from other factors
Non-IT stakeholders have more direct relationship to outcomes
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
30
IT-Centric Budgets Convey Impression Spending is “For” IT
Source: EAB interviews and analysis.
Good for Bean Counting, Bad for Value Demonstration
Promotes view of IT as a cost center
Hard for non-IT people to understand
Obscures IT investment in customer concerns
Shortcomings of the IT-Centric Budget
“What does IT spend on teaching and learning?”
Budgets Built Around IT Functions…
…Can’t Answer Important QuestionsIT Budget
Category Line Item
Enterprise Systems
HR Systems
Student Systems
Reporting and BI
Network andInfrastructure
Network Services
Systems Admin
Database Admin
Client Services Service Desk
Equipment Services
Training Services
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
31
ABC1 Enables Refined Cost Estimation, But Challenging to Apply
Activity-Based Costing
Source: EAB interviews and analysis.
1) Activity-based costing.
Detailing the Cost of Service Delivery
Fully-Loaded Service Cost Analysis
Activity-Based CostingA cost accounting practice that identifies and assigns costs to overhead activities and then assigns those costs to services
+ =Examples:
• Project FTEs
• Project hardware• Cloud license fees
Examples:
• Administrative overhead
• Network infrastructure• Utilities
Sample analyses informed:
• Cost of delivery at the transactional level
• Cost of service at customer level
• Rationale for budget request
Applying ABC in Higher Ed IT
Difficult to allocate some direct costs accurately
Does not account for service quality or ROI
Quantify and Allocate Direct Costs to Services
Quantify and Allocate Indirect Costs to Services
Elaborate analysis required for complex processes
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
32
Provided data support for revised RCM IT cost recovery model to replace telecom charges
Informs insource/outsource investment decisions and deferred maintenance impact
University of Illinois at Chicago Makes Multi-Year Commitment to ABC
Source: EAB interviews and analysis.
Better Data, But at a Price
UIC’s Activity-Based Costing Model
ABC Implementation in the IT Organization
Implementation Requires Significant Investment in Process Change and Support
Services broken into cost segments; granular IT staff time allocation starts process
1 IT finance staff FTE dedicated to data collection and analysis
Years to mature implementation including service segmentation and expense tracking4-5
“It was apparent we had to change our funding model. ABC laid the foundation for being able to say, this is what it costs to provide services. Not only do we use it to determine new per FTE funding model, but it also informs where we are investing and areas where we need to invest more.”
Cynthia Herrera Lindstrom, CIO
Cost to deliver 38 services calculated down to the transactional level
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
33
Developing Value Heuristics
Source: EAB interviews and analysis.
Join the Credit-Claiming Crowd
Strategic Enrollment Initiative
Task Force Report
Admissions Redesign
Brand Refresh
Facilities Upgrade
New CRM System
New Degree Programs
Enrollment Growth!
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
34
Source: EAB interviews and analysis.
Targeted Claims, Broad Implications
Making the Case for IT Value at Florida International University
Student Success
Automated attendance tracking in 100 classrooms links IT to high-impact success practice
Student Services Efficiency
In first three days of semester, chatbot successfully fields 7000 inquiries with 65% first-call success, freeing up staff for higher-value work
Improved Student Experience
Analytics project reviewing queuing system and call center data leads to 45% year-over-year drop in student walkouts at the One-Stop services center
“When academic units graduate more students in a timely way, the deans will always get credit for that. As a CIO you have to understand where you made the contribution.”
Robert Grillo, CIO
High-Impact IT Value Stories
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
35
University of Nevada Reno Links HPC Investment to Discovery and Learning
Two Birds With One Stone
Source: EAB interviews and analysis.
“Not all institutional goals have a specific IT mention in them. However, in many cases there is a secondary agenda that has an IT component. We look for where the IT piece is that helps achieve those goals.”
Steve Smith, CIO
Discovery: Create new knowledge…
Learning: Prepare graduates to compete…
HPC cluster bought with central funds
IT measures dollar value of HPC-related grants and
undergrad participation in funded research.
Demonstrates contribution to two strategic goals
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
36
Where Perfect Isn’t Possible, Aim for Good Enough
Heuristic (n.) - Any approach to problem solving, learning, or discovery that employs a practical method, not guaranteed to be optimal, perfect, logical, or rational, but instead sufficient for reaching an immediate goal.”
Wikipedia
Source: EAB interviews and analysis.
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
37Structure of a Value Story
Quantifiable
IT Claimable
Impact on Stakeholder Interests
Strategic Relevance
Claim is verified by customer, sponsor, consultant, etc.
Craft IT Value Claims to Eliminate Credit Noise and Subjectivity
ExternalValidation
Can be quantified at least to a rough approximation
IT contribution predominates
Substantially affects a process or success measure meaningful to stakeholder
Clear relevance to institutional strategic objective(s)
Source: EAB interviews and analysis.
Ensure the foundation, aim for the pinnacle
IT Value Heuristic Development Process
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
38
FILTER
• IT can claim most or all credit
• Contribution can be credibly quantified (within 20%)
• External party will validate claim
Develop Short List of Value Claims
Define Strategic Concern
• Enabled new customer process
• Expanded usage or access
• Made workflow faster, easier, cheaper
• Improved ability to assess or analyze
• Improved security or compliance
Find IT Value Contributions
• Determine area of strategic concern
• Identify stakeholders and their objectives
• List related IT projects and services
IDENTIFY
Rate Value Claims
• Relevance to stakeholder objective
• Magnitude of impact
• Ongoing v. point impact
PRIORITIZE
Source: EAB interviews and analysis.
A Process for Selecting Value Heuristics
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
39
Refer to IT Value Heuristics Worksheet
Source: EAB interviews and analysis.
Value Heuristics Exercise
1. Choose an area of strategic concern
2. List strategic objectives in that area
3. Consider IT contributions to objectives and list relevant initiatives
4. Identify initiatives that are IT-claimable and quantifiable
5. Use the Value Story Score tool (page 2) to rate each initiative as a potential value story
6. Be ready to share your good stories and how you will communicate them!
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
40Vox Populi
Strategic SignificanceAdaptable to NeedCapture Outputs, Not Inputs
• Help answer whether investments are effective
• Avoid self-interested judgment of service deliverers and project participants
• Can capture point information or continuous feedback
• Questions and respondent groups can change as needed
• Quantitative or qualitative data
• Always of leadership interest
• Indicates customer relationship strength
• May directly address strategic goals (e.g., student experience)
Customer Satisfaction Surveys a Critical Element of Value Demonstration
Source: EAB interviews and analysis.
Methodology and process affect results
Results subject to interpretation
Requires administrative and analytical resources
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
41
Higher Education IT Enjoys Diverse Range of Standard Instruments
Source: EAB interviews and analysis.
Customer Satisfaction Survey Options
Survey Core Focus Instrument Customizable
Process Support Service Cost
MISO Library and IT Service Importance, Use, Satisfaction
$2200-$3000
TechQual+ IT Service Expectations and Outcomes
Free
ETRAC (EDUCAUSE)
Student/Faculty Technology Preferences and Attitudes
Free
Homegrown As Desired “Free”
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
42
Declining Response RatesAccess Channels Increasingly Filtered
• “Too many surveys”• Shift from paper to online processes
reduces response rates
• Increased screening of phone calls
• Low email usage
Rising Desire to SurveyMany Competitors for Student Data
• Retention and engagement initiatives
• Learning outcomes assessments
• Service improvement efforts
25Number of surveys requested of students per year at one UK university
-22% Drop in response rates among first-year students from first to third survey in semester
Inst
itutio
nal D
rive
rsStu
dent
Res
pons
e
Sources: Manja Klemencic and Igor Chirikov, “How Do We Know How Students Experience Higher Education? On the Use of Student Surveys,” The European Higher Education Area(2015); Stephen Porter et al., “Multiple Surveys of Students and Survey Fatigue,” New Directions for Institutional Research (2004); “Student Surveys Are A Waste of Everyone’s Time,” The Guardian, 1 July 2016.
Enough AlreadyRelentless Data Demands Meet Survey Resistance
“Administering multiple surveys in one academic year can significantly suppress response rates.”
Porter et al. (2004)
“Proliferation of student surveys is part of the evidence-based movement in higher education [and] a reflection of growing competition.”
Klemenčič and Chirikov (2015)
Fatigue-Fighting Customer Survey Tactics
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
43
Source: EAB interviews and analysis.
Trading Length for Frequency and Comparability
Fast Turnaround
From survey close to results availability
2 DaysPrivate tuition-dependent institution with strong data-driven culture.
ChallengeSatisfy need for unit performance data without over-burdening campus community.
SolutionSix administrative units jointly conduct one annual customer satisfaction survey:• Information Technology• Facilities• Human Resources• Finance• Communications• Public Safety
Joint Approach Permits Annual Surveying
31%Response rate, format signals respect for respondent time
Structured for Easy Response
Limited to six multi-part questions per unit:• Frequency of unit use• Rate staff• Rate services• Core values alignment• Overall satisfaction • Comments
Top-level questions and response scales are same; detail items differ
Team-Oriented Results Reporting
Leadership team takes turns reporting results in campus newsletter
Common format makes comparisons easier
CIO use in deans’ meetings, etc. sets up feedback loop
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
44
Four Ways Coppin State University Reaches Students Where They Are
CLASSROOM MINI-SURVEYSFaculty allocate 7 minutes at beginning of class for short trainings and surveys
1
SPECIAL EVENT SURVEYSTechMania and Tech@Night events attract 300-400 students, yield survey responses
2
CAMPUS BOOTHSTechKNOW-staffed tables in high traffic areas provide information about IT services
3
The TechKNOW Squad
ChallengeA commuter HBCU, Coppin State struggles to get satisfactory response rates on student surveys.
SolutionVolunteer team of about 25 students conducts on-the-spot peer trainings and short surveys on IT preferences and service satisfaction.
Source: EAB interviews and analysis.
A Tiger Team for Peer-to-Peer Outreach
FOCUS GROUPSquad members are available for formal and informal consultation
4
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
45Information Is Power
Work with Faculty and Other Partners for Maximum Survey Impact
IT reviews question pool, creates first draft.
IT and survey provider monitor progress, call on faculty groups for response help as needed.
Pre-briefed partners assist with dissemination and response.
Design Field Analyze Plan
IT consults with faculty to finalize instrument.
Faculty and leadership get sneak peak of early results, help develop action plan.
“It was a heavy lift having faculty engaged in the survey conversation, but that was actually a benefit. We talked about what we needed to know versus what was interesting but not actionable. You need to be transparent and share the results, warts and all. In fact, the problem areas are your friends. Engaged faculty can advocate for improvement.’”
Eric Behrens, CIO, Widener University
Source: EAB interviews and analysis.
Publish
Survey partners help
design projects, seek resources
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
ROAD MAP46
Choose the Right Metrics for the IT Organization1
2 Establish IT’s Value to the Institution
3 Communicate IT Value Effectively
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
47
Users and Leaders Fail to Recognize IT’s Contributions to the Campus
Source: EAB interviews and analysis.
No Understanding Without Communication
What the Campus Doesn’t See, the Campus Can’t Appreciate
IT Defensiveness and Isolation Make Matters Worse
A Preponderance of Negative AssociationsThe campus only sees or hears from IT when something is wrong or broken—tempers run high and patience runs low
1
Successes Taken for Granted or Limited in RecognitionBreakdowns are more memorable than reliable services, and good project outcomes only know to a few, even in IT
2
Breaking Down IT’s Image Problem
“Us” versus “Them” Mentality Taking RootIT staff increasingly wary of communicating with campus—perpetuating the vicious cycle of negative perceptions
3
“I couldn’t name a single person in IT!”
“It sure would be nice if IT people knew how to speak plain English”“The only time I see IT is when they’re fixing yet another broken thing”
“We have so much to do that there’s not much time for communication.”
“The only time campus talks to us is to yell at us”
“No one appreciates how hard it is to deliver reliable services to so many different customers”
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
48
Professional Communications Benefits Are Real But Hard to Quantify
IT Communications Director
Source: EDUCAUSE, The Higher Education IT Workforce Landscape 2016; U.S. Bureau of Labor Statistics; EAB interviews and analysis.
A Dose of Soft Skills for the IT Organization
Important Skills at a Relative Bargain
Importance of communicating effectively among CIO skills
#1Mean salary of writers and editors in higher education
$63,310Cost of writer/editor relative to developer
82%
Skills Contributed
• Ability to communicate complex concepts in accessible language
• Segmented messaging to different groups via appropriate channels
Benefits to IT
• Better customer awareness of services and projects
• Refined communication of IT value and performance
• Listening and learning from a non-IT standpoint
• News judgment: ability to recognize newsworthiness and explain it
• More recognition for IT staff• Lessened workload, greater
effectiveness for IT leadership
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
49Low-Effort, High Impact Communication
How IT contributes to strategic goals
Scope of IT activities What IT is doing for you
How IT has made a difference
Dense IT strategic plan materials
Diffuse data, narrow user view
Project knowledge limited to a few
Successes stay local
Strategic initiatives roadmapMulti-dimensional representation of strategic initiative impact
IT-at-a-glance infographicsEasily consumed representations of IT activity accessible to a wide range of audiences
Impact-based project promotionTemplated project awareness stories inviting inquiry and participation
IT Success Stories
Systematic efforts to capture and relate IT accomplishments
Source: EAB interviews and analysis.
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
50
Project Prioritization at Embry-Riddle Assigns Risk and Impact Scores
The Last Mile of IT Strategic Planning
STRATEGIC REVIEW IT REVIEW LEADERSHIP
APPROVALCOLLECT
REQUESTS
IT asks for strategic project requests; requesters provide goals, cost and impact estimates
Appropriate Strategic Implementation Committee reviews for alignment with strategic domain
IT reviews for completeness, estimates cost, assigns risk score and impact score based on:
• Breadth of service
• Potential for revenue increase
• Efficiency/cost reduction
Core leadership team reviews inputs, prioritizes high impact/low effort projects
Project roadmap with timeline, impact, and risk information for all approved projects
Source: EAB interviews and analysis.
Strategic Initiatives Roadmap
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
51
Embry-Riddle Summarizes IT Direction in Multi-Dimensional Graphic
Source: EAB interviews and analysis.
A University-Wide Technology Road Map
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
52
Source: Embry-Riddle Aeronautical University Technology Roadmap 2018; EAB interviews and analysis.
Information Density Allows a Richer Story to Be Told
Concentric circles indicate project completion timeframes
Projects grouped by stakeholder categories
Project dot size represents impact, color represents risk
Embry-Riddle Technology Roadmap
Roadmap Identifies Strategic Projects by Category, Time, Impact, and Risk
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
53
Given Context, Measures of IT Scope and Scale Suggest User Impact
Source: EAB interviews and analysis.
The Case for Volume Metrics
The Virtues of Volume
• Big enough to include everybody
• Inform community of breadth of services
• Show IT is working hard
• Lend themselves to visualization
• Low effort to collect
University of AlbertaTexas A&M
University of Florida
Embry-Riddle Aeronautical University
University of Iowa
IT-at-a-Glance Infographics
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
54
Embry-Riddle Supplements Roadmap with Performance Facts Infographic
Source: Embry-Riddle Aeronautical University Information Technology; EAB interviews and analysis.
What Have You Done For Me Lately?
Infographic summarizes IT activity touching a wide range of interests, including:
• Project achievements
• Customer satisfaction
• Security interventions
• Budget information
• IT staff development
A Fresh Snapshot of IT Activity
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
55
DesignPredesigned template built around 9 metrics categories makes twice-annual update easier
Wash, Rinse, Repeat
ProductionMarketing and publications staff update the template to produce new infographic
AssemblyOver a two-week
window, CIO direct reports collect updated
data, consider changing reporting
categories Circulation
Infographic posted to portal, circulated
during meetings and town halls
Quick Production Eases Refresh of Embry-Riddle Performance Facts
Source: EAB interviews and analysis.
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
56Complementary Quick-Review Perspectives
Strategic Roadmap
Performance Facts Infographic
Relatable Customer Info
“The infographic provides a snapshot of what we have accomplished, and we simultaneously share our roadmap. Most importantly, we put our energy into where we go from here. To me, the infographic and roadmap are conversation tools. Any time I bring them to groups, dialog happens.”
Becky Vazquez, Vice President and CIO
Strategic Vision Plus Ongoing Activity Equals IT Contribution
Source: EAB interviews and analysis.
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
57
Loyola University Chicago Annual IT Summary Tells a Year’s Story
Source: Loyola University Chicago Information Technology Services; EAB interviews and analysis.
An IT Organization On a Page
The IT Annual Summary• Two sided trifold• Staff collects material• Leadership team writes• 2-3 month cycle time
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
58
Source: EAB interviews and analysis.
Projects Make It PersonalImpact-Based Project Promotion
Communicate IT Contributions to Influence Campus Perspectives
Showcase Creative Problem Solving
Foster a Narrative of Progress
Highlight Novel Technology Capabilities
Demonstrate Service to Campus Beneficiaries
Publicize IT projects to…
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
59
University of Iowa Project News Items Raise Cross-Community Awareness
Source: https://oneit.uiowa.edu/news/project-news; EAB interviews and analysis.
IT Projects as IT Promotion
Projects screened for high impact are described in a scrolling panel on the ITS website
Postings are built from a template
filled out by project creator
“Everybody in the IT community gets a chance to see there’s a new project, and can say, ‘I’m interested, can I join your project?”
Steve Fleagle, CIO
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
60
Documentation Reduces Need for Reporting, But Writing Still A Value Add
Project Management Informs Communication
Project Management Info
StoryElement
Sponsor, customer, team
Who
Project description and impact
What
Project plan and status
When
Intended outcomes and success measures
Why
Project management processes capture essentials
Skills and workload issues call for writer
handoff
Quick completion and publication
Source: https://oneit.uiowa.edu/news/project-news; EAB interviews and analysis.
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
61Personalizing Project Impact with Metrics
Headline establishes the need for protection in face of threat
1
Body uses volume metrics to show that IT is on the job
2
Conclusion shows that user still must “prove it’s you” to be safe
3
West Virginia University Infographic Prepares Campus for Two-Factor Authentication
“When we started communicating what IT was doing at WVU, we did a monthly update with two-paragraph summaries and an infographic. People would spend 10 minutes on the infographic but wouldn’t read the paragraphs.”
Vicki Smith, Communications
Director
Source: West Virginia University Information Technology Services; EAB interviews and analysis.
©2018 EAB Global, Inc. • All Rights Reserved • eab.com
62
Once Established, Success Stories Incentivize Staff Contributions
IT Success Stories
Source: EAB interviews and analysis.
Success May Be Hiding In Plain Sight
Quantifiable
IT Claimable
Impact on Stakeholder Interest
Strategic Relevance
ExternalValidation
“At first, people didn’t get what I was looking for with success stories. Once we wrote a few, the IT staff had an ‘Aha’ moment and started coming to me. It’s a form of recognition. They’re excited to show their work made a difference.”
Nicole Dahya, Communications Manager, University of Iowa
Source Stories
Story Publication
Staff Recognition
Leadership Pump Priming