Chapter 9

Evaluation of the IT Function

Management of Computer System

Performance

2

Agenda: Chapter 9

Objective: Students should be able: to explain the

implications arising from the analysis.

Evaluation of the IT Function

3

Evaluation of the IT Function

The evaluation of IT services and functions requires applying the proper metrics against actual and perceived effectiveness.

There are several levels of this including concepts previously discussed such as; providing appropriate targets and measuring

them against actual data. Measuring actual and targeted data against

industry and competitive data. Ex-post and ex-ante evaluation methodologies.

4

Evaluation of the IT Function

To measure effectiveness of an IT Function (department wide), it is necessary to view the IT department from a fixed perspective. Does the IT department; Provide desktop support and workgroup server support or Does it manage back-office systems, CRM, ERP/ERM,

SCM, POS E-Commerce and LAN, MAN, WAN and WAP services while steadily increasing operational efficiencies at lower departmental costs directly contributing to the bottom line of the companies revenue base.

5

Evaluation of the IT Function Effectiveness also entails the quantification of soft

and hard estimates or direct and indirect factors. As part of that consideration of services offered.

Examples of this include: Is it meeting its objectives? Are its projects on-time, to budget and meet

specifications? Is the IT department a key factor in reducing costs and

adding revenues? Do they maintain a constant level of service and support? If these are accurate, does executive management

recognize these traits?

6

Evaluation of the IT Function - Goals

There are two generic approaches to these view of IT Functions, A Goal Centered view and, A Service Centered view.

Goals are usually set at either strategic or tactical level.

They are implemented by the successful completion of a series of tasks or projects.

The Goal is usually associated with changing the steady state of the business or IT Function.

Change is the primary approach to this concept.

7

Evaluation of the IT Function - Goals

A Goals centered evaluation system is most commonly associated with the Program or Project Management perspectives. Scope it – Define the requirements and understand what

you will need to do. Plan it – design a solution and plan its implementation to

solve for the requirements identified in the scoping activity.

Execute it – Execute the Plan. Close it – measure the success against the original goals.

Do lessons learned, etc. Was the project successful?

8

Evaluation of the IT Function - Services A service centered view of the IT Function is

targeted toward maintaining or increasing the level of service currently provided.

This requires that a minimum level of service be maintained. Service levels are agreed to in advance by the IT

department and representatives of the user community. Metrics include tracking up-time and service availability.

This is reduced to writing in the form of an SLA or Service Level Agreement.

9

Evaluation of the IT Function - Services An SLA is usually measured by quantifying the

time the system is available or service is rendered and systems or service response time. Both of these elements must be factored into the SLA

baseline. As an example, if a system that is “available” but has a 30 second response time, it might as well be down.

There are variations to this theme but a 99.5% SLA on a system’s availability is common .

This means that within the 99.5 SLA, an E-Business solution can be “down” for about 3.6 hours a month [(30 day x 24 hours x 60 minutes) x .005]

10

Evaluation of the IT Function - Services 3.6 hours can be a significant amount of time.

However, consider that a three tier UNIX solution may require each server to be booted in series (First one, then the next and then the last one.)

A UNIX box can take 20 minutes to boot if it was a clean shut down. Therefore, each reboot can cost and hour.

There are many reasons for a system to be unavailable over the course of a month. This include everything from installing patches to fixing software problems.

3.6 hours is not enough time if a system is down late Friday night and your technicians are not available (been there and done that too).

11

Evaluation of the IT Function - Services There are numerous levels of SLAs

DD Hours ExtensionAllowable Downtime

ReliabilityDowntime/ Hours Per

Month

Downtime/ Minutes Per

Month

Data Center Classification

30 24 720 0.05 95.00% 36 21601

30 24 720 0.01 99.00% 7.2 4322

30 24 720 0.005 99.50% 3.6 2163

30 24 720 0.001 99.90% 0.72 43.24

30 24 720 0.0005 99.950% 0.36 21.65

30 24 720 0.00001 99.9990% 0.0072 0.432

12

Evaluation of the IT Function - Services Availability is the function of the users needs, the

system design and the negotiated baselines. The minimum levels of service required are documented.

This approach to Services evaluation greatly simplifies the quantification of the value of IT Functions. It either meets the requirements or it doesn’t

Another approach to evaluating IT Functionality is through the use of user satisfaction surveys.

13

Evaluation of the IT Function - Surveys Surveys vary in form and content but if users

complete them, they can be valuable resources. In either case, the establishment of goals or levels of

services, the satisfaction of the users needs to be taken into consideration.

The most common of which is through the use of the Satisfaction Survey.

These provide both hard data and direct responses but assumes certain factors.

To be successful, there are some basic premises that must be met.

14

Evaluation of the IT Function - Surveys

Is the sample base of users providing data sufficient in number to be statistically sound.

Is the raw data collected non-biased. There are many, many factors which can skew data

results. These factors include

getting the users to fill out the complete survey in a thoughtful manner without outside distractions.

That the questions do not skew the results. That the survey monitors do not skew the results.

How would users react when being interviewed for their user satisfaction by out of work actresses and actors (pretty people) who would be be hired to interview them to document their satisfaction with the services received.

15

Evaluation of the IT Function - Surveys

Do the surveys collect the correct data? Several examples from the book state “If Yours

answer was Good or Excellent, please give details”. Any problems here? It depends on your perspective. For corrective action, you also need to know what didn’t

work. The use of surveys also includes gathering data on

performance (good/bad) or perceptions of the services importance to the firm (Critical/Major). This type of data provides both a performance

perspective and a value to the company perspective of the user community.

16

Evaluation of the IT Function - Surveys In the case of data collected on Performance

or Importance, corrective action can be undertaken from the data collected. In these cases, the quantification of the results of

the surveys, SLA compliance or meeting goals through effective project management can be quantified.

Over time, and if the baseline for measurement remains the same, trends can and should be plotted.

17

Evaluation of the IT Function - Surveys The methodology includes

obtaining a statistically valid samples, determining the mean, median and standard deviation of that data set and,

comparing that data with the appropriate measures.

To obtain this data requires time and effort which does cost money.

18

Evaluation of the IT Function - Surveys As a result of the cost to compile the data, Surveys

are often reduced to a level that generates less than optimal statistical data sets.

Accordingly, the in-depth (data mining) analysis of the data is not performed for a variety of sound mathematical reasons.

The reviews that are performed provide a rating that is similar to the surveys themselves. The users are: Very Satisfied, Satisfied,dissatisfied or very dissatisfied.

19

Evaluation of the IT Function - Surveys Taken alone, this can give some value but there

could be significantly more obtained. It is appropriate to undertake complex analytical

reviews of collected data but the mathematical requirements that apply to any other data set that is to be analyzed should apply to IT evaluations as well. Don’t change the standards because the data is difficult

to obtain. Within the parameters identified herein, there are

many subcomponents that can be examined the result in the data collected.

20

Evaluation of the IT Function - Data The data collected should be reviewed and

quantified. The relationship and weighting of each elements

should be reviewed. Statistical normalization may be a valid approach as well.

Each approach to the valuation should be documented and agreed to by all parties in advance.

First get your facts; then you can distort them at your leisure. - Mark Twain

21

Evaluation of the IT Function - Data An on going theme of this class is to

quantify the data for decision making actions based on financial reasoning.

The compiled Survey data, the Service performance metrics and/or the achievement of Goals should then be matched with an economic value of the offering.

Measure the benefit and then the cost and make the decisions from that analysis.

22

Summary

SLAs measure response time or availability. They serve to confirm the expected value

received from the project. Surveys, if properly constructed, can

provide valid information on the satisfaction of the IT system or services customer. The construction is critical.

23

Homework