Evaluating health informatics projects

•Reasons for and problems of evaluation

•Objective model

•Subjective model

Definitions

• Evaluate – to determine the value of (Chambers)

• To examine and judge carefully (Dictionary.com)

Reasons for evaluation(Friedman/ Wyatt)

• Promotional – – encouraging people to use systems

• Scholarly – – study of the impact etc. of HI systems

• Pragmatic (practical) – – finding out what is good and bad, improving future

systems

• Ethical –– like any medical intervention, safe and effective

• Legal – – same reason. Also to inform users so they know when

and when not to use it

Perspectives

• Stakeholders– Developers– Users– Patients– Managers– Sources of funding

Effects

• Structure – environment, staff, money

• Processes – diagnosis, investigation, treatments

• Outcomes – success of treatment, survival, continuing health

Complexity

• Combination of– Medicine & health care– Information systems / IT– Evaluation methodology

• Each of these is a huge area

• Arguably IT is the simplest, or at least the most structured

Of Medicine• Extremely large and growing area of

knowledge• Complex structure

– Equipment, staff, regulation

• Processes– Treatments etc.

• Outcomes– Long term, difficult to measure– Knock-on effects of innovation– Effect of IT particularly hard to measure

Of Information systems

• Difficult to fully test– Combinatorial explosion

• Multi-function– Has a range of effects

• System itself vs. impact on health care

Of Evaluation

• Have to measure impact– This means impact on people - difficult to study

• Need patients and staff to perform tests– May not be enough willing to cooperate

• Range of things that can be evaluated, ranging from– ‘Does it work?’ to– ‘Does it help patients?’

Evaluation

• In theory, – study situation before & after

• In practice, – don’t know what changes would have occurred

without innovation– don’t know what interesting questions will arise

during study

Tips

• Tailor study to problem– Not research – specific to this project

• Collect useful data– Data which inform final decision

• Look for side-effects– Effects not related to intended purpose

• Formative & summative– Study during & after development

Tips (continued)

• In vitro vs. in vivo– Evaluate on-site & off-site

• Don’t accept developer’s view

• Take account of environment – context

• Let questions appear during study

• Be prepared to use a range of methods

What can be studied• Need for resource

– What does it give us that we didn’t have before

• Development process– What methods do developers use to design their

solution?

• Structure of resource– What does the program & spec look like?

• Functions of resource– How well does it work?

• Impact– How does it affect HCPs and patients?

Study features• Focus

– As previous slide

• Setting– Laboratory or hospital

• Data– Real or simulated

• Users– Developers, evaluators, end-users

• Decisions– None, simulated or real

Types of study

• Need validation• Design validation• Structure validation• Laboratory function• Field function• Laboratory user impact• Field user impact• Clinical impact

Objectivist or quantitative approach

• Can measure things objectively and without affecting thing being measured

• What to measure can be agreed rationally

• Can use numerical data

• Draw definite conclusions

Objectivist approaches

• Comparison-based– Like randomised clinical trial

• Objectives-based– Does it do what the designers said?

• Decision facilitation– Answers questions posed by managers

• Goal-free approach– Evaluators not aware of project goals

Methods

• Measurement

• Demonstration studies– Descriptive– Comparative– Correlational

• Statistical analysis

Measurement studies

• Terminology for measurements– Object e.g. patient– Object class e.g. patient group– Attribute e.g. temperature– Instrument e.g. thermometer– Observation e.g. temperature at one time

• Validation – calibration of thermometer

Demonstration studies

• Demonstrate effect– ‘Do patients who have been inoculated have a

higher temperature?’

• Object -> subject (patient)

• Attribute -> variable (temperature)

Descriptive

• ‘The patients in this study have a rather high temperature’.

• Mean, standard deviation etc.

Comparative

• ‘The patients in this study have a higher temperature than a control group’

• Controlled environment (usually)

• T-test etc.

Correlational

• ‘We are seeing more patients with fever since we introduced inoculation’

• Live situation

• Could still be a t-test

• Trying to associate one factor with another in a real situation

Subjectivist or qualitative

• Observations depend on observer

• Observations only meaningful in context

• Different points of view may be valid

• Descriptions as valuable as numbers

• Discussion of results

Subjectivist approaches

• Quasi-legal– Cf ethical debate

• Art criticism– Expert review

• Professional review– Site visit

• Responsive/illuminative– Immersion in environment– Questions evolve over time

Qualitative approach

• Attempts to understand why as well as measure differences e.g.– Is system working as intended?– How can it be improved?– Does it make a difference– Are differences beneficial?– Are the effects those expected?

Stages in qualitative study

• Negotiation of ground rules

• Immersion into environment

• Initial data collection to focus questions

• Iteration

• Report and feedback

• Final report

Methods in qualitative study

• Observation

• Interviews

• Document analysis

• Others, e.g. structured questionnaires

Mixed study

• Can combine qualitative and quantitative approaches