Evidence-Based Websites

Douglas Badenoch

Minervation Ltd

About me

• Strathclyde University– Lecturer, 1990-95

• Oxford CEBM– Programme Manager,

1996-2002

• Minervation Ltd– Founding Director,

2002-

About Minervation

Evidence-based health care consultancy

• Information science• Usability testing• Design, hosting and

development

Our Work

Content• J&J, NLH

Evaluation• CRUK, NHS Evidence

Design, hosting and development

• CEBM, EQUATOR Network, ScottishDental

EBW could mean…

1. Websites supporting EBHC

2. Websites about EBHC

3. Embedding evidence in websites

4. Evidence about (the effectiveness of) websites

This session mostly about…

1. Websites supporting EBHC

2. Websites about EBHC

3. Embedding evidence in websites

4. Evidence about (the effectiveness of) websites

“We can have more impact on health by doing what we already know than by any device or technology likely to become available in the next decade.”Muir Gray, 2005

We know what they should look like

1 There’s good evidence that we are not getting the evidence to where it’s needed, but that when we do, it has a positive impact upon practice.– Accessible, user-friendly– Full text, evidence-based summaries

• Clinicians have about 5 minutes to look for evidence– About half their questions go unanswered– They perform better when they have access to

evidence-based summaries

Websites about EBHC

4. The “maverick” traditions of EBHC innovation show no signs of abating

• Reporting guidelines

• Clinical research online

• Uncertainties

www.equator-network.org

Worldwide initiative to bring together all that we know about how to report the results of research.

• By study type• By clinical specialty• By publication type• By document structure

www.equator-network.org

Audience:• Researchers• Editors• RG developers

Current developments:• Implementing a database of RGs• Linking RGs to the underlying

evidence base• Supporting the implementation of

RGs“Take me to your

Reporting Guidelines”

We’re starting to do more about…

3. Embedding evidence-based methods into content production

• User involvement in requirements, comprehensive literature searching, critical appraisal and transparent evidence synthesis

• Review and updating protocols

www.scottishdental.org

Set up in 2003

Relaunched 2007

Taxonomy-based

User centred design

UK Prostate Link www.prostate-link.org.uk

Quality-assessed information about prostate cancer

• Regular updates and alerts

• Popular with users, not with providers!

Instruments

• No lack of them (over 270 in our 2005 review)!– Searched health and information science

databases– Very few instruments assessed all of the

elements that mattered to our users– Of those that did, none had been validated– “Levels of evidence” impractical– Evidence of problems with policing

Implementation

The assessment method had to:

• Address what matters to end-users

• Encompass validated, evidence-based methods of evaluating information

• Be valid and transparent

• Produce a quantitative score for use in search ranking

Implementation

Usability Questions: 1. Is the site accessible without a

login? 2. Does the site conform to web

accessibility standards? 3. Is the site design clear and

transparent? 4. Is the site design consistent

from one page to another? 5. Can users find what they need

on the site? 6. Is the format of information

clear and appropriate for the audience?

Reliability Questions: 7. Is it clear who has developed

the web site and what their objectives are?

8. Does the site report a robust quality control procedure?

9. Is the page content checked by an expert?

10. Is the page updated regularly?

11. Does the page cite relevant sources where appropriate?

Validation

Blind, independent comparison of two trained assessors:

• Information specialists

• Ranked 42 websites• Measured the rank

order correlation• P < 0.0001

Prostate Cancer Support Federation

http://prostatecancerfederation.org.uk/

A small user-led charity which provides unique information on a minimal budget.

Usability Questions: Score out of 3

1. Is the site accessible without a login? 3

2. Does the site conform to web Accessibility standards? 2

3. Is the site design clear and transparent? 3

4. Is the site design consistent from one page to another? 2

5. Can users find what they need on the site? 1

6. Is the format of information clear and appropriate for the audience? 2

Reliability Questions:

1. Is it clear who has developed the web site and what their objectives are? 3

2. Does the site report a robust quality control procedure? 0

3. Is the page content checked by an expert? 0

4. Is the page updated regularly? 1

5. Does the page cite relevant sources where appropriate? 2

Total 19/33

Embedding quality

• Greater public awareness– e.g Bad Science

• Various quality initiatives must be joined up

• Role of the IS must be central to quality

• Will people start looking at the About Us pages?

From prevalence to incidence

• Having identified the existing best available evidence, the challenge is to communicate it effectively and keep it up to date.

We have a real problem with..

4. Working out whether they work or not– Standards of proof are lower than normal NHS

interventions– EBHC websites (type 1) don’t even use the

traditional methods well– User modelling is only now taking centre stage in

evaluating website performance– Political imperatives will override all of that

anyway