Standards for National Clinical Audits- · PDF filenational clinical audit, ... For purposes...

Standards for a NationalClinical Audit or a QualityImprovement Study

HEALTHCAREHEALTHCAREQUALITYUALITY QUESTUEST

HQQ

Standards for a NationalClinical Audit or a QualityImprovement Study

20 April 2012

HEALTHCAREHEALTHCAREQUALITYUALITY QUESTUEST

HQQ

First draft issued on 7 November 2011

Healthcare Quality Quest LtdShelley FarmShelley LaneOwerRomseyHampshireSO51 6AS

Tel: 00 44 (0)23 8081 4024Fax: 00 44 (0)23 8081 4020

Email: [email protected]: www.hqq.co.uk

Copyright © Healthcare Quality Quest Ltd, 2012.

All rights reserved. No part of this document may be reproduced or transmitted in any formor by any means, electronic or mechanical, including photocopying, recording or by anyinformation storage or retrieval system, without permission in writing from Healthcare QualityQuest Limited.

Contents

Preface 1

Executive summary 2

Background 4

Purposes 5

Method 6

Search strategy 6Information included and abstracted 6Limitations of the search and the literature 6Review of drafts 7

Definitions of terms 8

Standards for a national clinical audit 10

Summary of literature supporting the standards 13

Structural aspects of a national clinical audit or equivalent activity 13Ethical basis 13Governance 13Stakeholder involvement 14Resources 15Roles and responsibilities and project plan 15Process aspects of a national clinical audit or equivalent activity 16Recruitment 16Improvement–driven aims and objectives 16Population or sample 16Data collection strategy 17Quality-of-care measures 17Data elements and data sources 19Data collection protocol or manual 19Protection of patient identity 20Identification and handling of ethics issues 20Training and support for data collectors 20Pilot testing 21Reliability testing 21Data linkages 21Data quality management, analysis and assurance 21Preliminary data and peer review of cases 22Identification of good practice and shortcomings in quality of patient care 22Analyses of causes of shortcomings in quality 23

Contents i

Facilitation of improvements 23Timely, complete and informative reports 24Communication among all stakeholders 23Outcome aspects of a national clinical audit or equivalent activity 24Level of participation 24Reliability of data 24Timeliness of reports on preliminary data 24Timeliness of complete reports 24Evidence of improvements in quality 24

Summary 26

References 27

Acknowledgements 34

Standards for a National Clinical Audit or a Quality Improvement Studyii

Preface 1

Preface

Healthcare Quality Quest (HQQ) was asked to evaluate an individual national clinical auditbeing carried out in the UK. We are aware of the limitations of a subjective assessment of anational clinical audit, even if carried out by an organization with considerable, recognizedexpertise in clinical audit.

However, we were unable to locate published standards against which a national clinical auditcould be compared. We checked with recognized international and academic experts in theconduct of international or national quality measurement. They stated that they were notaware of internationally accepted standards on the conduct of a national clinical audit or anational quality improvement project or equivalent activity.

Therefore, for purposes of completing the evaluation of the national clinical audit for whichwe were commissioned, we needed to establish a more objective basis for making judgementsabout the conduct of a national clinical audit. Thus, we searched literature describing aspectsof good or best practice relating to a national activity intended to measure and improve thequality of patient care, and from the literature, derived what we consider to be standards forthe conduct of a national clinical audit, quality improvement study or equivalent activity.

Standards for a National Clinical Audit or a Quality Improvement Study2

Executive summary

Large-scale studies or clinical audits of the quality of patient care provided in healthcareorganizations have been carried out in the UK for nearly two decades. Consensus or evidence-based national or international standards for the operation of national clinical auditsor equivalent activities have not been published. In the absence of agreement on best practicefor such activities, variation in the management and potential effectiveness of national clinicalaudits is inevitable. It can be uncertain if an individual national clinical audit or equivalentactivity is beneficial to the healthcare system and the patient population served.

Literature has been searched to identify aspects of current practice in the conduct of nationalor international clinical audits, quality improvement studies, quality indicator measurementsor equivalent large-scale initiatives that have the purpose of driving improvement in thequality of care provided in one or more healthcare systems. From the literature, 30 ‘standards’for the conduct of a national clinical audit have been identified. The standards are onstructural, process and outcome aspects of a large-scale activity that involves measuring andimproving healthcare services. The standards identified relate to the following aspects of anational clinical audit or equivalent activity:

• Structural aspects– Ethical basis – Governance – Stakeholder involvement– Resources– Roles and responsibilities and project plan

• Process aspects– Recruitment– Improvement–driven aims and objectives– Population or sample– Data collection strategy– Quality-of-care measures– Data elements and data sources– Data collection protocol– Protection of patient identity– Identification and handling of ethics issues– Training and support for data collectors– Pilot testing – Reliability testing – Data linkages– Data quality management, analysis and assurance– Preliminary data and peer review– Identification of good practice and shortcomings in quality of care– Analysis of causes of shortcomings in quality– Facilitation of improvements– Reports– Communication

Executive summary 3

• Outcome aspects– Level of participation– Reliability of data– Timeliness of preliminary reports– Timeliness of complete reports– Evidence of improvements

It is hoped that these standards for the conduct of a national clinical audit will facilitate debateon the value of adopting such standards for the operation of national clinical audits orequivalent activities.

Background

National clinical audits have been carried out in the UK since 1994.1 Equivalent activities havebeen carried out in other countries under names such as national (and international) qualityindicator projects, national quality improvement (QI) studies or national performancemeasurement projects. In the UK, the purpose of national clinical audits has been to seek toimprove patients’ experiences and health through the systematic review of healthcare deliveryand to ensure that all patients receive the most effective, up-to-date and appropriatetreatment, delivered by clinicians with the right skills and experience.2 Considerable financialsupport is needed for these audits and related activities. In the UK, public funding hassupported many national clinical audits.

Best practice for the conduct of clinical audits has been described in the UK.3–7 However, thepublications on best practice are focused on clinical audits carried out by clinicians or clinicalteams working in an individual healthcare organization, so-called local clinical audits. Guidanceon assessing and improving quality in national clinical audits has been published recently inthe UK.8 The guidance is not explicitly based on national or international evidence related tobest practice in the design and conduct of a national clinical audit or equivalent activity. Also,the guidance does not provide explicit standards against which all aspects of a national clinicalaudit can be assessed objectively.

In the absence of agreement on best practice for such activities, variation in the managementand potential effectiveness of national clinical audits is inevitable. In view of the cost ofnational clinical audit projects in England funded directly by public funding,9 the currentfinancial pressures on the National Health Service,10 and the review of data returns for NHSorganizations,11 it is timely to expect that organizations that operate national clinical auditswork to standards that are intended to assure best practice in the conduct of such audits.Adoption of best-known practice in the conduct of a national clinical audit should enabledemonstrating the effectiveness of a national clinical audit in achieving improvements in thequality of patient care as well as the cost-benefit of participation in national clinical audits.


Purposes

The purposes of the work presented in this report were to:

• identify from available literature explicit or implicit standards for the design and operation of a national activity that involves measuring and facilitating the improvement of the quality of healthcare services delivered to patients

• find examples of best practice in the performance of such activities that could be used as benchmarks for national clinical audits or equivalent activities

• develop a set of standards for the design and conduct of a national clinical audit.

The purposes of this report are to:

• provide a synthesis of available literature on explicit or implicit standards for the design and operation of a national activity that involves measuring and facilitating the improvement of the quality of patient care and examples of best practice

• present a draft of standards for the design and conduct of a national clinical audit for debate on the value of adopting such standards for the operation of national clinical audits or equivalent activities.

Purposes 5

Method

Search strategy

For purposes of identifying sources related to standards for national clinical audits, any activitythat involves the measurement of an aspect of the quality of patient care, including clinicalprocesses or outcomes, was included. The types of activities referred to included nationalclinical audits, national outcome studies, national performance indicator monitoring, nationalor international quality indicators measurement, national quality improvement studies, andregistries.

The following sources were used to find information: Medline and CINAHL databases usingPubMed, Ovid and EBSCOHost as search engines; websites for the Agency for HealthcareResearch and Quality (AHRQ), National Institutes of Health (NIH) and for selectedorganizations known to operate a national clinical audit or quality improvement study;journals known to publish studies related to measuring and improving quality; professionalorganizations known to be concerned with measuring and improving the quality of healthcareservices; and Google and Yahoo search engines.

The terms used for the searches included: criteria or standards or indicators or measures withclinical audit or audit or quality improvement; performance or quality indicators or measures;scope of work or statement of work and national quality indicators; and registry. Where limitscould be set, the years searched were limited to 2000 to 2011. Reference lists in publicationsretrieved or on websites searched were scanned for additional relevant sources.

Information included and abstracted

Any document that described or specified an expectation related to the design or operationof a national clinical audit or equivalent activity was identified from the source material. Alldocuments were screened and relevant documents were abstracted by one reviewer and thescreening and abstracts were validated by a second reviewer.

Much of the material published about what should be included in a clinical audit is written forclinicians carrying out their own clinical audits in an individual clinical service in a single healthcareorganization. This information was included selectively in the review if a point was thought by thereviewers of the literature to be relevant to the conduct of a national clinical audit.

Limitations of the search and the literature

The literature search was deliberately focused to identify defined aspects of practice in relationto the conduct of national clinical audits or equivalent national or international activities.Descriptions or reports of national clinical audits were not included unless they made pointsabout the performance of key aspects of a large-scale audit or equivalent activity or topotential standards for the conduct of a national audit.


The literature on the conduct of large-scale studies of the quality of care is not uniform orsystematic in the description of methodology used in carrying out the work. This shortcomingis recognized through the publication of the SQUIRE (Standards of Quality ImprovementReporting Excellence) guidelines for quality improvement reporting.12 In addition, some of theprojects carrying out this type of work do not make available detailed descriptions of theirmethods due to proprietary concerns.

Finally, the literature focuses extensively on processes involved in carrying out a nationalclinical audit or quality improvement study, for example, the quality of the measures used andmany aspects of the measurement process. A good faith assumption is made that high qualitymeasurement and related processes will make a significant contribution to the improvementof the quality of clinical care provided to patients.

Review of drafts

Individuals who are actively involved in clinical audits or quality-of-care measurement atnational or international levels reviewed drafts of the standards and changes were made inresponse to reviewer comments.

Method 7


Definitions of terms

A national clinical audit is defined in the UK as a project that meets all of the followingcriteria:2

• achieved or intended coverage at national level

• main focus is the quality of clinical practice

• evaluates practice against clinical criteria or guidelines and/or collects outcomes data

• applies the audit cycle and/or monitors clinical or patient outcomes data in an ongoing way

• is prospective, that is, does not include retrospective reviews of adverse outcomes such as is carried out for confidential enquiries

• includes patients in their governance and includes patient provided data.

The purpose of national clinical audits is to engage all relevant healthcare professionals insystematic evaluation of their clinical practice against standards and to support and encourageimprovement in the quality of treatment and care.2

For purposes of formulating the standards, the definition of clinical audit used is as follows:4

a quality improvement process that seeks to improve patient care and outcomes throughsystematic review of care against explicit criteria and the implementation of change. Aspectsof the structure, processes, and outcomes of care are selected and systematically evaluatedagainst explicit criteria. Where indicated, changes are implemented at an individual, team orservice level and further monitoring is used to confirm improvement in healthcare delivery.

The use of this definition has major implications for the conduct of a national clinical audit.The SQUIRE guidelines define intended improvement as ‘changes or improvements in careprocesses and patient outcomes’.12 Guidance for large-scale improvement initiatives in ahealthcare system emphasize the importance of the following factors:13–14

• Motivation of the stakeholders — Founding stakeholders have high levels of intellectual and emotional engagement in the initiative and are explicit about what they want to accomplish.

• Foundation and leadership — Stakeholders understand the place of the initiative in the context of the healthcare system and what is needed to achieve desired improvements and they ensure there is respected leadership of the initiative and support of key opinion leaders for the initiative.

• Quantifiable aim and specific timeframe — The intended outcome of the initiative is stated explicitly and there is a clear timeframe for the achievement of the outcome.

• Planning and management, with appropriate resources — The initiative is carefully planned, considering the range of changes that may be needed, including system levers and policy changes that may be needed to effect change in participating organizations. Resources are appropriate at both central and local levels.

Definitions of terms 9

• Performance measures and evaluation — The impact of change intended to achieve improvement is measured through the collection of reliable and valid data linked to the change initiative.

• Defined changes or actions to be adopted — Participating organizations know what change has to take place and how to achieve the change. It is likely that participating organizations will adopt the intervention intended to achieve improvement.

• A healthcare system environment that broadly supports the initiative — The healthcare system in which the initiative is being carried out is fully understood by the leaders of the initiative and the system can cope with the initiative.

• A structured process for spreading changes throughout participating organizations — The initiative includes mechanisms for disseminating learning across participating organizations, that is, builds a learning network among the organizations involved.

Recognized features of a quality improvement process, as judged by an expert panel, includethe following:15

• Iterative process — The initiative involves iterative design and implementation of a set of specific changes for improving care, ie, a change package.

• Feedback at meetings involving participant leaders — Leaders of the improvement initiative at participating organizations meet at local level to review information on its implementation.

• Feedback of systematically collected data — The improvement initiative includes feedback of systematically collected data on implementation of change for improving care.

• Recognized change method — One or more recognized change methods are used in the improvement initiative.

• Data-driven — The design and/or implementation of a set of changes for improving care is driven by data collected systematically during implementation.

• Local conditions — Local conditions at participant organizations are taken into account in the design and/or implementation of the set of specific changes for improving care.


Standards for a national clinical audit

A summary of standards for national clinical audits or equivalent activities derived fromavailable literature is in the table. The standards cover structure, process and outcomesaspects of a national clinical audit.

1

2

3

4

5

Standards for a national clinical audit

Structural aspects

No.

The subject of a national clinical audit has an ethically basedjustification for the conduct of the audit as a means ofachieving improvement in the quality of care provided topatients.

There is continuous and effective governance of a nationalclinical audit by a group that assumes accountability for theeffectiveness of the audit.

All key stakeholders in a national clinical audit are identifiedand actively involved in all aspects of the design and conductof the audit. Key stakeholders include patients or service usersor their representatives.

Resources allocated to a national clinical audit at national levelare appropriate for the valid and reliable measurement andproactive facilitation of improvement of the quality of patientcare for the subject of the audit.

Roles and responsibilities for the design and conduct of anational clinical audit at national and participating site levelsare described clearly and there is a published project plan forthe execution of all aspects of the audit.

Ethical basis

Governance

Stakeholder involvement

Resources

Roles and responsibilitiesand project plan

Topic Standard

6

7

8

9

10

Process aspects

No.

The national clinical audit is continuously proactive inrecruiting the participation of all sites intended for inclusion.

The stated aims and objectives of a national clinical auditreflect an intention to drive improvements in the quality ofpatient care.

The population or sample selected for a national clinical auditis suitable for drawing valid conclusions in relation to thestated objectives of the audit and for minimizing bias in thefindings.

The strategy selected for collecting data for a national clinicalaudit is suitable for the stated objectives of the audit.

Quality-of-care measures used in a national clinical audit meetcriteria for such measures that are established by thegovernance board and are capable of producing valid andreliable data about the quality of care.

Recruitment

Improvement–driven aimsand objectives

Population or sample

Data collection strategy

Quality-of-care measures

Topic Standard


11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

Data specified for collection for a national clinical audit permitjudging compliance with quality-of-care measures in accordancewith the objectives of the audit, using currently available data tothe extent possible to avoid an unreasonable data collectionburden on participating sites.

A comprehensive protocol describes in detail all aspects ofdata collection and handling for a national clinical audit and isaccessible by all stakeholders and all participants in the audit.

Systems and processes are designed and fully implemented toprotect the identity of patients or service users whose care ismeasured in a national clinical audit consistent with applicablenational laws and regulations.

Any ethics-related issues in a national clinical audit arerecognized quickly and acted on appropriately.

Data collectors for a national clinical audit are trained tofollow the data collection protocol and are supported tosupply complete, accurate, reliable and timely data.

The design, data collection process, protocol and tools to beused in a national clinical audit are pilot tested before allintended sites are involved in participation and action is takenon the lessons learned from the pilot test.

The reliability of the data collected for a national clinical auditis tested and the findings are included in reports.

If a national clinical audit relies on linkages with electronicrecords or databases that contain information about individualpatients, systems are implemented to ensure the security of thelinkages.

Data submitted for a national clinical audit are managed,analysed appropriately and quality assured. Findings related todata quality and their implications are reported.

Preliminary data are available on a timely basis to participatingsites to enable a peer review and feedback process inparticipating sites before findings are published. The local peerreview includes analysis of findings and any cases notconsistent with good practice to identify and feed back to thenational organization any clinically acceptable exceptions notpreviously acknowledged in data collection.

Areas of good practice and shortcomings between what isdone and what should be done in the delivery of care areidentified and reported.

Participating sites are supported in identifying root causes ofany shortcomings in quality of care.

A strategy is adopted and implemented for proactivelyfacilitating the implementation of improvements in quality inparticipating sites.

Complete, timely and useful reports are provided for allstakeholders.

Communication with all stakeholders is maintainedthroughout all stages of the clinical audit.

Data elements and datasources

Data collection andhandling protocol ormanual

Protection of patientidentity

Identification and handlingof ethics issues

Training and support fordata collectors

Pilot testing

Reliability testing

Data linkages

Data quality management,analysis and assurance

Preliminary data and peerreview

Identification of goodpractice and shortcomingsin quality of care

Analysis of causes ofshortcomings in quality

Facilitation ofimprovements

Reports

Communication

Process aspects

No. Topic Standard


26

27

28

29

30

Outcome aspects

No.

A target is set and met for the number of eligible organizationsor services that participate in a national clinical audit.

Reliability testing or independent validation of data collectiondemonstrates that the reliability of data collected for theclinical audit meets or exceeds an established target.

Reports of preliminary data collected are supplied toparticipating sites promptly following the completion of datasubmission, that is, within a short number of weeks of thedeadline for submission of the data.

Reports of final data collected are provided promptly, that is,within a short number of weeks of the deadline for submissionby participating sites of any additional data following review ofpreliminary findings and cases that did not appear to meetquality-of-care measures.

Improvements in the quality of care from one time period, forexample, one year, to the next of the clinical audit are sufficientto justify continuation of a national clinical audit.

Level of participation

Reliability of data

Timeliness of reports onpreliminary data

Timeliness of completereports

Evidence of improvementsin quality of care

Topic Standard


Summary of literature supporting thestandards

Structural aspects of a national clinical audit or equivalent activity

Ethical basis

The justification for the conduct of a national clinical audit or equivalent activity needs toinclude analysis of benefits and risks to patients in the current healthcare delivery system andwhether or not the proposed clinical audit or equivalent activity responds to the risk–benefitanalysis.16–17

Considerations for a risk–benefit analysis could include the following:8, 18

• the incidence or prevalence of the condition in the intended population

• the impact the condition has for patients, families or carers, healthcare and/or social services

• evidence that current care is sub-optimal

• evidence of variation in the quality of care among providers and populations

• evidence available from which good quality care is defined such as a national guideline or systematic review

• evidence that significant improvements in health or social care might be achievable

• support for the audit from relevant clinical professions

• evidence that an audit might reduce inequalities in care

• support for national health policy objectives.

The rationale for carrying out a clinical audit should include a statement recognizing theethical basis for the audit, and specifically relate to the need to improve or assure the qualityof patient care.

Governance

For purposes of this work, governance refers to direction, oversight, high-level decision-making and accountability for a national clinical audit. Decision-making by the governancefunction should include design, funding, execution, reporting and dissemination ofinformation concerning the clinical audit. A goal of proper governance and oversight shouldbe transparency to all stakeholders in the audit on its operation, accountability for decision-making and reporting of results.19–20

There should be a named governance board for a national clinical audit, the board consistingof representatives of all the key stakeholders in the clinical audit including patients or theirimmediate representatives.20 The roles of the governance board should be defined explicitly,


reviewed periodically and revised as needed.20 The roles should include the assumption ofresponsibility and accountability for at least the following:20

• Executive or steering functions — Making major scientific, design, liaison, financial, resource, ethical and legal decisions concerning the objectives and operation of the audit. These decisions should be made with appropriate input from clinical, clinical audit and quality improvement experts and others as needed.

• Scientific functions — Overseeing the technical design, execution and reporting relating to the audit. The design of the audit, including the quality-of-care measures to be used, should reflect the input of experts in the clinical subject of the audit, clinical audit and quality improvement, information systems and possibly statistics and change management.

• Liaison and communication functions — Maintaining relationships with stakeholders in the audit, including monitoring the satisfaction of participants with the operation of the clinical audit; health care organizations and services participating and not participating in the audit; patients or service users and their representatives; funders; and other key stakeholders.

• Data access, use and publication functions — Overseeing access to and use of the data created through the clinical audit, presentation of the findings at scientific meetings and publication about the audit in scientific journals.

• Improvement implementation — Overseeing the effectiveness of the clinical audit in facilitating improvements in the quality of patient care in comparison to an intended level of improvement agreed by the governance board, and acting as needed to increase the effectiveness of the audit in achieving improvements in the quality of care.21

Stakeholder involvement

Stakeholders in a national clinical audit include clinicians, including doctors, nurses and otherhealthcare professionals; clinical audit and quality improvement specialists; patients or serviceusers and their representatives; managers of healthcare provider organizations; commissionersor funders of healthcare services; professional bodies; public health specialists; and potentiallyresearchers, regulators, and representatives of government or other groups. Stakeholders in anational audit should include representative participants in the audit and users of the datacollected.

All relevant stakeholders, including patients or service users and their representatives, shouldbe involved from the beginning of the clinical audit through to completion of the auditprocess,6, 22 including the proactive facilitation of improvements as needed. The views of allstakeholders in a national clinical audit should have equal weight in decision-making by thegroup responsible for governance of the audit.

When people are stakeholders in a national clinical audit, but do not have knowledge orprevious experience with clinical audit, appropriate training on clinical audit and their rolesand responsibilities in the national audit should be provided.6


Resources

Finite resources are available to support national clinical audits. The resources needed tooperate a national clinical audit at national level and the burden on participating sites shouldbe carefully considered when the audit is designed.19–20 The most efficient approaches ormethods that are likely to be effective in meeting the audit objectives should be used. Inparticular, entire population-based audits are not justified when extensive time is required fordata collection and validation, especially when appropriate statistically determined samplescould be used. Resources allocated to a national clinical audit should consider needs for:

• effective IT systems for collecting, quality controlling and reporting data

• facilitating changes in practice to achieve improvements in the quality of patient care in participating sites.

Roles and responsibilities and project plan

The roles and responsibilities of all people who are involved in the conduct of a nationalclinical audit at national and participating site levels should be defined.20 Defined roles for thenational organization operating the clinical audit should cover at least the following:

• governance functions related to the clinical audit

• design of all stages of the audit

• specification of the standards to be used as the basis for measuring quality

• data collection and submission, including preparation of data collectors

• data collation, analysis and management, including data security and data quality

• identification of good practice and shortcomings in care

• facilitation of analysis of shortcomings and action intended to achieve improvement

• liaison and ongoing communication with participants in the audit

• project management and regular performance reviews against the published project plan.

Defined roles for the national clinical audit at participating site level should cover at least thefollowing:

• leadership and accountability for participating in the clinical audit, including participating in any training required, submission of data and acting on the findings

• data collection, reliability testing of data collection and validation of the completeness and accuracy of data submitted

• action on any problems associated with the collection and submission of data

• review of preliminary data and review of cases in which care has not been consistent withgood practice as measured by the clinical audit

• analysis of the root causes of shortcomings in the provision of care at site level

• action to achieve improvements in care as indicated by the audit findings.


Individuals with responsibility for a national audit at national organization and participatingsite levels should actively demonstrate commitment to the audit and provide necessaryleadership for participation in the audit.6 Commitment should include accepting responsibilityto implement change as needed.6

The project plan should be clear about all timelines for the national organization andparticipating sites and services and the published timelines should be routinely met.22

Process aspects of a national clinical audit or equivalent activity

Recruitment

Factors that may motivate clinicians or organizations to participate in a national clinical auditinclude the perceived relevance, importance and scientific credibility of the audit; the abilityto compare a site’s data with national data and use the information at site level; the risks andburdens of participation; and the presence of any incentives for participation,20, 23 includingavailability of technical support. Recruitment and retention can affect how well an auditrepresents the target population; therefore, well-planned strategies for enrollment andretention are critical.20

Goals and processes for recruitment and retention should be described clearly and theireffectiveness assessed periodically. Variations between intended and actual levels ofrecruitment and actual participation should be continuously evaluated for their risk of introducing bias in the audit findings.20 Reasons for non-participation and non-continuation in the audit should be identified and acted on by the governance board forthe audit.

The recruitment process should be fully transparent providing for early and full disclosure ofthe requirements for participation to potential participating sites, including access to thecomplete project plan, defined roles and responsibilities, data collection tools to be used anda realistic estimate of resources needed for participation.

Improvement–driven aims and objectives

The stated aims and objectives of a national clinical audit should be explicit about theintention to have participating sites use the audit to drive improvements in the quality of care,where the audit demonstrates the need for improvement. The aims and objectives also shouldbe explicit about the intention of the organization operating the audit to lead and support theimplementation of any needed improvements in quality.

The design and conduct of the audit should use several strategies that are likely to facilitateimprovements in the quality of care in participating sites.21

Population or sample

The population or sample for a national clinical audit should be clearly defined6 and selectedto approximate the relevant characteristics of the target population as much as possible.20

Cases to be included in and excluded from the audit should be well specified to ensure that


findings are valid,21 and where relevant, the specification should be tested to ensure thatintended cases are not excluded, for example, by poor coding.24

The number of cases to be included, how cases are to be selected including confirmation ofa diagnosis or condition when relevant,25 and the length of time over which data are to becollected, including when data collection should stop, should be planned in accordance withthe overall purpose of the audit.21 The desired number of cases should be selected consideringthe magnitude of an expected, clinically meaningful effect; practicality; cost; and the intentionto show the effects of interventions to achieve improvement from one data collection periodto another.20 Also, the population or sample, along with the data collection strategy and datasources, should be selected to enhance representativeness and to minimize potential sourcesof bias,20 and if consistent with the audit’s objectives, enable international comparisons. Thenumber of cases in the population or sample should not place an unnecessary burden of datacollection on participating sites.21

The organization carrying out a national clinical audit should seek information on thepopulations or samples used in similar audits in other countries to enable comparisons offindings.

Provision should be made for reporting missing cases to enable judging the potential of biasin the findings.20

Data collection strategy

The strategy selected for collecting data, that is, retrospective, concurrent or prospective, forthe audit should be suitable for the stated objectives of the clinical audit, the availability andaccessibility of data required, and the timing of the audit. The strategy should not put anundue burden on those supplying data.

Quality-of-care measures

Clinical audit measures enable numeric quantifications, usually as percentages or ratios, ofhealthcare quality for the subject and objectives of a clinical audit.26

Criteria should be established by the governance board for the national clinical audit for theselection of quality-of-care measures to be used in the audit. Criteria could include any or allof the following:26–33

• Evidence based — Each measure must be based on a strong foundation of research. A measure of clinical process must be based on research showing that the process addressed by the measure, when performed correctly, leads to improved clinical outcomes.27

Any process-focused measure should address a clinical process quite proximate to the desired outcome, with relatively few intervening processes.19, 27 The beneficial effect of processes that are far upstream from outcomes can be nullified if important processes closer to the outcome are not performed effectively. Therefore, the measurement of processes not proximate to outcomes can be of little value.27 An exception can be for conditions for which there is a low level of scientific certainty of the link between processes of care and outcomes such as the care of people with Parkinson’s disease.34


If there is no clear valid published evidence of best practice available for the subject of the clinical audit, the measures used in the audit should be developed through a consensus-building process that involves key stakeholders.35–39

• Scientifically acceptable — The measure is fully specified. It accurately captures whether intended care has been delivered.27 The measure is tested and demonstrated to produce credible (valid) and consistent (reliable) information about the quality of care.35, 40–43

• Important to measure and report — The measure is important to enable making significant gains in healthcare quality. There is evidence available that demonstrates that current practice and/or health outcomes are less than optimal and that there is an obvious opportunity to improve quality of care through the use of the measure.

• Usable and relevant — Intended users of the national clinical audit findings can understand the results of the measure and are likely to find the results useful for quality improvement and decision-making.

• Feasible to collect — Data needed to demonstrate compliance with the measure is readily available and retrievable without undue burden on participants in the data collection process, including patients.

• Efficient — Measures used in a national clinical audit do not duplicate measurement of current practice or outcomes that may be undertaken through another national audit, programme or activity in the healthcare system. If the national clinical audit needs such data, safe and secure arrangements for data linkages should be used to avoid duplication of effort in capturing data.

• Measures the patient experience of care — Depending on the subject and objectives of the clinical audit, measurement should include patient perceptions of the care experience. 44–45

• Avoids risks to patients — The measure has minimal or no unintended adverse consequences (such as giving an antibiotic before laboratory confirmation of the need for an antibiotic to meet an imposed time limit for treatment).27

Each measure specification should describe at least the following:26–28, 46–48

• the exact aspect of care that is being measured about the provision of quality of care for the target group, often called the criterion

• complete description of the target group to be included for the measure, including reference to the time period in which cases are eligible for inclusion

• any clinically acceptable reasons or explanations that would provide justification to a clinical peer group of the failure to provide quality of care as defined, often called exceptions

• detailed explanation of how to collect the information needed, including definitions of terms used

• exactly how the numerator and denominator will be expressed and calculated for the measure

• if ratings are used, exactly how a rating is to be assigned

• if a measure is based on a survey, for example, of patient experience, the instructions for obtaining the sample, conducting the survey and guidance on minimum sample size.


The quality-of-care measures used in a national clinical audit should have at least contentand/or face validity, which should be established by the organization leading the audit.42, 49–53

In tests of validity, the sensitivity or positive predictive value (PPV) of the measures should bedetermined.54–61 Cases identified as ‘false positives’ should be identified and subject toindividual case review to identify how the measures should be improved.54–61

Measures used in a national clinical audit should be reviewed annually to ensure that eachmeasure is still clinically relevant and continues to meet the criteria established for inclusion inthe national clinical audit.

Data elements and data sources

Data elements are derived from the quality-of-care measures and their specifications. All termsused in the measures should be carefully defined including exact specifications of ranges ofany laboratory values if relevant, findings of investigations and synonyms and acceptableabbreviations.46, 62–63

Data specified for collection should permit judging compliance with quality-of-care measuresin accordance with the objectives of the audit, and precise directions should be provided formaking decisions concerning whether or not data available is consistent with the measures.46

A driving force for specifying data elements should be that the data could be used to supportquality improvement efforts at local level.21

The collection of extraneous, that is, ‘nice to know’ or optional, data should be collected fora clear purpose related to the audit objectives and kept to a minimim.20

Preferred sources for data to be retrieved or abstracted should be specified, especially whenthere are duplicate sources, for example, electronic and paper patient records. Directions forselecting the more valid data source should be provided. If secondary data sources, such asexisting databases, are to be used, the reliability of the data in the secondary source shouldbe tested and confirmed.42, 46, 64

Measurement of patient-reported outcomes should use only tools that have been designedand validated for this purpose.19 Measurement of patient experiences with the process of careshould use only tools that have been designed and validated for this purpose.

Data collection protocol or manual

A complete protocol should describe all aspects of data collection for a national clinical audit,6,

19, 35 including at least the following: data definitions; instructions for making decisions aboutcompliance with quality-of-care measures used in the audit; data sources to be used; intendeddata collectors and the competences needed to collect data correctly and completely; howcases to be included in the audit are to be selected; how data are to be recorded or captured;time frames for collection and submission of data; how the identity of patients involved in theaudit is to be protected; and how data are to be maintained at local level in case data haveto be validated or individual cases need to be identified for some reason; and how data areto be managed, collated, analysed and quality assured.46


The data collection protocol should be easily accessible to all stakeholders in an audit prior tothe need for participating sites to make a decision on participation in the audit.

Protection of patient identity

The governance board of a national clinical audit must assure that the systems and processesdesigned to protect the identity of patients or service users whose care is measured in theaudit are robust and consistent with applicable national laws and regulations, and that theyare continuously implemented in all sites participating in the audit. Data submitted byparticipating sites should be anonymized.

Identification and handling of ethics issues

Ethics issues in a clinical audit arise at three stages in the conduct of a clinical audit andprovision should be made to handle ethics issues that emerge at each stage:17, 65–66

• At the design stage — The governance board for a national clinical audit should ensure that all aspects of the design of a national clinical audit are checked against criteria that would flag a potential ethics issue in the design of the clinical audit and enable appropriate judgement about the issue.66

• When the findings of data collection for a national clinical audit are available — Findings of a national clinical audit should be reviewed by the governance board to identify if there was any risk for patients in the audit if their care was not provided consistent with good practice; if there are any patients in the audit for whom a life-threatening or quality-of-life-threatening shortcoming in care occurred; if data that could be used to identify any patient was disclosed; or if the audit revealed any clinically significant departure from usual clinical care. The board should assure that relevant sites are aware if any of these situations occurred and take appropriate action.

• When findings from repeat data collection for a national clinical audit are available — A clinical audit that does not address needed changes in practice should be reviewed by the governance board because of the potential failure to meet the ethical responsibilities of healthcare professionals and organizations to improve quality. If an audit indicates that effective practice is not being provided to patients, it is unethical to continue to provide substandard care and to withhold improvements from patients.

Training and support for data collectors

At each participating site, an individual should be named to be accountable for thecompletion and submission of data for an audit.20

Training on carrying out data collection and submission for a national clinical audit should beavailable for all clinical or non-clinical staff who will be responsible for collecting andsubmitting data.20–21, 35, 42, 46, 67–69 The training can be provided in a variety of ways includingonline training or case studies, e-learning, at network or other regular meetings, or liveworkshops. Training or operational manuals for all people with a role in the audit, butparticularly for people doing data collection and submission, should be easily accessible.20

Web-based, electronic and live support, with prompt response times, should be routinely andeasily accessible for data collectors for a national clinical audit.


Feedback from data collectors about the value of the training, materials and support providedshould regularly be sought and the findings acted on.

Pilot testing

The entire design, data collection process, protocol and tools to be used in a national clinicalaudit should be thoroughly pilot tested in several sites eligible for participation in the auditbefore a national clinical audit involves all intended sites in the audit.20, 35, 42–43, 67, 70 Pilot testingshould consider at least the effectiveness of the audit design and data collection process; thetime and cost involved in collecting and submitting data; the feasibility of collating data todemonstrate compliance with quality-of-care measures; and the likelihood that data collectedcan be acted on by the range of intended participant sites.46 Action should be taken on thelessons learned from the pilot test.

Reliability testing

Data collected for a national clinical audit should be tested for reliability using a methodologyintended to produce statistical confidence in the consistency of the data reported, sometimesreferred to as reabstracting data.35, 42, 68–69, 71–79 Reliability of data collection should be tested at eachparticipating site,68 which can be done by staff working at the site. In addition, data should beindependently reliability tested by the national clinical audit programme.19, 68–69, 71–73, 76–78, 80

The reliability of data for a national clinical audit (both determined at site and national levels)should be reported routinely, along with the findings of the audit.69, 72, 75

Data linkages

When a national clinical audit is making use of linkages with electronic records or databasesto capture information about the care provided to individual patients, robust systems andprocesses must be implemented to ensure the continuous security of the linkages.8, 19

To minimize the cost of duplicating data collection and collation by participating sites, whennational clinical audits require the same or highly similar patient-specific information, theorganizations operating the national audits should arrange to share the information throughappropriate secure data linkages.

Data quality management, analysis and assurance

The quality of data submitted for a national clinical audit should be monitored and assured.19

Systems that use front-end logic to check data completeness, duplication and values at thetime of data entry and provide data quality information to sites as they submit data should beused.19–22 The data entry system should enable sites to display and download their own data.21

The data collection protocol or manual should describe how missing, invalid, erroneous orcontradictory data are handled and how data collected are analysed.19–20 Participating sites inthe audit should be provided with timely feedback on the quality of data submitted.20


Quality assurance procedures concerning the quality assurance of data, procedures followedin all stages in the audit, and electronic systems used to support the audit should be includedin the data collection protocol and evidence of their routine implementation should beprovided to the governance board.20

The findings of quality assurance checks should be used to make changes in the conduct ofthe audit as needed.20

Analysis of data should use appropriate methods to consider sampling variability81 and riskadjustment, as appropriate, including reliability adjusting using Bayesian methods,82–83 asappropriate.

Observations about data quality should be included in reports.

Preliminary data and peer review of cases

Online, preferably real-time, reports of performance should be available for each site forparticipants to review site-specific performance in comparison to overall nationalperformance; performance in equivalent care settings; and best performance, for example,the performance of the top 10% or the top quartile of participating sites.21

Data collected should be formally reviewed by participating sites through a local peer groupprocess prior to the publication of data.19, 68 Participant sites should be able to carry out ad hocanalyses of their data.19 The peer review process at site level should review the analysis of risk-adjusted results and cases not consistent with good practice at local level and shouldinvolve preparation for implementation of improvements.84–85

Participating sites should be able to correct or modify the data held in the national clinical auditrelated to their performance prior to publication, if evidence of error or inappropriate judgementin data collection is supplied to the national clinical audit. Although there can be concern aboutgaming of results if participating clinicians review their own data prior to publication of nationalclinical audit findings, a study of clinicians identifying 650 medical exceptions to quality-of-caremeasures concluded through a peer review panel that 93.6% of the exceptions identified wereappropriate, 3.1% were inappropriate and 3.3% were of uncertain appropriateness.48 Afterclinical staff received direct feedback about inappropriate exceptions, 42% changedmanagement. The peer review process took less than 5 minutes per case.48

Identification of good practice and shortcomings in quality of patient care

In addition to identifying areas of good practice as demonstrated through the audit, a nationalclinical audit should identify explicitly the gaps between the quality of care being delivered topatients or service users and what was agreed as best practice as represented in the quality-of-care measures used in the audit.5, 19, 86–88 Participating sites should easily be able todetermine their levels of performance in relation to the identified shortcomings in quality ofcare.


Analyses of causes of shortcomings in quality

A national clinical audit should help participating sites to understand the causes of theirshortcomings in care, through provision of examples of analysis of variation89–90 and root causeanalysis of audit findings and case studies. The contributions of existing processes and systems;tools, including electronic prompts; policies (or the absence of needed policies or outdatedpolicies); availability and allocation of staff with appropriate knowledge and skills; the careenvironment and facilities; lack of multi-professional teamwork approach to care; failures indocumentation or information provision; breakdowns in communication, including with patientsor service users and their carers; aspects of the organizational or clinical service culture; or otherpotential causes of shortcomings in quality should be recognized explicitly in order to guideparticipating sites to implement the right actions to achieve improvements in care.41, 91

Facilitation of improvements

The Cochrane systematic review on audit and feedback concluded that feedback tohealthcare professionals about how their clinical practice compares with that of their peers oraccepted guidelines may not be effective in leading to improvements in practice.92 Healthcareorganizations that have high levels of compliance with good practice guidelines tend toprovide timely, individualized, non-punitive feedback of clinical audit findings to clinical staff.93

A national clinical audit should adopt and implement a framework for spread of good practiceamong participating sites.94–95 The sites should have access to a robust quality improvementframework and tools for facilitating a culture of improvement of patient care. The nationalaudit programme should identify what works best among improvement initiatives andencourage the rapid adoption of those initiatives among participating sites.8, 21, 68–69, 96 Asneeded, the national clinical audit programme should assume responsibility for creating andmarketing practical tools for participating sites to use to help improve their performance,including case studies, templates, risk stratification tools, pocket cards, drug dosingnomograms, discussion forums and educational materials available through webinars andslidesets.21, 38, 97

The national audit should promote examples of improved or sustained excellent performanceso that continually updated best practices are disseminated to all participating sites, includingthrough access to existing professional meetings and conferences and the use of awardsrecognizing substantially improved practice.21, 68–69, 98 Poster and oral presentations of localimprovement success stories at professional meetings and national conferences based on thenational clinical audit experience should be encouraged.

Generic recommendations that relate to the need to implement best practice, as representedby the quality-of-care measures used in an audit, should be avoided.

Timely, complete and informative reports

The national clinical audit should produce complete and useful reports in accordance with astrict timeline.8, 19 Reports should reflect the SQUIRE guidelines for reporting on qualityimprovement studies to the extent feasible.12


Reports to participating sites should be designed to help sites quickly and easily identify theirtop areas of needed improvement in relation to their peers.21 Reports on national clinicalaudits should be provided for groups of identified stakeholders, including patients and serviceusers.8

Communication among all stakeholders

Communication with all stakeholders should be maintained throughout every stage of theclinical audit,6, 8 particularly with management, clinicians and data collectors in participatingsites. Communication should be provided in the manner and within timeframes specified inthe governance arrangements, roles and responsibilities and project plan.

Outcome aspects of a national clinical audit or equivalent activity

Level of participation

The governance board should establish a target level of participation in a national clinicalaudit by eligible organizations or services and publish the participation rate in relation to thetarget.

Reliability of data

The governance board should establish a target level of reliability of data collected and publishthe findings of reliability testing of data collected. Reliability testing or independent validationof data collection should demonstrate that the reliability of data collected for the clinical auditis at least 90% or an equivalent kappa score.69, 99

Timeliness of reports on preliminary data

Reports of preliminary data collected should be supplied to participating sites for local reviewin real time or within a deadline of weeks following submission of the data, the deadline tobe established by the governance board.

Timeliness of complete reports

Complete reports of national clinical audits are supplied to participating sites and otherstakeholders within a deadline of weeks of the deadline for review of preliminary data byparticipating sites, the deadline to be established by the governance board.

Evidence of improvements in quality

Evidence of improvements in the quality of care from one time period to the next of theclinical audit must be sufficient to justify continuation of a national clinical audit, in view ofthe resources committed to the audit. The governance board should establish and applymeasures of success in judging the effectiveness of the clinical audit.

If the need for improvement is still evident but the level of improvement is not significant, thegovernance board should act to determine how data are being used to drive qualityimprovement. The board should make changes in the operation of the audit to influence theimplementation of actions needed in participating sites to achieve needed improvement.



Summary

This report is intended to identify possible standards for the design and operation of a nationalclinical audit or equivalent activity, using available literature on national audits or equivalentactivities and examples of best practice.

The conduct of a national clinical audit is dominated by a large number of process stages,each of which needs to be carried out in the best possible way in order to provide the bestquality of data about patient care. In addition, there are key structural aspects of designingand operating a national clinical audit that are critical to the success of the audit. A nationalclinical audit should be accountable for the outcomes the activity actually achieves, and ashort list of possible outcome measures is identified.

It is hoped that the possible standards in this report will be debated among stakeholders innational clinical audits, and that such standards become the basis for undertaking nationalclinical audits or equivalent projects in the future.

References 27

References

1. Amess M, Walshe K, Shaw C, Coles J. The audit activities of the medical Royal Colleges and their Faculties in England. London: CASPE Research; 1995.

2. What is national audit and patient outcomes measurement? National Clinical Audit and Outcomes Programme. London: Healthcare Quality Improvement Partnership; 2011. Available at: www.hqip.org.uk/national-clinical-audit-and-patient-outcomes-programme. Last accessed 14 April2012.

3. NCCA Criteria for Clinical Audit. London: National Centre for Clinical Audit; 1997.

4. National Institute for Clinical Excellence. Principles for Best Practice in Clinical Audit. Abingdon: Radcliffe Medical Press; 2002.

5. Copeland G. A Practical Handbook for Clinical Audit. NHS Clinical Governance Support Team; March 2005. Available at: www.hqip.org.uk/guidance-from-independent-organisations#handbook. Last accessed 14 April 2012.

6. Criteria and indicators of best practice in clinical audit. London: Healthcare Quality Improvement Partnership; 2009. Available at: www.hqip.org.uk/criteria-of-best-practice-in-clinical-audit. Last accessed 14 April 2012.

7. Burgess R, editor. New Principles of Best Practice in Clinical Audit. Oxford: Radcliffe Publishing; 2011.

8. Assessing and improving quality in national clinical audits. London: Healthcare Quality Improvement Partnership; October 2011. Available at: www.hqip.org.uk/assessing-quality-in-national-clinical-audit-guidance-goes-live. Last accessed 14 April 2012.

9. Healthcare Quality Improvement Partnership Report and accounts, 31 March 2011. Available at: www.hqip.org.uk/our-accounts. Last accessed 14 April 2012.

10. Boseley S. The great NHS funding mystery. If the NHS budget is being protected, why are hospitals facing huge cuts? The Guardian, Wednesday 8 June 2011.

11. Fundamental Review of Data Returns. A Consultation on the Recommendations of the Review. Department of Health; 2011. Available at: www.dh.gov.uk/en/Consultations/Liveconsultations/ DH_129725. Last accessed 14 April 2012.

12. Ogrinc G, Mooney SE, Estrada C, Foster R, Goldmann D, Hall LW, Huizinga MM, Liu SK, Mills P, Neily J, Nelson W, Pronovost PJ, Provost L, Rubenstein LV, Speroff T, Splaine M, Thomson R, Tomolo AM, Watts B. The SQUIRE (Standards for Quality Improvement Reporting Excellence) guidelines for quality improvement reporting: explanation and elaboration. Qual Saf Health Care 2008;17(Suppl I):i13–i32.

13. McCannon CJ, Schall MW, Perla RJ. Planning for Scale: A Guide for Designing Large-Scale Improvement Initiatives. IHI Innovation Series white paper. Cambridge, Massachusetts: Institute for Healthcare Improvement; 2008. Available at: www.ihi.org/knowledge/Pages/IHIWhitePapers/ PlanningforScale WhitePaper.aspx. Last accessed 14 April 2012.

14. Perla RJ, Bradbury EW, Gunther-Murphy C. Large-scale improvement initiatives in healthcare: A scan of the literature. J Healthc Qual 2011; doi:10.1111/j.1945–1474.2011.00164.x.

15. O’Neill SM, Hempel S, Lim YW, Danz MS, Foy R, Suttorp MJ, Shekelle PG, Rubenstein LV. Identifying continuous quality improvement publications: what makes an improvement intervention ‘CQI’? BMJ Qual Saf 2011. doi:10.1136/bmjqs.2010.050880.


16. Casarett D, Fox E, Tulsky JA. Recommendations for the Ethical Conduct of Quality Improvement. A Report of the National Ethics Committee of the Veterans Health Administration. National Center for Ethics in Health Care, Veterans Health Administration; May 2002.

17. Dixon N. Review of Ethics Issues related to Clinical Audit and Quality Improvement Activities. London: Healthcare Quality Improvement Partnership; 2009. Available at: www.hqip.org.uk/ethics-and-clinical-audit-and-quality-improvement. Last accessed 14 April 2012.

18. Criteria for selecting and prioritising topics for national clinical audits to be funded by DH NCAPOP. Department of Health; October 2009. Available at: www.dh.gov.uk/en/Publicationsandstatistics/ Publications/PublicationsPolicyAndGuidance/DH_112270. Last accessed 14 April 2012.

19. Strategic principles for a national approach to clinical quality registries and Operating principles for Australian clinical quality registries. Australian Commission on Safety and Quality in Healthcare. Available at: www.safetyandquality.gov.au. Last accessed 14 April 2012.

20. Gliklich RE, Dreyer NA, editors. Registries for Evaluating Patient Outcomes: A User’s Guide. 2nd edition. (Prepared by Outcome DEcIDE Center [Outcome Sciences, Inc. d/b/a Outcome] under Contract No. HHSA29020050035I TO3) AHRQ Publication No.10-EHC049. Rockville, MD: Agency for Healthcare Research and Quality. September 2010. Available at: www.effectivehealthcare.ahrq.gov. Last accessed 14 April 2012.

21. Peterson ED, Roe MT, Rumsfeld JS, Shaw RE, Brindis RG, Fonarow GC, Cannon CP. A Call to ACTION (acute coronary treatment and intervention outcomes network): a national effort to promote timely clinical feedback and support continuous quality improvement for acute myocardial infarction. Circ Cardiovasc Qual Outcomes 2009;2(5):491–9.

22. Pearson M. Lessons from the management of acute myocardial infarction. Heart 2005;91 Suppl 2:ii28–30. doi:10.1136/hrt.2005.062075.

23. Cornish JA, Tilney HS, Tan E, Thompson MR, Smith JJ, Tekkis PP. The National Bowel Cancer Audit Project: What do trusts think of the National Bowel Cancer Audit and how can it be improved? Tech Coloprocto 2011;15(1):53–9.

24. Harris AH, Reeder RN, Ellerbe LS, Bowe TR. Validation of the treatment identification strategy of the HEDIS addiction quality measures: Concordance with medical record review. BMC Health Serv Res 2011;11:73.

25. Jones RC, Dickson-Spillmann M, Mather MJ, Marks D, Shackell BS. Accuracy of diagnostic registers and management of chronic obstructive pulmonary disease: The Devon primary care audit. Respir Res 2008;9:62.

26. Remus D, Fraser I. Guidance for using the AHRQ Quality Indicators for hospital-level public reporting orpayment. Rockville, MD: Agency for Healthcare Research and Quality; 2004. AHRQ Pub. No. 04-0086-EF.Available at: www.leg.state.nv.us/74th/Interim_Agendas_Minutes_Exhibits/Exhibits/HealthCare/E030608J-6.pdf. Last accessed 14 April 2012.

27. Chassin MR, Loeb JM, Schmaltz SP, Wachter RM. Accountability measures — Using measurement to promote quality improvement. New Engl J Med 2010;363(7):683–8.

28. Criteria for Evaluation. Measure Evaluation Criteria. National Quality Forum; January 2011. Available at: www.qualityforum.org/docs/measure_evaluation_criteria.aspx. Last accessed 14 April 2012.

29. Kesmodel US, Jølving LR. Measuring and improving quality in obstetrics — The implementation of national indicators in Denmark. Acta Obstet Gynecol Scand 2011;90(4):295–304. doi:10.1111/j/1600–0412.2011.01078.x. Epub 2011 Mar 4.

30. Krumholz HM, Keenan PS, Brush JE Jr, Bufalino VJ, Chernew ME, Epstein AJ, Heidenreich PA, Ho V, Masoudi FA, Matchar DB, Normand SL, Rumsfeld JS, Schuur JD, Smith SC Jr, Spertus JA, Walsh MN, American Heart Association Interdisciplinary Council on Quality of Care and Outcomes Research,

References 29

American College of Cardiology Foundation. Standards for measures used for public reporting of efficiency in health care: A scientific statement from the American Heart Association Interdisciplinary Council on Quality of Care and Outcomes Research and the American College of Cardiology Foundation. J Am Coll Cardiol 2008;52(18):1518–26.

31. Maggard MA, McGory ML, Ko CY. Development of quality indicators: Lessons learned in bariatric surgery. Am Surg 2006;72(10):870–4.

32. Mainz J. Developing evidence-based clinical indicators: A state of the art methods primer. Int J Qual Health Care 2003;15 Suppl 1:i5–11.

33. Wollersheim H, Hermens R, Hulscher M, Braspenning J, Ouwens M, Schouten J, Marres H, Dijkstra R,Grol R. Clinical indicators: Development and applications. Neth J Med 2007;65(1):15–22.

34. van der Geer E, van Tuijl HF, Rutte CG. Performance management in healthcare: Performance indicator development, task uncertainty, and types of performance indicators. Soc Sci Med 2009;69(10):1523–30. Epub 2009 Sep 24.

35. Travaglia J, Debono D. Clinical audit: a comprehensive review of the literature. Sydney Australia: Centre for Clinical Governance Research, University of New South Wales; 2009. Available at: www.health.vic.gov.au/clinicalengagement/downloads/pasp/literature_review_clinical_audit.pdf. Last accessed 14 April 2012.

36. Cryer C. Injury outcome indicators — Validation matters. Int J Inj Contr Saf Promot 2005:12(4):219–24.

37. Ko DT, Wijeysundera HC, Zhu X, Richards J, Tu JV. National Expert Panel. Canadian quality indicators for percutaneous coronary interventions. Can J Cardiol 2008;24(12):899–903.

38. Lee DS, Tran C, Flintoft V, Grant FC, Liu PP, Tu JV. Canadian Cardiovascular Outcomes Research Team/Canadian Cardiovascular Society Heart Failure Quality Indicator Panel. CCORT/CCS quality indicators for congestive heart failure. Can J Cardiol 2003;19(4):357–64.

39. Mian S, Wagg A, Irwin P, Lowe D, Potter J, Pearson M. National audit of continence care: Laying the foundation. J Eval Clin Pract 2005:11(6):533–43.

40. Hearnshaw H, Harker R, Cheater F, Baker R, Grimshaw G. A study of the methods used to select review criteria for clinical audit. Health Technol Assess 2002;6(1):1–78.

41. Campbell SM, Hann M, Hacker J, Dueie A, Thapar A, Roland MO. Quality assessment for three common conditions in primary care: validity and reliability of review criteria developed by expert panels for angina, asthma and type 2 diabetes. Qual Saf Health Care 2002;11(2):125–30.

42. Dixon N, Pearce M. Guide to ensuring data quality in clinical audits. London: Healthcare Improvement Partnership; 2010. Available at: www.hqip.org.uk/hqip-issues-guide-to-ensuring-data-quality-in-clinical-audit. Last accessed 14 April 2012.

43. Wagg A, Mian S, Lowe D, Potter J. Pearson M. Continence Programme Working Party. National audit of continence care for older people: Results of a pilot study. J Eval Clin Pract2005;11(6):525–32.

44. High Quality Care for All. NHS Next Stage Review Final Report. Department of Health; June 2008, p. 47. Available at: www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAnd Guidance/DH_085825. Last accessed 14 April 2012.

45. Equity and excellence: Liberating the NHS, July 2010. Available at: www.dh.gov.uk/en/ Publicationsandstatistics/Publications/PublicationsPolicyAndGuidance/DH_117353. Last accessed 14 April 2012.

46. Dixon N, Pearce M. Clinical Audit Manual. Romsey, Hampshire UK: Healthcare Quality Quest; 2010.


47. What are quality measures? Overview. Centers for Medicare and Medicaid Services. Available at: www.cms.gov/QualityMeasures. Last accessed 21 February 2012. Also see Scope of Work, Quality Improvement Organizations. Available at: www.cms.gov/QualityImprovementOrgs/downloads/ 9thSOWBaseContract_C_08-01-2008_2_.pdf. Last accessed 14 April 2012.

48. Persell SD, Dolan NC, Friesema EM, Thompson JA, Kaiser D, Baker DW. Frequency of inappropriate medical exceptions to quality measures. Ann Intern Med 2010;152(4):225–31.

49. Davies S, McDonald KM, Schmidt E, Schultz E, Geppert J, Romano PS. Expanding the uses of AHRQ’s prevention quality indicators: Validity from the clinician perspective. Med Care 2011;49(8):679–85.

50. Harboe KM, Anthonsen K, Bardram L. Validation of data and indicators in the Danish Cholecystectomy Database. Int J Qual Health Care 2009;21(3):160–8. Epub 2009 Mar 21.

51. Head BJ, Aquilino ML, Johnson M, Reed D, Maas M, Moorhead S. Content validity and nursing sensitivity of community-level outcomes from the Nursing Outcomes Classification (NOC). J Nurs Scholarsh 2004;36(3):251–9.

52. Idvall E, Hamrin E, Siöström B, Unosson M. Quality indicators in postoperative pain management: A validation study. Scand J Caring Sci 2001;15(4):331–8.

53. Maas M, Johnson M, Moorhead S, Reed D, Sweeney S. Evaluation of the reliability and validity of nursing outcomes classification patient outcomes and measures. J Nurs Meas 2003;11(2):97–117.

54. Borzecki AM, Cevasco M, Chen Q, Shin M, Itani KM, Rosen AK. How valid is the AHRQ Patient Safety Indicator “postoperative physiologic and metabolic derangement”? J Am Coll Surg 2011; 212(6):968–76.e1–2. Epub 2011 Apr 13.

55. Borzecki AM, Kaafarani H, Cevasco M, Hickson K, Macdonald S, Shin M, Itani KM, Rosen AK. How valid is the AHRQ Patient Safety Indicator “postoperative hemorrhage or hematoma”? J Am Coll Surg 2011;212(6):946–53.e1–2. Epub 2011 Apr 7.

56. Borzecki AM, Kaafarani HM, Utter GH, Romano PS, Shin MH, Chen Q, Itani KM, Rosen AK. How valid is the AHRQ Patient Safety Indicator “postoperative respiratory failure”? J Am Coll Surg 2011;212(6):935–45. Epub 2011 Apr 7.

57. Cevasco M, Borzecki AM, Chen Q, Zrelak PA, Shin M, Romano PS, Itani KM, Rosen AK. Positive predictive value of the AHRQ Patient Safety Indicator “postoperative sepsis”: Implications for practice and policy. J Am Coll Surg 2011;212(6):954–61. Epub 2011 Apr 7.

58. Cevasco M, Borzecki AM, McClusky DA 3rd, Chen Q, Shin MH, Itani KM, Rosen AK. Positive predictive value of the AHRQ Patient Safety Indicator “postoperative wound dehiscence”. J Am Coll Surg 2011;212(6):962–7. Epub 2011 Apr 13.

59. Cevasco M, Borzecki AM, O’Brien QJ, Chen Q, Shin MH, Itani KM, Rosen AK. Validity of the AHRQ Patient Safety Indicator “central venous catheter-related blood stream infections”. J Am Coll Surg 2011;212(6):984–90. Epub 2011 Apr 13.

60. Chen Q, Rosen AK, Cevasco M, Shin M, Itani KM, Borzecki AM. Detecting patient safety indicators: How valid is “foreign body left during procedure” in the Veterans Health Administration? J Am Coll Surg 2011;212(6):977–83. Epub 2011 Apr 13.

61. Scanlon MC, Harris JM 2nd, Levy F, Sedman A. Evaluation of the Agency for Healthcare Research and Quality pediatric quality indicators. Pediatrics 2008;121(6):e1723–31. Epub 2008 May 12.

62. Clarke RT, Warnick J, Stretton K, Littlewood TJ. Improving the immediate management of neutropenic sepsis in the UK: Lessons from a national audit. Br J Haematol 2011;153(6):773–9.

63. Sketcher-Baker KM, Kamp MC, Connors JA, Martin DJ, Collins JE. Using the quality improvement cycle on clinical indicators — improve or remove? Med J Aust 2010;193 (8 Suppl):S104–6.

References 31

64. Kelly M, Lamah M. Evaluating the accuracy of data entry in a regional colorectal cancer database: Implications for national audit. Colorectal Dis 2007:9(4):337–9.

65. Dixon N. Ethics and clinical audit. In Burgess R, editor. New Principles of Best Practice in Clinical Audit. Oxford: Radcliffe Publishing; 2011.

66. Dixon N. Ethics and clinical audit and quality improvement (QI) — A guide for NHS organisations. London: Healthcare Quality Improvement Partnership; 2009. Available at: www.hqip.org.uk/clinical-audit-resources-3/#LCA. Last accessed 14 April 2012.

67. Indicators. International Quality Indicator Project. Available at: www.internationalqip.com/ indicators.aspx. Last accessed 14 April 2012.

68. Bridgewater B, Keogh B, Kinsman R, Walton P. Sixth National Adult Cardiac Surgical Database Report 2008, Demonstrating Quality. The Society for Cardiothoracic Surgery in Great Britain and Ireland and Dendrite Clinical Systems Ltd; 2009. Available at: www.e-dendrite.com. Last accessed 14 April 2012.

69. Shiloach M, Frencher SK Jr, Steeger JE, Rowell KS, Barzokis K, Tomeh MG, Richards KE, Ko CY, Hall BL. Toward robust information: Data quality and inter-rater reliability in the American College of Surgeons National Surgical Quality Improvement Program. J Am Coll Surg 2010;210:6–16.

70. Roberts CM, Lowe D, Barnes S, Pearson MG. A prospective study of the practical issues of local involvement in national audit of COPD. J Eval Clin Pract 2004;10(2):281–90.

71. Centers for Medicare and Medicaid Services and The Joint Commission. Data quality (Joint Commission Performance Measurement Systems Only), Chapter 5 in Specifications Manual for National Hospital Quality Measures (Specifications Manual); August 2005. Available at: www.qualitynet.org/dcs/ContentServer?c=Page&pagename=QnetPublic%2FPage%2FQnetTier 4&cid=1141662780097. Last accessed 18 April 2012.

72. Gompertz PH, Irwin P, Morris R, Lowe D, Rutledge Z, Rudd AG, Pearson MG. Reliability and validity of the Intercollegiate Stroke Audit Package. J Eval Clin Pract 2001;7(1):1–11.

73. Harrison DA, Brady AR, Rowan K. Case mix, outcome and length of stay for admissions to adult, general critical care units in England, Wales and Northern Ireland: The Intensive Care National Audit and Research Centre Case Mix Programme Database. Crit Care 2004;8(2):R99–111.

74. Hart S, Berqquist S, Gajewski B, Dunton N. Reliability testing of the National Database of Nursing Quality Indicators pressure ulcer indicator. J Nurs Care Qual 2006;21(3):256–65.

75. Herrett E, Smeeth L, Walker L, Weston C. MINAP Academic Group. The Myocardial Ischaemia National Audit Project (MINAP). Heart 2010;96(16):1264–7.

76. Scientific Applications International Cooperation. NPCR Education and Training Series (NETS), Module 3: Quality Control for Central Registries, Part 3: Reabstracting Audit Procedures. Availableat: www.cdc.gov/cancer/npcr/training/nets/module3/nets3_3.pdf. Last accessed 18 April 2012.

77. Standards Q&A. The Joint Commission DSC Update, Issue 3, 2010. Available at: http://manual.jointcommission.org/bin/view/Manual/Questions/UserQuestionId03Stk100888. Last accessed 18 April 2012.

78. To T, Estrabillo E, Wang C, Cicutto L. Examining intra-rater and inter-rater response agreement: a medical chart abstraction study of a community-based asthma care program. BMC Med Res Methodol 2008;8:29. doi: 10.1186/1471-2288-8-29. Available at: www.biomedcentral.com/1471-2288/8/29. Last accessed 18 April 2012.

79. Willis CD, Jolley DJ, McNeil JJ. Cameron PA, Phillips LE. Identifying and improving unreliable items in registries through data auditing. Int J Qual Health Care 2011;23(3):317–23. Epub 2011 Mar 25.


80. Kragsterman B, Pärsson H, Lindbäck J, Bergqvist D, Björck M. Swedish Vascular Registry (Swedvasc). Outcomes of carotid endarterectomy for asymptomatic stenosis in Sweden are improving: Results from a population-based registry. J Vasc Surg 2006;44(1):79–85.

81. Denton AS, Bond SJ, Matthews S, Bentzen SM, Maher EJ, UK Link Gynaecology-Oncology group. Treatment-related morbidity and hospital league tables: experience from a national audit of radiotherapy-induced morbidity in cervical carcinoma. Clin Oncol (R Coll Radiol) 2002;14(1):40–2.

82. Gajewski BJ, Mahnken JD, Dunton N. Improving quality indicator report cards through Bayesian modelling. BMC Med Res Methodol 2008;18:77.

83. Osborne NH, Ko CY, Upchurch GR Jr, Dimick JB. The impact of adjusting for reliability on hospital quality rankings in vascular surgery. J Vasc Surg 2011;53(1):1–5.

84. Mainz J, Krog BR, Bjørnshave B, Bartels P. Nationwide continuous quality improvement using clinical indicators: the Danish National Indicator Project. Int J Qual Health Care 2004;16(Suppl 1):i45–i50.

85. Wightman R, Fielding J, Green S. Audit of antipsychotic prescribing in dementia: Cambridgeshire results and lessons learnt. Psychiatr Danub 2011;23 Suppl 1:S126–9.

86. Seddon M, Buchanan J, EPIQ. Quality improvement in New Zealand healthcare. Part 3: achieving effective care through clinical audit. N Z Med J 2006;119(1239):U2108.

87. Gale CP, Roberts AP, Batin PD, Hall AS. Funnel plots, performance variation and the Myocardial Infarction National Audit Project 2003–2004. BMC Cardiovas Disord 2006;6:34.

88. Qvist P, Rasmussen L, Bonnevie B. Gjørup T. Repeated measurement of generic indicators: A Danish national program to benchmark and improve quality of care. Int J Qual Health Care2004;16(2):141–8.

89. Barter S, Drinkwater K, Remedios D. National audit of provision of MRI services 2006/07. Clin Radiol2009;64(3):2384–90.

90. Tawn DJ, Squire CJ, Mohammed MA, Adam EJ. National audit of the sensitivity of double-contrast barium enema for colorectal carcinoma, using control charts for the Royal College of Radiologists Clinical Radiology Audit Sub-Committee. Clin Radiol 2005;60(5):558–64.

91. Dixon N. Getting Clinical Audit Right to Benefit Patients. Romsey, Hampshire UK: Healthcare Quality Quest;2011.

92. Jamvedt G, Young JM, Kristoffersen DT, O’Brien MA, Oxman A. Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database of Systematic Reviews 2006, Issue 2. Art. No.:CD000259. doi:10.1002.14651858. CD000259.pub2.

93. Hysong SJ, Best RG, Pugh JA. Audit and feedback and clinical practice guideline adherence: Making feedback actionable. Implement Sci 2006;28;1:9.

94. Massoud MR, Nielsen GA, Nolan K, Nolan T, Schall MW, Sevin C. A Framework for Spread: From Local Improvements to System-wide Change. IHI Innovation Series white paper. Cambridge, Massachusetts: Institute for Healthcare Improvement; 2006. Available at: www.IHI.org/knowledge/Pages/IHIWhitePapers/AFrameworkforSpreadWhitePaper.aspx. Last accessed 14 April 2012.

95. Nolan K, Schall MQ, Erb F, Nolan T. Using a framework for spread: The case of patient access in the Veterans Health Administration. Jt Comm J Qual Patient Safe 2005; Jun;31(6):339–47.

96. Rudd AG, Lowe D, Irwin P, Rutledge Z, Pearson M, Intercollegiate Stroke Working Party. Qual Health Care 2001;10(3):141–51.

97. Grant RL, Batty GM, Aggarwal R, Lowe D, Potter JM, Pearson MG, Oborne A, Jackson SH. National sentinel clinical audit of evidence-based prescribing for older people: Methodology and development. J Eval Clin Pract 2002;8(2):189–98.

References 33

98. Husk J. Achieving changes in practice from national audit: National audit of the organization of services for falls and bone health in older people. J Eval Clin Pract 2008;14(6):974–8.

99. Williams SC, Watt A, Schmaltz SP, Koss RG, Loeb JM. Assessing the reliability of standardized performance indicators. Int J Qual Health Care 2006;18(3):246–55.

Acknowledgements

We wish to acknowledge and thank the following people for their extensive reviews andcomments on earlier drafts of this document and their support of this work.

Dr M. Rashad Massoud, Director, USAID Health Care Improvement Project, Senior VicePresident, Quality & Performance Institute, University Research Co, Bethesda, Maryland, USA

Dr Nikolas Matthes, Vice President, Research and Development, Press Ganey, AssistantProfessor (adjunct), Johns Hopkins University, Elkridge, Maryland, USA

Professor Sandy Middleton, Director, Nursing Research Institute, St Vincent’s and Mater HealthSydney, Australian Catholic University and Director, National Centre for Clinical OutcomesResearch (NaCCOR), Sydney, Australia

Dr Elke Pieper, Clinical Effectiveness Manager, King’s College NHS Foundation Trust, London

Sarah Powell, Clinical Audit and Effectiveness Manager, Papworth Hospital NHS FoundationTrust, Cambridge

Sue Venables, Senior Clinical Audit Facilitator and NICE Facilitator, Kent and MedwayPartnership Trust, West Malling, Kent

Professor Ulrich Wienand, University of Ferrara, Ferrrara, Italy

In addition, we acknowledge the support and positive feedback provided for this work by thechair and members of the National Audit and Governance Group (NAGG) in England.


Shelley Farm, Shelley Lane, Ower, Romsey, Hampshire SO51 6ASTelephone: 023 8081 4024 Fax: 023 8081 4020Email: [email protected] Web: www.hqq.co.uk

Standards for National Clinical Audits- · PDF filenational clinical audit, ... For purposes...

Documents

Transcript of Standards for National Clinical Audits- · PDF filenational clinical audit, ... For purposes...