DOCUMENT 1 Critical Appraisal of Clinical Performance Measures … · physicians (Article 137e of...

Ärztliche Zentralstelle Qualitätssicherung __________________________________________________________________________________________

DOCUMENT 1

Critical Appraisal of Clinical PerformanceMeasures for Quality Management Quality Criteriaand Quality Indicators in Health CareConsensus Paper presented by the German Medical Association, the National Associationof Statutory Health Insurance Physicians and Association of the Scientific Medical Societies in Germany (AWMF)

DOCUMENT 2

Prioritisation of Topics Related to Health Problemsor Health Care Delivery Problemsfor the Guideline-Clearing-Projectof the German Guideline Clearinghouse

___________________________________________________________________________________________________

Agency for Quality in Medicine – Joint Institution of the German Medical Association and the

National Association of Statutory Health Insurance Physicians

___________________________________________________________________________________________________

URBAN & FISCHER VERLAG • JENA

DOCUMENT 1

Critical Appraisal of Clinical PerformanceMeasures for Quality Management Quality Criteriaand Quality Indicators in Health CareConsensus Paper presented by the German Medical Association, the National Associationof Statutory Health Insurance Physicians and Association of the Scientific Medical Societies in Germany (AWMF)

MASTHEAD ________________________________________________________________________

EDITOR:

Agency for Quality in Medicine (AQuMed) Joint Institution of the German Medical Association and the National Associa-tion of Statutory Health Insurance Physicians

WORKING GROUP QUALITY INDICATORS OF THE AQUMED EXPERT PANEL ON GUIDELINES

Dr. L.Altenhofen, ZI (Central Institute for Public Health System Physicians)Prof. Dr.W. Brech, Physicians’ Association of South WuerttembergDr. G. Brenner, ZI (Central Institute for Public Health System Physicians)Prof. Dr. M. Geraedts, University of DuesseldorfE. Gramsch, Physicians’ Association of Lower Saxony, GermanyProf. Dr. F.-W. Kolkmann, Regional Medical Association of Baden-WuerttembergDr. H. G. Krumpaszky, German Medical AssociationProf. Dr.W. Lorenz,AWMFDr. U. Oesingmann, Physicians’ Association of Westphalia-LippeProf. Dr. Dr. G. Ollenschlaeger,AQuMedDr. P. Rheinberger, Federal Association of SHI PhysiciansProf. Dr. H.-K. Selbmann, University of TuebingenDr. D. von Stillfried, Federal Association of SHI PhysiciansF. Stobrawa, German Medical AssociationH.Thole,AQuMed (Editorial Office)

PLEASE CONTACT:

Aerztliche Zentralstelle Qualitaetssicherung (Agency for Quality in Medicine):Aachener Strasse 233–237D 50931 Cologne (Koeln)

Phone: +49 - 221 - 4004 - 501; Fax: +49 - 221 - 4004 - 590 e-mail: [email protected] ·Internet: http://www.azq.de

© 2001 Agency for Quality in Medicine

2

TABLE OF CONTENTS

1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2.1 Clinical Performance Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2.2 Aspects of Care . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2.3 Criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2.4 (Quality) Indicators / Performance Thresholds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2.5 Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3. Objectives of Clinical Performance Measures for Quality Assurance and Quality Management . .

3.1 General Aims . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4. Quality, Practicability and Financiability of Programmes for Clinical Performance Measures . . .

4.1 General Criteria of Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5. Prerequisites for the Realisation of Programmes for the Development and Application of ClinicalPerformance Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5.1 General Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5.2 Systematic Development, Implementation, Evaluation and Maintenance of Clinical Performance Mea-sures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6. Specification and Assessment of the Quality/Practicability of Clinical Performance Measures . . .

6.1 Specifying the Characteristics of Clinical Performance Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6.2 Appraising the Quality and Practicability of Clinical Performance Measures . . . . . . . . . . . . . . . . . . . . . . .

7. Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7.1 Checklist to Appraise the Quality of Clinical Performance Measures (CPMs) for Quality Management

7.2 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3

4

4

4

4

5

5

6

6

6

7

7

9

9

9

10

10

10

13

13

14

2. DEFINITIONS

2.1 Clinical PerformanceMeasures

• Clinical performance measuresmeasure attributes of health care as-pects (e.g. structures, processes oroutcomes) to be assessed within thescope of quality improvement activ-ities (according to [4]). • The internationally used term“clinical performance measure” is ageneric term that includes both “cri-terion” and “(quality) indicator”.

2.2 Aspects of Care• Individual structures, processes oroutcomes of clinical care that areconsidered suitable for assessmentin the context of quality manage-ment may also be described as as-pects of clinical care.• A number of suggestions havebeen made for selecting health careaspects that should be incorporatedinto clinical quality management.Aspects of care to be considered forappraisal include procedures• “with a high frequency of occur-rence, a high risk for patients, orproblem-prone procedures” [ac-cording to 5], – that are characterized by overuse,underuse or misuse [according to 1

and Advisory Council report],

1. INTRODUCTION

With the amendment of the FifthBook of Social Code (SGB V) in1999 the German legislator has madethe “„consideration of criteria forassessing appropriate and efficienthealth care services on the basis ofevidence-based guidelines” imme-diately binding upon sickness funds,approved hospitals and contractingphysicians (Article 137e of the FifthBook of Social Code).The Co-ordinating Committee 1. is expected to determine criteriafor assessing the appropriate and ef-ficient management of at least 10 dis-eases per year, in particular on thebasis of evidence-based clinical prac-tice guidelines. Diseases should beselected according to indications thatdiagnostic or therapeutic approachesto these diseases were underused,overused or misused and that elimi-nating these deficiencies could havea strong impact on morbidity andmortality in the population;2. will give recommendations forthe procedures required for the im-plementation and evaluation of thecriteria set out under 1, in particularregarding the documentation to beconducted by health care providers.The decisions are immediately bind-ing upon sickness funds, hospitalsand contracting physicians. The Co-ordinating Committee will give rec-ommendations for the rules of reim-bursement of documentation ser-vices, especially regarding the docu-mentation to be conducted by healthcare providers.The above-mentioned law [1] con-tains a number of unspecific termsthat are not easy to interpret andwhich seem to require clarification[2]. The aim of this consensus paperis to establish and specify both defi-nitions and objectives for guideline-supported criteria or indicators that

need to be determined for the Ger-man area of care and internationallyaccepted quality requirements forthese tools of quality promotion andquality control of health care ser-vices by means of the following“Evaluation Criteria for ClinicalPerformance Measures in QualityManagement”. Form and content of this list followthe “Evaluation Criteria for Clini-cal Practice Guidelines” developedby the German Medical Associationand the National Association of Statu-tory Health Insurance Physicians [3],which have formed the basis for thedevelopment of the German Guide-line Clearinghouse in recent years. Similar to the approach that hasbeen adopted for guidelines it rec-ommended to critically appraise thecriteria and indicators of quality aswell. As experience with GuidelineClearinghouses suggests this is oneway of avoiding cost-intensive andinappropriate over-regulation inhealth care. According to the authors, the suc-cessful dissemination and applicationof these evaluation criteria will befostered by adequate information, bytaking patient interests into accountand by involving the general public inthe debate concerning the criteria forclinical performance measures.

4

Fig. 1. Hierarchy of terms for quality assessment in health care: suggested by theAQuMed Working Group on Quality Indicators (acc. to Geraedts)

1Medical care is considered inadequate if the kind or extent of services fail to achieve the medical treatment target or if parts ofthe population do not have access to necessary health services. Treatment is regarded as incorrect if it causes more harm thanbenefit. Excessive treatment must be assumed if medically unnecessary services or services are provided that fail to have any im-pact on the treatment target.

– “with proven inter-provider vari-ability; processes of care that haverecently undergone major changes;procedures with a strong financialimpact; processes considered to bemeasurable or adaptable or proce-dures whose providers support theintended changes of their health careservices” [according to 6], – “that carry the possibility of im-proving health care delivery and out-comes that have a high potential ofconsumer interest or might be used toimprove decision-making processes(for or against specific providers ofcare)” [according to 7].• As regards the selection of the as-pects of care that are to be integratedinto clinical quality management,the authors of the present paper rec-ommend that evidence to prove theirvalidity be presented. • Example of a specific aspect ofcare: “Diagnosis in patients withsuspected stroke”.

2.3 Criteria• Criteria for assessing specific as-pects of health services are those at-tributes that are usually considered es-sential for high quality medical care. • The Joint Commission on Accred-itation of Healthcare Organizations(JCAHO) has developed an interna-tionally used list of such criteria.The criteria suggested by JCAHO[8] include: • Accessibility of care: the ease withwhich patients can obtain the carethat they need when they need it• Appropriateness of care: the degreeto which the correct care is provided,given the current state of the art• Continuity of care: the degree towhich the care needed by patients isco-ordinated among practitioners andacross organisations and time)• Effectiveness of care: the degreeto which care (for example. a proce-dure) is provided in the correct man-ner (that is, without error) given thecurrent state of the art• Efficacy of care: the degree towhich a service has the potential tomeet the need for which it is used

• Efficiency of care: the degree towhich the care received has the de-sired effect with a minimum of ef-fort, expense or waste• Patient perspective issues: the de-gree to which patients (and theirfamilies) are involved in the deci-sion-making processes in matterspertaining to their health, and thedegree to which they are satisfiedwith their care• Safety of the care environment:the degree to which the environmentis free from hazard or danger• Timeliness of care: the degree towhich care is provided to patientswhen it is needed.• As regards the formulation of alist of criteria applicable to the Ger-man health care system, the abovelist will have to be supplemented,reduced or otherwise altered.• A criterion which is consideredparticularly relevant to a specific as-pect of care and which should there-fore be prioritised for appraisal with-in the scope of quality management,is sometimes called a “tracer” (e.g.ZI positional paper [9]). As the term“tracer” has been applied in a similarway in the context of both quality in-dicators (see below) and aspects ofhealth care (see above), the authorsrather recommend against its usehere. • Example of a specific criterion:“Timeliness of the diagnosis in pa-tients with suspected stroke”.

2.4 (Quality) Indicators/Performance Thresholds

• Quality indicators are measures thatallow distinguishing between highand low quality of structures, process-es, or outcomes of medical care. • Quality indicators are ancillaryparameters which indirectly reflectthe quality of a unit by means ofnumbers and ratios between num-bers. They could also be describedas quality-related indices. [10]• The quality of health care is acomplex phenomenon which gener-ally can only be described by apply-ing several indicators, since any sin-

gle indicator can measure no morethan single aspects of quality. There-fore it is reasonable to arrange sev-eral indicators in the form of indica-tor profiles in order to evaluate aspecific aspect of health care or aspecific criterion.• Indicators are performance assess-ment tools that help to direct attentionto those aspects of care that needmore intensive monitoring withinanyone organisation [according to 5].• According to the definition of theJoint Commission on Accreditationof Healthcare Organizations [5], in-dicators and/or clinical performancemeasures are suitable for, amongother things, monitoring and assess-ing the quality of relevant leading,controlling and management activi-ties and of clinical and ancillary ac-tivities, which have an impact on pa-tient outcomes of care. • Performance thresholds includethose intervals in which the actualparameter values of specific qualityindicators are considered good or in-significant A performance valuerepresents the performance thresh-old whose upper and lower limit co-incide [according to 10]. • Different performance thresholdsare conceivable for a quality indica-tor – depending on specific influenc-ing factors. In the first place patient-related (e.g. case mix, especiallywhen influenced by co-morbidityand severity of disease) and organi-sation-related (esp. structural fea-tures) influencing factors should betaken into account [5]. In case theseinfluencing factors may be eliminat-ed statistically (adjustment) a singleperformance threshold should besufficient.• The discriminating ability of qual-ity indicators and their respectiveperformance thresholds largely de-pends on their sensitivity and speci-ficity. An optimally selected perfor-mance threshold will identify almostany quality problems (high sensitiv-ity) without raising too many falsealarms (high specificity). In thiscontext, positive and negative pre-dictive values are affected by theprevalence of quality problems.

5

• Example of a quality indicator:The proportion of patients with sus-pected stroke (in a hospital, in a re-gion) who have been diagnosed with-in a critical time frame (2 to 6 hoursafter the first signs of a stroke) out ofall suspected patients of this hospi-tal/region (performance threshold:>80%).• Example of a quality indicatorwhich might have a negative impacton patient care when considered inisolation: The proportion of patientswith diabetes in some area of care,whose HbA1c value is below the ref-erence value. This indicator wouldhave to be supplemented with anoth-er indicator measuring the inci-dence of hypoglycaemia; otherwiseovercorrection of blood glucose lev-els with potentially harmful effectsto the patient might create a particu-larly positive impression of thepractice or network in question. In the meantime a number of lists ofclinical performance measures havebeen available internationally. As faras the development of clinical per-formance measures for the Germanhealth care system is concerned, itmight be worthwhile to make use ofthese lists. Some of the most rele-vant sources of information havebeen listed in Table 1.

2.5 GuidelinesClinical practice guidelines are sys-tematically developed statementsthat aim to assist both physiciansand patients with decisions aboutappropriate health care for specificclinical problems. [11,12]Guidelines represent a consensusthat has been achieved by severalexperts from different backgrounds,representatives of professional asso-ciations and working groups,methodologists and patients accord-ing to defined and transparent proce-dures. Guidelines that meet certainmethodological requirements(Guideline Manual, [12]) may beused to derive clinical performancemeasures. Methodological require-ments for the systematic develop-

ment of clinical performance mea-sures from guidelines will be ad-dressed in Chapter 5.2. In general, these measures need tobe further specified in terms of per-formance values and performancethresholds. The criteria that apply tothis specification have been sum-marised in the “Checklist to Ap-praise Clinical Performance Mea-sures for Quality Management”.An example of how quality indica-tors can be derived from guidelinesis the aspect of care of preoperativediagnosis and surgical treatment ofrectal carcinoma [Guideline of theGerman Cancer Society (DKG) andquality indicators according to Her-manek for the Information Centrefor Standards in Oncology (ISTO)of the German Cancer Society]. Forthe sake of transparency a meticu-lous comparison between guidelinerecommendations and quality indi-cators is considered necessary [13].

3. OBJECTIVES OF CLINICALPERFORMANCE MEASURESFOR QUALITY ASSURANCEAND QUALITY MANAGEMENT

3.1 General Aims

Clinical performance measures forma basis for • securing and improving the provi-sion of health care services to thepopulation,• avoiding unnecessary and outdat-ed clinical measures and unneces-sary costs,• reducing unwanted quality varia-tions in the field of medical care,• implementing systematically de-veloped aids for decision-making inmedical practice,• encouraging clinical managementthat is scientifically accounted for,economically adequate and takingpatients’ needs and preferences intoaccount,

6

Table 1. Examples of Lists of Clinical Performance Measures and Performance Indica-tor Sets.

Name Source of Information

Computerized Needs Oriented Quality Agency for Healthcare Research andMeasurement Evaluation System Quality (www.ahcpr.gov)(CONQUEST)

National Library of Quality Indicators and Joint Commission on Accreditation ofORYX Programme Healthcare Organizations

(www.jcaho.org)

Health Plan Employer Data and National Commission for QualityInformation Set (HEDIS®) Assurance (www.ncqa.org)

FACCT Quality Measures Foundation for Accountability (www.facct.org)

Quality Indicator Project® Association of Maryland Hospitals & Health Systems (www.qiproject.org)

Acute Health Clinical Indicator Project Department of Human Services Victoria,Australia (www.dhs.vic.gov.au/ahs/quality/clinical.htm)

NHS Performance Indicators National Health Service Executive (www.doh.gov.uk)

Clearing House Databases UK Clearinghouse on Health Outcomes(www.leeds.ac.uk/nuffield/infoservices/UKCH/home.html)

Zuericher Indikatoren-Set Verein Outcome Zuerich(www.vereinoutcome.ch)

• informing the general public (pa-tients, health insurers, regulators,health care professionals etc.) aboutnecessary and common medical pro-cedures for specific health risks andmedical disorders [15, modified ac-cording to 15].The aim of clinical performancemeasures is • to continuously improve treat-ment outcomes and patient care [ac-cording to 16]• the objective and quantitative as-sessment of the clinical effective-

ness and cost-effectiveness of healthcare [according to 17]• their application within the frame-work of internal quality manage-ment and external quality assurance(specification of quality), [18]• to provide the possibilities for in-forming the general public (patients,health insurers, regulators, healthcare professionals etc.) about thequality of health care services or ofanyone health care organisation [18– modified version].

4. QUALITY, PRACTICABILITYAND FINANCIABILITY OFPROGRAMMES FORCLINICAL PERFORMANCEMEASURES

4.1 General Criteria of Quality

The realisation of programmes for thedevelopment and application of clini-cal performance measures – just as ofguidelines [12, 19] – is largely depen-dent on their quality, practicabilityand financiability [20, 21, 22, 23, 24].

7

Table 2. The relation between guideline recommendations and clinical performance measures (practice example). Aspect of care: Di-agnosis and therapy in patients with suspected rectal carcinoma.

Guideline Recommendation (2) Quality Criterion Quality Indicator(Performance Threshold) (3)

Pretherapeutic diagnosis Appropriateness of care Proportion of bioptic examinations in Necessary examinations include: patients who had rigid rectoscopy as a

percentage of all electively operated patients with carcinoma of the middle and lower thirds of the rectum (> 90%)

Pretherapeutic diagnosis Appropriateness of care Proportion of patients with endoluminalExaminations that may prove helpful sonography out in the individual case:of all the patients with rectal carcinoma who had local excision (> 90%)– endosonography is mandatory prior

to local excision

The treatment of rectal carcinoma should Appropriateness of care Proportion of patients with preoperativealways be planned on the basis of the results of a histological confirmation of cancer out ofhistological examination. Apart from securing all the patients operated for rectal carcinomathe diagnosis of carcinoma pretherapeutically (approx. 95%)the aim should be to achieve a tumourclassification according to the World Health Organization (WHO) guidelines.

Surgical treatment with curative intent: Patient perspective issues Proportion of patients with continence-savingIf possible, continence-saving surgical protectomy out of all the curatively (R0)techniques should be preferred. treated patients (< 30%)

Surgical treatment with curative intent: Appropriateness of care Proportion of patients with total mesorectumFor tumours of the lowe two thirds of the excision down to the pelvic floor out of all rectum the mesorectum should be removed the patients with complete radical excisioncompletely down to the puborectal sling of rectal carcinoma of the middle and lower(total mesorectal excision). third of the rectum (R0) (> 90%)werden.

2 Eigler FW, Gabbert H, Herfarth Ch, Hermanek P, Hohenberger W, Hossfeld DK, Junginger Th, Kruck P, Meyer HJ, Pichlmaier H, Sauer R,Stock W: Rektumkarzinom. In: Deutsche Krebsgesellschaft e.V. (Hrsg.): Qualitätssicherung in der Onkologie – Diagnose und Therapiemaligner Erkrankungen. Interdisziplinäre Leitlinien 1999. Koordination: Informationszentrum für Standards in der Onkologie (ISTO).München, Zuckerschwerdt 1999: 123–1383 Hermanek, P: Qualitätsmanagement bei Diagnose und Therapie kolorektaler Karzinome. Leber, Magen, Darm 1996; 26: 20–24

8

Table 3. Minimum Requirements for Clinical Performance Measures [according to 21].

• Collection of data for clinical performance measures must be easy.

• A relation will have to be demonstrated between the clinical performance measure and the quality of health care services. Ideally,there should be evidence from well-designed trials to prove this relation.

• Clinical performance measures are intended to provide the most precise information possible about the quality of health care services.

• The results of programmes for quality assessment by means of performance measures should generally be uninfluencable by subjec-tive assessments which otherwise might confound an unbiased and correct documentation (problem of bias). This is particularly im-portant if incentives or sanctions are tied up with the respective parameter values of clinical performance measures.

• Data of clinical performance measures must be adjustable with regard to potential influencing factors that bear no relation to thequality of health care services.

• For each clinical performance measure as well as the quality evaluation programme derived from it clear objectives should be estab-lished. A clinical performance measure programme targeted at providers’ interest, knowledge and motivation regarding quality ofcare in the sense of internal and confidential quality management – must definitely meet other requirements than projects of exter-nal control or projects in a context of competition between different providers. etern.

Table 4. Quality Criteria for Clinical Performance Measures.

A Importance of Clinical Performance Measures (CPMs) for Health Care Systems

Consensus on the National programs for the development and use of CPMs (e.g. in accordance with Article 137e of the Fifth Bookrelative importance of Social Code) should be established and realised only for those aspects of care for which consensus existsof aspects of between stake-holders about their impact on the overall quality of medical care. Legal requirements have to be clinical care taken into account.

Controllability of CPMs should refer to those aspects of care that have a high potential of quality improvement when qualityaspects of clinical care management techniques are employed.

Relevance CPMs and their performance thresholds must be relevant to health care practice.Target groups and aspects ofcare being focused by CPMs should be clearly defined and described.

Balance CPMs and the respective quality evaluation programmes should assess the equality of health care in a balancedmanner, i.e. their effects and side effects on medical care should be taken into account. Side effects could result from emphasising certain processes while at the same time neglecting other equally important but unmeasured processes. Therefore, sets of CPMs or CPM profiles should be used to assess aspects of care if possible.

Cost-benefit ratio The purpose of CPMs /CPM programs is to improve health care delivery. Cost-effectiveness of individual CPMsand programs used to assess quality by means of CPMs should be taken into account in comparison to othermethods affecting the envisaged aspects of care.

Updating/Systematic CPM programs should include information about the date and mode for reviewing, the methodological quality of Reviewing CPMs as well as information about the individual(s)/body/ies) performing the review.

B Quality Requirements for Clinical Performance Measures

Comprehensibility CPMs must be clearly and distinctly defined by using easily comprehensible language and precise terminology.

Validity CPMs are considered valid if there is sufficient scientific evidence that they truly measure quality of care (internal validity) and that this measurement and subsequent orientation towards respective performance thresholdsactually lead to an improvement of medical care delivery to the target group and/or their outcomes (externalvalidity).

Reproducibility CPMs are considered reproducible if different groups of independent experts develop comparable CPMs whenusing the same scientific evidence and methods.

Reliability CPMs are considered reliable if different independent users using the same database and methods at different times attain comparable parameter values for given quality indicators.

Discrimination CPMs and related performance thresholds should help to distinguish between “good” and “bad” providersof care.To this end, CPMs must be both sensitive, i.e. able to measure even slight quality differences, and specific,i.e. only react to those cases where true variability exists in quality of care.

Adjustability CPMs should be formulated in such a way that they measure the quality of specific aspects of care of comparable units (regions, organisations, providers of care). If there is only limited comparability all possible confounders should be identified and statistically adjusted.

Table 4. (Continued).

C Quality Specification for Clinical Performance Measures

Documentation of Procedures, participants, materials and premises used to develop a CPM program should be documented exactly.CPM program The link between CPMs and the best available scientific evidence should also be documented clearly.development

Transparency CPM programs should only be implemented if their objectives, methods used and underlying scientific findings,sources, authors and target groups are clearly identified.

Stakeholder and Representatives of target groups should be involved in CPM program development. In particular, CPM programspatient measuring outcomes of care involvement should involve affected patients.

Application of CPMs should be complemented with tools that help achieve the objectives of qualitymeasurement, i.e. improvemeasuring results quality of care.Tools include e.g. user-friendly publications of quality assessments, assessment interpretation

guides, educational and training material.

For this reason “quality criteria”have also been described for clinicalperformance measure programmes,which in part correspond to the qual-ity criteria that have been definedfor guidelines. A list of minimum re-quirements for these clinical perfor-mance measures goes back to theAustralian Council on HealthcareStandards [21, Table 3].On the basis of these minimum re-quirements Table 4 presents a morecomprehensive list of quality criteriafor clinical performance measures.These quality criteria follow theevaluation criteria for guidelines es-tablished by the German Medical As-sociation and the Federal Associationof SHI Physicians [3]. In part, theyare based on proposals from organi-sations in the United States [18, 20],Australia [21] and Scotland [16].

5. PREREQUISITES FOR THEREALISATION OFPROGRAMMES FOR THEDEVELOPMENT ANDAPPLICATION OF CLINICALPERFORMANCE MEASURES

5.1 General Prerequisites

The successful implementation ofquality criteria and quality indica-tors is bound by specific prerequi-sites that may be classified as eithergeneral or problem-specific.

9

5.2 Systematic Development,Implementation,Evaluation andMaintenance of ClinicalPerformance Measures

The requirements described abovecan only be met by establishing asystematic programme for the de-velopment, implementation, evalua-tion and maintenance of criteria/in-dicators, as the examples of theAcute Health Clinical Indicator Pro-ject (Australia) [21] or the StandardProgramme of the Clinical Stan-dards Board for Scotland [16] illus-trate. Another example of implementationis the German Diabetes Contract be-tween the sickness funds ofNorthrhine Westphalia and thePhysicians’ Association of Nor-drhein that has been conducted withapproximately 120,000 diabetic per-sons. It is a systematic implementa-tion of guidelines for the manage-ment of diabetes by means of opera-tionalised quality indicators for themeasurement of the processes andoutcomes of diabetes managementusing a uniform evaluation methodand reporting of its results in theform of benchmark comparisonback to the medical practices in-volved.

Table 5. Prerequisites for the success-ful realisation of programmes for the de-velopment and application of clinicalperformance measures [according to21].

• Clear definition and specification ofprogramme objectives

• General availability of data recordingsystems

• Evidence of and monitoring for theavailability, correctness and reliabilityof necessary data

• Instruction and training of the staffthat is supposed to process indicator-related data

• Procedures for adjustment of patient-related and organisation-related fea-tures that may confound the evalua-tion of quality – based on the knowl-edge and experience of localproviders and consented by all indi-viduals / bodies involved (health in-surers, providers etc.)

• Methods/procedures for solving/min-imising numerous interface problemsof health care and/or data recordingand data transmission by means ofstandardised and established inter-faces

• Integration of indicators into a com-prehensive programme of quality as-sessment and quality promotion.

characteristics and recommenda-tions. Table 7 outlines quality indi-cators defined by the Joint Commis-sion on Accreditation of HealthcareOrganizations. A further example isgiven in Table 8.

6.2 Appraising the Qualityand Practicability ofClinical PerformanceMeasures

Faced with the vast number of clini-cal performance measures that havebeen described internationally, we

6. SPECIFICATION ANDASSESSMENT OF THEQUALITY/PRACTICABILITYOF CLINICALPERFORMANCE MEASURES

6.1 Specifying the Characteristics of ClinicalPerformance Measures

As in the case of guidelines [12],transparency regarding the aims,content and evidence of clinical per-formance measures will be fosteredby a structured specification of their

recommend – as in the case ofguidelines – a systematic and criticalappraisal of these tools based onconsideration of the quality criteriaoutlined in the present paper.To that end an appraisal tool called“Checklist to Appraise ClinicalPerformance Measures for Quali-ty Management” has been madeavailable.The format of the appraisal proce-dure follows the one that was de-signed for the AQuMed GuidelineClearinghouse.

10

Table 6. Proposal for the development and implementation of clinical performance measures based on AHCPR guidelines [accordingto 20].

A. Steps in the Planning Phase

1. Describe the objectives of quality assessment.2. Identify relevant guidelines.3. Identify the patient target group of the guideline.4. Identify relevant guideline recommendations from which proposals for quality indicators (outcome measures) or possibly

performance thresholds may be derived.

B. Steps in the Development Phase

5. Specify groups of individuals (physicians, other health care professionals etc.) and areas of care that will be affected by the evaluation process.

6. Specify intended quality indicators, additional outcome measures, evaluation periods and the evaluation plan.7. Identify sources of data for intended quality indicators 8. Define outcome measures, specify sources of data, give explanations of outcome measures (e.g. time frames).9. Check specifications of outcome measures (see under 8) (among other things, describe procedures for the validation of the

indicator and specify its validity)10. Design and establish evaluation tools (documentation forms or electronically assisted documentation modules).11. Define uniform electronic interfaces for system-independent data export from practice administration systems (e.g. the

German BDT interface for the transmission of performance measure data within the scope of quality management of chronic diseases).

12. Describe the procedures of data analysis and evaluation methods.13. Conduct a pilot test with this evaluation tool, then review outcome measures, forms and procedures, if necessary.

C. Steps in the Implementation Phase

14. Collect data, check quality of data.15. Compile the results, check for quality presentation of results.16. Interpret results, perform quality assessment.17. Check the results when quality is inadequate. Discuss unclear quality deficits in a peer review procedure. Modify the

evaluation tool in the case of false positive or false negative findings.18. Establish a centre for evaluation and control.19. Define measures of quality improvement.20. Repeat quality assessment.

11

Table 7. Diagram indicating the development and specification of quality indicators defined by the Joint Commission on Accredita-tion of Healthcare Organizations [6].a

Indicator Development Form Format

I. Indicator Statement

II. Definition of Terms (define terms contained in the indicator that need further explanation for data collection purposes)

III. Type of indicator

A. Indicate whether this indicator is a (1.) rate-based indicator; or (2.) sentinel event indicator.

B. Indicate whether this indicator primarily addresses: (1.) a process of care; or (2.) an outcome of care.

IV. Rationale

A. Explain why this indicator is useful and the specific process or outcome.

B. Identify supportive references used to develop the above rationale.

C. Identify the components of quality that are assessed by this indicator.

V. Description of Indicator Population

A. Indicator numerator:

Indicator denominator:

B. Subcategories (identify patient subpopulations by which the indicator data will be separated for analysis)

VI. Indicator Data Collection Logic

A. List the data elements and corresponding data sources from which data elements may be retrieved.

B. Describe the sequence of data element aggregation through which the numerator events and denominator events are identified by

the indicator.

VII. Underlying Factors

List factors that may explain variation in indicator data and thereby direct quality improvement activities.

A. Patient factors (factors outside the health care organization’s control contributing to patient outcomes)

(1.) Severity of illness (factors related to the degree of illness or stage of disease prior to treatment)

(2.) Comorbid conditions (disease factors, not intrinsic to the primary disease that may influence the frequency of the even

identified by the indicator

(3.) Other patient factors (nondisease factors that may have an impact on the frequency of the event, such as age, sex, refusal to

consent)

B. Practitioner factors (factors, usually controllable by the organization, related to specific practitioners, e. g. nurses, physicians,respiratory therapists)

C. Organization factors (factors, usually controllable by the organization, that contribute to either specific aspects of patient care or

to the general ability of care givers to provide services)

12

Table 8. Example of a structured specification of characteristics of and recommendations for the clinical performance measure of“Thrombolysis after Myocardial Infarction“ – [according to 21]

1. Topic / Name Example:Thrombolysis for acute myocardial infarction (MI)

2. Background / Thrombolysis for myocardial infarction is a life-saving treatment that must be provided as soon Objective of the indicator as possible.programme

Description of the For the purposes of the indicator programme, the problem of care is defined as follows:guideline-supported – Myocardial infarction requiring thrombolysis is defined as:recommendation • Chest pain greater than 30 minutes

• New ST segment elevation or left bundle branch block (LBBB)– Thrombolytic therapy is defined as the application of e.g. intravenous streptokinase/T-PA etc.for the purpose of enhancing clot lysis

3. Numerator Number of patients with MI requiring thrombolysis who received thrombolytic therapy withinone hour after hospital admission.

4. Denominator Number of patients with MI requiring thrombolysis who received thrombolytic therapy duringtheir hospital stay.

5. Potential data sources Numerator: medical records Denominator: ICD-10-AM: Diag – I21.0, I21.1, I21.2, I21.3, I21.4, I21.9.

6. Indicator strengths Commonly accepted indicator for all hospitals which meets all the requirements for an indicator of internal quality management.This indicator affects a care problem of significant extent.

7. Indicator weaknesses • Data about treatment timing are not available from routine administrative data.• Data recording may be affected by numerous factors.Data recording and data adjustment is difficult.

8. Applicability of the programme The programme is applicable across hospitals in all areas of care.

9. Prerequisites for its realisation Hospitals need to realise the following measures:• Standardised documentation of the timing of treatment strategies

10. Frequency of data recording Every six (6) months

11. Desirable quality of care Not specified (e.g. n % of affected patients)

12. Programme justification Not specified (e.g. evidence to prove care deficits)

13. References or sources Not specified (e.g. guideline N.N.)underlying the programmeand/or its justification

14. Evaluation programme Not specified

15. Estimated costs of Not specifiedprogramme realisation

16. Validity of the programme Not specified

7. APPENDIX

7.1 Checklist to Appraise the Quality of Clinical Performance Measures (CPMs) for QualityManagement

1. Questions concerning the Development of CPMs

13

Serial y n uc nano.

Responsibility for the development of clinical performance measures1.1 Is the institution responsible for the development of clinical performance measures clearly identified?1.2 Are the individuals and/or bodies involved in the development of clinical performance measure

(specialist groups, stakeholders, user and patient groups) clearly identified as to their function andkind of involvement?

1.3 Are details provided about any financial or any other forms of funding received from a third partyfor the development of clinical performance measures?

1.4 If there was any sponsoring by commercial stakeholders or any indication of potential obligationsand/competing interests, has its potential impact on the clinical performance measures beendiscussed?

Selection of quality-relevant aspects of clinical care1.5 Has a detailed analysis and documentation been performed of the aspect of clinical care for which

clinical performance measures are to be developed?1.6 Are the sources of information and methods described that were used to retrieve, identify and

select clinical practice guidelines for these quality-relevant aspects of health care?1.7 Are the methods described for obtaining consensus on the prioritisation of quality-relevant aspects

of care?

Identification, formulation and selection of clinical performance measures including related performance thresholds

1.8 Are the sources of information and methods described that were used to search and identify bothpotential clinical performance measures and related performance thresholds?

1.9 Are the methods for selecting existing clinical performance measures described?1.10 Have the methodological characteristics of clinical performance measures and performance

thresholds been systematically reviewed?1.11 Are the sources of information and methods described that were used to search, identify and select

the evidence underlying clinical performance measures and performance thresholds?1.12 Are the methods used to assess the strength of the evidence specified?1.13 Are the methods used to obtain consensus on the clinical performance measures and performance

thresholds described?

External validation and pilot studies1.14 Were the clinical performance measures independently reviewed prior to publication?1.15 Are the methods, commentaries and consequences of the review described?1.16 Have the clinical performance measures been tested or piloted?1.17 If so, is there explicit information provided about both the methods of the pilots used and the result

adopted?

Updating of clinical performance measures1.18 Are the individuals or the body responsible for initiating and performing the reviewing and/or

updating process clearly identified?1.19 Are the methods (evaluation plan) for reviewing and/or updating precisely described?1.20 Is the date for reviewing and/or updating precisely described?

Transparency of the development of clinical performance measures1.21 Is a summary given of the methodology used to develop clinical performance measures (e. g. in the

form of a clinical performance measures report)

7.2. References1. KKF-Verlag (2000) SGB V – Hand-

buch. GKV-Gesundheitsreform-gesetz 2000. Altötting: KF-Verlag,pp. 195–197

2. Hart D (2001) Health TechnologyAssessment (HTA) und gesundheit-srechtliche Regulierung. MedR2001 (Heft 1) 1–8

14

2. Questions concerning the Content and Format of Clinical Performance Measures

3. Questions concerning the Application of Clinical Performance Measures

3. Bundesärztekammer, KassenärztlicheBundesvereinigung (1997) Be-urteilungskriterien für Leitlinien inder medizinischen Versorgung.Dtsch Ärztebl 94: A2154–2155, B-1622–1623, C-1754–1755

4. Joint Commission on Accreditationof Healthcare Organizations (JCA-HO). National Library of HealthcareIndicators – Health Plan and Net-

work Edition. Oakbrook Terrace,IL: JCAHO, 1997

5. Joint Commission on Accreditationof Healthcare Organizations (JCA-HO). Primer on indicator develop-ment and application. Measuringquality in health care. OakbrookTerrace: JCAHO, 1990

6. Crombie IK, Davies HTO, AbrahamSCS. Audit Handbook – Improving

Serial y n uc nano.

Data management3.1 Are practicable and evaluated tools available for collecting the data necessary to estimate the

parameter value of clinical performance measures (e. g. documentation forms containing instructions,electronic data processing programs etc.)?

3.2 Is a clear description provided of the data pooling processes that may be required for estimatingthe parameter value of clinical performance measures?

3.3 Are informational and/or training programs available for those involved in the data collection process?3.4 Are the procedures used to ensure the quality of data described?3.5 Are the procedures used to ensure technical data security and data protection explicitly stated?

Application of clinical performance measures3.6 Is information available about the intended dissemination of quality evaluation results, which is based

on the parameter values of clinical performance measures?3.7 Is specific information available about the application and/or ownership of data and the results

of the analysis?3.8 Is an estimate available of the financial resources needed for collecting and analysing the data

for routine use?3.9 Are plans available for evaluating the impact that the application of clinical performance measures

has on health care?

Serial y n uc nano.

Systematic explanation of clinical performance measures2.1 Are the clinical performance measures described in unambiguous terms and comprehensible language?2.2 Are all the terms used in the description clearly defined?2.3 Are the reasons for applying the clinical performance measures explicitly stated?2.4 Are the target group(s) of the clinical performance measures identified?2.5 Is (Are) the patient target group(s) of the clinical performance measures clearly defined (e. g. as to sex,

age, stage of the disease, concomitant diseases, organisational unit, area of health care, region)?2.6 Are the factors influencing the parameter value of clinical performance measures described? 2.7 Are corrective adjustment procedures described that may help to improve the inter-performer

comparability of the parameter values of clinical performance measures?2.8 Are the performance thresholds of the clinical performance measures explicitly identified?2.9 Are the sources used to collect the data for clinical performance measures and their influencing

factors clearly specified?

For each question of the checklist, you may choose one out of a set of four possible answers:y = yes, n = no, uc = unclear, na = not applicable

einer kreisbezogenen Quali-tätssicherungsstudie. Chirurg, inpress – accepted for publication 9/01

14. Bundesärztekammer, Kassen-ärzt-liche Bundesvereinigung (1997):Beurteilungskriterien für Leitlinienin der medizinischen Versorgung.Dtsch Ärztebl 94: A2154–55;B1622–1623; C1754–1755

15. Bernstein SJ, Hilborne LH (1993).Clinical indicators: the road to qual-ity care? Joint Commission Journalon Quality Improvement 1993;19(11): 501–509

16. Clinical Standards Board for Scot-land (2000) CSBS-Standard Coro-nary Heart Disease (CHD) – Sec-ondary Prevention following AcuteMyocardial Infarction – Rehabilita-tionhttp://www.clinicalstandards.org

17. Gold L, Parry D, Raftery J, StevensA, Walshe K, Woolley M (1998). Anassessment of potential indicators ofclinical effectiveness. Birmingham:University of Birmingham, 1998

18. Conquest (1999). ComputerisedNeeds-Orientated Quality Manage-ment Evaluation System.http://www.ahcpr.gov/qual/conquest.htm

19. Bundesärztekammer, Kassenärzt-liche Bundesvereinigung (1999) DasLeitlinien-Clearingverfahren vonBundesärztekammer und Kassen-ärztlicher Bundesvereinigung inZusammenarbeit mit der Deutschen

health care through clinical audit.Chichester, NY: John Wiley andSons, 1993

7. Foundation for Accountability(FACCT). The facts about FACCT.Accountability Action. 1996; 1: 5–8

8. Joint Commission on Accreditationof Healthcare Organizations (JCA-HO). Guide to Quality Assurance.Chicago: JCAHO, 1988

9. ZI (2000) Positionspapier des Zen-tralinstituts für die kassenärztlicheVersorgung in der BundesrepublikDeutschland zur inhaltlichen Aus-gestaltung des § 137e SGB V undzur Auswahl von Krankheitsbildernfür eine evidenzbasierte Leitlinien-implementation im Gesundheitswe-sen (released by the Central InstituteBoard on July 4, 2000) – unpub-lished

10. GMDS-Arbeitsgruppe Qualitäts-sicherung in der Medizin. Infor-matik, Biometrie und Epidemiolo-gie in Medizin und Biologie. 1996;4: 200–230

11. Lohr K, Institute of Medicine USA,WHO Treffen für Leitlinien in derMedizin, Schloss Velen, Germany1997

13. Das Leitlinien-Manual von AWMFund ÄZQ. Z. ärztl. Fortbild. Qual.sich. 95, Suppl I, 1–84, 2001

13. Kopp I, Koller M, Stinner B et al.:Chirurgische Therapie des Rek-tumkarzinoms: Abbildung der realenVersorgungssituation im Rahmen

Krankenhausgesellschaft und denSpitzenverbänden der GesetzlichenKrankenversicherungen Ziele undArbeitsplan. Dtsch Ärztebl 96 (Heft33): A-2105–2106

20. Agency for Health Care Policy andResearch (AHCPR) (1995). Usingclinical practice guidelines to evalu-ate quality of care. Volume 2: Meth-ods. US DHHS – AHCPR, Bethesda1995. AHCPR Pub. No. 95-0046

21. ACHS (Australian Council in HealthCare Standards) Care EvaluationProgram (Collopy BT, Campbell JT,Williams JW, Portelli RJ, IbrahimJE, Cicuttini FM, Majoor JW, Mc-Neil JJ) (1999) Acute Health ClinicalIndicator Project, Dep. of HumanServices Victoria and Monash Uni-versity Department of Epidemiologyand Preventive Medicine July 1999.http://hna.ffh.vic.gov.au/ahs/quality/clinical.htm

22. Eddy DM. Performance measure-ment: problems and solutions.Health Affairs 1998; 17: 7–25

23. Nadzam DM, Turpin R, Hanold LS,White RE (1993). Data-driven perfor-mance improvement in health care:the Joint Commission’s IndicatorMeasurement System (IM System).Joint Commission Journal on QualityImprovement 993; 19 (11): 492–500.

27. Sheldon T (1998) Promoting healthcare quality: what role performanceindicators? Quality in Health Care1998; 7 (Suppl): S45–S50

15

16

DOCUMENT 2

Prioritisation of Topics Related to Health Problemsor Health Care Delivery Problemsfor the Guideline-Clearing-Projectof the German Guideline Clearinghouse

MASTHEAD ________________________________________________________________________

EDITOR:

Agency for Quality in Medicine Joint Institution of the German Medical Association and the National Association ofStatutory Health Insurance Physicians

WORKING GROUP “PRIORITISATION”

Dr. Martin Braun, German Hospital FederationProf. Juergen Fritze,Association of Private Health InsuranceDr. Elke Herz,VdaK (Association of all seven German statutory health insurance funds for employees)Dr. Ina Kopp,Association of the Scientific Medical SocietiesDr. Hans Krumpaszky, German Medical AssociationProf. Dr.Wilfried Lorenz,Association of the Scientific Medical SocietiesProf. Dr. Dr. Guenter Ollenschlaeger,AQuMedDr. Kirsten Reinhard, Federal Association of the General Regional Sickness Funds (AOK)Dr. Paul Rheinberger, Federal Association of SHI PhysiciansDr. Nicole Schlottmann, German Hospital FederationHenning Thole, Medical Adviser,AQuMedEva Volke, Dipl. Soz.Wiss., Federation of German Pension Insurance InstitutesProf. Dr. Juergen Windeler, Medical Review Board of the Statutory Health Insurance Funds

Translated by Karin Beifuss

PLEASE CONTACT:

Aerztliche Zentralstelle Qualitaetssicherung (Agency for Quality in Medicine)Aachener Strasse 233–237D 50931 Cologne (Koeln)

e-mail: [email protected]: +49 - 221 - 4004 - 501; Fax: +49 - 221 - 4004 - 590 Internet: http://www.azq.de, http://www.leitlinien.de

Released: 19 Dec. 2001: By: Extended Planning Group of the German Guideline Clearinghouse at the Agency for Qualityin Medicine (AQuMed)

© 2001 Agency for Quality in Medicine

17

TABLE OF CONTENTS

1. Background and Mission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2. Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3. The Problem-Related Approach to Prioritisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4. Practical Procedures and Methods of Prioritisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.1 Conducting Groups and Timeframes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.1.1 Preparatory work undertaken by AQuMed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.1.2 The Extended Planning Group’s prioritisation committee . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.1.3 Timeframes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.2 Collection of Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.2.1 Points of orientation for the nomination of topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.2.2 Themenrecherche . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.2.3 Open proposal process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.3 Analysis of Topic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.3.1 Screening and structuring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.3.2 Guideline screening . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.3.3 Creating a list of topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.4 Consensus Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.5 Compilation of a List of proposals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6. Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6.1 Consensus Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6.1.1 Delphi-Technik . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6.1.2 Delphi Technique . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6.1.3 Comparison of essential features of the two consensus processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6.2 Presentation of the Process of Prioritisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6.3 Groups that may be Invited to Participate in an Open Proposal Process . . . . . . . . . . . . . . . . . . . . . . . . .

6.4 Further Readings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

18

18

18

19

19

19

19

19

19

19

19

20

20

20

2

20

20

21

21

21

21

21

22

22

23

24

24

1. BACKGROUND ANDMISSION

In 1999 the German Medical Asso-ciation, the National Association ofStatutory Health Insurance Physi-cians, the German Hospital Federa-tion and the Federal Association ofthe Statutory Sickness Funds (here-after called partners) established theGerman Guideline Clearinghouse(GGC) as an institutionalised frame-work for the critical appraisal ofclinical practice guidelines. In a contract effective May 1, 1999the partners to the GGC have agreedthat the Clearinghouse among otherobjectives• should enable the appraisal of rel-evant guidelines (Sect. 1 of the con-tract),• should submit a list of proposalsfor the appraisal of guidelines to theExtended Planning Group (Sect. 3of the contract).As a further development, the prag-matic approach to prioritisation (1)that has been adopted earlier is to besupplemented with a problem-relat-ed approach to identifying topics.This refined approach which is in-tended to fully disclose to the gener-al public the objectives, general con-ditions as well as the overall processof topic identification involves theapplication of explicit, systematicand reproducible methods and pro-cedures. Therefore, information about theideas and priorities of interested par-ties such as guideline users, devel-opers or patients shall be includedinto this instrument through an ap-propriate proposal process. The GGC program at AQuMedmakes use of a prioritisation processthat is described in the followingsections. It helps to prepare the deci-sions of the Extended PlanningGroup, who constitute the Clearing-house Steering Committee for topics

to be specifically explored as part ofthe GGC program.

2. OBJECTIVES

The process of setting priorities fortopics to be explored within theGGC is aimed to produce:

• AcceptanceOne of the outcomes of a repro-ducible process, whose internal log-ic and decision structure has beenapproved by involved and interestedparties, is that decisions which logi-cally derive from this process aremore likely to be accepted (e.g. bygiving a lower priority to a certaintopic). Consequently, the difficultprocess of filtering and identifyingpriority health care topics will notonly be more acceptable to thegroup of individuals both directlyand indirectly involved but also tothe vast number of interested indi-viduals and/or parties.

• Transparency A transparent, reproducible processprovides a common basis of under-standing among both involved andinterested parties (internal and ex-ternal transparency).

• Efficiency and FeasibilityThe prioritisation process describedhere employs methodological defi-nitions that are intended to allow ef-ficient implementation within thelimits described.In addition, the prioritisation pro-cess is also determined by aspects offeasibility including issues of re-source utilisation and timeframes. Itdoes not, however, explicitly ex-clude an extension of the process(such as e.g. the involvement of oth-er interested parties).

3. THE PROBLEM-RELATEDAPPROACH TOPRIORITISATION

The aim of the German GuidelineClearinghouse (GGC) is to analyse,

appraise and report on the quality ofhealth care guidelines. To guidelinedevelopers the outcomes of thisprocess in the form of GuidelineClearing Reports serve as orienta-tional aids for the development ofnew and improved guidelines. Inthe face of the overall social rele-vance of health care delivery andthe considerable time and expenseinvolved in the development ofguidelines and the evaluative pro-cess of quality assurance (e.g. in theform of the GGC programme), thechoice of an adequate approach totopic prioritisation is of great sig-nificance. In the past, topic selec-tion has been based on a pragmaticapproach to a diagnosis-orientedidentification of health care deliv-ery problems. Within the scope of political discus-sions and shop talk among physi-cians health care problems are usu-ally addressed and specified in termsof disease. But such discussions of-ten reveal that problems do not nec-essarily concern the whole spectrumof a disease, but rather isolated, indi-vidual aspects such as e.g. areas ofcare, specific decision situations orthe problematic use of drugs and/orother therapies.Basically, guidelines may have dif-ferent starting points. They can be • disease-/diagnosis-oriented (e.g.for diabetes mellitus), • method-oriented (e.g. about coro-nary angiography), • symptom-oriented (e.g. aboutburning upon miction) or • problem- or target-related (e.g.about the impact of breast cancer onquality of life). A guideline addressing a specifichealth care delivery problem is in-tended to support improvementstrategies. Consequently, it mustclearly focus on problem-solvingmeasures.Therefore, it is reasonable to supple-ment the GGC prioritisation processwith a problem-related approach,which enables the Clearinghouse to• address actual problems of healthcare delivery,

18

1Helou A, Perletz M, Schwartz FW: Prior-itätensetzung bei der Entwicklung medizinis-cher Leitlinien: Z Ärztl Fortbild Qual sich(ZaeFQ) 2000, 94: 53–60

• deal with topics comprising sever-al different areas of care,• deal with problematic primarycare situations that have a high rele-vance and high demand for solu-tions,• cut across diagnoses and/or be di-rected towards treatment cases.In particular, we must systematical-ly take into account different prefer-ences for various interests of nomi-nators that serve as baseline for indi-vidual rankings (for further readingssee Appendix, Chapter 6.4).Concerning the target groups of theGerman Guideline Clearinghouse, aproblem-related approach to settingpriorities for the selection of topicsfor GGC projects can be expected toproduce an even higher level of ac-ceptance and relevance.

4. PRACTICAL PROCEDURESAND METHODS OFPRIORITISATION

4.1 Conducting Groups andTimeframes

4.1.1 Preparatory workundertaken by AQuMed

The following preparatory steps (fora complete presentation see Chap-ters 4.2 and 4.3) are to be taken byAQuMed for the Extended PlanningGroup’s prioritisation committee(see Chapter 4.1.2):• searching for topics• co-ordinating the open proposalprocess and evaluating its results,• analysing topics and creating a listof topics containing all the proposalsto be included in the consensus pro-cess.Responses and further inquiriesshould be discussed with the Ex-tended Planning Group’s prioritisa-tion committee as they arise.

4.1.2 The Extended PlanningGroup’s prioritisationcommittee

In accordance with the contract per-taining to the German Guideline

Clearinghouse the Extended Plan-ning Group’s role as policy maker isa tool designed to organise GGCprojects. The Extended Planning Group’s pri-oritisation committee prepares topicprioritisation decisions. Their task isto • monitor the process as a wholeand settle questions and further in-quiries of AQuMed arising duringwork,• conduct a formalised consensusprocess (see Chapter 4.4 and Ap-pendix 6) resulting in • develop a list of priority topics(see Chapter 4.5) containing all theinformation required for a decisionabout the Extended PlanningGroup’s final determination.

4.1.3 Timeframes

The timeframes for the implementa-tion of the prioritisation process de-pend upon a number of requirementsincluding • the sequence of Extended Plan-ning Group meetings,• the topics to be explored withinthe framework of GGC projects atAQuMed,• updating requirements for the top-ics proposed, if demanded by any ofthe partners, • the potential for utilising the re-sults for other projects conducted byother committees of the medicalself-governing bodies.Depending on the requirements list-ed above the prioritisation processcan be conducted once a year.

4.2 Collection of Topics

4.2.1 Points of orientation for thenomination of topics

The process of collecting topics,which in turn consists of the searchfor topics and the open proposalprocess, is intended to identify po-tential topics for GGC projects. Thecollection of topics might be guidedby the following points of orienta-tion.

4.2.2 Search for topicsThe specific search for topics mayrefer to the following sources frompreviously mentioned areas:• health care delivery data,• documented problems with healthcare delivery,• information from additional liter-ature searches.The search for health care deliverydata and documented problems withhealth care delivery may include in-formation from the followingsources:• health care providers, • health care purchasers, • scientific medical societies, • the Statutory Health InsuranceFunds’ Medical Review Board,• quality assurance project offices,• Federal Health Monitoring by thegovernment and other federal au-thorities, • outcome of health care deliveryresearch,

19

Table 1. Points of Orientation for thePrioritisation of Topics (listed in a ran-dom order with no intended priorityranking).

• prevalence/incidence of the healthproblem,

• health problem associated with highcosts,

• differences in practice patterns (prac-tice variation patterns),

• health problems with relevant, avoid-able differences of quality in medicalcare,

• individual burden of disease,

• indications of underuse, misuse oroveruse,

• priority of the health problem fromthe population’s point of view,

• a guideline’s potential for improvingquality of life,

• a guideline’s potential for improvinghealth-related outcomes,

• a guideline’s potential for improvingquality of care,

• ethical and social issues.

• committees and registers,• third party insurers.The supplementary search for refer-ences is performed systematicallyby qualified and skilled librariansand documentarists at AQuMed. Specific requirements for the entiresearch process may be separatelydefined for each project by the Ex-tended Planning Group’s prioritisa-tion committee or the ExtendedPlanning Group itself.

4.2.3 Open proposal processTo complement the procedures out-lined in Chapter 4.2.2 topic nomina-tions should be submitted through aproposal process, which is e.g. opento the following groups:• scientific medical societies and professional associations of healthcare providers,• health care providers,• health care purchasers,• patients,• other interested parties.As some time will be required toprepare the decisions (see Chapter4.1.3) of the Extended PlanningGroup’s prioritisation committee, astructured and time-limited processshould be employed to enable adap-tation of topics to the decision pro-cess. Therefore we propose to con-duct:• Internet-based public surveys viaE-mail,• queries via the platforms or pro-fessional media of target groups(e.g. specialist journals),• targeted queries in order to locateproposals from interested parties,e.g. the Pension Insurance Institutesor the members and organisationalpartners of the German Associationfor Quality Assurance in HealthCare (AQS according to Sect. 137bof the Fifth Book of Social Code(see Appendix, Chapter 6.3).Specific information that should ac-company the nominations submittedin the open proposal process in-clude:• specification of the topic whileproviding the most precise problem-related description possible (not on-

ly general aims such as “betterhealth” or “cancer treatment”),• specification of the reasons fornominating this specific topic,• an evaluation of its relevance,• specification of the targetedchange (potential aim).

4.3 Analysis of TopicsAll the topics compiled as a result ofboth the search for topics (see Chap-ter 4.2.2) and the open proposal pro-cess (see Chapter 4.2.3) should beincluded in the consensus process inthe form of a list of proposals. Thepreparation of this process to be un-dertaken by AQuMed comprises thefollowing steps.

4.3.1 Screening and structuringThe first step is a screening of re-ceived proposals for double cita-tions or topics that have alreadybeen explored elsewhere. In the nextstructuring step nominations are tobe categorised according to specifictopic areas.

4.3.2 Guideline screeningIn a systematic search, which is sim-ilar to the one used to identify guide-lines within the scope of the Clear-inghouse for Clinical PracticeGuidelines (see e.g. 2), a standard-ised search strategy is applied tocheck nominated topics against ex-isting guidelines. In general, thesearch covers a period of 10 years.Due to costs, the search is restrictedto German and English. The screening for guidelines isbased on guideline databases thatare accessible via the AQuMedGuideline Information System(http://www.leitlinien.de).This step may also be used to identi-fy topics for which no guidelines areavailable yet. These topics may thenbe referred back to the Clearing-

house partners or, via the associationof scientific medical societies(AWMF), to the scientific medicalsocieties in order to encourage thedevelopment of new guidelines.

4.3.3 Creating a list of topicsAs a result of the preceding steps,topic lists will be compiled and fur-ther refined by means of a consensusprocess. To that end, all proposalsare collected and presented in a stan-dardised list so that it will no longerbe possible to identify the nominatoror nominating organisation.

4.4 Consensus ProcessThe overall process of prioritisationincludes a methodological part(search, analysis and evaluation ofthe list of proposals), a consensusprocess (in which the prioritisationcommittee produces a sequential listof topics to be dealt with in the formof a list of proposals) and the finaldecision about the sequence of pri-ority topics in the order in whichthey should be explored by the Ex-tended Planning Group. The resultsof this decision will be summarisedin a priority list.The consensus process is designedto provide the means for setting pri-orities of topics by both involvingdifferent groups in this process ofcategorisation on an equal basis andby ensuring that this process betransparent and reproducible. Concerning the consenting activitiesof the GGC prioritisation process atAQuMed it needs to be investigatedwhich of the common consentingprocedures are considered practica-ble. Current prioritisation processes,above all Delphi and NominalGroup Technique, are known fortheir methodological or logistic ad-vantages. As regards the prioritisa-tion of topics for Clearinghouse pro-jects, however, experiences with andexpectations for these techniquesare highly variable among the mem-bers of the committees involved.From the point of view of the au-thors of the present draft paper a

20

2The English Version of the Guideline-Clear-ingreport Hypertension is available on theAQuMed-Homepage at:http://www.leitlinien.de/englhypertension.htm/Content

external conditions (e.g. time and lo-cation).• All relevant groups should be in-volved in the consensus process,which should be checked regularlyif necessary.• There should be an introduction toeach new process.• Consensus processes should beconducted by individuals who adopta neutral position (moderators) andwho have mastered the technique tobe used. • A long enough period of timeshould be allowed for conducting, inparticular, the Delphi technique.

6.1.1 Delphi Technique

The Delphi technique (Delphi pro-cedure, Delphi conference) is de-fined as a multistage, interactive in-quiry process that is designed toachieve consensus with controlledfeedback and anonymous question-ing of participants. The Delphi technique proceeds inthe following phases:1. Topics are collected. This step in-cludes the search for topics and theopen proposal process (see Chapter4.2).2. Several query rounds are con-ducted for setting priorities in theprioritisation committee. Aftereach round the results are com-piled and then anonymously re-turned to the prioritisation com-mittee for re-reviewing. This en-ables systematic modification andcritique of the aggregate anony-mous responses.3. Group consensus is achieved bysummarising (mostly statistically)individual opinions during a finaldiscussion round in the prioritisationcommittee.4. The Delphi technique shouldbe stopped if it becomes clear thatopinions converge to a consensusor participants begin to lose inter-est.Due to the time required for the pri-oritisation process it might becomenecessary to restrict the Delphi tech-nique to a few rounds.

specific technique should thereforebe adopted only after all the partiesinvolved have had ample opportuni-ty for testing and experiencing theconcepts in question.Methods and prerequisites of theconsensus process are outlined inthe Appendix (Chapter 6.1).

4.5 Compilation of a List ofproposals

The outcome of the consensus pro-cess is a list of proposals which inturn serves the Extended PlanningGroup as the basis for making a fi-nal decision on the ranking of top-ics for the GGC programme. Thelist of proposals is intended to dis-close the following in a repro-ducible manner:• the sequence of topics (showingthe ranking),• the topic; narrow the range clearlyif necessary,• the reason for nominating this top-ic, • the identification of the nominatoror of the area from which the pro-posal or topic derives (e.g. literaturesearch, open proposal process).Also, this list should include allthose topics (resulting e.g. from anopen proposal process) that cannotbe incorporated into the Clearing-house project for methodologicalreasons (absence of a guideline) butmay be used for feedback to theClearinghouse partners to encouragefurther projects.

5. SUMMARY

The aim of the prioritisation processfor topics to be explored within theframework of the German GuidelineClearinghouse is to facilitate accep-tance, transparency as well as effi-ciency and feasibility. In this way,topics for the development of guide-line clearing reports may be deter-mined.The previous diagnosis-oriented ap-proach to the process of topic selec-tion is supplemented with a prob-

lem-related approach; in addition,interested parties (e.g. guidelineusers, developers and patients) willbe more strongly involved in theprocess of topic nomination. Areas that might be investigated forGGC topics range from informationabout health care delivery data ofrelevant groups, documented prob-lems of health care delivery to infor-mation received through an openproposal process. The following aspects might beconsidered when nominating a top-ic: its health impact, the impact tobe expected from a Clearinghouseproject, the expected user accep-tance, economic issues, possiblyexisting practice variation patterns,the individual burden of disease,the topic’s relevance from the pop-ulation’s point of view as well asethical and social and quality of lifeissues. Topic collection is followed by ananalysis of topics, where guidelinescreening is used to identify topicsfor which guidelines already existand which are thus suitable for aClearinghouse project. In this context topics for which noguidelines exist can be identifiedand reported to the respective organ-isations in order to emphasise theneed for and invite the developmentof new guidelines.Using a formalised consensus pro-cess topics are ranked by priority.The outcome of this process is a listof priority proposals that will besubmitted to the Extended PlanningGroup.

6. APPENDIX

6.1 Consensus Process

General requirements for the plan-ning and realisation of consensusprocesses• In every respect, the entire con-sensus process needs to be plannedand pre-determined as regards bothinternal (e.g. planned targets and in-dividual steps in the process) and

21

6.1.2 Nominal Group Technique(NGT)

The Nominal Group Technique is agroup decision process that was de-veloped in 1968 by Delbecq et al. Itsresult is a list of proposals and state-ments ranked according to their rele-vance. The process works as fol-lows: 1. To start with, each member of theprioritisation committee will indi-

vidually write down his/her ideasabout given problems and questionswithout a preceding discussion.2. Answers are collected and pre-sented to the entire prioritisationcommittee without disclosing theauthors’ identities. Collection pro-ceeds until there are no more pro-posals.3. In a joint round the individualmembers of the prioritisation com-

mittee one after the other give theirviews on the answers received.Steps 1–3 may be iterated severaltimes.4. After several rounds the mem-bers of the prioritisation committeefinally achieve consensus on the or-der of priorities of topics on the listof proposals by voting or ranking.

22

Delphi Technique Nominal Group Technique (NGT)

Method structured series of questionnaires structured meeting

Group contact anonymous personal

Communication in writing direct

Time to administer months days

Administrative expenditure relatively high relatively low

Administrative costs relatively high relatively low

Costs per participant low depending on travel expense

Number of participants unlimited limited (–9), larger numbers of participantsare possible when groups are divided

6.1.3 Comparison of essential features of the two consensus processes

6.2 Presentation of the Process of Prioritisation

23

• representatives from the Associa-tion of the Scientific Medical Soci-eties (AWMF),• representatives from the Confer-ence of State Health Ministers,• representatives from the Bundes-geschaeftsstelle für Qualitaets-sicherung (which is a national co-or-dination office for quality assuranceactivities).

6.4 Further ReadingsAtteslander P (2000) Methoden der em-

pirischen Sozialforschung. Berlin,New York: Walter de Gruyter

Black N et al (1999) Consensus devel-opment methods: a review of bestpractice in creating clinical guide-lines. J Health Serv Res Policy 1999(Oct); 4 (4): 236–48

Delbecq AL, Van de Ven AH, GustafsonDH: Group techniques for programplanning. Glenview, Illinois: Scott,Foresman and Company, 1975

Dey I (1993) Qualitative data analysis. Auser-friendly guide for social scien-tists. London, New York: Routledge

Gollier C, Eeckhoudt L, Levasseur M(1994) The economics of addingand subdividing independent risks. JRisk Uncertainty 8: 325–337

Gross A et al (2001): Optimal methodsfor guideline implementation. Con-clusions from Leeds Castle meeting.Medical Care 39: Suppl. 2, 85–92

Kunz R, Ollenschlaeger G, Raspe H, etal (eds) Lehrbuch evidenzbasierteMedizin in Klinik und Praxis,Cologne: Dt. Ärzte-Verl., 2000, IS-BN 3-7691-0383-1

Phillipchalk RP (1995) Conformity,compliance and obedience. In: Invi-tation to social psychology. FortWorth: Harcourt Brace College Pub-lishers, 247–279

Wulff HR (1981) Diseases – the vehiclesof clinical experience. In: Rationaldiagnosis and treatment. Oxford:Blackwell Scientific Publications,68-79

6.3 Groups that may beInvited to Participate inan Open Proposal Process

In chapter 4.2.3, the organisationalpartners to the AQS (in accordancewith Sect. 137b of the Fifth Book ofSocial Code) are taken as an exam-ple for groups that might be invitedto propose topics. These include:• German Medical Association,• German Hospital Federation,• National Association of StatutoryHealth Insurance Physicians,• Federal Association of SicknessFunds,• Professional associations of thenursing professions,• Association of Private Health In-surance,as well as permanent guests of theAQS committees:• representatives from the FederalMinistry of Health,

24

DOCUMENT 1 Critical Appraisal of Clinical Performance Measures … · physicians (Article 137e of...

Documents

Transcript of DOCUMENT 1 Critical Appraisal of Clinical Performance Measures … · physicians (Article 137e of...