7/30/2019 ABC Work Based Assessment

1/4

doi:10.1136/bmj.326.7392.753

2003;326;753-755BMJJohn J NorciniWork based assessmentABC of learning and teaching in medicine:

http://bmj.com/cgi/content/full/326/7392/753Updated information and services can be found at:

These include:

References

http://bmj.com/cgi/content/full/326/7392/753#otherarticles

4 online articles that cite this article can be accessed at:http://bmj.com/cgi/content/full/326/7392/753#BIBLThis article cites 3 articles, 2 of which can be accessed free at:

Rapid responseshttp://bmj.com/cgi/eletter-submit/326/7392/753

You can respond to this article at:

serviceEmail alerting

box at the top right corner of the articleReceive free email alerts when new articles cite this article - sign up in the

Topic collections

(112 articles)Teaching(241 articles)Undergraduate

(198 articles)Continuous professional developmentArticles on similar topics can be found in the following collections

Notes

http://www.bmjjournals.com/cgi/reprintformTo order reprints of this article go to:

http://bmj.bmjjournals.com/subscriptions/subscribe.shtmlgo to:BMJTo subscribe to

on 9 January 2007bmj.comDownloaded from

http://bmj.com/cgi/content/full/326/7392/753http://bmj.com/cgi/content/full/326/7392/753http://bmj.com/cgi/content/full/326/7392/753#otherarticleshttp://bmj.com/cgi/content/full/326/7392/753#otherarticleshttp://bmj.com/cgi/content/full/326/7392/753#otherarticleshttp://bmj.com/cgi/content/full/326/7392/753#BIBLhttp://bmj.com/cgi/content/full/326/7392/753#BIBLhttp://bmj.com/cgi/eletter-submit/326/7392/753http://bmj.com/cgi/eletter-submit/326/7392/753http://bmj.com/cgi/collection/teachinghttp://bmj.com/cgi/collection/teachinghttp://bmj.com/cgi/collection/teachinghttp://bmj.com/cgi/collection/undergradhttp://bmj.com/cgi/collection/medical_careers:continuous_professional_developmenthttp://bmj.com/cgi/collection/undergradhttp://bmj.com/cgi/collection/medical_careers:continuous_professional_developmenthttp://bmj.com/cgi/collection/medical_careers:continuous_professional_developmenthttp://www.bmjjournals.com/cgi/reprintformhttp://www.bmjjournals.com/cgi/reprintformhttp://bmj.bmjjournals.com/subscriptions/subscribe.shtmlhttp://bmj.bmjjournals.com/subscriptions/subscribe.shtmlhttp://bmj.bmjjournals.com/subscriptions/subscribe.shtmlhttp://bmj.bmjjournals.com/subscriptions/subscribe.shtmlhttp://bmj.com/http://bmj.com/http://bmj.com/http://bmj.bmjjournals.com/subscriptions/subscribe.shtmlhttp://www.bmjjournals.com/cgi/reprintformhttp://bmj.com/cgi/collection/teachinghttp://bmj.com/cgi/collection/undergradhttp://bmj.com/cgi/collection/medical_careers:continuous_professional_developmenthttp://bmj.com/cgi/eletter-submit/326/7392/753http://bmj.com/cgi/content/full/326/7392/753#otherarticleshttp://bmj.com/cgi/content/full/326/7392/753#BIBLhttp://bmj.com/cgi/content/full/326/7392/753

7/30/2019 ABC Work Based Assessment

2/4

ABC of learning and teaching in medicineWork based assessmentJohn J Norcini

In 1990 psychologist George Miller proposed a framework forassessing clinical competence. At the lowest level of the pyramidis knowledge (knows), followed by competence (knows how),performance (shows how), and action (does). In this framework,Miller distinguished between action and the lower levels.Action focuses on what occurs in practice rather than whathappens in an artificial testing situation. Work based methods ofassessment target this highest level of the pyramid and collectinformation about doctors performance in their normalpractice. Other common methods of assessment, such asmultiple choice questions, simulation tests, and objectivestructured clinical examinations (OSCEs) target the lower levelsof the pyramid. Underlying this distinction is the sensible but

still unproved assumption that assessments of actual practiceare a much better reflection of routine performance thanassessments done under test conditions.

MethodsAlthough the focus of this article is on practising doctors, workbased assessment methods apply to medical students andtrainees as well. These methods can be classified in many ways,

but this article classifies them in two dimensions. The firstdimension describes the basis for making judgments about thequality of performance. The second dimension is concerned

with how data are collected.

Basis for judgmentOutcomesIn judgments about the outcomes of their patients, the qualityof a cardiologist, for example, might be judged by the mortalityof his or her patients within 30 days of acute myocardialinfarction. Historically, outcomes have been limited to mortalityand morbidity, but in recent years the number of clinical endpoints has been expanded. Patients satisfaction, functionalstatus, cost effectiveness, and intermediate outcomesforexample, HbA1c and lipid concentrations for diabeticpatientshave gained acceptance.

Patients outcomes are the best measures of the quality ofdoctors for the public, the patients, and doctors themselves. For

the public, outcomes assessment is a measure of accountabilitythat provides reassurance that the doctor is performing well inpractice. For individual patients, it supplies a basis for deciding

which doctor to see. For doctors, it offers reassurance that theirassessment is tailored to their unique practice and based on real

work performance.Despite the fact that an assessment of outcomes is highly

desirable, at least four substantial problems remain. These areattribution, complexity, case mix, and numbers.

Firstly, for a good judgment to be made about a doctorsperformance, the patients outcomes must be attributable solelyto that doctors actions. This is not realistic when care isdelivered within systems and teams. Secondly, patients with thesame condition will vary in complexity depending on the

severity of their illness, the existence of comorbid conditions,and their ability to comply with the doctors recommendations.Although statistical adjustments may tackle these problems,

This article explains what is meant bywork based assessment and presents aclassification scheme for current methods

Three aspects of doctors performancecan be assessedpatients outcomes,process of care, and volume of practice

Does

Shows how

Knows how

Knows

Millers pyramid for assessing clinical competence

Clinical records

Administrative data

Diaries

Observation

Basis for judgment

Methods of collecting data

Outcomes of care

Process of care

Practice volume

Classification for work based assessment methods

Care is delivered in teams, so judging a doctors performance throughoutcomes is not realistic

Clinical review

753BMJ VOLUME 326 5 APRIL 2003 bmj.com

on 9 January 2007bmj.comDownloaded from

http://bmj.com/http://bmj.com/http://bmj.com/http://bmj.com/

7/30/2019 ABC Work Based Assessment

3/4

they are not completely effective. So differences in complexitydirectly influence outcomes and make it difficult to comparedoctors or set standards for their performance. Thirdly,unevenness exists in the case mix of different doctors, againmaking it difficult to compare performance or to set standards.Finally, to estimate well a doctors routine performance, a

sizeable number of patients are needed. This limits outcomesassessment to the most frequently occurring conditions.

Process of careIn judgments about the process of care that doctors provide, ageneral practitioner, for example, might be assessed on the basisof how many of his or her patients aged over 50 years have

been screened for colorectal cancer. General process measuresinclude screening, preventive services, diagnosis, management,prescribing, education of patients, and counselling. In addition,condition specific processes might also serve as the basis formaking judgments about doctorsfor example, whetherdiabetic patients have their HbA1c monitored regularly andreceive routine foot examinations.

Measures of process of care have substantial advantagesover outcomes. Firstly, the process of care is more directly in thecontrol of the doctor, so problems of attribution are greatlyreduced. Secondly, the measures are less influenced by thecomplexity of patients problemsfor example, doctorscontinue to monitor HbA1c regardless of the severity of thediabetes. Thirdly, some of the process measures, such asimmunisation, should be offered to all patients of a particulartype, reducing the problems of case mix.

The major disadvantage of process measures is that simplydoing the right thing does not ensure the best outcomes forpatients. That a physician regularly monitors HbA1c, forexample, does not guarantee that he or she will make thenecessary changes in management. Furthermore, althoughprocess measures are less susceptible to the difficulties ofattribution, complexity, and case mix, these factors still have anadverse influence.

VolumeJudgments about the number of times that doctors haveengaged in a particular activity might include, for example, thenumber of times a surgeon performed a certain surgicalprocedure. The premise for this type of assessment is the large

body of research showing that quality of care is associated withhigher volume.

Data collection

Clinical practice recordsOne of the best sources of information about outcomes,process, and volume is the clinical practice record. The externalaudit of these records is a valid and credible source of data. Twomajor problems exist, however, with clinical practice records.

Firstly, judgment can be made only on what is recordedthis may not be an accurate assessment of what was actuallydone in practice.

Secondly, abstracting records is expensive and timeconsuming and is made cumbersome by the fact that they areoften incomplete or illegible.

Widespread adoption of the electronic medical record maybe the ultimate solution, although this is some years away.Meanwhile, some groups rely on doctors to abstract their own

records and submit them for evaluation. Coupled with anexternal audit of a sample of the participating doctors, this is acredible and feasible alternative.

For a sound assessment of an individualdoctors process of care, a sizeablenumber of patients need to be included

Advantages of volume based assessment over assessment ofoutcomes and process

x Problems of attribution are reduced substantiallyx Complexity is eliminatedx Case mix is not relevant

However, such assessment alone offers no assurance that the activity wasconducted properly

Unevenness in case mix can reduce usefulness of using patients outcomesas a measure of doctors competence

Judgments on process of care might include foot examinations for diabeticpatients

Traditional m edical records may g ive way to widespread use of electronicrecords, making data co llection easier and qu icker

Clinical review

754 BMJ VOLUME 326 5 APRIL 2003 bmj.com

on 9 January 2007bmj.comDownloaded from

http://bmj.com/http://bmj.com/http://bmj.com/http://bmj.com/

7/30/2019 ABC Work Based Assessment

4/4

Administrative databasesIn some healthcare systems large computerised databases areoften developed as part of the process of administering andreimbursing for health care. Data from these sources areaccessible, inexpensive, and widely available. They can be usedin the evaluation of some aspects of practice performance

such as cost effectiveness

and of medical errors. However, thelack of clinical information and the fact that the data are oftencollected for invoicing purposes makes them unsuitable as theonly source of information.

DiariesDoctors, especially trainees, may use diaries or logs to recordthe procedures they perform. Depending on the purpose of thediary, entries can be accompanied by a description of thedoctors role, the name of an observer, an indication of whetherit was done properly, and a list of complications. This is areasonable way to collect data on volume and an acceptablealternative to the abstraction of clinical practice records untilmedical records are kept electronically.

ObservationData can be collected in many ways through practiceobservation, but to be consistent with Millers definition of work

based assessment, the observations need to be routine or covertto avoid an artificial test situation. They can be made in anynumber of ways and by any number of different observers. Themost common forms of observation based assessment areratings by supervisors, peers, and patients. Other examples ofobservation include visits by standardised patients (lay peopletrained to present patient problems realistically) to doctors intheir surgeries and audiotapes or videotapes of consultationssuch as those used by the General Medical Council.

PortfoliosDoctors typically collect from various sources the practice datathey think pertinent to their evaluation. A doctors portfoliomight contain data on outcomes, process, or volume, collectedthrough clinical record audit, diaries, or assessments by patientsand peers. It is important to specify what to include inportfolios as doctors will naturally present their best work, andthe evaluation of it will not be useful for continuing qualityimprovement or quality assurance. In addition, if there is adesire to compare doctors or to provide them with feedbackabout their relative performance, then all portfolios mustcontain the same data collected in a similar fashion. Otherwise,there is no basis for legitimate comparison or benchmarking.

The photograph of a surgical team is from Philippe Plailly/Eurelios/SPL;the photographs illustrating case mix are from Photofusion, by David Tothill(left) and Pete Addis (right); the photograph of the foot is from Ray Clarke

(FRPS) and Mervyn Goff (FRPS/SPL); and the medical records photographis from Michael Donne/SPL.

BMJ2003;326:7535

Databases for clinical audit are becomingmore available and may provide moreuseful information relating to clinicalpractice

Peer evaluation rating forms

Below are the aspects of competence assessed with the peer ratingform developed by Ramsey and colleagues.* The form, given to10 peers, provides reliable estimates of two overall dimensions ofperformance: cognitive and clinical skills, and professionalism.

Cognitive and clinical skillsx Medical knowledgex Ambulatory carex Management of complex problemsx Management of hospital inpatientsx Problem solvingx Overall clinical competence

Professionalismx Respectx Integrityx Psychosocial aspects

of illnessx Compassionx Responsibility

*Ramsey PG et al. Use of peer ratings to evaluate physician performance. JAMA1993;269:1655-60

Patient rating form*

The form is given to 25 patients and gives a reliable estimate of adoctors communication skills. Scores are on a five point scalepoorto excellentand are related to validity measures. The patients must

be balanced in terms of age, sex, and health status. Typical questionsare:x Tells you everything?x Greets you warmly?x Treats you as if youre on the same level?x Lets you tell your story?x Shows interest in you as a person?x Warns you what is coming during the physical examination?x Discusses options?x Explains what you need to know?

x Uses words you can understand?

*Webster GD. Final report of the patient satisfaction questionnaire study. AmericanBoard of Internal Medicine, 1989

Process

of care

Portfolio

Patient

outcomes

Practice

volume

Clinical

records

Administration

databaseDiary Observation

How portfolios are compiled

John J Norcini is president and chief executive officer of theFoundation for Advancement of International Medical Education andResearch, Philadelphia, Pennsylvania.

The ABC of learning and teaching in medicine is edited by PeterCantillon, senior lecturer in medical informatics and medicaleducation, National University of Ireland, Galway, Republic of Ireland;Linda Hutchinson, director of education and workforce developmentand consultant paediatrician, University Hospital Lewisham; andDiana F Wood, deputy dean for education and consultantendocrinologist, Barts and the London, Queen Marys School ofMedicine and Dentistry, Queen Mary, University of London. Theseries will be published as a book in late spring.

Further reading

x McKinley RK, Fraser RC, Baker R. Model for directly assessing andimproving competence and performance in revalidation ofclinicians. BMJ2001;322:712-5.

x Miller GE. The assessment of clinical skills/competence/performance. Acad Med1990:S63-7.

x Norcini JJ. Recertification in the United States. BMJ1999;319:1183-5.

x Cunningham JPW, Hanna E, Turnbull J, Kaigas T, Norman GR.Defensible assessment of the competency of the practicingphysician. Acad Med1997;72:9-12.

Clinical review

755BMJ VOLUME 326 5 APRIL 2003 bmj.com

on 9 January 2007bmj.comDownloaded from

http://bmj.com/http://bmj.com/http://bmj.com/http://bmj.com/