Journal Club

29/11/13Jessica Griffith

Chosen Paper

• ‘Measuring the quality of anaesthesia from a patient’s perspective: development, validation, and implementation of a short questionnaire’

• Hocking et al• Br J Anaesth 2013 Dec; 111(6) 979-89

Background

• An important aspect of the quality of anaesthetic care is the satisfaction of the patient with their care

• Need a suitable instrument to measure this• Simple and easily administered• Validated

Quality Of health care Through the patient’s Eyes (QuOTE)

• Series of studies performed in Netherlands Institute of Health Services Research

• Items generated and ranked for importance by patients

• No input from experts

Validity

• Definition: degree to which a research study measures what it intends to measure

• Compare with existing psychometric tool• Use statistics e.g. Cronbach’s α

Novel Test of Validity

“a valid satisfaction instrument should contain information about the concept that is being measured – the patient’s perception of the quality of anaesthesia. If this is true, effective feedback of this information to anaesthetists might be expected to change the patient’s perception of quality of anaesthesia”

Aims of Study

1. Develop and validate a short instrument the Perception of Quality in Anaesthesia (PQA)

2. Identify the most important aspects of anaesthesia quality from the patient’s perspective

Methods

• Ethical approval• Single centre - Sir Charles Gairdner Hospital,

Perth – university affiliated tertiary institution• October 2008 – Oct 2009

Part 1: Derivation and Validation

Generating of Items

• Asked patients and members of the public to describe any attributes they consider important for ‘high quality anaesthesia’

• Face-to-face and by e-mail• Recruitment continued until no new attributes

were generated

Ranking of Items

• Attributes were printed onto a pack of cards• A different set of participants ranked them in

order of importance• 12 top ranked attributes used in pilot study• Pilot study assessed wording, understanding

and practicality

Perception of Quality in Anaesthesia Questionnaire

1. If you had a general anaesthetic, do you remember anything during surgery?

2. If you did not have a general anaesthetic (had a nerve block or epidural, etc.) did you feel any pain during the operation?

3. How would you rate the management of your pain after the operation?

4. Did you have any nausea (feeling sick) or vomiting after the operation?

5. Did your anaesthetist address any concerns you had regarding your anaesthetic?

6. Did your anaesthetist talk to you in a way you could understand?

7. Did you feel confident in the ability of your anaesthetist?8. How would you rate the amount of information your

anaesthetist gave you?9. How gentle was your anaesthetist?10.How would you rate your anaesthetist’s technical skills?

(needles, drips etc.)11.How friendly was your anaesthetist?12.How would you rate the amount of time you had with

your anaesthetist?

Comparative Questions

• Overall satisfaction:– How would you rate your anaesthetic experience

overall?– Would you recommend your anaesthetist to a

good friend?• Overall anxiety and understanding of risk:– Were you nervous about having this anaesthetic?– What level of risk did you think you were taking in

undergoing anaesthesia?

Validating

• Included 714 patients over 4 week period• Participants invited to suggest additional

questions more important than those on questionnaire

• Test-retest reliability assessed on 100 patients• Content validity study performed with

separate sample of post-op patients and medical experts

Execution of Questionnaire

• Interview conducted on day after surgery by one of 4 trained interviewers

• Inclusion criteria: • adult patients undergoing elective and

emergency surgery requiring anaesthetic care• Excluded: • patients ventilated post-op • intellectual impairment requiring permanent care

Scoring• Each question has a five point scale e.g. very poor to very

good– Unsatisfactory patient response defined as score ≤ 3

• Performance Score = proportion of patients with unsatisfactory response

• Importance Score = mean rank for each attribute

• Quality Index = importance score x performance score

Additional Data

• Data forms filled in by anaesthetist included:– Patient characteristics– Anaesthetic details– Perioperative details

• Data forms cross-checked against anaesthetic record and operating theatre management systems

Part 2: Evaluation of the PQA in a Feedback Study

• 48 week before and after study• To determine if providing anaesthetists with

information from the PQA was associated with any change in patient perception of quality of anaesthesia

• 6 month period of individualised feedback• After this PQA performance scores were re-

measured

Feedback

Statistical Analysis

• Principle component analysis– Used in exploratory data analysis to determine the

structure of correlations within data• Content validity study – Aiken V coefficients

were calculated• Poisson regression – to see if responses to PQA

questions were associated with various aspects of medical care and patient characteristics

Results: Part 1Generation, Ranking and Pilot Testing

• 100 participants generated 52 items• Ranked in order of importance by further 20

participants• Top 12 items included

Generation and Ranking

Patient Validation of the PQA

• 714 patients completed questionnaire out of 799 eligible (89% response rate)

• Questionnaire completed within 3 mins• Pearson correlation coefficient of test-retest was

0.88• Highest quality index questions:– PONV question (2.3)– Pain management question (1.04)– Communication with anaesthetist (0.3)

Content Validity Study

• 30 post-op patients– Mean Aiken V coefficient of 0.84 (P<0.0001)

• 54 medical experts– Mean Aiken V coefficient of 0.72 (P<0.0001)

• Indicate satisfactory content validity

• Cronbach’s α for all questions was 0.7

Results: Part 2Evaluation of the PQA in a

Feedback Study

Pre-feedback group

• Studied weeks 1 – 14• 2359 patients eligible and 2046 consented

(87% response rate)• Unsatisfactory responses– ≥ 1 = 45%– ≥ 2 = 12%– ≥ 3 = 3%

Feedback group

• Studied in weeks 15-39• 4251 patients returned questionnaire

Post-feedback group

• Weeks 40-49• 1721 eligible with 1421 returning

questionnaires (83% response rate)• Unsatisfactory responses:• ≥ 1 = 35% (45%)• ≥ 2 = 6% (12%)• ≥ 3 = 1% (3%)

Classification of Components of PQA

PQA Responses Indicating Unsatisfactory Performance

Results Summary• Similar pre- and post-feedback differences were observed

if the PQA completed by interview, or left at bedside• Patients in pre-feedback group had 45% higher rate of

unsatisfactory responses• Differences in anaesthetic management in post-feedback

group:– Increased use of 5HT3 anti-emetics– More inductions of anaesthesia where consultant assisted

trainee– Reduced use of morphine as intra-op opioid

Conclusions

• “We have developed a short instrument to measure the patient’s perception of the quality of anaesthesia and demonstrated that it is valid and reliable”

• “Individualised, repeated feedback of patient satisfaction-related performance to health practitioners can improve the patient’s experience”

Anaesthesia Quality

• Communication with anaesthetist– Gentleness and attention– Information and confidence– Addressing the patient’s concerns

• Avoidance of PONV• Post-op pain management

Discussion

• Validity?• Difficult to determine if any psychometric

instrument truly measures what is intended• No gold standard as reference• Conventional methods used• In addition tested PQA in before and after

study

Limitations

• No power calculation used before starting the investigation

• No control group for before and after study (parallel group of patients treated by anaesthetists not exposed to feedback)

• Confounded by patient characteristics and surgical differences between pre-and post-feedback groups

Additional Factors Affecting PQA

• Patients less likely to give an adverse rating of their experience if they:– underwent shorter surgery– male– older age– less pre-operative anxiety

How is this tool different?

• Several instruments already developed to measure patient satisfaction

• Aim to measure all aspects of satisfaction – opinions of patients and experts

• Lengthy to perform• Factors often included:– Waiting/delays– Recovery nursing care– Continuity of care– Emotional support

Is this relevant to Arrowe Park Hospital?

Any questions or comments?