Teaching the Art of Literature Analysis

Evan Williams, PharmD, BCPSAssistant Professor of Pharmacy Practice

Husson University School of Pharmacy

Why literature evaluation is important and why people hate it

How to make literature less terrible for students and preceptors

What parts of the study are important and what is beyond essential

Overview

Students should focus on the statistical methods used as it relates to internal validity of the findings more so than the external validity of the study when discussing journal articles.

  True or False

Pre Test Questions

Journal Article Presentations should a) Focus on an area of practice relevant to the

practice siteb) Tie into some other project or problem at the

practice sitec) Be flexible enough to fit into the amount of

time a practice site or preceptor can devote to the assignment

d) All of the above

Pre Test Questions

The following study for gorillastatin produced the following results after 5 years of follow up. Based on the results in the table, about how many patients do I need to treat with gorillastatin for 5 years to prevent one CAD event?a) 5 b) 8 c) 12 d) 16

Pre Test Questions

Discuss reasons literature evaluation is an essential skill for all pharmacists

Identify components of a quality “journal club” format

Describe ways to make student literature review useful to your practice

Briefly distinguish between useful and confounding statistical methods and demonstrate ability to calculate NNT and NNH

Objectives

Are your students this last guy?

Essential skill for any clinician Everyone wants to follow “evidence-based

medicine”◦ How do you know if recommended treatment is

evidence based? Needed to keep up with emerging therapies

◦ New drugs◦ New guidelines

If you don’t do this, the next generation of pharmacists will be followers instead of leaders regarding drug therapy

Literature Evaluation is Important!

Dry, boring reading◦ 50 Shades of Primary Literature?

Time considerations

Perceived lack of utility◦ This is all summarized in Pharmacists’ Letter anyway

Fear of statistics◦ What the devil is the difference between a t-test, f-

test, and a z-test, anyway?

Why People Hate Literature

There is nothing “clubby” about it◦ We force them to present some random article and

belittle them for not knowing stuff we never taught them

It is detached from other learning outcomes◦ “Pick a recent article that has something to do with

your internal medicine rotation.”◦ They pick terrible articles because of this

They lack good examples of what we are expecting

Why Students Hate “Journal Club”

Decide what the desired learning outcome is◦ Presentation skills?◦ A refined, grammatically perfect handout?◦ Identify all of the trial’s strengths and

weaknesses? ◦ Distinguish important clinical implications of the

trial presented?

Once the outcome is identified, make sure this is communicated to the student

How to Make This Better

Selection of Articles◦ Student vs. Preceptor Selection

◦ Ensure that the article relates to something happening on rotation! This will make it relevant and useful

◦ Type of Article Original Research Review Article Guideline Evaluation

How to Make Lit Eval Better

Selection of Articles◦ Good vs. Bad Articles

Clinical vs. Surrogate Outcomes Sketchy statistics Primary outcomes vs. Secondary outcomes

◦ New vs. Old Older articles aren’t bad or irrelevant!! If the student doesn’t know it, it is a learning

opportunity

How to Make Lit Eval Better

Identify what you’d like the student to present◦ Background

Do you need this? Needs to be quick

◦ Methods Brief

These are fairly standard Breeze through the stats

How to Make Lit Eval Better

Identify what you’d like the student to present◦ Results

Best presented in a table format Identify primary vs secondary outcomes Distinguish between clinical and statistical

significance Does power matter?

Calculate NNT and NNH

How to Make Lit Eval Better

Identify what you’d like the student to present◦ Discussion should take the most time

Is the outcome valid based on the test used? Confounders

Are they looking at the right groups of patients? Influence of uncontrolled variables?

Statistical vs. clinical significance? Does this change my practice?

◦ Conclusions Do you agree with the author’s conclusion?

How to Make Lit Eval Better

Presentation Setting◦ Big time presentation and roast?◦ Conversational?◦ Panel Discussion? ◦ CE style?

Do what makes sense for your practice site!◦ Make this useful to you

How to Make Lit Eval Better

Give students enough time◦ Check in to make sure progress is being made◦ Suggest addressing questions prior to the

presentation to improve the quality of the presentation

Read the article!

Show the student a clear example of a good handout to follow◦ Many are available, so pick one that you like

Students tend to do better with bullet points Make them create their own tables

Other Points For Improvement

In order to properly evaluate the student’s critique, you need to have identified the key points that should have pointed out

◦ What are the results?

◦ Are the results valid?

◦ How can I use the results in my practice?

What You Need to Know

Any statistically significant differences?

◦ p-value vs confidence intervals Little p-values don’t make results more significant

Confidence intervals can give an impression of how precise the results are Confidence intervals that cross unity are not significant

◦ Clinical significance?

What Are the Results?

Number Needed to Treat◦ Good way to quantify clinical significance

◦ 1/difference in the event rate between groups 25 of 500 people in the placebo group died - (5%) 10 of 550 people in the drug group died – (1.8%) The difference between the groups is 3.2%, or 0.032 1/0.032 = 31.25 ~ 31 If I treat 31 people with the drug, I will prevent one

death

◦ Compare this to the Number Needed to Harm

What Are the Results?

Number Needed to Harm◦ Same calculation, but for adverse events

10 of 500 people had major bleeding in the placebo group – (2%)

22 of 550 people had major bleeding in the drug group – (4%) The difference between the groups is 2%, or 0.02 1/0.02 = 50 If I give 50 people the drug, I will cause 1 major bleed

◦ Does the risk vs benefit ratio seem reasonable?

What Are the Results?

Was the primary endpoint clearly stated and reported?◦ Was it relevant? ◦ Surrogate vs clinical endpoints◦ Composite endpoints

Was the designed to reveal the primary endpoint?◦ Powered enough to detect a difference if one

wasn’t found?◦ Does the discussion focus on subgroup analyses?

Are the Results Valid?

Where the right people included?◦ Intent to Treat (ITT) or Per Protocol Analysis?◦ Inclusion and exclusion criteria

Do they control for confounders? Do they limit external validity?

Was bias effectively dealt with?◦ Blinding◦ Valid control groups

Placebo vs active control?

Are the Results Valid?

NNT vs NNH

Are the patients studied similar to my patients?

How does this evidence jive with other studies regarding the same subject?

Is this study important enough to shift clinical practice?

How Can I Apply the Results?

New Drugs◦ What studies got this approved?◦ Who was looked at?◦ Side effect profiles?

New Guidelines◦ Check the references◦ Things that have strong/low evidence

Things you don’t know/always get questions about◦ When is aspirin indicated?◦ Evidence that vaccines don’t cause autism?◦ Does LDL really correlate with MI?◦ Why do we still use warfarin?

Make This Useful to You

Pharmacists need to be able to evaluate literature

We should be showing students why this is important

Make sure journal article presentations focus on preparing pharmacists that can critically evaluate literature◦ Presentation style, format, handouts, length can all vary and

should be adapted to the practice site

Focus on the key issues within the studies presented◦ Don’t get bogged down in statistics◦ Consider the clinical impact

Make the exercise beneficial for you and the student

Summary

Students should focus on the statistical methods used as it relates to internal validity of the findings more so than the external validity of the study when discussing journal articles.

  True or False

Post Test Questions

Journal Article Presentations should a) Focus on an area of practice relevant to the

practice siteb) Tie into some other project or problem at the

practice sitec) Be flexible enough to fit into the amount of

time a practice site or preceptor can devote to the assignment

d)All of the above

Post Test Questions

The following study for gorillastatin produced the following results after 5 years of follow up. Based on the results in the table, about how many patients do I need to treat with gorillastatin for 5 years to prevent one CAD event?a) 5 b) 8 c) 12 d) 16

Post Test Questions

Thank you!

Questions?