Assembling a Systematic Review Team: Balancing Expertise and Potential

Conflicts of Interest

Avoiding bias in systematic reviewsAHRQ annual meeting

Bethesda North Marriott Conference CenterSeptember 20, 2011

IOM recommendations

STANDARD 2.1 Establish a team with appropriate expertise and experience to conduct the systematic review

2.1.1 Include expertise in the pertinent clinical content areas

STANDARD 2.2 Manage bias and conflict of interest (COI) of the team conducting the systematic review

2.2.3 Exclude individuals whose professional or intellectual bias would diminish the credibility of the review in the eyes of the intended users

STANDARD 2.4 Manage bias and COI for individuals providing input into the systematic review

2.4.2 Exclude input from individuals whose COI or bias would diminish the credibility of the review in the eyes of the intended users

STANDARD 2.7 Submit the protocol for peer review 2.7.1 Provide a public comment period for the protocol and publicly report on

disposition of comments

Definitions

Bias– a systematic error or deviation from the truth, in results or

inferences (Higgins JPT, et al. Cochrane handbook 2009).– inclination to present or hold a partial perspective at the expense of

(possibly equally valid) alternatives

Conflict of interest (COI) – a set of conditions in which professional judgment concerning a

primary interest (such as the validity of research) tends to be unduly influenced by a secondary interest (such as financial or professional gain) (Thompson, 1993).

Expertise– Extensive knowledge based on research, experience, or

occupation and in a particular area of study. Expert is someone who is a widely recognized authority by peers and public.

Potential biases in systematic reviews

Holding an established opinion on the answers to the research questions of the review; the inclination to present or hold a partial perspective at the expense of (possibly equally valid) alternatives– Individuals in a particular specialty may have an unbalanced

familiarity for one type of intervention or technology.– Individuals livelihood may depend on finding greater benefit of one

type of intervention or technology over a competitor.– Experts may hold a previously formed opinion and be unlikely to

change. Holding an established opinion on the quality of studies to be

included. – Individuals who conducted a study or trial may not be able to

assess the quality of their own or other “competitor” studies in an unbiased manner.

My take home points

We are concerned with bias, but this is difficult to measure. Non-financial conflicts of interest can result in bias.

– Perceived conflicts of interest may still reduce the credibility of the review in the eyes of the intended users

– Conflicts due to familiarity or livelihood based on an intervention or technology can be managed by balancing the team and input from different specialty areas.

Expertise can be a form of bias– May be difficult to discern unless they have published an editorial or

commentary on the subject. – Not including appropriate expertise may reduce the credibility of the

review in the eyes of the intended users– It is important to hear opinions from all sides, but the core team

should be without bias. Individuals should not be involved in grading their own study, either at

the individual study level or at the body of evidence level.

Introductions and Acknowledgements

Susan Norris, Oregon Health and Science University

Gillian Sanders Schmidler, Duke University

AHRQ staff Elise Berliner Christine Chang Yen-Pin Chiang Supriya Janakiraman Elisabeth Kato

Mary Nix

EPC workgroup Eric Bass Stanley Ip Melissa Mcpheeters Sydney Newberry Susan Norris Margaret Piper Paul Shekelle Meera Viswanathan Evelyn Whitlock Renee Wilson Michael White

Objectives

Provide a framework for thinking about intellectual conflict of interest

Review of the existing literature on the potential effects of intellectual conflict of interest

Work through examples of potential conflicts of interest when conducting a systematic review

Case Examples

Procedure specialist

A review is comparing surgical, medical or alternative treatments. Should a specialist who makes a living doing only surgical interventions be the principal investigator?

1. Yes2. Yes, but with restrictions3. No4. It depends

Procedure specialist

A review is comparing surgical, medical or alternative treatments. Should a specialist who makes a living doing only surgical interventions be part of the core team?

1. Yes2. Yes, but balance with other perspectives3. Yes, but with restrictions4. No 5. It depends

Professional society staff

Should a scientist who works for a specialty professional society be the principal investigator of a review?

1. Yes2. Yes, but with restrictions3. No 4. It depends

Professional society staff

Should a scientist who works for a specialty professional society be a co-author of a review?

1. Yes2. Yes, but balance with other

perspectives3. Yes, but with restrictions4. No 5. It depends

Professional society staff

Should a scientist who works for a specialty professional society be on the technical expert panel?

1. Yes2. Yes, but balance with other

perspectives3. Yes, but with restrictions4. No 5. It depends

Professional society staff

Should a scientist who works for a specialty professional society be a peer reviewer?

1. Yes2. Yes, but balance with other

perspectives3. Yes, but with restrictions4. No 5. It depends

Active research in area

A scientist has federal grants to study one of the intervention of interest. Should this person be the principal investigator of the review?

1. Yes2. Yes, but with restrictions3. No 4. It depends

Active research in area

A scientist has federal grants to study one of the intervention of interest. Should this person be a co-author of the review?

1. Yes2. Yes, but balance with other

perspectives3. Yes, but with restrictions4. No 5. It depends

Active research in area

A scientist has federal grants to study one of the intervention of interest. Should this person be on the technical expert panel?

1. Yes2. Yes, but balance with other

perspectives3. Yes, but with restrictions4. No 5. It depends

Active research in area

A scientist has federal grants to study one of the intervention of interest. Should this person be a peer reviewer?

1. Yes2. Yes, but balance with other

perspectives3. Yes, but with restrictions4. No 5. It depends

Published

A scientist was an author of a study that would potentially be included in the review. Should this person be the principal investigator?

1. Yes2. Yes, but with restrictions3. No 4. It depends

Published

A scientist was an author of a study that would potentially be included in the review. Should this person be on the core team?

1. Yes2. Yes, but balance with other

perspectives3. Yes, but with restrictions4. No

Published

A scientist was an author of a study that would potentially be included in the review. Should this person be on the technical expert panel?

1. Yes2. Yes, but balance with other

perspectives3. Yes, but with restrictions4. No

Published

A scientist was an author of a study that would potentially be included in the review. Should this person be a peer reviewer?

1. Yes2. Yes, but balance with other

perspectives3. Yes, but with restrictions4. No 5. It depends

Involved in study

A scientist coordinated one center of a multi-center trial that would potentially be included in a review. Should this be the principal investigator?

1. Yes2. Yes, but with restrictions3. No 4. It depends

Involved in study

A scientist coordinated one center of a multi-center trial that would potentially be included in a review. Should this be part of the core team?

1. Yes2. Yes, but balance with other

perspectives3. Yes, but with restrictions4. No 5. It depends

Published review or guidelines

A scientist has previously published a review or guideline on the subject of the review. Should this person be the principal investigator?

1. Yes2. Yes, but with restrictions3. No 4. It depends

Published review or guidelines

A scientist has previously published a review or guideline on the subject of the review. Should this person be a co-author?

1. Yes2. Yes, but balance with other

perspectives3. Yes, but with restrictions4. No 5. It depends

Scientific advisor or editor

A scientist serves on an advisory or editorial board without financial compensation on a related topic to the review. Should this person be the principal investigator?

1. Yes2. Yes, but with restrictions3. No 4. It depends

Scientific advisor or editor

A scientist serves on an advisory or editorial board without financial compensation on a related topic to the review. Should this person be a co-author?

1. Yes2. Yes, but balance with other perspectives3. Yes, but with restrictions4. No 5. It depends

Scientific advisor or editor

A scientist serves on an advisory or editorial board without financial compensation on a related topic to the review. Should this person be on the technical expert panel?

1. Yes2. Yes, but balance with other perspectives3. Yes, but with restrictions4. No 5. It depends

Scientific advisor or editor

A scientist serves on an advisory or editorial board without financial compensation on a related topic to the review. Should this person be a peer reviewer?

1. Yes2. Yes, but balance with other perspectives3. Yes, but with restrictions4. No 5. It depends