PCORI’s Process for Peer Review of Primary Research and ......Standards before results are...

i

PCORI’s Process for Peer Review of Primary

Research and Public Release of Findings

Key Findings from Public Comments and

Proposed Final Revisions

Prepared for PCORI by American Institutes for Research

February 17, 2015

ii

This report was prepared under PCORI RFQ # PCO-CVDLDSR2013.004

Acknowledgments

PCORI and the American Institutes for Research (AIR) wish to thank the members of the

Stakeholder Advisory Group for their guidance throughout the public comment process, as well

as the stakeholders who participated in PCORI’s live and online events about the proposed

process during the public comment period.

PCORI is appreciative of the efforts of Thomas Workman, PhD and Marla L. Clayman, PhD, MPH

and the team at AIR for their work on collecting and analyzing the public comments.

iii

Overview

On September 15, 2014, the Patient-Centered Outcomes Research Institute (PCORI) Board of Governors

approved for posting for public comment PCORI’s draft Proposed Process for Peer Review of Primary

Research and Public Release of Findings. On the same day, PCORI opened a 54-day public comment

period for the proposed process, developed in response to the section of PCORI’s authorizing legislation

that states in part that “The institute shall ensure that there is a process for peer review of primary

research” and provides a set of additional requirements for the release of research findings.

From September 15 to November 7, 2014, the American Institutes for Research® (AIR®), as contracted by

PCORI, collected public comments on the proposed process. A total of 63 unique submissions and 832

unique comments were received in addition to comments from the National Institutes of Health (NIH)

and the PCORI Methodology Committee.

Researchers, patients, patient or caregiver advocacy organizations, and clinicians made up the majority

of those making comments, accounting for 86% of the total submissions (32%, 11%, 21%, and 22%,

respectively). No other group contributed more than 10% of total comments submitted; additional

comments came from professional journals, the life sciences industry, and individual family members or

caregivers.

PCORI also received comments and recommendations from the PCORI Methodology Committee and the

NIH. These comments were processed separately as they were not part of the public-comment-

collection process. Several of these comments, however, echoed sentiments expressed by the collected

public comments.

Although there was a substantial degree of agreement with the proposed process across all stakeholder

groups, five major themes emerged from an analysis of the comments following a systematic

assessment process, resulting in nine recommendations for revision that were reflected across the

comments.

Revisions to the draft process document were made based on these recommendations and the updated

draft presented to the PCORI Board of Governors on February 24, 2015 to consider its adoption.

http://www.pcori.org/events/2015/board-governors-meeting-0

iv

Contents

Overview ................................................................................................................................................ iii

Introduction ............................................................................................................................................ 1

Methods of Analysis………………………………………………………………………………………………………………………………..1

Stakeholder Advisory Group…………………………………………………………………………………………………………………….4

Major Findings......................................................................................................................................... 6

Quantity and Source of the Comments ................................................................................................ 6

Comments per Proposal Section .......................................................................................................... 8

Areas of General Agreement................................................................................................................ 8

Themes from the Comments of Disagreement or Concern ................................................................. 11

Suggestions for Revision and PCORI Responses………………………………………………………………………………….13

PCORI’s Process for Peer Review of Primary Research and Public Release of Findings 1

Introduction

The Patient-Centered Outcomes Research Institute (PCORI) drafted its Proposed Process for Peer Review

of Primary Research and Public Release of Findings to meet its statutory mandate—which states in part

that “The institute shall ensure that there is a process for peer review of primary research.”

The dual requirements in PCORI’s authorizing law for peer review of primary research and making

research findings publicly available within a specific timeframe are not directly linked in the legislation

and pose challenges that must be resolved if PCORI is to comply with both. The proposed process

drafted by PCORI and revised in response to public comment is intended to balance the challenges of

ensuring the timely release of research results while assessing their scientific integrity in a way that

respects the interests and needs of patients, other healthcare stakeholders, our awardees, and the

general public, as well as recognizes the crucial role of peer-reviewed journals in disseminating new

evidence to the scientific and clinical communities.

The mandate for peer review underscores the importance of ensuring that primary research funded by

PCORI be critically appraised for scientific integrity as well as for adherence to PCORI’s Methodological

Standards before results are released. Such a peer-review process adds to the credibility, authority, and

trustworthiness of PCORI-funded research findings.

On September 14, 2014, the PCORI Board of Governors approved for posting for public comment the

draft proposed process document. On the same day, PCORI opened a 54-day public comment period for

the proposed process. Public comments were collected from September 15 to November 7, 2014.

PCORI hosted a live public forum on September 29, 2014 and a webinar on October 29, 2014 featuring

stakeholder feedback on the proposed process.

PCORI contracted with the American Institutes for Research® (AIR®) to create the web-based tool used to

collect public comment and provide the analysis of the comments on the proposed process.

A total of 63 unique submissions and 832 unique comments were received in addition to comments

from the National Institutes of Health (NIH) and the PCORI Methodology Committee.

Methods of Analysis Submissions were pulled as they appeared from the online tool (pcori.airprojects.org/node/1/webform-

results/download) and entered into a coding table. Repeated submissions (two submissions with the

same content) were combined into a single entry, and repeated comments within a single submission

were eliminated for a final cleaned copy of the comment table. Elements of the letterhead (when used),

PCORI or return address, salutation, or signature line were removed. This table was used for the purpose

of coding and analysis. All counts of submissions and comments reflect the combination of duplicate

files.


General Review Comments as a whole were reviewed in their entirety. Analysts also reviewed the autocoded results of

all Likert-scale responses and compared them to the comments provided by individual respondent and

by stakeholder type. Analysts were able to get a clear feel of the comments in general to guide the

creation of a code book to organize and process comments.

Coding A Code Book and Coding Scheme were developed to help organize the comments and identify the

suggestions for revision found in the comments as a whole. A team of four qualitative researchers coded

each of the comments. A set of comments was coded by multiple staff members to test agreement

across the coding staff. This helps determine how reliable the coding of comments is across the entire

set. Agreement across coding staff exceeded 90%, so no changes or clarifications to the code book were

necessary.

All comments were coded by the coding team for 10 general classifications:

1. Unique Identification. A number was used to count individual comments by submission. Comments were separated and numbered by the order in which they were received.

2. Stakeholder Self-Identification. Codes were created for each of the stakeholder categories provided in the online collection tool. A second code was created in cases where a stakeholder category was not selected to indicate if the comment did or did not reflect a knowledge/familiarity of the research process; however, this code was not necessary and therefore was not used.

3. Relevance to the Report. Comments were coded to classify their relevance to (1) the report as a whole, (2) a specific section, or (3) the overall process. Comments that did not relate to the report in any of these ways were also coded appropriately.1 Comments were also autocoded with the question in which the response was submitted.

4. Level of Agreement. All comments that related to the report were coded, based on their stated or interpreted agreement with the section of the Proposal. Review of the entire set of comments found that an additional code was needed, as a number of comments stated general agreement but also mentioned a small concern. These comments were coded as “Agree With Caution.” Coders also identified comments where agreement or disagreement was not made clear.

5. Affect. All comments were also coded for the affect, or emotional direction, of the comment: positive, negative, or indifferent.

Additional codes were used to organize the comments expressing disagreement or concern so that a

clear set of revision recommendations could be identified. The following codes were used for this

purpose:

1 Comments considered unrelated to the report may still be related to PCORI in general or be of interest to its work.


6. Suggested Action/Direction. Comments were coded to classify the suggested action directly stated or implied within the comment using the following distinctions:

a. Add something: The comment stated or implied that an additional factor or item be added to a section of the process.

b. Remove something: The comment stated or implied that some factor or item be removed from a section of the process.

c. Modify something: The comment stated or implied that a factor or item in a section of the process needed to be revised or modified.

d. No action indicated. This code was used when no stated or implied action was provided in the comment.

e. Action suggested is unrelated to the process. This code was used when a comment stated or implied an action that was unrelated to any section of the process.

f. Action is unclear. This code was used when an action was implied but written in such a way that its direction could not be interpreted.

7. Feasibility. Initial review of the comments as a whole found that issues of feasibility regarding the process were often implied. Therefore, coders identified comments where feasibility appeared to be the central tenet of the disagreement. Comments relating to the feasibility of the proposed process were coded with the feasibility code as:

a. Process as a whole is not feasible.

b. Specific section of process is not feasible.

c. Legal mandate is not feasible.

i. The feasibility concern focused on peer-review system capacity.

ii. The feasibility concern focused on PCORI staffing capacity.

iii. The feasibility concern focused on the capacity of academic journals.

iv. The feasibility concern focused on researcher capacity.

8. Language and Writing Style. Comments focused on the style of the proposed process, including organization, format, length, or word choice/phrasing.

Two additional codes were used to identify comment content important to PCORI:

9. Compliment or Support. All comments that indicated support for PCORI or included compliments were coded and marked as support for PCORI, the process as a whole, or a specific section.

10. Inquiry. All specific requests for clarification were also coded.

Additional Analysis to Determine Themes

The complete set of separated and coded comments was entered into NVivo 10 software. Reports of

comments per code were reviewed to determine:

General word frequency (without codes)


Comments by agreement and disagreement codes

Comments by direction and affect of disagreement (recommended action and feasibility)

Comments of appreciation or complimentary to the process

Coded lists of comments were then reviewed for additional analysis. A conceptual map was developed

to further clarify the nature of the comments and their relationship to stakeholders’ interests. Analysts

then began the process of developing specific themes on the basis of four criteria:

1. The prominence of the theme across comment submissions. A theme was considered prominent if it was raised in three or more of the submissions, particularly if repeated multiple times in comments throughout each submission.

2. The clarity of the comment toward providing a recommendation for revision. Analysts were particularly mindful of the suggested action or intent of the comments, and they used thematic analysis to focus the nature of the direction of revisions, especially as revisions related to specific stakeholder interests.

3. Tensions between stakeholder perspectives. Tensions—differing perspectives or interests, usually aligned with specific stakeholder groups that appear to be oppositional to one another—are an important aspect of comment analysis. As is typical in the collection and analysis of public comments, tensions between stakeholder groups often emerge and need to be highlighted so that revision recommendations can be developed to bridge differing perspectives.

4. The “weight” of the comment’s source. “Weight” refers to the level of importance that a comment has in the formation of revision recommendations. A comment is given additional weight when it represents the collective voice of a larger constituency, such as an organization or society. Although AIR believes that each comment should be considered, regardless of its source, analysts also appreciate that comments coming from an individual or single corporation must be weighted differently from those of a professional association representing multiple (and in some cases, many) stakeholders. A full table of comments with sources identified has been made available to PCORI should the Committee choose to consider source more explicitly in determining revisions.

Stakeholder Advisory Group Nine stakeholders representing different stakeholder groups were invited to serve as stakeholder

advisors for the project. Group members were chosen from PCORI contacts who had participated in

public events about the proposed process or who PCORI knew were planning to submit comments about

the process document. This group was assembled to ensure an additional level of detailed stakeholder

input into the analysis of the public comments.

On December 15, 2014, members of the advisory group reviewed the synthesis of the public comments

and major recommendation themes, participated in an adapted Affinity Diagramming exercise, and

offered both feedback and suggestions for their improvement. These suggestions have been

incorporated in this report. Members of the Stakeholder Advisory Group are listed in Table 1.


Table 1. Participants in the PCORI Methodology Report Patient Panel

Participant Information Brief Biography

Marc Boutin, JD Mr. Boutin is the Executive Vice President and Chief Operating Officer of the National Health Council—an organization which provides a united voice for millions of people with chronic diseases and disabilities and their family caregivers. He provides guidance to patient organizations on various association issues, including corporate structure, government relations, fundraising, and outreach, and he is a regular spokesperson before the media, Congress, and policy makers on major issues of interest to the patient community.

Rachel Hess, MD, MS Dr. Hess is a professor of internal medicine at the University of Utah. Her research focuses on understanding and improving patient-centered outcomes, such as health-related quality of life. She also directs the University of Utah’s Health System Innovation and Research Program.

Rebecca Burkholder, JD Ms. Burkholder is Vice President of Health Policy at the National Consumers League—a national, nonprofit organization that has represented consumers and workers since 1899. Ms. Burkholder coordinates the League’s work on various healthcare issues, including safe use of medication, patient safety, doctor–patient communication, and direct-to-consumer advertising. She is responsible for research on these issues, producing consumer information, and advocating for system changes.

John Schall, MPP Mr. Schall is Chief Executive Officer of the Caregiver Action Network. He is a public policy and communications professional with expertise in a wide range of policy fields, including health care, labor, education, economic development, taxation, and budget policy.

David Bates, MD, MSc Dr. Bates is the Medical Director of Clinical and Quality Analysis at Partners HealthCare. He also serves as Chief of the Division of General Internal Medicine and Primary Care in the Department of Medicine at Brigham and Women’s Hospital and is a professor of medicine at Harvard Medical School. His primary informatics focus concerns the use of information technology to measure and improve the quality of care within the field of medication safety.

Suzanne Belinson, PhD, MPH

Dr. Belinson is the Executive Director for Clinical Effectiveness at the BlueCross BlueShield Association (BCBSA). She is part of the leadership team in BCBSA’s Office of Clinical Affairs, which focuses on developing emerging programs and services that enhance the clinical effectiveness for the independent BCBS Plans. Before joining the BCBSA, Dr. Belinson was at Northwestern University, where she focused on community-based cervical cancer prevention strategies for the United States and new screening and diagnostic test development for the developing world.


Participant Information Brief Biography

Jennifer Graff, PharmD Dr. Graff is the National Pharmaceutical Council’s (NPC) Vice President of Comparative Effectiveness Research (CER). Dr. Graff works to advance NPC’s CER and evidence-based medicine policy research initiatives assessing the methods, interpretation, and application of CER.

Christine Laine, MD, MPH, FACP

Dr. Laine is Editor in Chief of Annals of Internal Medicine and a Senior Vice President at the American College of Physicians. She is a practicing physician in Philadelphia and is board-certified in internal medicine. She is active in the field of medical journalism and holds leadership positions in the International Committee of Medical Journal Editors, the Council of Science Editors, and the ethics committee of the World Association of Medical Editors. She has been instrumental in the development of editorial policy on issues such as authorship and conflicts of interest in medical research.

Chandra Branham, JD Ms. Branham is Vice President of Payment and Health Care Delivery at the Advanced Medical Technology Association (AdvaMed). She leads AdvaMed’s policy work in a number of areas, including diagnostics payment, coverage issues, and comparative effectiveness.

Major Findings

Quantity and Source of the Comments A total of 63 individuals or organizations submitted comments online at the close of the open comment

period (midnight ET, November 7, 2014). Comments were provided in a formal letter by 25

organizations. PCORI also received formal comments and suggestions from the NIH and the PCORI

Methodology Committee; these were not included in the public comment count or statistical analyses

and were analyzed separately.

Collectively, the 63 unique submissions comprised a total of 832 unique comments. Comments were

identified by stakeholder category as indicated by the individual or organization submitting comments.

Figure 1 provides a chart of the percentages of total comments by self-identified stakeholder category.


Figure 1. Number and Percentage of Submissions by Stakeholder Category2

The following identifiers were not selected by any of the individuals’ or organizations’ submissions:

Payer/Insurer

Training Institution

Hospital/Health System

Purchaser/Business

Organizational Provider

Policy Maker

2 Stakeholder identification as it is referenced throughout this report is self-identified. However, several classifications were changed to better reflect the perspective of the submission. In several submissions, the name of the individual or organization provided within the comment indicated a contradicting classification. Specifically, several submissions representing an individual’s perspective were self-identified as coming from a patient-advocacy organization or a training institution. These submissions were carefully scrutinized and corrected. AIR wishes to honor the self-identification process but also recognizes the importance of actual representation of stakeholder categorization when providing weights for thematic analysis.

Researcher, 20, 32%

Patient, 7, 11%Patient/Caregiver Advocacy

Organization, 13, 21%

Clinician/Clinician Society, 14, 21%

Journal/Publisher, 3, 5%

Family member or Caregiver, 1, 2%

Life Sciences Industry,

5, 8%

Submissions by Stakeholder Category (n=63)

Legend:

The total number of submissions

for each stakeholder group is

followed by the percentage.


Comments per Proposal Section Comments were coded to identify the section of the process. Coding staff assessed the autocoding

function and made adjustments via manual coding if the comment was not truly relevant to the section

of the process as it was entered into the intake form. Table 2 reflects the accurate accounting of

comments received, per section of the Proposal.

The proposed peer-review process received the highest number of comments (119).

Table 2. Comment Totals by Section

Section Submissions Comments

Related to Process as a Whole 35 68

Related to a Specific Section 60 466

Registration at ClinicalTrials.gov, RoPR, or PROSPERO 36 65

Production of a Draft Final Report, Scientific Abstract, and Stand-Alone Results Table

33 76

Proposed Peer-Review Process 45 119

Providing a Lay-Friendly Summary 40 72

Submission of Results to ClinicalTrials.gov 21 25

Posting of Full Final Reports on PCORI’s Website 37 76

Notification to PCORI of Submission to Journals and Publication 24 33

Related to the Process Overall 47 253

Timeline of Process 31 59

How Well the Proposal Meets PCORI’s Obligations 19 26

How Well the Proposal Provides a Clear Understanding of the Process

14 14

Feasibility of the Proposal 18 25

Sequencing of the Proposal 26 36

Patient Centeredness 7 27

Research Prioritization 7 16

Translation Table and Registries 1 2

Dissemination of Research 15 48

Areas of General Agreement The majority of the submissions responded in agreement to the statements provided at the end of the

comment collection form, with an average of 57% of responses indicating the first and second highest

indicators of agreement. Responses to each Likert-scale assessment are provided in Figures 2 through 4.


Figure 2. Responses to Rating Scale #1 (Meets Obligations)

Figure 3. Responses to Rating Scale #2 (Provides Clarity)

Very Well, 20, 32%

Somewhat Well, 15, 24%

Neither Well nor Poorly, 9, 14%

Somewhat Poorly, 3, 5%

Very Poorly, 1, 1%

No Response, 15, 24%

How Well the Proposal Meets PCORI's Obligations Under its Authorizing Legislation

Very Clear, 25, 40%

Somewhat Clear, 12, 19%

Neither Clear nor Unclear, 5, 8%

Somewhat Unclear, 4, 6%

Very Unclear, 0, 0%


How Well the Proposal Provides a Clear Understanding of the Peer Review and Public Release Process

Legend:

The total number of submissions for each

stakeholder group is followed by the percentage.

Legend:

The total number of submissions for each stakeholder group is followed by the percentage.


Figure 4. Agreement with Sequencing

In addition, the majority of actual comments collected demonstrate a strong sense across stakeholder

groups of general agreement with the proposed process, as Figure 5 shows.

Figure 5. Indications of Agreement per Comment

Strongly Agree, 22, 35%

Somewhat Agree, 15, 24%

Neither Agree nor Disagree, 5, 8%

Somewhat Disagree,

5, 8%

Strongly Disagree, 0, 0%


Do You Agree with the Sequencing of the Peer Review and Public Posting of Results Process

Indicated Agreement, 239,

49%

Indicated Agreement with

Caution, 189, 39%

Indicated Disagreement, 57,

12%

Indications of Agreement Per Comment

Legend:




Legend:





Themes

Emerging Themes/Overarching Issues Themes from the public comments were developed to better understand the comments that expressed

disagreement or concern about one or more sections of the process.

In general, the comments were respectful, expressing recognition and appreciation for PCORI’s efforts

to make research publicly available. Some differences did emerge, however, between members of the

research community and those who identified as patients/caregiver advocacy organizations. These

differences, reflected in the conceptual map of the comment themes, are shown in Figure 6.

Emerging Themes Surrounding Caution or Disagreement

Five unique themes emerged from the analysis of the comments identified as indicating caution or

disagreement. As described above, the themes were derived by considering the source, frequency, and

intensity.

1. Unclear details concerning the conduct of peer review

2. Feasibility and threats to accuracy

3. Excessive burdens on researchers

4. Dissemination of the research findings

5. Lay-abstract authorship, readability, and utility

These five themes are explored in detail with PCORI’s responses in Table 3. These themes, however, do

not represent the entirety of all comments, which spanned a wide range of individual reactions across a

variety of subjects. Differences between comments from researchers and comments from

patients/caregivers/advocacy organizations were found with respect to composition of reviewer lists,

writing of lay abstracts, and dissemination of findings.

Table 3. Themes from the comments of Disagreement or Concern

Aggregate Theme Components of the Theme

Unclear details concerning the conduct of peer review

• The peer-reviewer selection process and concerns about what determines “content” expertise

• The ways in which the peer-review process will be standardized across multiple reports

• The ways in which peer reviews will be quality controlled • Role of patients, caregivers, or their advocates in the process • The inclusion of a Methodological Review in the process


Feasibility and threats to accurate interpretation of results

• The resolution of conflicts between peer reviewers, the review panel and the authors, or between the PCORI peer review and a publisher’s peer review

• Potential misinterpretation or inconsistent reporting of study findings between different abstracts and publications

Excessive burdens the process may place on researchers

• Potential for multiple rounds of review and revisions • Potential for additional work requested after the end of the

funding period

Dissemination of the research findings

• Public release process is limited for widespread dissemination • Unclear connection of process to broader PCORI

Dissemination Plan

Lay abstract authorship, readability, and utility

• Proposed reading level of 8th grade too high • Role of patients, caregivers, or their advocates in the

authorship process • Length of abstract may be insufficient

PCORI’s Response to Public Comments: Summary and Conclusion

Suggestions for revision were organized by theme and reviewed to determine a list of nine specific areas for revision of the proposed process. Table 4 provides a summary of the most significant revision suggestions, with PCORI's responses and proposed revisions to the draft language.

Table 4. Suggestions for Revision and PCORI’s Response

What We Heard PCORI’s Response Draft Revision Language

Incorporate patients and other stakeholders into the peer review, lay abstract, and dissemination processes.

PCORI is committed to including patients and other stakeholders at every stage of the research and dissemination processes. We have modified the document accordingly.

Reviewers for a particular draft final report will include the Methodologist and/or biostatistician, External subject matter experts, Patient/caregiver and other stakeholder reviewers: At least one patient and/or caregiver with experience with the disease or condition relevant to the study will be invited to serve as a reviewer, commenting on the study’s relevance and usefulness. The goal will be to determine if patient/caregiver perspectives, values, and preferences were adequately considered in the report’s summary of the study’s results and to confirm that the results are useful to them in making


decisions about care options. As appropriate, a representative of another stakeholder group, such as payers, employers, the life sciences industry, or policy makers, also may be invited to comment as part of the review process.

Creating and posting the 500-word (lay) abstract for patients, consumers, and the general public addresses… a. PCORI will select a qualified and

experienced contractor to develop all of the lay abstracts in a manner and format that is consistent.

b. The contractor will be required to conduct cognitive and focus testing, ensure that the abstracts are written at an appropriate reading level, and include patients to ensure that the findings are comprehensible.

Establish a clear process for the resolution of potential disagreements or differences in interpretation between required changes to a final report by the PCORI peer review and those required by a journal review. Make the entire review process transparent so that reviewer comments and researcher responses are available to the public.

PCORI recognizes that the conclusions in the peer-reviewed final report may differ from what appears in journal publications, for a variety of reasons. PCORI will work closely with investigators to update material posted on PCORI.org to note any updates in findings based on publications. To the fullest extent possible, PCORI’s process will be coordinated with a journal’s review process. PCORI agrees that for purposes of transparency, anonymized reviewer comments, and Awardees’ responses, will be posted as part of PCORI’s review process.

There may be times when there is a material difference of opinion between the Awardee’s Principal Investigator and PCORI about reviewer comments or proposed revisions … the final report as revised by agreement between PCORI and the Awardee will be posted on PCORI.org, along with the anonymized comments of the peer reviewers and the Awardee’s response.


Create a standardized approach to the selection of peer reviewers and the peer-review process as a whole. Include details in the process about how PCORI will select the peer reviewers or use a vendor to provide peer review of the reports.

PCORI agrees that more detail about the review process is useful and warranted.

PCORI shall engage a qualified contractor, who will be closely managed by PCORI staff, to administer the peer review of draft final reports.

Reviewers for a particular draft final report will include the Methodologist and/or biostatistician, External subject matter experts, Patient/caregiver and other stakeholder reviewers.

The Principal Investigator shall recommend up to two subject matter experts for PCORI to consider inviting to participate in the review of the final report. Clinical, scientific, and technical experts from drug and device manufacturers may be among those chosen as methodologists or content experts.

Clarify or revise the process to indicate the relationship between this process and PCORI’s broader plan and efforts to disseminate research findings.

PCORI agrees that it would be helpful to clarify how the peer review and public release process fits within PCORI’s broader plans for disseminating the results of its funded studies and has added such language. Currently a draft dissemination and implementation framework and toolkit is posted on PCORI.org; additional updates and planning documents will posted as they are available.

This draft document outlines PCORI’s proposed process for fulfilling its statutory mandate to develop and implement a process for peer-reviewing its primary research and making research findings publicly available in a form and format useful to patients, clinicians, and others. This process is envisioned as one element within a far more extensive and integrated effort to disseminate the results of PCORI-funded research to stakeholders across the healthcare community. We are planning and will implement this broader dissemination and implementation strategy in close collaboration with the Agency for Healthcare Research and Quality (AHRQ), as outlined in our authorizing legislation, as well as through the community of healthcare stakeholders, both


individuals and organizations, with whom we and our funded investigators have been engaged since early in PCORI’s existence.

Clarify the purpose and approach to the methodological review of the report.

PCORI agrees and has added such language.

The methodological review is intended to provide final confirmation of the validity of the ongoing PCORI staff review of each project for adherence to PCORI’s Methodology Standards, a process undertaken throughout the life of a particular study. If the review finds that the study has methodological flaws not previously identified and addressed, the Awardee will be asked to revise the conclusions or other aspects of the research report to reflect that fact.

Reduce the required reading level of the lay abstract to below 8th grade.

PCORI recognizes the need to disseminate information that is accessible by those with limited health literacy and has modified its requirement for readability accordingly.

Following formal acceptance of the final report, PCORI will create a standardized summary of the study’s results for patients and general public with readability at the 6th grade level, which will be reviewed and approved by the Awardee.

Assure that the abstract and tables are not prior publication.

Harmonize the processes of producing the final report, reporting on ClinicalTrials.gov and publication in the peer-reviewed literature.

Minimize the burden to Awardees.

PCORI recognizes the importance of these concerns and has taken several steps to address them. These include clarifying that the International Committee of Medical Journal Editors considers the 500-word abstract and results tables referenced in the peer-review process document not to be prior publication and that PCORI will work closely with Awardees to see that PCORI’s process does not interfere with their fulfilling any other legal or regulatory reporting requirements or impede their ability to publish papers resulting from their funded projects.

PCORI’s plan to include within its proposed process an approach developed by the National Institutes of Health to implement the Food and Drug Administration Amendments Act (FDAAA) through the National Library of Medicine’s clinical trials registry (ClinicalTrials.gov), one that journal editors generally accept as not constituting prior publication.

Registration is a documented milestone in the contract between the Awardee and PCORI.

The date when results tables are submitted is a recorded milestone in the Awardees contract with PCORI.


The Awardee Institution must submit a draft final report to PCORI on a date established and recorded as a milestone in the contract with PCORI. The date may not exceed 13 months from the primary completion date.

Add the requirement of a formal evaluation of the process in one year.

PCORI agrees with this suggestion and has added such language to the process document.

This process will undergo a formal review one year after adoption, with additional review and revision as appropriate in future years to assess how well it is functioning and consistent with PCORI’s authorizing law.

PCORI’s Process for Peer Review of Primary Research and ......Standards before results are...

Documents

Transcript of PCORI’s Process for Peer Review of Primary Research and ......Standards before results are...