i
PCORI’s Process for Peer Review of Primary
Research and Public Release of Findings
Key Findings from Public Comments and
Proposed Final Revisions
Prepared for PCORI by American Institutes for Research
February 17, 2015
ii
This report was prepared under PCORI RFQ # PCO-CVDLDSR2013.004
Acknowledgments
PCORI and the American Institutes for Research (AIR) wish to thank the members of the
Stakeholder Advisory Group for their guidance throughout the public comment process, as well
as the stakeholders who participated in PCORI’s live and online events about the proposed
process during the public comment period.
PCORI is appreciative of the efforts of Thomas Workman, PhD and Marla L. Clayman, PhD, MPH
and the team at AIR for their work on collecting and analyzing the public comments.
iii
Overview
On September 15, 2014, the Patient-Centered Outcomes Research Institute (PCORI) Board of Governors
approved for posting for public comment PCORI’s draft Proposed Process for Peer Review of Primary
Research and Public Release of Findings. On the same day, PCORI opened a 54-day public comment
period for the proposed process, developed in response to the section of PCORI’s authorizing legislation
that states in part that “The institute shall ensure that there is a process for peer review of primary
research” and provides a set of additional requirements for the release of research findings.
From September 15 to November 7, 2014, the American Institutes for Research® (AIR®), as contracted by
PCORI, collected public comments on the proposed process. A total of 63 unique submissions and 832
unique comments were received in addition to comments from the National Institutes of Health (NIH)
and the PCORI Methodology Committee.
Researchers, patients, patient or caregiver advocacy organizations, and clinicians made up the majority
of those making comments, accounting for 86% of the total submissions (32%, 11%, 21%, and 22%,
respectively). No other group contributed more than 10% of total comments submitted; additional
comments came from professional journals, the life sciences industry, and individual family members or
caregivers.
PCORI also received comments and recommendations from the PCORI Methodology Committee and the
NIH. These comments were processed separately as they were not part of the public-comment-
collection process. Several of these comments, however, echoed sentiments expressed by the collected
public comments.
Although there was a substantial degree of agreement with the proposed process across all stakeholder
groups, five major themes emerged from an analysis of the comments following a systematic
assessment process, resulting in nine recommendations for revision that were reflected across the
comments.
Revisions to the draft process document were made based on these recommendations and the updated
draft presented to the PCORI Board of Governors on February 24, 2015 to consider its adoption.
iv
Contents
Overview ................................................................................................................................................ iii
Introduction ............................................................................................................................................ 1
Methods of Analysis………………………………………………………………………………………………………………………………..1
Stakeholder Advisory Group…………………………………………………………………………………………………………………….4
Major Findings......................................................................................................................................... 6
Quantity and Source of the Comments ................................................................................................ 6
Comments per Proposal Section .......................................................................................................... 8
Areas of General Agreement................................................................................................................ 8
Themes from the Comments of Disagreement or Concern ................................................................. 11
Suggestions for Revision and PCORI Responses………………………………………………………………………………….13
PCORI’s Process for Peer Review of Primary Research and Public Release of Findings 1
Introduction
The Patient-Centered Outcomes Research Institute (PCORI) drafted its Proposed Process for Peer Review
of Primary Research and Public Release of Findings to meet its statutory mandate—which states in part
that “The institute shall ensure that there is a process for peer review of primary research.”
The dual requirements in PCORI’s authorizing law for peer review of primary research and making
research findings publicly available within a specific timeframe are not directly linked in the legislation
and pose challenges that must be resolved if PCORI is to comply with both. The proposed process
drafted by PCORI and revised in response to public comment is intended to balance the challenges of
ensuring the timely release of research results while assessing their scientific integrity in a way that
respects the interests and needs of patients, other healthcare stakeholders, our awardees, and the
general public, as well as recognizes the crucial role of peer-reviewed journals in disseminating new
evidence to the scientific and clinical communities.
The mandate for peer review underscores the importance of ensuring that primary research funded by
PCORI be critically appraised for scientific integrity as well as for adherence to PCORI’s Methodological
Standards before results are released. Such a peer-review process adds to the credibility, authority, and
trustworthiness of PCORI-funded research findings.
On September 14, 2014, the PCORI Board of Governors approved for posting for public comment the
draft proposed process document. On the same day, PCORI opened a 54-day public comment period for
the proposed process. Public comments were collected from September 15 to November 7, 2014.
PCORI hosted a live public forum on September 29, 2014 and a webinar on October 29, 2014 featuring
stakeholder feedback on the proposed process.
PCORI contracted with the American Institutes for Research® (AIR®) to create the web-based tool used to
collect public comment and provide the analysis of the comments on the proposed process.
A total of 63 unique submissions and 832 unique comments were received in addition to comments
from the National Institutes of Health (NIH) and the PCORI Methodology Committee.
Methods of Analysis Submissions were pulled as they appeared from the online tool (pcori.airprojects.org/node/1/webform-
results/download) and entered into a coding table. Repeated submissions (two submissions with the
same content) were combined into a single entry, and repeated comments within a single submission
were eliminated for a final cleaned copy of the comment table. Elements of the letterhead (when used),
PCORI or return address, salutation, or signature line were removed. This table was used for the purpose
of coding and analysis. All counts of submissions and comments reflect the combination of duplicate
files.
PCORI’s Process for Peer Review of Primary Research and Public Release of Findings 2
General Review Comments as a whole were reviewed in their entirety. Analysts also reviewed the autocoded results of
all Likert-scale responses and compared them to the comments provided by individual respondent and
by stakeholder type. Analysts were able to get a clear feel of the comments in general to guide the
creation of a code book to organize and process comments.
Coding A Code Book and Coding Scheme were developed to help organize the comments and identify the
suggestions for revision found in the comments as a whole. A team of four qualitative researchers coded
each of the comments. A set of comments was coded by multiple staff members to test agreement
across the coding staff. This helps determine how reliable the coding of comments is across the entire
set. Agreement across coding staff exceeded 90%, so no changes or clarifications to the code book were
necessary.
All comments were coded by the coding team for 10 general classifications:
1. Unique Identification. A number was used to count individual comments by submission. Comments were separated and numbered by the order in which they were received.
2. Stakeholder Self-Identification. Codes were created for each of the stakeholder categories provided in the online collection tool. A second code was created in cases where a stakeholder category was not selected to indicate if the comment did or did not reflect a knowledge/familiarity of the research process; however, this code was not necessary and therefore was not used.
3. Relevance to the Report. Comments were coded to classify their relevance to (1) the report as a whole, (2) a specific section, or (3) the overall process. Comments that did not relate to the report in any of these ways were also coded appropriately.1 Comments were also autocoded with the question in which the response was submitted.
4. Level of Agreement. All comments that related to the report were coded, based on their stated or interpreted agreement with the section of the Proposal. Review of the entire set of comments found that an additional code was needed, as a number of comments stated general agreement but also mentioned a small concern. These comments were coded as “Agree With Caution.” Coders also identified comments where agreement or disagreement was not made clear.
5. Affect. All comments were also coded for the affect, or emotional direction, of the comment: positive, negative, or indifferent.
Additional codes were used to organize the comments expressing disagreement or concern so that a
clear set of revision recommendations could be identified. The following codes were used for this
purpose:
1 Comments considered unrelated to the report may still be related to PCORI in general or be of interest to its work.
PCORI’s Process for Peer Review of Primary Research and Public Release of Findings 3
6. Suggested Action/Direction. Comments were coded to classify the suggested action directly stated or implied within the comment using the following distinctions:
a. Add something: The comment stated or implied that an additional factor or item be added to a section of the process.
b. Remove something: The comment stated or implied that some factor or item be removed from a section of the process.
c. Modify something: The comment stated or implied that a factor or item in a section of the process needed to be revised or modified.
d. No action indicated. This code was used when no stated or implied action was provided in the comment.
e. Action suggested is unrelated to the process. This code was used when a comment stated or implied an action that was unrelated to any section of the process.
f. Action is unclear. This code was used when an action was implied but written in such a way that its direction could not be interpreted.
7. Feasibility. Initial review of the comments as a whole found that issues of feasibility regarding the process were often implied. Therefore, coders identified comments where feasibility appeared to be the central tenet of the disagreement. Comments relating to the feasibility of the proposed process were coded with the feasibility code as:
a. Process as a whole is not feasible.
b. Specific section of process is not feasible.
c. Legal mandate is not feasible.
i. The feasibility concern focused on peer-review system capacity.
ii. The feasibility concern focused on PCORI staffing capacity.
iii. The feasibility concern focused on the capacity of academic journals.
iv. The feasibility concern focused on researcher capacity.
8. Language and Writing Style. Comments focused on the style of the proposed process, including organization, format, length, or word choice/phrasing.
Two additional codes were used to identify comment content important to PCORI:
9. Compliment or Support. All comments that indicated support for PCORI or included compliments were coded and marked as support for PCORI, the process as a whole, or a specific section.
10. Inquiry. All specific requests for clarification were also coded.
Additional Analysis to Determine Themes
The complete set of separated and coded comments was entered into NVivo 10 software. Reports of
comments per code were reviewed to determine:
General word frequency (without codes)
PCORI’s Process for Peer Review of Primary Research and Public Release of Findings 4
Comments by agreement and disagreement codes
Comments by direction and affect of disagreement (recommended action and feasibility)
Comments of appreciation or complimentary to the process
Coded lists of comments were then reviewed for additional analysis. A conceptual map was developed
to further clarify the nature of the comments and their relationship to stakeholders’ interests. Analysts
then began the process of developing specific themes on the basis of four criteria:
1. The prominence of the theme across comment submissions. A theme was considered prominent if it was raised in three or more of the submissions, particularly if repeated multiple times in comments throughout each submission.
2. The clarity of the comment toward providing a recommendation for revision. Analysts were particularly mindful of the suggested action or intent of the comments, and they used thematic analysis to focus the nature of the direction of revisions, especially as revisions related to specific stakeholder interests.
3. Tensions between stakeholder perspectives. Tensions—differing perspectives or interests, usually aligned with specific stakeholder groups that appear to be oppositional to one another—are an important aspect of comment analysis. As is typical in the collection and analysis of public comments, tensions between stakeholder groups often emerge and need to be highlighted so that revision recommendations can be developed to bridge differing perspectives.
4. The “weight” of the comment’s source. “Weight” refers to the level of importance that a comment has in the formation of revision recommendations. A comment is given additional weight when it represents the collective voice of a larger constituency, such as an organization or society. Although AIR believes that each comment should be considered, regardless of its source, analysts also appreciate that comments coming from an individual or single corporation must be weighted differently from those of a professional association representing multiple (and in some cases, many) stakeholders. A full table of comments with sources identified has been made available to PCORI should the Committee choose to consider source more explicitly in determining revisions.
Stakeholder Advisory Group Nine stakeholders representing different stakeholder groups were invited to serve as stakeholder
advisors for the project. Group members were chosen from PCORI contacts who had participated in
public events about the proposed process or who PCORI knew were planning to submit comments about
the process document. This group was assembled to ensure an additional level of detailed stakeholder
input into the analysis of the public comments.
On December 15, 2014, members of the advisory group reviewed the synthesis of the public comments
and major recommendation themes, participated in an adapted Affinity Diagramming exercise, and
offered both feedback and suggestions for their improvement. These suggestions have been
incorporated in this report. Members of the Stakeholder Advisory Group are listed in Table 1.
PCORI’s Process for Peer Review of Primary Research and Public Release of Findings 5
Table 1. Participants in the PCORI Methodology Report Patient Panel
Participant Information Brief Biography
Marc Boutin, JD Mr. Boutin is the Executive Vice President and Chief Operating Officer of the National Health Council—an organization which provides a united voice for millions of people with chronic diseases and disabilities and their family caregivers. He provides guidance to patient organizations on various association issues, including corporate structure, government relations, fundraising, and outreach, and he is a regular spokesperson before the media, Congress, and policy makers on major issues of interest to the patient community.
Rachel Hess, MD, MS Dr. Hess is a professor of internal medicine at the University of Utah. Her research focuses on understanding and improving patient-centered outcomes, such as health-related quality of life. She also directs the University of Utah’s Health System Innovation and Research Program.
Rebecca Burkholder, JD Ms. Burkholder is Vice President of Health Policy at the National Consumers League—a national, nonprofit organization that has represented consumers and workers since 1899. Ms. Burkholder coordinates the League’s work on various healthcare issues, including safe use of medication, patient safety, doctor–patient communication, and direct-to-consumer advertising. She is responsible for research on these issues, producing consumer information, and advocating for system changes.
John Schall, MPP Mr. Schall is Chief Executive Officer of the Caregiver Action Network. He is a public policy and communications professional with expertise in a wide range of policy fields, including health care, labor, education, economic development, taxation, and budget policy.
David Bates, MD, MSc Dr. Bates is the Medical Director of Clinical and Quality Analysis at Partners HealthCare. He also serves as Chief of the Division of General Internal Medicine and Primary Care in the Department of Medicine at Brigham and Women’s Hospital and is a professor of medicine at Harvard Medical School. His primary informatics focus concerns the use of information technology to measure and improve the quality of care within the field of medication safety.
Suzanne Belinson, PhD, MPH
Dr. Belinson is the Executive Director for Clinical Effectiveness at the BlueCross BlueShield Association (BCBSA). She is part of the leadership team in BCBSA’s Office of Clinical Affairs, which focuses on developing emerging programs and services that enhance the clinical effectiveness for the independent BCBS Plans. Before joining the BCBSA, Dr. Belinson was at Northwestern University, where she focused on community-based cervical cancer prevention strategies for the United States and new screening and diagnostic test development for the developing world.
PCORI’s Process for Peer Review of Primary Research and Public Release of Findings 6
Participant Information Brief Biography
Jennifer Graff, PharmD Dr. Graff is the National Pharmaceutical Council’s (NPC) Vice President of Comparative Effectiveness Research (CER). Dr. Graff works to advance NPC’s CER and evidence-based medicine policy research initiatives assessing the methods, interpretation, and application of CER.
Christine Laine, MD, MPH, FACP
Dr. Laine is Editor in Chief of Annals of Internal Medicine and a Senior Vice President at the American College of Physicians. She is a practicing physician in Philadelphia and is board-certified in internal medicine. She is active in the field of medical journalism and holds leadership positions in the International Committee of Medical Journal Editors, the Council of Science Editors, and the ethics committee of the World Association of Medical Editors. She has been instrumental in the development of editorial policy on issues such as authorship and conflicts of interest in medical research.
Chandra Branham, JD Ms. Branham is Vice President of Payment and Health Care Delivery at the Advanced Medical Technology Association (AdvaMed). She leads AdvaMed’s policy work in a number of areas, including diagnostics payment, coverage issues, and comparative effectiveness.
Major Findings
Quantity and Source of the Comments A total of 63 individuals or organizations submitted comments online at the close of the open comment
period (midnight ET, November 7, 2014). Comments were provided in a formal letter by 25
organizations. PCORI also received formal comments and suggestions from the NIH and the PCORI
Methodology Committee; these were not included in the public comment count or statistical analyses
and were analyzed separately.
Collectively, the 63 unique submissions comprised a total of 832 unique comments. Comments were
identified by stakeholder category as indicated by the individual or organization submitting comments.
Figure 1 provides a chart of the percentages of total comments by self-identified stakeholder category.
PCORI’s Process for Peer Review of Primary Research and Public Release of Findings 7
Figure 1. Number and Percentage of Submissions by Stakeholder Category2
The following identifiers were not selected by any of the individuals’ or organizations’ submissions:
Payer/Insurer
Training Institution
Hospital/Health System
Purchaser/Business
Organizational Provider
Policy Maker
2 Stakeholder identification as it is referenced throughout this report is self-identified. However, several classifications were changed to better reflect the perspective of the submission. In several submissions, the name of the individual or organization provided within the comment indicated a contradicting classification. Specifically, several submissions representing an individual’s perspective were self-identified as coming from a patient-advocacy organization or a training institution. These submissions were carefully scrutinized and corrected. AIR wishes to honor the self-identification process but also recognizes the importance of actual representation of stakeholder categorization when providing weights for thematic analysis.
Researcher, 20, 32%
Patient, 7, 11%Patient/Caregiver Advocacy
Organization, 13, 21%
Clinician/Clinician Society, 14, 21%
Journal/Publisher, 3, 5%
Family member or Caregiver, 1, 2%
Life Sciences Industry,
5, 8%
Submissions by Stakeholder Category (n=63)
Legend:
The total number of submissions
for each stakeholder group is
followed by the percentage.
PCORI’s Process for Peer Review of Primary Research and Public Release of Findings 8
Comments per Proposal Section Comments were coded to identify the section of the process. Coding staff assessed the autocoding
function and made adjustments via manual coding if the comment was not truly relevant to the section
of the process as it was entered into the intake form. Table 2 reflects the accurate accounting of
comments received, per section of the Proposal.
The proposed peer-review process received the highest number of comments (119).
Table 2. Comment Totals by Section
Section Submissions Comments
Related to Process as a Whole 35 68
Related to a Specific Section 60 466
Registration at ClinicalTrials.gov, RoPR, or PROSPERO 36 65
Production of a Draft Final Report, Scientific Abstract, and Stand-Alone Results Table
33 76
Proposed Peer-Review Process 45 119
Providing a Lay-Friendly Summary 40 72
Submission of Results to ClinicalTrials.gov 21 25
Posting of Full Final Reports on PCORI’s Website 37 76
Notification to PCORI of Submission to Journals and Publication 24 33
Related to the Process Overall 47 253
Timeline of Process 31 59
How Well the Proposal Meets PCORI’s Obligations 19 26
How Well the Proposal Provides a Clear Understanding of the Process
14 14
Feasibility of the Proposal 18 25
Sequencing of the Proposal 26 36
Patient Centeredness 7 27
Research Prioritization 7 16
Translation Table and Registries 1 2
Dissemination of Research 15 48
Areas of General Agreement The majority of the submissions responded in agreement to the statements provided at the end of the
comment collection form, with an average of 57% of responses indicating the first and second highest
indicators of agreement. Responses to each Likert-scale assessment are provided in Figures 2 through 4.
PCORI’s Process for Peer Review of Primary Research and Public Release of Findings 9
Figure 2. Responses to Rating Scale #1 (Meets Obligations)
Figure 3. Responses to Rating Scale #2 (Provides Clarity)
Very Well, 20, 32%
Somewhat Well, 15, 24%
Neither Well nor Poorly, 9, 14%
Somewhat Poorly, 3, 5%
Very Poorly, 1, 1%
No Response, 15, 24%
How Well the Proposal Meets PCORI's Obligations Under its Authorizing Legislation
Very Clear, 25, 40%
Somewhat Clear, 12, 19%
Neither Clear nor Unclear, 5, 8%
Somewhat Unclear, 4, 6%
Very Unclear, 0, 0%
No Response, 17, 27%
How Well the Proposal Provides a Clear Understanding of the Peer Review and Public Release Process
Legend:
The total number of submissions for each
stakeholder group is followed by the percentage.
Legend:
The total number of submissions for each stakeholder group is followed by the percentage.
PCORI’s Process for Peer Review of Primary Research and Public Release of Findings 10
Figure 4. Agreement with Sequencing
In addition, the majority of actual comments collected demonstrate a strong sense across stakeholder
groups of general agreement with the proposed process, as Figure 5 shows.
Figure 5. Indications of Agreement per Comment
Strongly Agree, 22, 35%
Somewhat Agree, 15, 24%
Neither Agree nor Disagree, 5, 8%
Somewhat Disagree,
5, 8%
Strongly Disagree, 0, 0%
No Response, 16, 25%
Do You Agree with the Sequencing of the Peer Review and Public Posting of Results Process
Indicated Agreement, 239,
49%
Indicated Agreement with
Caution, 189, 39%
Indicated Disagreement, 57,
12%
Indications of Agreement Per Comment
Legend:
The total number of submissions
for each stakeholder group is
followed by the percentage.
Legend:
The total number of submissions
for each stakeholder group is
followed by the percentage.
PCORI’s Process for Peer Review of Primary Research and Public Release of Findings 11
Themes
Emerging Themes/Overarching Issues Themes from the public comments were developed to better understand the comments that expressed
disagreement or concern about one or more sections of the process.
In general, the comments were respectful, expressing recognition and appreciation for PCORI’s efforts
to make research publicly available. Some differences did emerge, however, between members of the
research community and those who identified as patients/caregiver advocacy organizations. These
differences, reflected in the conceptual map of the comment themes, are shown in Figure 6.
Emerging Themes Surrounding Caution or Disagreement
Five unique themes emerged from the analysis of the comments identified as indicating caution or
disagreement. As described above, the themes were derived by considering the source, frequency, and
intensity.
1. Unclear details concerning the conduct of peer review
2. Feasibility and threats to accuracy
3. Excessive burdens on researchers
4. Dissemination of the research findings
5. Lay-abstract authorship, readability, and utility
These five themes are explored in detail with PCORI’s responses in Table 3. These themes, however, do
not represent the entirety of all comments, which spanned a wide range of individual reactions across a
variety of subjects. Differences between comments from researchers and comments from
patients/caregivers/advocacy organizations were found with respect to composition of reviewer lists,
writing of lay abstracts, and dissemination of findings.
Table 3. Themes from the comments of Disagreement or Concern
Aggregate Theme Components of the Theme
Unclear details concerning the conduct of peer review
• The peer-reviewer selection process and concerns about what determines “content” expertise
• The ways in which the peer-review process will be standardized across multiple reports
• The ways in which peer reviews will be quality controlled • Role of patients, caregivers, or their advocates in the process • The inclusion of a Methodological Review in the process
PCORI’s Process for Peer Review of Primary Research and Public Release of Findings 12
Feasibility and threats to accurate interpretation of results
• The resolution of conflicts between peer reviewers, the review panel and the authors, or between the PCORI peer review and a publisher’s peer review
• Potential misinterpretation or inconsistent reporting of study findings between different abstracts and publications
Excessive burdens the process may place on researchers
• Potential for multiple rounds of review and revisions • Potential for additional work requested after the end of the
funding period
Dissemination of the research findings
• Public release process is limited for widespread dissemination • Unclear connection of process to broader PCORI
Dissemination Plan
Lay abstract authorship, readability, and utility
• Proposed reading level of 8th grade too high • Role of patients, caregivers, or their advocates in the
authorship process • Length of abstract may be insufficient
PCORI’s Response to Public Comments: Summary and Conclusion
Suggestions for revision were organized by theme and reviewed to determine a list of nine specific areas for revision of the proposed process. Table 4 provides a summary of the most significant revision suggestions, with PCORI's responses and proposed revisions to the draft language.
Table 4. Suggestions for Revision and PCORI’s Response
What We Heard PCORI’s Response Draft Revision Language
Incorporate patients and other stakeholders into the peer review, lay abstract, and dissemination processes.
PCORI is committed to including patients and other stakeholders at every stage of the research and dissemination processes. We have modified the document accordingly.
Reviewers for a particular draft final report will include the Methodologist and/or biostatistician, External subject matter experts, Patient/caregiver and other stakeholder reviewers: At least one patient and/or caregiver with experience with the disease or condition relevant to the study will be invited to serve as a reviewer, commenting on the study’s relevance and usefulness. The goal will be to determine if patient/caregiver perspectives, values, and preferences were adequately considered in the report’s summary of the study’s results and to confirm that the results are useful to them in making
PCORI’s Process for Peer Review of Primary Research and Public Release of Findings 13
decisions about care options. As appropriate, a representative of another stakeholder group, such as payers, employers, the life sciences industry, or policy makers, also may be invited to comment as part of the review process.
Creating and posting the 500-word (lay) abstract for patients, consumers, and the general public addresses… a. PCORI will select a qualified and
experienced contractor to develop all of the lay abstracts in a manner and format that is consistent.
b. The contractor will be required to conduct cognitive and focus testing, ensure that the abstracts are written at an appropriate reading level, and include patients to ensure that the findings are comprehensible.
Establish a clear process for the resolution of potential disagreements or differences in interpretation between required changes to a final report by the PCORI peer review and those required by a journal review. Make the entire review process transparent so that reviewer comments and researcher responses are available to the public.
PCORI recognizes that the conclusions in the peer-reviewed final report may differ from what appears in journal publications, for a variety of reasons. PCORI will work closely with investigators to update material posted on PCORI.org to note any updates in findings based on publications. To the fullest extent possible, PCORI’s process will be coordinated with a journal’s review process. PCORI agrees that for purposes of transparency, anonymized reviewer comments, and Awardees’ responses, will be posted as part of PCORI’s review process.
There may be times when there is a material difference of opinion between the Awardee’s Principal Investigator and PCORI about reviewer comments or proposed revisions … the final report as revised by agreement between PCORI and the Awardee will be posted on PCORI.org, along with the anonymized comments of the peer reviewers and the Awardee’s response.
PCORI’s Process for Peer Review of Primary Research and Public Release of Findings 14
Create a standardized approach to the selection of peer reviewers and the peer-review process as a whole. Include details in the process about how PCORI will select the peer reviewers or use a vendor to provide peer review of the reports.
PCORI agrees that more detail about the review process is useful and warranted.
PCORI shall engage a qualified contractor, who will be closely managed by PCORI staff, to administer the peer review of draft final reports.
Reviewers for a particular draft final report will include the Methodologist and/or biostatistician, External subject matter experts, Patient/caregiver and other stakeholder reviewers.
The Principal Investigator shall recommend up to two subject matter experts for PCORI to consider inviting to participate in the review of the final report. Clinical, scientific, and technical experts from drug and device manufacturers may be among those chosen as methodologists or content experts.
Clarify or revise the process to indicate the relationship between this process and PCORI’s broader plan and efforts to disseminate research findings.
PCORI agrees that it would be helpful to clarify how the peer review and public release process fits within PCORI’s broader plans for disseminating the results of its funded studies and has added such language. Currently a draft dissemination and implementation framework and toolkit is posted on PCORI.org; additional updates and planning documents will posted as they are available.
This draft document outlines PCORI’s proposed process for fulfilling its statutory mandate to develop and implement a process for peer-reviewing its primary research and making research findings publicly available in a form and format useful to patients, clinicians, and others. This process is envisioned as one element within a far more extensive and integrated effort to disseminate the results of PCORI-funded research to stakeholders across the healthcare community. We are planning and will implement this broader dissemination and implementation strategy in close collaboration with the Agency for Healthcare Research and Quality (AHRQ), as outlined in our authorizing legislation, as well as through the community of healthcare stakeholders, both
PCORI’s Process for Peer Review of Primary Research and Public Release of Findings 15
individuals and organizations, with whom we and our funded investigators have been engaged since early in PCORI’s existence.
Clarify the purpose and approach to the methodological review of the report.
PCORI agrees and has added such language.
The methodological review is intended to provide final confirmation of the validity of the ongoing PCORI staff review of each project for adherence to PCORI’s Methodology Standards, a process undertaken throughout the life of a particular study. If the review finds that the study has methodological flaws not previously identified and addressed, the Awardee will be asked to revise the conclusions or other aspects of the research report to reflect that fact.
Reduce the required reading level of the lay abstract to below 8th grade.
PCORI recognizes the need to disseminate information that is accessible by those with limited health literacy and has modified its requirement for readability accordingly.
Following formal acceptance of the final report, PCORI will create a standardized summary of the study’s results for patients and general public with readability at the 6th grade level, which will be reviewed and approved by the Awardee.
Assure that the abstract and tables are not prior publication.
Harmonize the processes of producing the final report, reporting on ClinicalTrials.gov and publication in the peer-reviewed literature.
Minimize the burden to Awardees.
PCORI recognizes the importance of these concerns and has taken several steps to address them. These include clarifying that the International Committee of Medical Journal Editors considers the 500-word abstract and results tables referenced in the peer-review process document not to be prior publication and that PCORI will work closely with Awardees to see that PCORI’s process does not interfere with their fulfilling any other legal or regulatory reporting requirements or impede their ability to publish papers resulting from their funded projects.
PCORI’s plan to include within its proposed process an approach developed by the National Institutes of Health to implement the Food and Drug Administration Amendments Act (FDAAA) through the National Library of Medicine’s clinical trials registry (ClinicalTrials.gov), one that journal editors generally accept as not constituting prior publication.
Registration is a documented milestone in the contract between the Awardee and PCORI.
The date when results tables are submitted is a recorded milestone in the Awardees contract with PCORI.
PCORI’s Process for Peer Review of Primary Research and Public Release of Findings 16
The Awardee Institution must submit a draft final report to PCORI on a date established and recorded as a milestone in the contract with PCORI. The date may not exceed 13 months from the primary completion date.
Add the requirement of a formal evaluation of the process in one year.
PCORI agrees with this suggestion and has added such language to the process document.
This process will undergo a formal review one year after adoption, with additional review and revision as appropriate in future years to assess how well it is functioning and consistent with PCORI’s authorizing law.
Top Related