Kenneth Wilson, Alan SchreierEast Carolina University

David ResnikNIEHS/NIH

And

Amanda DrozdowskiEast Carolina University

2009 RESEARCH CONFERENCE ON RESEARCH INTEGRITYNIAGARA FALLS, NY

May 16, 2009The ideas and opinions in this presentation do not represent the views of the NIEHS, the NIH or the US government

Includes All Media Hypotheses Paper Protocols Electronic Original Data Photographic Processed Data Other

Methods Analyses Future Plans Manuscripts/Articles

What are the problems?From the Research Misconduct Perspective

What are the current standards or best practices that will prevent problems?

How well do active scientists adhere to these best practices?

Phase I – Interview Officials who conduct research misconduct investigations

Findings:1. In almost half of the cases where research records were

examined, there were problems with the research records.

2. Almost 80 percent of the officials who reported problems with research records had at least one case where the problems were so serious that some issues were never resolved.

3. Over 10 percent of the officials who reported problems with research records reported that the problems with the research records in their typical cases were so serious that some issues were never resolved.

“Research Records and the Resolution of Misconduct Allegations at Research Universities” by Kenneth Wilson, Alan Schreier, Angel Griffin and David Resnik Accountability in Research 2007 14:57–71

Phase II – Conduct focus groups Identify best practices

The best practices differentiate and provide separate standards for three different levels within the university: • the individual researcher, • the research group leader, • the department/institution.

The authors believe these best practices constitute a “snapshot” of the current normative standards for research records within the academic research community..

“Academic Research Record-Keeping: Best Practices for Individuals, Group Leaders, and Institutions” by AA Schreier, KR Wilson, and D. Resnik Academic Medicine 2006 81:42-47.

Phase II – Conduct focus groupsDevelop phase III survey instrument

These two phases have been funded by NSF.

Phase II – Develop phase III survey instrument Person assigned to manage research records and data. Process to assure the smooth transfer and retention of research information I set standards for good record keeping. A written document that explains how records are to be kept New members were clearly told who owns the research records. New members were clearly told who has access rights to the collected data . New members were clearly told how the research records are to be recorded. New members get models of good record keeping practices. Workers provided with record keeping materials A common storage area Supervisors review the research records Steps taken to see that record keeping plan followed. Periodically backup data and other research records. Copies of the supporting records were stored with the manuscript. Documenting and sharing important basic research information Plan to store the research records for at least 3 years. Emails and conference calls documented and saved Able to reconstruct what leavers had done and understand the data Most research records would survive a disaster

These two phases have been funded by NSF.

Phase III- Conduct a national survey of 600 federally funded researchers to document their record keeping practices (Funded by NIH --Completed results presented today)

Phase IV- Survey students/post-docs of the Phase III researchers (Funded by NIH – in the field) to assess the “intergenerational” transmission of research values and ideals

Phase V- In five years, repeat Phase III & IV to determine changes.

Focus groups strongly recommend a combination of web surveys and telephone interviews.

From a random sample of senior faculty at universities who were awarded NIH grants we completed 688 interviews (over 50% response rate).

Respondents were asked about their professional characteristicsProblems they experienced finding

research records in the past 3 yearsTheir actual research practicesTheir “ideal” research practices (those

they would recommend to junior colleagues just starting their own research)

Names of advanced graduate students and post-docs working with them

Real and Ideal research practices were based on the best practices we had developed in Phase II.

Since we were interviewing researchers in many different disciplines, they were formulated at the level of the research group leader.

For example, all group leaders need to set standards but the nature of the standards they set will be different if they are doing biological research or social research.

The focus group told us that at the level of the group leader, the practices were or should be universal.

Respondents were in the Biological Sciences (59%) Social/Behavioral Sciences (20%) Clinical Sciences (11%) Physical Sciences (10%)

The average length of time since their first academic appointment was 19 years.

Think about those times in the last 3 years when you had to go to your research records to locate important pieces of information for a final report, a grant application, a manuscript for publication or for planning a new study. Please indicate if the following problems have 1) almost never; 2) rarely; 3) sometimes; 4) usually; or 5) almost always happened.

Answers recorded as these numbers (1 to 5).

Asked about 12 different problems – for example

I found the information I needed quickly and easily.

I found the information I needed after a significant search.

I know that the information was recorded somewhere but I could not find it.

Another person had the records and it took a while to get them.

I found some of the records had been altered with no explanation.

84% of respondents "usually” or “almost always” found the information they were looking for quickly and easily.

Only 37 percent of our respondents reported that they “almost never” or “rarely” had to conduct a significant search to find a piece of information.

36% of the respondents reported that someone else had the records (sometimes, usually or almost always).

25 percent reported that they found some critical detail that had not been recorded (sometimes, usually or almost always).

How often have the following happened in research that you have participated in during the last 3 years?  Would you say  that these 1) almost never; 2) rarely; 3) sometimes; 4) usually; or 5) almost always happened?

There was a written document that explains how records are to be kept.

Most of my research records would survive in a useable form if a disaster (like a fire, flood, or computer crash) were to strike my research space.

The people working on a research project were provided with record keeping materials (notebooks, binders, specialized software, etc.).

There was a plan to periodically backup data and other research records.

Mean ratings for real practices ranged from 2.81 to 4.61.

7 of the 19 practices had mean rating of 4.00 (usually) or higher.

Only 1 (There was a written document that

explains how records are to be kept) had a mean rating less than 3.00 ( sometimes).

When answering these questions, imagine that young faculty members approach you for advice. They are new to your department and they are about to begin work on their first project where they supervise graduate students and/or staff members. Which of the following would you recommend? Please indicate if it is a practice that you would 1) almost never; 2) rarely; 3) sometimes; 4) usually; or almost always recommend.

The mean rating for the 19 ideal practices ranged from 3.86 to 4.79

18 of 19 ideal practices have average ratings between "usually"(4) and "almost always"(5)

Only 1 (There was a written document that

explains how records are to be kept) had a mean rating less than 4.00 (usually).

13 of 19 have average ratings closest to "almost always" (above 4.5).

These results show that the best practices are widely shared among top researchers.

By comparing what researchers report actually happens in their research (ratings of real practices) with the standards they would recommend to new researchers (ratings of ideal standards), we can identify those standards to which researchers routinely adhere and those that are more problematic.

All 19 ideal practices were rated higher than the corresponding real practice.

The smallest differences were found for:The people working on a research project

should be provided with record keeping materials (notebooks, binders, specialized software, etc.).

There should be a common storage area (physical and/or virtual) where people can store the records they are currently using.

There should be a plan to store or archive the research records in a secure manner for at least 3 years.

The largest differences were found for:There should be a written document that

explains how records are to be kept. Your standard practices should allow you to

reconstruct what research workers had been doing and understand the data they collected even if they leave abruptly.

These results show that while best practices are widely shared among top researchers they have a difficult time living up to their ideal standards.

There is a significant negative correlation between Real Practices and having Problems (r = -.302***).

The correlations between Ideal Practices and having Problems is NOT statistically significant.

There is a significant correlation between the Difference between researchers’ Real and Ideal practices and their Problems (r= 3.51***). This remains significant when the level of Real Practices are controlled.

There is no significant relationship between the year of the researcher’s first faculty appointment and:Their real record keeping practicesTheir ideal record keeping practicesThe difference between their real and ideal

record keeping practicesThe problems caused by their record

keeping practices

While all disciplines have similar standards, there are significant differences between the primary academic disciplines and:Their real record keeping practicesTheir ideal record keeping practicesThe difference between their real and ideal

record keeping practicesThe problems caused by their record

keeping practices

The Best Practices that we identified during our Phase II focus groups are widely shared by federally funded researchers .

“Academic Research Record-Keeping: Best Practices for Individuals, Group Leaders, and Institutions” by AA Schreier, KR Wilson, and D. Resnik Academic Medicine 2006 81:42-47.

While most elite researchers share an understanding of the best practices, they are not able to live up to their own ideal standards.

Lower levels of Real Practices and greater deviations from Ideal Practices create additional work for many researchers.

Lower levels of Real Practices and greater differences between a researcher’s real and ideal practices created time consuming problems for researchers.

While all disciplines have similar standards, the standards in the physical sciences are lowest and the standards in the clinical sciences are highest.

The physical sciences report the most problems created by research record keeping while the social/behavioral sciences report the fewest.

Length of time that researchers have been practicing does not influence their real or ideal practices or the problems they encounter.

For more information please contact

Ken Wilson PhD Department of

Sociology East Carolina

University Greenville, NC

27858 wilsonk@ecu.edu

For a copy of the Best Practices , contact

Al Schreier PhD Research and

Graduate Studies East Carolina

University Greenville, NC 27858 schreiera@ecu.edu