APSA Workshop on Data Access and Research Transparency (DA-RT) in Political Science

Hosted by the Inter-university Consortium for Political and Social Research

Sponsored by the Center for Qualitative and Multi-Method Inquiry, Syracuse University, and

the Center for Political Studies, University of Michigan

• Introductions

• Thanks to Michelle Overholser and ICPSR for organizing the workshop logistics

Premise

• Limited introspection and documentation

• threatens the credibility and legitimacy of social scientific research

Definitions of Progress

CREDIBILITY LEGITIMATE

• the quality of being believable or trustworthy

• in accordance with recognized principles

DA-RT History

• Origin: APSA Council

• Multi-epistemic & multi-method

• First actions: – Two year consultation and discussion – Ethics guide changes– PS Symposium on Transparency (January 2014)

• Now– We seek to support journal editors, archivists, and researchers

DA-RT Components

• Data Access

• Research Transparency– Production Transparency– Analytic Transparency

A. Data Access

• To the extent that researchers’ evidence-based knowledge claims rely on data they themselves generated or collected, they should – provide access to those data – or explain why they cannot.

B. Production Transparency

• Researchers providing access to data should offer a full account of the procedures used to collect or generate the data.

C. Analytic Transparency

• Researchers making evidence-based knowledge claims should provide a full account of how they drew their analytic conclusions from the data

“First Use” Principle

• Researchers who collect or generate data have the right to use those data first.

• Scholars may postpone data access for– for one year after publication or – for a period that publishers/funders specify.

• Journals often require availability on publication.

DA-RT Principle

• Evidence based knowledge claims need‐– citations to the data, and – an explanation of how the data and claims are

connected.

• If data are in author’s charge, the author must – provide access, and – documentation describing how they were

generated/collected.

DA-RT is part of a conversation about transparency occurring in all sciences

6. Researchers have an ethical obligation to facilitate the evaluation of their evidence based knowledge claims through data access, production transparency, and analytic transparency so that their work can be tested or replicated.

6.1 Data access: Researchers making evidence-based knowledge claims should reference the data they used to make those claims. If these are data they themselves generated or collected, researchers should provide access to those data or explain why they cannot.6.2 Production transparency: Researchers providing access to data they themselves generated or collected, should offer a full account of the procedures used to collect or generate the data.6.3 Analytic Transparency: Researchers making evidence-based knowledge claims should provide a full account of how they draw their analytic conclusions from the data, i.e., clearly explicate the links connecting data to conclusions.

Political Science is leader in adopting research transparency as a standard

Transparency as a Meta-Standard

A broader goal

Transparency as a Meta-Standard

• We approach transparency as an “upstream” meta-standard, which applies regardless of the type of social inquiry being conducted.

• Distinguish functional approach, where particular epistemic requirements recommend a particular fix. All part of the same conversation.

Transparency as a Meta-Standard

• All social inquiry is process-dependent. Conduct impacts the quality of results.

• Transparency is a core constitutive element of social science.

For all rule-based social inquiry

• Scholarly communities hold shared and stable beliefs that research designed and conducted in particular ways possesses certain characteristics.

• The conduct of social inquiry and the written products that represent its conclusions are designed to capture those characteristics.

• Openness corollary: For any given piece of research in a particular tradition, the ability of scholars to claim the underlying warrants depends on their showing that it was designed and conducted in accordance with those rules.

Think Globally, Act Locally

• The meta-standard is a general obligation, not a norm. It applies wherever scholars use a shared logic of inquiry to reach evidence-based conclusions.

• The justification for openness (establishing a knowledge claim’s legitimacy) and its structure (showing both evidence and analysis) apply universally.

• DA-RT recognizes that the optimal means of achieving openness respect the challenges and opportunities that characterize various research traditions.

Goals

• Pursue points of agreement on key issues, including:

– Data citation practices

– Data access requirements (timing, embargoes, exemptions)

– Research transparency requirements (e.g. sharing code, analysis plans)

• Find ways to provide support and create incentives for effective implementation.

Workshop

• Now: General introduction

• This morning: 5 presentation topics

• This afternoon, tomorrow morning: Breakouts followed by consolidation sessions

• Conclusion

Presentation Topics

1. Persistent identifiers, trusted repositories, and better citation practices;

2. Pre-registration of research designs;

3. Integrating transparency standards into journal workflow

4. Active citation

5. The Center for Open Science’s “badges” initiative.

First Breakouts

• The purposes for which data are shared (for replication, secondary analysis, and/or pedagogical purposes), and what authors need to provide to satisfy those purposes.

• The timing of sharing data and supporting materials. Possible triggers include manuscript submission, requests by reviewers, electronic publication, paper publication, and within a specified time after publication. Should reviewers have access to data?

• Satisfying openness standards when the data authors use are under ethical or proprietary constraint.

Second Workshop Breakouts

• Providing incentives for data sharing, including insisting on appropriate citation and increasing disciplinary credit for data sharing as a value-adding practice.

• Ensuring that the discipline establishes adequate training facilities to prepare scholars to engage in data management.

• Addressing concerns of authors regarding the potential implications of data transparency for their work, including the issues of confidentiality and data access.

Organization of the Breakouts

• Three parallel sessions, small group discussions.

• Survey responses used to form groups.

• Each group should discuss questions and exchange information. Interested to gauge difference/convergence.

Instructions• Bring to consolidation meetings a brief description of

convergence, concerns, proposals.

• If a consensus emerges, please provide a short (2 or 3 para) text describing the areas of agreement.

• If there are areas of disagreement, please indicate what those were.

• The purpose of the consolidation meetings is to bring broader knowledge to debates and provide greater support for consensus ideas.

Thank you for participating!