The August 2016 newsletter of the African Politics Conference Group includes a series of contributions to the DA-RT debate, with a focus on Africanist studies. Access the full newsletter here.
Conversations and debates on issues of data access and research transparency will continue at this week’s APSA convention at various panels and roundtables, some filled with JETS advocates, others with critics, others with both. These conversations may affect decisions regarding journals’ policies. We urge you to participate in these discussions.
We append a list of events below, with the list of panelists and excerpts from panel descriptions (from the APSA online program).
APSA 2016 Panels and Roundtables
Data Access, Transparency, and Replication in Political Science
Thu, September 1, 8:00 to 9:30am, PCC, 114
Panelists will discuss emerging norms, standards, and ongoing concerns in how the discipline handles data access, transparency, and replication.
Chair William G. Jacoby, Michigan State University
Raymond Duch, CESS Nuffield
Nicole Janz, University of Nottingham
Arthur Lupia, University of Michigan, Ann Arbor
Neil Malhotra, Stanford University
Colin Elman, Syracuse University
The Qualitative Transparency Deliberations: An Interim Report
Thu, September 1, 10:00 to 11:30am, PCC, 204-C
This roundtable will provide an interim report and discussion on the unfolding Qualitative Transparency Deliberations (QTD) sponsored by the APSA’s Organized Section on Qualitative and Multi-Method Research. Since April 2016, the QTD has involved both far-reaching consultation and intensive deliberation about the benefits, costs, and practicalities of achieving transparency for diverse forms of qualitative empirical research, including process tracing, interpretive inquiry, comparative-historical research, and ethnography. On this panel, members of the QTD Steering Committee will discuss the structure and nature of the deliberative process and the key issues on which the QTD working groups have been focusing their attention, and will provide a preview of some of the approaches to transparency and recommendations emerging from the working-group discussions.
Chair Kimberly J. Morgan, George Washington University
Tim Buthe, Duke University
Alan M. Jacobs, University of British Columbia
Kimberley S. Johnson, Barnard College
Sarah E. Parkinson, University of Minnesota
Andrew Bennett, Georgetown University
DA-RT and Qualitative Methods: The Case of Process Tracing
Thu, September 1, 4:00 to 5:30pm, PCC, 204-B
The purpose of this roundtable is to stimulate a focused conversation on what DA-RT might look like in a key part of the qualitative toolkit – process tracing.
It features participants all of whom have made significant contributions to the theory and practice of process tracing. Coming from different epistemological and substantive perspectives, they will address a set of questions surrounding DA-RT.
1) Are common community standards of DA-RT/process-tracing possible and desirable?
2) At a practical/operational level, what does DA-RT look like when applied to process tracing?
3) Given that process tracing in many contexts raises foundational ethical/human-subject concerns, how can these be reconciled with demands for greater data access?
4) What are the trade-offs involved – for graduate training; for theory development; for submissions to leading journals – in implementing greater levels of transparency?
5) Given that process tracing is already a time and resource intensive endeavor, how can we raise transparency standards without simultaneously creating strong disincentives for researchers (especially younger scholars) to use the method?
6) Can formalization – through Bayesian analysis, say – enhance the transparency of process tracing? If so, are there trade-offs to consider?
The roundtable’s goal is not consensus. Rather, we hope to stimulate a discussion that gives us a better net assessment of the pluses and minuses of implementing DA-RT.
Chair Jeffrey T. Checkel, Simon Fraser University
Andrew Bennett, Georgetown University
James Mahoney, Northwestern University
Tasha A. Fairfield, London School of Economics
Regina A. Bateson, MIT
Peter A. Hall, Harvard University
Elisabeth Jean Wood, Yale University
Roundtable on the CPS Special Issue on Transparency in the Social Sciences
Fri, September 2, 2:00 to 3:30pm, Marriott, Room 406
…As the first journal issue in political science based on results-blind peer review, the CPS Special Issue on Transparency in the Social Sciences offers a unique opportunity to critically reflect on these questions. The CPS Special Issue includes three articles that were accepted on either the basis of a prospective research design (i.e. a pre-analysis plan) or on a results-blind basis, with any discussion of the results stripped out of the manuscript. Authors of articles included in the Special Issue have also committed to make available all background information, including opening up field notes and last-minute alterations to the research design.
This roundtable brings together the participants in the Special Issue, including the Special Issue editors, CPS editors, and article authors, along with other leaders in the movement for greater transparency in the social sciences, to reflect on and debate the tradeoffs of journal publication via a results-blind process.
Chair Lauren Prather, University of California, San Diego
Aaron S. Erlich, McGill University
- Daniel Hidalgo, Massachusetts Institute of Technology
Dominika Kruszewska, Harvard University
Arthur Lupia, University of Michigan, Ann Arbor
Thomas Pepinsky, Cornell University
David J. Samuels, University of Minnesota, Twin Cities
Yael Zeira, University of Mississippi
New Tools and Standards for Research Transparency
Sat, September 3, 10:00 to 11:30am, Marriott, Franklin 11
The transparency revolution underway across many of the social sciences promises to improve qualitative, quantitative, and experimental research. This push is motivated in part by concerns around high levels of researcher discretion and perverse career incentives to publish striking and novel results. Deliberate or inadvertent adjustment of analysis strategies ex-post can give rise to spurious results. Despite intense interest in research transparency, the gains from this movement are not yet fully realized, due in part to the lack of shared standards and practices. This panel comprises four concrete proposals for improving ex-ante and ex-post transparency that cut across research traditions and empirical subfields.
Lin and Green address the problem that even detailed research plans inevitably fail to address unforeseen complications. Results are often heavily dependent on the choices researchers make in these circumstances. Lin and Green propose that researchers develop standard operating procedures in order to reduce researcher discretion.
Elman, Kapiszewski, and Lupia provide criteria that researchers can use to demonstrate that they followed the rules of their respective logics of inquiry. Importantly, they clarify the distinction between transparency at the production stage and at the analytic stage.
Transparency standards are particularly underdeveloped for computer-assisted text analysis methods, a relatively new area of inquiry. Romney, Stewart, and Tingley outline a new set of tools that will increase the confidence in and reproducibility of text analysis from data collection to statistical analysis.
Blair, Cooper, Coppock, and Humphreys build a framework for completely characterizing the formal features of a research design: the population, sampling strategy, potential outcomes, assignment mechanism, estimand, and estimation strategy. The tools, including software, they propose can be used to specify, improve, and pre-register designs.
Standard Operating Procedures: A Safety Net for Pre-Analysis Plans – Donald P. Green, Columbia University; Winston Lin, Columbia University
Transparent Social Inquiry and the Meaning of Political Science – Colin Elman, Syracuse University; Diana Kapiszewski, Georgetown University; Arthur Lupia, University of Michigan, Ann Arbor
Plain Text? Transparency in Computer-Assisted Text Analysis – David Alexander Romney, Harvard University; Brandon Michael Stewart, Harvard University; Dustin Halliday Tingley, Harvard University
Formally Characterizing Research Designs to Encourage Diagnosis and Registration – Graeme Blair, UCLA; Jasper Jack Cooper; Alexander Coppock, Yale University; Macartan Humphreys, Columbia University
Chair Andrew Bennett, Georgetown University
Andrew Bennett, Georgetown University
Justin Grimmer, Stanford University
Who Publishes, Who Perishes? Challenges in Research with Vulnerable Populations
Sat, September 3, 4:00 to 5:30pm, Loews, Commonwealth C
Scholars across the various fields of political science and those involved in interdisciplinary research with colleagues in anthropology, history, sociology, social work, and related disciplines here and abroad face a range of ethical constraints when studying highly vulnerable individuals and groups.
Scholars who conduct research with highly vulnerable populations are bound to safeguard the integrity of their data in order to ensure the wellbeing of people who could be targeted for reprisals if their identity were revealed. Yet these same scholars are faced with the prospect of making interview and focus group transcripts, field notes and other research data accessible and/or more public and transparent in order to fulfill the DA-RT requirements and to disseminate their work through the publication of scholarly articles (a particular concern for junior scholars).
This roundtable brings together scholars to discuss the ethical implications of this new research and policy landscape. We will pay particular attention to the potential effects these twin pressures may have on scholars involved in interdisciplinary work and the gendered and/or racialized implications of this controversy for scholars themselves as well as the people whose reality we study at the grassroots level.
Chair: Shareen Hertel (University of Connecticut)
Elisabeth Wood (Yale University)
Lisa Wedeen (U-Chicago)
Jane Mansbridge (Harvard JFK School)
Sarah Parkinson (U-Minnesota Humphrey School)
Deborah Yashar (Princeton University)
Structures of Political Science Revolutions: DA-RT and Audit Cultures
Sunday, September 4, 8-9.30 a.m., Loews, Commonwealth A2
Roundtable participants’ shared concern for the future of political scientific practices leads them to consider the following sorts of questions. What is DA-RT’s transformational potential vis à vis institutional forms and strategic relationships? What is the political science future that its proponents envision? Instead of uniting the discipline behind (or under) a single procedural banner, does DA-RT have the potential to split the discipline into, say, “DA-RT journals” versus “traditional journals”; or to split the Association itself because of concerns over how it has been introduced (neither of these notions so farfetched in light of the UK’s experience; see Shore and Wright 2000)? Is the current framing of DA-RT a part of the neoliberal reforms ongoing in higher education worldwide (journal impact scores, measures of faculty “productivity” [e.g., the US’s Academic Analytics, the UK’s Research Excellence Framework]), replacing discussions of quality with depoliticized systems to monitor it (Pels 2000:135)? And what about its relationship to philosophy of science debates concerning the unity of science or to the actual status of replication in natural science, a status presumed by policy proponents?
Roundtable participants will speak to these and other questions. In keeping with the minutes from both 2012 and subsequent Executive Council meetings calling for discussion of DA-RT with the membership and calls across the discipline to have the conference meetings this year enable further, explicit discussion of the framing and desirability of such research and editorial policies, the roundtable is intended to develop a conversation that opens up new issues for exploration.
Chair: Dvora Yanow (Wageningen/Kate Hamburger Kolleg, Duisburg)
Mary Hawkesworth (Rutgers)
Ido Oren (Florida)
Timothy Pachirat (UMass Amherst)
Peregrine Schwartz-Shea (Utah)
Featuring commentary from Karen Alter, Giovanni Capoccia, Eric Grynaviski, Jeffrey Isaac, Andrew
Moravcsik, James A. Morrison, and Jelena Subotic, among others. You can find the complete newsletter here.
Starting immediately, AJPS will award “open data” and “open materials” badges to articles that make publicly available “the digitally-shareable data necessary to reproduce the reported results” and “the components of the research methodology needed to reproduce the reported procedure and analysis,” respectively. These are two of the “Badges to Acknowledge Open Practices” from the Center for Open Science (COS). See announcement here.
In a post on his blog, Tom Pepinsky raises questions about DA-RT and expresses concerns that the QTD process is reifying the dichotomy between “quantitative” and “qualitative” research. Andy Moravcsik writes a long response affirming that active citation right now amounts to the “only viable default approach for qualitative work.” Active citation, he claims, is a very conservative “back to the future” proposal” that would make journal articles in qualitative political science resemble articles of three decades ago, with “discursive footnotes, textual references, longer word-limits, and interpretive analysis.”
From ASN website:
The Association for the Study of Nationalities (ASN) raises concerns regarding the Data Access and ResearchTransperancy Initiative originating within the American Political Science Association (APSA).
The Association for the Study of Nationalities (ASN) has recently learned of the DA-RT discussions originating within the American Political Science Association (APSA). We are concerned that a set of stipulations regarding research signed by 27 journals (JETS) was enacted on January 15th 2016 when so few of those who will be affected by these stipulations are aware of their requirements. We share the concerns outlined by the 1,173 petition signatories to delay DA-RT implementation to more fully consider its implications for a broad range of research.
The DA-RT stipulations establish conditions in which editors and reviewers can request that materials produced in the course of research should be made available on a public domain website. These materials can include databases but also interview transcripts, recordings, and fieldnotes. We understand that exemptions can be requested from these requirements. However, we remain concerned that there is not sufficient clarity in the policies as outlined to protect from the potential harms to sources and researchers in the conduct of research, as well as to long-term implications of harm for the conduct of research on sensitive topics. Nor do we find IRB / ethics board statements to be sufficient protection, given the diversity of access to such protections for our members across the globe. We note many of the concerns DA-RT raises for research on violence and autocratic regimes as well as those using qualitative approaches, as outlined in the APSA Qualitative and Multi-Methods Research newsletter in 2015 as well as the APSA Comparative Politics Newsletter in Spring 2016. As researchers on sensitive topics, we share many of these concerns in our own work. We are particularly disturbed that the serious problems outlined by researchers in these forums have not yet produced changes to the policy appropriate to the potential harms.
We urgently request a re-thinking of the DA-RT stipulations to incorporate better policies to address concerns of researchers working on sensitive topics, including those using qualitative and historical research.
The ASN Board of Directors and the Advisory Board