In a post, AJPS affirms that, having already “expressed a full commitment to the general principles of data access and research transparency….the Journal will not compromise this position in any way,” but also that “we currently are revising the replication policy, to incorporate guidelines for information drawn from qualitative research.”
ISSUES IN THE PRODUCTION OF NEW DATABASES UNDER DA-RT
I’m a general support of the DA-RT idea, but I’m not yet convinced that the downstream effects have been completely analyzed.
Some of my grad students have expressed dismay that they will have to release files on new databases they have created and have coded using the policy agendas system.
So I wonder if the DA-RT initiative may inadvertently create a potentially serious incentive problem here: undermining the production of collective goods (new high-quality databases) because of premature release. I’m not sure temporary embargoes would solve this.
One of the students raising this issue has coded every instance of how provisions in laws end their lives (laws are almost never repealed in toto). This involved analysis of some 300,000 provisions in the US Code, all coded by the policy agendas data system, since 1789. You can imagine what could happen if one journal published the major findings and required release. Right now, this could be negotiated, I guess (anyway that is what I told her). Her findings are dynamite: both politics and information matter: the size and diversity of the enacting coalition (politics) AND whether the law is concentrated in one policy agendas major topic, the number of committees holding hearings on the provision, and whether it was considered with an open rule. So one can imagine the potential demand for this new database.
DA-RT has a “first use” principle, allowing for delay of up to one year (which may be modified by the journal in which the publication is to occur). This certainly could alleviate the pressure, but I am not sure it solves it. DA-RT also requires only analysis decks to be produced. But in the example above, this second provision probably would not help.
The DA-RT system may well work well for much of what we do: experiments, drawing from various archived data sources, maybe even data from new surveys. Issues of qualitative research have already been raised. But DA-RT may undermine our discipline’s infrastructure development in unintended ways. I’m hoping we can think this through.
“All of these people trusted me with personal stories…. Deserving this trust called for great sympathy, sensitivity and – above all – going the extra mile to protect the speaker’s integrity and privacy.”
In this short post, Kristin Monroe argues that data access requirements must be flexible: ethical and privacy concerns may demand modification of testimony or destruction of data.
At 13.01 ET on Tuesday Nov. 10, 1000 people had signed petition to journal editors requesting a delay in implementation of DA-RT.
Andrew Schrank’s short essay says DA-RT advocates: 1) need to be more clear about what counts as “cited data” (“cleaned and coded survey data or field notes? Actual survey or interview transcripts? Audio or video tapes of the surveys or interviews themselves? Or something else entirely?”); and 2) do a better job defending the value of replication, which really should mean “the re-collection as well as the re-analysis of data,” for there are other means of insuring validity and social control in science.
As of 9 am ET on November 10, some 980 people had signed the petition to delay implementation of DA-RT.
SPPQ editors write about their decision to adopt DA-RT–“simply a codification of…[our] longstanding replication policies”–and why they won’t agree to a delay, here.