Preregistration in Practice: General Challenges
CHALLENGES (based on https://osf.io/2dxu5/)
A) Changes to procedure during study administration
-> Deviations are common.
-> Unexpected things can always happen during data collection (e.g. participants might behave drastically different than expected, making
changes in your plans inevitable)
--> This does not make your preregistration obsolete!
SOLUTION
-> Document changes to preregistry. Transparantly report where and why changes were made
-> Other researchers will benefit from described changes
B) Discovery of assumption violations during analysis
-> Your data might be differently distributed than expected; you might need to change analysis
SOLUTION
-> Preregister incrementally for pre-defined stages (e.g. first
preregister evaluation of distributional forms, then preregister model,
...).
-> OR Blind dataset (i.e. scramble outcomes) until all assumptions are tested.
->
OR Preregister decision-trees (e.g. specify testing a normality
assumption and, depending on outcome, selection of test A or B).
->
OR Design a standard operating procedure (SOP) = describe decision
rules for handling data, include more general rules than decision trees.
C) Data are pre-existing
->
You want to use data collected by other researchers?
-> You will not be able to prove that you did not see data before formulation of hypotheses
SOLUTION:
-> Register analysis plans and transparently provide insight about
potential biases and unfluences (What was and what was not known
before?)
D) Longitudinal studies and large, multivariate datasets
-> Project often involve large, multivariate datasets = impossible to preregister all design of all studies resulting from project
SOLUTION
-> Solutions to the first challenges apply here (see above).
-> Incremental Preregistration.
E) Many experiments
-> You conduct many experiments with similar
designs = data acquisition is so simple that any
effort that would go into a preregistration would be considered
inefficient
SOLUTION
->Establish preregistration templates for defining variables and parameters for the protocol
F) A program of research (i.e. multiple tests)
->
If an
area of research is high risk, most research outcomes are null results, and the
infrequent positive results have huge implications
SOLUTION
-> Preregistration enables a transparent assessment of multiple tests only
if preregistration and results of analysis plans are permanently
preserved and accessible for review
G) Few a priori expectations ("Discovery Research")
-> It seems useless to preregister when you don't have hypotheses
SOLUTION
-> Register a first study with a simple analysis plan for exploratory analysis with the goal of generating hypotheses.
-> This forms the basis for a subsequent preregistration for a second study to test the generated hypotheses.
->
OR preregister a cross-validation process (i.e. dataset it split in
two; one part is used for exploratory analysis, the other part remains
sealed until exploration is complete)
H) Competing predictions
-> This
is not really a challenge. Given a certain study design and analysis plan, you
can have multiple hypotheses and if you have preregistered those, the data will
provide trustworthy support for one or another.
SOLUTION
-> Pregerister multiple predictions simultaneously
I) Narrative inferences and conclusions (i.e. I want to stress the interesting results in the presentation of the paper)
->
To
honor the ultimate goal of preregistration and open science, it is also
important to keep it in mind when writing up your study conclusions. Even if a
researcher preregisters and reports the result according to the predictions,
they can still have exploratory extra-results in the paper and it might be
tempting to focus on surprising, interesting results in the discussions &
conclusions section.
-> Preregistration does not dictate the narrative design of a paper
(Interpretation and narrative conclusions are an important stimulus in
the development of theory!)
->
Preregistration merely facilitates transparency in interpretations (I
as a researcher display one opinion, other views/interpretations can be
applied)
This is a summary of the article "The preregistration revolution" (Nosek, Ebersole, DeHaven, & Mellor, (2017).
For more information: