Challenges & Solutions

Changes to procedure during study administration

• Deviations are common.
• Unexpected things can always happen during data collection (e.g. participants might behave drastically different than expected, making changes in your plans inevitable)
• This does not make your preregistration obsolete!

Solution

• Document changes to preregistry. Transparantly report where and why changes were made
• Other researchers will benefit from described changes
  

Discovery of assumption violations during analysis

• Your data might be differently distributed than expected; you might need to change analysis

Solution

• Preregister incrementally for pre-defined stages (e.g. first preregister evaluation of distributional forms, then preregister model, ...).
• OR Blind dataset (i.e. scramble outcomes) until all assumptions are tested.
• OR Preregister decision-trees (e.g. specify testing a normality assumption and, depending on outcome, selection of test A or B).
• OR Design a standard operating procedure (SOP) = describe decision rules for handling data, include more general rules than decision trees.
  

Data are pre-existing

• You want to use data collected by other researchers?
• You will not be able to prove that you did not see data before formulation of hypotheses

Solution

• Register analysis plans and transparently provide insight about potential biases and unfluences (What was and what was not known before?)

Longitudinal studies and large, multivariate datasets

• Project often involve large, multivariate datasets = impossible to preregister all design of all studies resulting from project

Solution

• Solutions to the first challenges apply here (see above).
• Incremental Preregistration.

Many experiments

• You conduct many experiments with similar designs = data acquisition is so simple that any effort that would go into a preregistration would be considered inefficient

Solution

• Establish preregistration templates for defining variables and parameters for the protocol

A program of research (i.e. multiple tests)

• If an area of research is high risk, most research outcomes are null results, and the infrequent positive results have huge implications

Solution

• Preregistration enables a transparent assessment of multiple tests only if preregistration and results of analysis plans are permanently preserved and accessible for review
  

Few a priori expectations ("Discovery Research")

• It seems useless to preregister when you don't have hypotheses

Solution

• Register a first study with a simple analysis plan for exploratory analysis with the goal of generating hypotheses.
• This forms the basis for a subsequent preregistration for a second study to test the generated hypotheses.
• OR preregister a cross-validation process (i.e. dataset it split in two; one part is used for exploratory analysis, the other part remains sealed until exploration is complete)

Competing predictions

• This is not really a challenge. Given a certain study design and analysis plan, you can have multiple hypotheses and if you have preregistered those, the data will provide trustworthy support for one or another. 

Solution

• Preregister multiple predictions simultaneously

Narrative inferences and conclusions (i.e. I want to stress the interesting results in the presentation of the paper)

• To honor the ultimate goal of preregistration and open science, it is also important to keep it in mind when writing up your study conclusions. Even if a researcher preregisters and reports the result according to the predictions, they can still have exploratory extra-results in the paper and it might be tempting to focus on surprising, interesting results in the discussions & conclusions section.
• Preregistration does not dictate the narrative design of a paper (Interpretation and narrative conclusions are an important stimulus in the development of theory!)
• Preregistration merely facilitates transparency in interpretations (I as a researcher display one opinion, other views/interpretations can be applied)

For more information: https://osf.io/2dxu5/