Multiverse Analysis
Garden of Forking Paths
All data analysis projects must choose between different ways to analyze data. This is known as the garden of forking paths:
Garden of Forking Data
In practice, this means we must choose between many possible versions of our dataset. Every data processing step leads to a different final dataset.
Enter the multiverse:
Why Care About the Multiverse?
Ignoring the multiverse can lead to undesirable actions/outcomes:
Cherry Picking
Pigeonholes
Why Care About the Multiverse?
Ignoring the multiverse can lead to undesirable actions/outcomes:
Cherry Picking
Selectively reporting the analysis that shows your preferred result
Pigeonholes
Becoming constrained by overly rigid analysis criteria
Multiverse Analysis
- Transparently and systematically analyze the whole multiverse
- Transparency reduces cherry-picking
- Systematically handling decisions reduces pigeonholing
