Reproducibility Challenge

2021 Reproducibility Challenge

The MIDAS Reproducibility Challenge II is open to researchers from any field that makes use of data, broadly construed. We seek entries in four categories:

Training materials for reproducible research. These should be well developed materials for courses or workshops, with a clear theme that can be easily adopted by other instructors.

Guides, processes and tools that can be readily adopted by researchers in one or multiple research areas. Other researchers, even if only within a narrowly defined research field, should be able to follow such guides, processes and tools to make their work reproducible.

A template for a reproducible project. The goal is to allow others to emulate and develop similar approaches and lower the burden of making their projects reproducible. We especially welcome submissions that seek to make analyses not only reproducible, but transparent. This might include especially well organized or well documented code (for example, using notebooks); saving intermediate datasets / results so that they can be easily inspected; code that draws attention to and allows easy modification of analytical choices such as statistical tuning parameters, data cleaning choices, etc.

An analysis of reproducibility in a research field, including meta-analysis or reproducing published works, to identify good practices for and challenges to replication and confirmation of published works, and proposals for new practices and tools to overcome barriers. A research field needs to be well defined, but can be broad or narrow, e.g. “single-cell RNAseq”, or “child development research with video data”.

A significant challenge across scientific fields is the reproducibility of research results, in both the narrow sense of repeating the calculations using the same data, and in the broader sense of deriving generalized insight from one or more investigations that bear on a common question. Ensuring that data science research results can be reliably reproduced is particularly challenging, especially as data science projects involve increasingly complex data and pipelines. In addition, when data science methodology is applied to vastly different research domains, the reproducibility of results hinges on domain-specific factors as well as components of the data science methodology.  

The 2021 MIDAS Reproducibility Challenge winners bring to us principled yet creative work. Krüger and Nordås provide a template for a principled approach to organize, document and share a computational project. Teplitskiy and Evans use an automated and rigorous method to assess the reproducibility and replicability of survey data research and delineate causes for the lack of replicability.. Zhao and Wen propose a Bayesian model criticism approach to assess the replicability of a body of research studies that address the same scientific question with variations in data sets and analytical methods. 

Through the 2020 and 2021 Reproducibility Challenges, MIDAS seeks exemplary work from our researchers to make data science projects more reproducible, and research outcomes more robust. We now have a wonderful collection of our researchers’ work to improve reproducibility in data science, and we are building collaboration with these researchers to enable the wide adoption of best practices. 

Each of the winning teams will receive a cash prize. Later in the semester, they will present their work through a series of video showcases. Their projects will also be added to the MIDAS Reproducibility Hub. 

Congratulations again to the winners.

MIDAS Reproducibility Challenge Winners

March 3rd, 2022 – MIDAS is excited to announce the 2021 Reproducibility Challenge winners and congratulates them for their outstanding work to make data science research more reproducible.

Planning Committee