A significant challenge across scientific fields is the reproducibility of research results, and third-party assessment of such reproducibility. Ensuring that results can be reliably reproduced is no small task: computational environments may vary drastically and can change over time, rendering code unable to run; specialized workflows might require specialized infrastructure not easily available; sensitive projects might involve data that cannot be directly shared; the robustness of algorithmic decisions and parameter selections varies widely; data collection methods may include crucial steps (e.g. wrangling, cleaning, missingness mitigation strategies, preprocessing) where choices are made but not well-documented. Yet a cornerstone of science remains the ability to verify and validate research findings, so it is important to find ways to overcome these challenges.
The first MIDAS Reproducibility Challenge was held in the first 8 months of 2020. Our goal was to highlight high-quality, reproducible work at the University of Michigan by collecting examples of best practices across diverse fields. Besides incentivizing reproducible workflows and enabling a deeper understanding of issues of reproducibility, the results of the challenge also provide templates that others can follow.
Judges
- Jake Carlson: Manager, Deep Blue Repositories and Research Data Services, U-M Libraries
- H.V. Jagadish: Director, MIDAS, and Professor, Computer Science and Engineering, CoE
- Matthew Kay: Assistant Professor, School of Information
- Jing Liu: Managing Director, MIDAS
- Josh Pasek: Assistant Professor, Communication and Media, LSA
- Brian Puchala: Assistant Research Scientist, Materials Science and Engineering, CoE
- Arvind Rao: Associate Professor, Computational Medicine and Bioinformatics, and Radiation Oncology, Med. School
