Activities

Launched in 2018: Structured challenge submission tool for better interpretability and reproducibility of MICCAI challenge results and for improving quality control of challenge proposals during a fully web-based review process.

Published to arXiv in 2019: Paper on how to publish a biomedical image analysis challenge [6].

Released in 2019: Visualization toolkit for benchmarking results [7].

MICCAI registered challenges

Similar to how clinical trials have to be registered before starting, the complete design of accepted MICCAI challenges will be put online before the challenges take place. Changes to the design (e.g. to the metrics or ranking schemes applied) must be well-justified and officially be registered online (as a new version of the challenge design). Registering challenges is a big step towards higher quality challenges. It not only has the potential to lead to more thoughtful challenge designs but also provides all the information necessary for challenge participants. Furthermore, all changes will be transparent to the community, ensuring increased quality control. Below, the registered challenges are listed.

MICCAI 2020

MICCAI 2021

Organizing a challenge based on an accepted challenge proposal may be time-consuming, especially when large-scale data annotation is necessary. To address the traditionally tight schedule between MICCAI challenge acceptance and organization, we offered an early review of MICCAI 2021 challenge proposals in the call for MICCAI 2020 challenges. Organizers interested to organize a challenge in 2021 were able to receive an early acceptance that leaves time for careful preparation of the challenge. Only convincing challenge proposals were accepted in this round. Other challenges were asked to resubmit in the next call for challenges.

Further reading:

1. Nichols, T. E. et al. (2017). Best practices in data analysis and sharing in neuroimaging using MRI. Nature Neuroscience, 20(3), 299. doi: 10.1038/nn.4500.
2. Maier-Hein et al. (2018). Why rankings of biomedical image analysis competitions should be interpreted with care. Nature Communications 9: 5217. doi: 10.1038/s41467-018-07619-7
3. Reinke et al. (2018, September). How to Exploit Weaknesses in Biomedical Challenge Design and Organization. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 388-395). Springer, Cham. doi: 10.1007/978-3-030-00937-3_45. 
4. Grand-Challenges. (2012) at https://grand-challenge.org/
5. Mendrik, A.M. and Aylward, S.R. (2018, December). Beyond the Leaderboard: Insight and Deployment Challenges to Address Research Problems. NIPS 2018 Challenges in Machine Learning (CiML) Workshop. https://arxiv.org/abs/1811.03014/
6. Maier-Hein et al. (2020). BIAS: Transparent reporting of biomedical image analysis challenges. Medical Image Analysis, 101796. doi: 10.1016/j.media.2020.101796
7. Wiesenfarth et al. (2019). Methods and open-source toolkit for analyzing and visualizing challenge results. https://arxiv.org/abs/1910.05121/

Acknowledgements

Many thanks to Annika Reinke, Sinan Onogur and Matthias Eisenmann (Div. Computer Assisted Medical Interventions, DKFZ Heidelberg) for implementing the structured challenge submission system.