Activities

Launched in 2018: Structured challenge submission tool for better interpretability and reproducibility of MICCAI challenge results and for improving quality control of challenge proposals during a fully web-based review process.

Published to arXiv in 2019: Paper on how to publish a biomedical image analysis challenge [6].

Released in 2019: Visualization toolkit for benchmarking results [7].

MICCAI registered challenges

Similar to how clinical trials have to be registered before starting, the complete design of accepted MICCAI challenges will be put online before the challenges take place. Changes to the design (e.g. to the metrics or ranking schemes applied) must be well-justified and officially be registered online (as a new version of the challenge design). Registering challenges is a big step towards higher quality challenges. It not only has the potential to lead to more thoughtful challenge designs but also provides all the information necessary for challenge participants. Furthermore, all changes will be transparent to the community, ensuring increased quality control. Below, the registered challenges are listed.

MICCAI 2020

MICCAI 2021

MICCAI 2022

Organizing a challenge based on an accepted challenge proposal may be time-consuming, especially when large-scale data annotation is necessary. To address the traditionally tight schedule between MICCAI challenge acceptance and organization, we offered an early review of MICCAI 2022 challenge proposals in the call for MICCAI 2021 challenges. Organizers interested to organize a challenge in 2022 were able to receive an early acceptance that leaves time for careful preparation of the challenge. Only convincing challenge proposals were accepted in this round. Other challenges were asked to resubmit in the next call for challenges.

MICCAI endorsed events

The number of submitted challenge proposals is steadily increasing. Due to limited room capacities, not every interesting challenge can be accepted for an ongoing MICCAI event. Therefore, we decided to accept challenges as MICCAI endorsed online-only events, which are not bounded to a specific conference, but organized under the MICCAI umbrella.

Further reading:

1. Nichols, T. E. et al. (2017). Best practices in data analysis and sharing in neuroimaging using MRI. Nature Neuroscience, 20(3), 299. doi: 10.1038/nn.4500.
2. Maier-Hein et al. (2018). Why rankings of biomedical image analysis competitions should be interpreted with care. Nature Communications 9: 5217. doi: 10.1038/s41467-018-07619-7
3. Reinke et al. (2018, September). How to Exploit Weaknesses in Biomedical Challenge Design and Organization. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 388-395). Springer, Cham. doi: 10.1007/978-3-030-00937-3_45. 
4. Grand-Challenges. (2012) at https://grand-challenge.org/
5. Mendrik, A.M. and Aylward, S.R. (2018, December). Beyond the Leaderboard: Insight and Deployment Challenges to Address Research Problems. NIPS 2018 Challenges in Machine Learning (CiML) Workshop. https://arxiv.org/abs/1811.03014/
6. Maier-Hein et al. (2020). BIAS: Transparent reporting of biomedical image analysis challenges. Medical Image Analysis, 101796. doi: 10.1016/j.media.2020.101796
7. Wiesenfarth, M. et al. (2021). Methods and open-source toolkit for analyzing and visualizing challenge results. Scientific Reports 11, 2369. doi: 10.1038/s41598-021-82017-6

Acknowledgements

Many thanks to Annika Reinke, Sinan Onogur and Matthias Eisenmann (Div. Computer Assisted Medical Interventions, DKFZ Heidelberg) for implementing the structured challenge submission system.