Statistical evaluation of the performance of machine learning models

Richard Dinga

Machine learning model validation is essential because any methodological progress cannot be trusted if the results are not properly evaluated using valid and reliable methods. Unfortunately, incorrect methods are common in scientific literature and business, leading to suboptimal model selection and incorrect conclusions. This tutorial will provide a practical guide to a wide range of everyday validation tasks. Namely statistically testing if a model performs better than chance, comparing the performance of two or more models, selecting performance measures with respect to their statistical properties such as reproducibility and statistical power, and model validation in the presence of confounds, validation of feature effects and validation of clustering methods, all with a strong focus on valid statistical inference. Participants will learn how to apply correct statistical tests in various situations.