Success stories of AI models are plentiful, but we have also seen prominent examples where the models behave in unexpected ways. For example, a typical failure mode of state-of-the-art prediction models is the inability to abstain from making predictions when the test data violate assumptions made during training, potentially resulting in highly confident but incorrect predictions. Hence, there is a renewed interest in improving the reliability and transparency of AI models.

Uncertainty quantification provides a vital diagnostic to access the reliability of an AI system. For developers, it can suggest strategies for improving the system. For example, high data uncertainty may point towards improving the data representation process, while a high model uncertainty may suggest the need to collect more data. For users, accurate uncertainties, especially when combined with effective communication strategies, can add a critical layer of transparency and trust, crucial for better AI-assisted decision making. Trust in AI systems is a necessary condition for their successful deployment in high-stakes applications spanning health care, finance, and the social sciences.

The research on uncertainty quantification (UQ) is long-standing but has enjoyed ever increasing interest in recent years due to the observations made above. Numerous approaches for improved UQ in AI models have been proposed. However, choosing a particular UQ method depends on many factors: the underlying model, type of machine learning task, characteristics of the data, and the user's goal. If inappropriately used, a particular UQ method may produce poor uncertainty estimates and mislead users. Moreover, even a highly accurate uncertainty estimate may be misleading if poorly communicated. The main goals of this tutorial are:

• to introduce a diverse set of algorithms to quantify uncertainties, metrics to measure them, methods to improve the quality of estimated uncertainties, and approaches to communicate the uncertainties effectively;
• to discuss a taxonomy and guidance for choosing these capabilities based on the user's needs; and
• to encourage further exploration of connections between UQ and other pillars of trustworthy AI such as fairness and transparency.