On assessing calibration of multivariate ensemble forecasts

A necessary condition for optimal use in decision making of probability forecasts in general, and ensemble forecasts in particular, is that they should be calibrated. Ensemble calibration implies that each observation is statistically indistinguishable from the members of its forecast ensemble. The dimensionality of multivariate forecast settings, where the observation is a vector of related quantities rather than a scalar, affords opportunities for a rich variety of types of ensemble miscalibration, and so requires more sophisticated metrics for detection and diagnosis. This article investigates the performance in an artificial data setting of five multivariate ensemble calibration metrics that have been proposed in the literature. It is found that none of these dominates the others for detecting covariance miscalibration across a variety of miscalibration types, and that each exhibits weak performance in some instances, implying that several of the metrics should be used simultaneously when multivariate ensemble miscalibration is to be investigated. Multivariate biases were detected more strongly and diagnostically using univariate rank histograms for the individual forecast vector components.