A data quality approach to the identification of discrimination risk in automated decision making systems

Automated decision-making (ADM) systems may affect multiple aspects of our lives. In particular, they can result in systematic discrimination of specific population groups, in violation of the EU Charter of Fundamental Rights. One of the potential causes of discriminative behavior, i.e., unfairness, lies in the quality of the data used to train such ADM systems. Using a data quality measurement approach combined with risk management, both defined in ISO standards, we focus on balance characteristics and we aim to understand how balance indexes (Gini, Simpson, Shannon, Imbalance Ratio) identify discrimination risk in six large datasets containing the classification output of ADM systems. The best result is achieved using the Imbalance Ratio index. Gini and Shannon indexes tend to assume high values and for this reason they have modest results in both aspects: further experimentation with different thresholds is needed. In terms of policies, the risk-based approach is a core element of the EU approach to regulate algorithmic systems: in this context, balance measures can be easily assumed as risk indicators of propagation - or even amplification - of bias in the input data of ADM systems.

Automated summary

The study indicates that automated decision-making systems may lead to systematic discrimination, with data quality being a fundamental factor. Utilizing data quality measurement methods defined in ISO standards along with risk management, balance indexes play a crucial role in identifying discrimination risk. In terms of policy-making, the risk-based approach is essential in regulating algorithmic systems.