A Horizontal Tuning Framework for Machine Learning Algorithms Using a Microservice-based Architecture

Usually, data collected through surveys or by means of sensors is prone to errors and inaccuracies, such as missing data and outliers. Such datasets consist of numerical and string variables, with a high variety of values. Emerging issues, for instance, missing or categorical data lead to errors in running most of the machine learning algorithms. Data analysis and pre-processing are usually more substantial and time-consuming than the implementation of the machine algorithms. Nevertheless, the obtained results are significantly influenced by the way missing data or outliers are approached. This paper presents various methods for coping with null and extreme values. Furthermore, it highlights the significance of encoding and scaling the analysed data and their impact on the performance of the machine learning algorithms. Thus, this paper proposes a methodology for a Missing, Outliers, Encoding & Scaling (MOES) horizontal tuning framework using microservices as applications for data processing in order to obtain the best combination of the employed methods. For exemplification purposes, a real data set from the banking sector is used. Furthermore, the proposed methodology was tested using a second real data set from the utilities sector and the results also showed that both the AUC (Area under the Curve) and execution time were better than in the case of employing the PyCaret Python library.

Automated summary

This paper presents methods for handling null and extreme values, as well as the impact of encoding and scaling on the performance of machine learning algorithms. It also proposes a horizontal tuning framework to obtain the best combination of methods.