Machine Learning-Based Fast Angular Prediction Mode Decision Technique in Video Coding

In this paper, we propose a machine learning-based fast intra-prediction mode decision algorithm, using random forest that is an ensemble model of randomized decision trees. The random forest is used to estimate an intra-prediction mode from a prediction unit and to reduce encoding time significantly by avoiding the intensive Rate-Distortion (R-D) optimization of a number of intra-prediction modes. To this aim, we develop a randomized tree model including parameterized split functions at nodes to learn directional block-based features. The feature uses only four pixels reflecting a directional property of a block, and, thus the evaluation is fast and efficient. To integrate the proposed technique into the conventional video coding standard frameworks, the intra-prediction mode derived from the proposed technique, called an inferred mode, is used to shrink the pool of the candidate modes before carrying out the R-D optimization. The proposed technique is implemented into the high efficiency video coding test model reference software of the state-of-the-art video coding standard and joint exploration model reference software, by integrating the random forest trained off-line into the codecs. Experimental results demonstrate that the proposed technique achieves significant encoding time reduction with only slight coding loss as compared with the reference software models.