Delivery of omnidirectional video using saliency prediction and optimal bitrate allocation

In this work, we propose and investigate a user-centric framework for the delivery of omnidirectional video (ODV) on VR systems by taking advantage of visual attention (saliency) models for bitrate allocation module. For this purpose, we formulate a new bitrate allocation algorithm that takes saliency map and nonlinear sphere-to-plane mapping into account for each ODV and solve the formulated problem using linear integer programming. For visual attention models, we use both image- and video-based saliency prediction results; moreover, we explore two types of attention model approaches: (i) salient object detection with transfer learning using pre-trained networks, (ii) saliency prediction with supervised networks trained on eye-fixation dataset. Experimental evaluations on saliency integration of models are discussed with interesting findings on transfer learning and supervised saliency approaches.

Automated summary

In this work, a user-centric framework for delivering omnidirectional video (ODV) on VR systems is proposed, utilizing visual attention models for bitrate allocation. The study formulates a new bitrate allocation algorithm considering saliency maps and nonlinear mapping, and explores saliency prediction using pre-trained networks and supervised networks. Experimental evaluations reveal interesting findings on the advantages of transfer learning and supervised saliency approaches in saliency integration.