Pyramid-VAE-GAN: Transferring hierarchical latent variables for image inpainting

Significant progress has been made in image inpainting methods in recent years. However, they are incapable of producing inpainting results with reasonable structures, rich detail, and sharpness at the same time. In this paper, we propose the Pyramid-VAE-GAN network for image inpainting to address this limitation. Our network is built on a variational autoencoder (VAE) backbone that encodes high-level latent variables to represent complicated high-dimensional prior distributions of images. The prior assists in reconstructing reasonable structures when inpainting. We also adopt a pyramid structure in our model to maintain rich detail in low-level latent variables. To avoid the usual incompatibility of requiring both reasonable structures and rich detail, we propose a novel cross-layer latent variable transfer module. This transfers information about long-range structures contained in high-level latent variables to low-level latent variables representing more detailed information. We further use adversarial training to select the most reasonable results and to improve the sharpness of the images. Extensive experimental results on multiple datasets demonstrate the superiority of our method. Our code is available at https://github.com/ thy960112/Pyramid-VAE-GAN.

Automated summary

This paper proposes a Pyramid-VAE-GAN network for image inpainting, which utilizes high-level latent variables to represent complex high-dimensional prior distributions and maintains rich details through a pyramid structure. A novel cross-layer latent variable transfer module is introduced to transfer structure information from high-level variables to more detailed low-level variables. Adversarial training is used to select the most reasonable results and improve image sharpness. Experimental results demonstrate the superiority of the proposed method.