QuickDedup: Efficient VM deduplication in cloud computing environments

Deduplication is one of the major storage optimisation techniques for Virtual Machines (VMs) in cloud environment. Usually, hashing of blocks helps in identifying duplicate data blocks. This paper proposes a novel deduplication approach, QuickDedup that reduces the overall deduplication time, metadata overhead and the number of hash computations, and subsequent comparisons for the VM disk images. In addition to minimising the deduplication related metadata, which is a necessary by-product useful in checking deduplication, QuickDedup, follows novel byte comparison scheme to prepare various block classes. This way, QuickDedup eliminates or minimises the need for hash calculation and subsequent comparisons. QuickDedup performs the calculation and comparisons of hashes within the respective categories only. QuickDedup saves the space required for hash storage during deduplication and makes deduplication of VM disk images much faster. We conducted a detailed evaluation of QuickDedup on various metrics with different kinds and sizes of VM images taken from publicly available datasets. The evaluation results show a substantial improvement of up to 96% in the overall deduplication time required to deduplicate VM images apart from significant savings in metadata and storage overhead. (C) 2020 Elsevier Inc. All rights reserved.