HPDedup: A Hybrid Prioritized Data Deduplication Mechanismfor Primary Storage in the Cloud

Huijun Wu∗†, Chen Wang∗, Yinjin Fu‡, Sherif Sakr∗†, Liming Zhu∗†, Kai Lu§∗Data61, CSIRO

†The University of New South Wales, Australia‡PLA University of Science and Technology, China

§Science and Technology on Parallel and Distributed Laboratory,State Key Laboratory of High Performance Computing,

State Key Laboratory of High-end Server & Storage Technology,College of Computer, National University of Defense Technology, Changsha, China

Abstract—Eliminating duplicate data in primary storage ofclouds increases the cost-efficiency of cloud service providers aswell as reduces the cost of users for using cloud services. Mostexisting primary deduplication techniques either use inlinecaching to exploit locality in primary workloads or use post-processing deduplication running in system idle time to avoidthe negative impact on I/O performance. However, neither ofthem works well in the cloud servers running multiple servicesor applications for the following two reasons: Firstly, thetemporal locality of duplicate data writes may not exist in someprimary storage workloads thus inline caching often fails toachieve good deduplication ratio. Secondly, the post-processingdeduplication allows duplicate data to be written into disks,therefore does not provide the benefit of I/O deduplicationand requires high peak storage capacity. This paper presentsHPDedup, a Hybrid Prioritized data Deduplication mechanismto deal with the storage system shared by applications runningin co-located virtual machines or containers by fusing aninline and a post-processing process for exact deduplication.In the inline deduplication phase, HPDedup gives a fingerprintcaching mechanism that estimates the temporal locality ofduplicates in data streams from different VMs or applicationsand prioritizes the cache allocation for these streams based onthe estimation. HPDedup also allows different deduplicationthreshold for streams based on their spatial locality to reducethe disk fragmentation. The post-processing phase removesduplicates whose fingerprints are not able to be cached due toweak temporal locality from disks. The hybrid deduplicationmechanism significantly reduces the amount of redundantdata written to the storage system while maintaining inlinedata writing performance. Our experimental results show thatHPDedup clearly outperforms the state-of-the-art primarystorage deduplication techniques in terms of inline cacheefficiency and primary deduplication efficiency.

Keywords-Data Deduplication; Cache Management; PrimaryStorage; Cloud Service

I. INTRODUCTION

Data deduplication is a technique that splits data intosmall chunks and uses the hash fingerprints of these datachunks to identify and eliminate duplicate chunks in order tosave storage space. Deduplication techniques have achievedgreat successes in backup storage systems [1]. However,

significant challenges remain to apply deduplication tech-niques in primary storage systems mainly due to the lowlatency requirement in primary storage applications [2].Recent studies show that duplicate data widely exists inthe primary workloads [2] [3] [4]. In the cloud computingscenario, the primary workloads of the applications runningon the same machine are observed having high duplicateratio as well [5]. For a cloud datacentre, there are significantincentives to remove duplicates in its primary storage forcost-effectiveness and competitiveness.

The existing data deduplication methods for primarystorage can be classified into two main categories based onwhen the deduplication is performed: inline deduplicationtechniques [6] [4] [5] and post-processing deduplicationtechniques [2] [7] [8]. The former performs data deduplica-tion on the write path of I/O requests to immediately identifyand eliminate data redundancy, while the latter removesduplicate data in background to avoid the performanceimpact on the I/O. However, challenges remain for both ofthese two methods.

For inline deduplication, fingerprint lookup is the mainperformance bottleneck due to that the size of a fingerprinttable often exceeds the size of the memory. While a backupstorage system may be able to tolerate the delay of diskbased fingerprint lookup, the deduplication system of aprimary storage system has to rely on caching to satisfy thelatency requirement of applications. The state-of-art tech-niques for inline primary deduplication [6] [4] [5] exploittemporal locality of primary workloads by maintaining an in-memory fingerprint cache to perform deduplication. Thesededuplication mechanisms do not ensure that all duplicatechunks are eliminated. We call them non-exact deduplica-tion. However, the temporal locality in primary workloadsdoes not always exist [9] [10]. For the workloads with weaktemporal locality, caching the unnecessary fingerprints notonly wastes the valuable cache space but also compromisesother workloads with good locality. iDedup [4] also exploitsspatial locality to alleviate data fragmentation on disks by

arX

iv:1

702.

0815

3v2

[cs

.DC

] 1

6 A

pr 2

017

only eliminating duplicate block sequences longer than afixed threshold. However, when data streams from differentsources have different spatial locality, a fixed threshold mayfail to achieve good deduplication ratio or read performance.For primary storage in clouds, the differences of localitybecome a severe problem. Firstly, the weak temporal lo-cality becomes more apparent in the cloud when multipleapplications running in virtualized containers sharing thesame physical primary storage system. Since deployingdeduplication in each virtual machine often fails to detect theduplicates among different virtual machines, deduplicationshould be deployed at the host physical machine. The datastreams from co-located VMs or applications may interferewith each other and destroy the temporal locality. It hassignificant impact on the fingerprint cache efficiency man-aged by existing caching policies. The stream interferenceproblem has been addressed in backup deduplication byresorting the streams [11]. However, it is not applicable forprimary storage systems because we cannot change the orderof requests of primary workloads. Secondly, as shown in ourexperiments on real-world traces (in Section III), differentworkloads show quite different spatial locality. Therefore, afixed global threshold is not optimal for alleviating the diskfragmentation in primary storage in the clouds.

For post-processing deduplication techniques [2] [7] [8].There are two main drawbacks: Firstly, duplicate chunksare written to disks before being eliminated. This makesdeduplication not effective in reducing peak storage use. ForSSD based primary storage in cloud architecture like hyper-converged infrastructure, this affects the lifetime of SSDdevices. Secondly, the competition between post-processingdeduplication process and foreground applications on usingresources such as CPU, RAM and I/O can be a problemwhen a large amount of duplicates has to be eliminated.

To avoid the limitations and exploit the advantages ofinline and post-processing deduplication, in this paper, wefuse the two phases together and propose a hybrid datadeduplication mechanism to particularly deal with dedupli-cation in virtualized systems running multiple services orapplications from different cloud tenants. The goal is toachieve a good balance between I/O efficiency and storagecapacity saving in primary storage deduplication. In theinline deduplication phase, we differentiate the temporal lo-cality of different data streams using a histogram estimationbased method. The estimation method periodically assessesthe temporal locality of the data streams from differentservices/applications. Based on the estimation, we proposea cache replacement algorithm to prioritize fingerprint cacheallocation to favor data streams with good temporal locality.The mechanism significantly improves cache efficiency ininline deduplication and reduces the workload in the post-processing deduplication phase. Moreover, we adjust thethreshold for different data streams dynamically to alleviatethe disk fragmentation while achieving high inline dedupli-

cation ratio. The post-processing deduplication phase onlydeals with relatively small amount of duplicate data blocksthat are missed in cache in the inline deduplication phase.Compared to systems that purely rely on post-processingdeduplication, a highly efficient inline deduplication pro-cess greatly reduces the storage capacity requirement andcontention in system resources.

Overall, this paper makes the following main contribu-tions:

1) We propose a novel hybrid deduplication mechanismthat fuses inline deduplication and post-processingdeduplication together for primary storage systemsshared by multiple applications/services. The mecha-nism is able to provide exact deduplication in compar-ison to many inline deduplication mechanisms whileavoiding drawbacks of purely post-processing dedupli-cation mechanisms.

2) We give a locality estimation based cache replacementalgorithm that significantly improves the fingerprintcache efficiency in primary storage deduplication sys-tems. The estimation method is able to exploit localityin individual data streams for cache hit rate improve-ment.

3) We evaluate our mechanism using traces generatedfrom real-world applications. The result shows that theproposed mechanism outperforms the state-of-art in-line and post-processing deduplication techniques. e.g.,HPDedup improves the inline deduplication ratio by upto 39.70% compared with iDedup in our experiments.It also reduces up to 45.08% disk capacity requirementcompared with the state-of-art post-processing dedupli-cation mechanism in our evaluation.

The remaining of this paper is organized as follows:Section II describes the background information and mo-tivations for our approach; Section III presents the designof HPDedup; Section IV introduces how to differentiatethe locality of data streams in deduplication; Section Vpresents the detailed results of our experimental evaluation;Section VI reviews related work and Section VII concludesthe paper.

II. BACKGROUND AND MOTIVATION

In this section, we present the background and key obser-vations that motivate this work.

A. Deduplication for Primary Storage in Clouds

Virtualization enables a cloud provider to accommodateapplications from multiple users to run on a single physicalmachine while providing isolation between these applica-tions. Recently, container techniques like Docker [12] furtherreduce the overhead of using virtual machines to isolateuser applications, thus support running more applicationssimultaneously on a physical machine.

0.0 0.5 1.0 1.5 2.0# of data chunks between two duplicates 1e7

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

# of data chunk

1e7

(a) FIU-mail trace

0 1 2 3 4 5# of data chunks between two duplicates 1e5

0.0

0.5

1.0

1.5

2.0

# of data chunk

1e5

(b) FIU-web trace

0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6# of data chunks between two duplicates 1e7

0.0

0.5

1.0

1.5

2.0

# of data chunk

1e5

(c) Cloud-FTP trace.

Figure 1: Temporal locality analysis for three I/O traces. The x-axis is the number of data blocks between two adjacentoccurrences of the same data block. i.e., for a I/O sequence ”abac”, each letter represents a data block. The number of datablocks between two adjacent occurrence of ”a” is 1.

0 50 100 150 200 250 300 350 400 450Request Sequence Count

0

20

40

60

80

100

Cache Sizes (%

of entries)

FIU-home FIU-mail FIU-web Cloud-FTP

(a) LRU

0 50 100 150 200 250 300 350 400 450Request Sequence Count

0

20

40

60

80

100

Cache Sizes (%

of entries)

FIU-home FIU-mail FIU-web Cloud-FTP

(b) LFU

0 50 100 150 200 250 300 350 400 450Request Sequence Count

0

20

40

60

80

100

Cache Sizes (%

of entries)

FIU-home FIU-mail FIU-web Cloud-FTP

(c) ARC

Figure 2: The size of cache entries occupied by each data stream. The x-axis is the request sequence count while the y-axisis the percentage of cache occupied by different streams. The area of different colors indicates the cache resources used byeach data stream.

Although providing much benefits in improving the re-source sharing efficiency, increasing number of applicationsfrom different users sharing the same machine raises chal-lenges to primary storage deduplication. In typical con-figurations, the cloud software stack such as OpenStack[13] map the data volumes of VMs to persistent blockstorage devices connected by networks. It is impractical toachieve deduplication within a container or a virtual machinedue to the overhead of storage device access. Moreover, itwould fail to identify the duplicates among different VMsor containers. It is only feasible to detect duplication in thehost’s hypervisor that manages I/O requests from VMs orcontainers. As the file information in VMs or containers isnot available in the underlying hypervisor, we design ourdeduplication mechanism to be based on the block level inthe hypervisor. A similar architecture has also been used inan existing post-processing primary deduplication technique[14].

B. Temporal locality Affects Efficiency of Fingerprint Cache

Existing primary storage deduplication techniques oftenexploit temporal locality through fingerprint cache in anattempt to detect most of duplicates in the cache. However,some recent studies reveal that the locality may be weak in

the primary storage workloads [15].We evaluate the temporal locality with real-world traces,

which contain a 24-hour I/O trace from a file server runningin the cloud as well as the FIU trace [16] commonly usedin deduplication research. The file server is used for datasharing among a research group consisting of 20 people.We denote the file server trace as Cloud-FTP, and FIU mailserver trace as FIU-mail and FIU Web server trace as FIU-web.

As shown in Figure 1, the average distance between twoadjacent occurrences of a data block in both FIU-mail andFIU-web trace is small and highly skewed, indicating goodlocality. For the Cloud-FTP trace, the temporal locality isweak. The temporal locality of duplicates in primary storagesystems varies among different applications.

We further evaluate the cache efficiency for the threedifferent workloads when they arrive at a storage systemwithin the same time frame. The cache replacement algo-rithms we use in our evaluation include LRU (Least RecentlyUsed), LFU (Least Frequently Used) and ARC (AdaptiveReplacement Cache). The three cache replacement policiesexploit the recency, frequency and the combination of bothof workloads, respectively.

We extract two-hour traces from the three FIU traces

Table I: Workload Statistics of the 2-hour traces.

Trace Request number Write request ratio Duplicate writes

Cloud-FTP 2293424 84.15% 387140FIU-mail 1961588 98.58% 1633424FIU-web 116940 49.36% 30534

FIU-home 293605 91.03% 32688

(10am-12am on the first day.) and the Cloud-FTP tracewe collect. The characteristics of the two-hour traces areshown in Table I. We mix these traces according to thetimestamps of requests to simulate an I/O pattern of multipleapplications on a cloud server. We set the cache size to 32Kentries. Figure 2 illustrates the actual percentage of cacheoccupied by each data stream.

Table II: Duplicates detected under different cache replace-ment algorithms.

Cache Policy FIU-home FIU-mail FIU-web Cloud-FTP

LRU 20568 399622 16667 11977LFU 19984 381157 16072 11072ARC 22248 1119355 12245 467

Table II shows the number of duplicate blocks detectedby each cache policy. Under the LRU and LFU cachereplacement algorithm, the cache allocated to Cloud-FTPstream is above 2/3 of the maximum cache capacity, but thenumber of duplicates detected in the stream is less than 2%of the overall duplicates. Under the ARC cache replacementalgorithm, the Cloud-FTP stream is allocated less portionof cache, however, only 467 duplicates are detected. Thisexperiment shows that data streams with weak temporallocality of duplicates result in poor cache efficiency andfail to detect most of duplicates in the inline deduplicationprocess.

When duplicates cannot be effectively detected throughcache lookup, they are written to disks, which results inextra storage space requirement in capacity planning. Itis therefore important to improve cache efficiency whenlocality of duplicates is not guaranteed.

III. THE DESIGN OF HPDEDUP

The inline or post-processing deduplication alone is diffi-cult to satisfy the deduplication ratio and latency requirementof a primary storage system. However, the two techniquescomplement each other. Fusing them together to form ahybrid deduplication system is able to achieve exact dedu-plication with satisfactory write performance. Particularly,the caching in inline deduplication not only speeds upfingerprint lookup, but also reduces the amount of datawritten to disks and relieves the burden of handling largeamount of duplicates in the post-processing phase. On theother hand, the post-processing deduplication is able todetect duplicates missed out in the inline cache, therefore

achieves exact deduplication. Including a post-processingphase potentially relaxes the inline cache size requirementas well.

In another word, a hybrid deduplication system is able toachieve a balance between the I/O performance and dedu-plication ratio, which is essential for inline deduplicationof primary storage. This motivates the architecture designof HPDedup. In the following, we first give the hybridarchitecture, and then describe the inline phase and post-processing phase of HPDedup.

A. HPDedup Architecture Overview

Host OS

VM(Domi) VM(Domj)

Hypervisor (Dom0)

HPDedup

Inline Deduplication

I/O streams

Storage Area Network (SAN)

Fingerprint Cache

Post-processing Deduplication

SSD

On-Disk

Fingerprint Table Data

Buffer

Garbage Collector

Container Container

Post-processing

Deduplication Engine

Inline

Deduplication

EngineFP PBA

FP PBA

Stream Locality Estimator

LBA Mapping Table

PBA

PBA

Reference Count

Table

PBA Count

LBA PBA

LBA PBA

PBA Count

FP LBA

FP LBA

PBA

PBA

On-Disk LBA

Mapping TableMapppping

LBA

gg Tablengg

PBA

LBA PBA

Figure 3: System architecture of HPDedup.

Virtualization is a core technology that enables the cloudcomputing. Running multiple virtual machines in a physicalmachine is a common practice in cloud datacentres. In cloudsoftware stack such as OpenStack [13], the storage volumesof virtual machines are often mapped to persistent blockstorage devices. It is impractical to implement deduplicationinside a virtual machine while data streams from co-locatedVMs are written to the same physical device. The mainreason is that performing deduplication in each VM is notable to remove the duplicates across VMs. We thereforeplace our deduplication mechanism at the block device level.

For the cloud scenario, the deduplication mechanism canbe implemented inside the hypervisor that manages the I/Orequests from VMs running on top of it. Some existing post-processing primary deduplication techniques [14] place theirdeduplication mechanisms at the same level.

Figure 3 illustrates the system architecture of HPDedup.A number of storage devices are connected to the servervia SAN or through similar storage environments. The

hypervisor (e.g., Xen) is responsible for translating LBA(logical block address) to PBA (physical block address) forblock I/O requests from VMs running on top of it. HPDedupworks at the hypervisor level to eliminate duplicate datablocks. For multiple containers running on the same host,HPDedup can be deployed at the block device level ofthe host machine. For simplicity, we will mainly use thehypervisor setting to describe the design of HPDedup in therest of this paper.

B. Inline Deduplication Phase

In the inline deduplication phase, HPDedup maintains anin-memory fingerprint cache that stores the fingerprint andPBA mapping to avoid slow disk-based fingerprint tablesearch, and an LBA mapping table that stores the mappingbetween LBAs and PBAs of blocks. The LBA mappingtable is stored in NVRAM to avoid the data loss. Theinline deduplication of data streams is performed in theinline deduplication engine: the fingerprint of each datablock is computed by a cryptographic hash function, likeMD5 or SHA-1. The deduplication engine then looks up theblock fingerprint in the fingerprint cache. The stream localityestimator is responsible for monitoring and estimating boththe temporal and spatial locality for the data streams comingfrom different VMs or containers. The temporal localityestimation is used for optimizing the hit rate of the finger-print cache while the spatial locality estimation adjusts thededuplication threshold for data streams to reduce the diskfragmentation.

When the fingerprint of the incoming data block is foundin the fingerprint cache, an entry of the LBA of the comingblock and the corresponding PBA will be created and addedinto the LBA mapping table if such an entry does not exist,otherwise nothing is done because it is a duplicate write. Ifthe block fingerprint is not found in the fingerprint cache,the data block is written to the underlying primary SANstorage. In this process, the data is staged in the data bufferin SSD for performance consideration. We use D-LRU [17]algorithm to manage the data buffer in SSD to store recentlyaccessed data by exploiting temporal locality.

When a data block is written to the underlying primarystorage, the corresponding metadata associated with this datablock including its fingerprint, LBA and PBA mapping aswell as the reference count is updated in three tables in theSSD: on-disk fingerprint table, on-disk LBA mapping tableand reference count table. The duplicates whose fingerprintsare not cached in fingerprint cache will be eliminated in thepost-processing phase.

C. Post-Processing Deduplication Phase

In the post-processing deduplication phase, the post-processing deduplication engine scans the on-disk fingerprinttable and identifies duplicates. Note that duplicates iden-tified in this phase are not in the fingerprint cache, thus

they are not processed by inline deduplication phase. Theentries with duplicate fingerprints are then removed whilethe corresponding LBAs are mapped to the same PBA inLBA mapping table. The reference count to the originalPBAs containing the same data is decremented and the diskspace is further claimed by the garbage collector. After post-processing, the unique data blocks in data buffer of SSD areorganized into fixed-sized coarse-grained objects and flushedto the underlying persistent store.

IV. DIFFERENTIATE DATA STREAM LOCALITY INDEDUPLICATION

In this section we describe how to differentiate the tem-poral and spatial locality among different data streams toimprove the efficiency of primary deduplication in the cloud.Both temporal and spatial locality estimation are performedin the stream locality estimator of the inline deduplicationmodule. Specifically, the temporal locality of duplicates ina data stream is used to guide the allocation of fingerprintcache to the data stream in order to achieve higher inlinededuplication ratio. The spatial locality of a stream is used toachieve a balance between the inline deduplication ratio andthe read performance. We first describe how to measure andestimate the temporal locality of duplicates in data streamsin IV-A, and then describe how to manage the fingerprintcache based on the temporal locality measurement in IV-B.Thirdly, we discuss how to handle the disk fragmentationbased on the difference of spatial locality among datastreams in IV-C.

A. Temporal Locality Estimation for data streams

The temporal locality of duplicates characterizes howsoon duplicates of a data block may arrive in the system ina data stream. A good temporal locality indicates duplicatedata blocks generally are close to each other while a weaklocality indicates that duplicate blocks are often far awayfrom each other or there are few duplicates in the datastream.

To measure the temporal locality of duplicates, we intro-duce a metric called Local Duplicate Set Size (LDSS). LDSSof a stream is defined as the number of duplicate fingerprintsin last n contiguous data blocks arriving before a given time.Here, we call n estimation interval.

To use LDSS to guide fingerprint cache allocation, weneed to predict the LDSS of the future arrivals of datablocks of the data stream. A common approach is to use thehistorical LDSS values to predict the future LDSS of the datastream. To obtain a historical LDSS value from a data stream,a naive way is to count all distinct fingerprints for each datastream and their occurrences within an estimation interval.However, this incurs a high memory overhead which is closeto the cache capacity because all the fingerprints need to berecorded. To address the problem, the stream locality esti-mator uses the reservoir sampling algorithm [18] to sample

fingerprints from a data stream, and then estimate LDSS fromthese samples using the unseen estimation algorithm [19].

Reservoir sampling algorithm assumes an unknown num-ber of fingerprints in a data stream and guarantees thateach fingerprint in the data stream has an equal chanceto be sampled. In our implementation, each element in thesampling buffer is a pair containing a fingerprint and itsoccurrence count.

The unseen estimation algorithm is able to estimate theunseen data distribution based on the histogram of the sam-ples of observed data. For HPDedup, the unseen estimationalgorithm is used to estimate LDSS of data streams based thesampled fingerprints from these streams. We refer readersto [19] for more details about the theoretical aspect ofestimation algorithms. Here, we give a high-level descriptionof using the unseen estimation algorithm to estimate theLDSS values for a data stream.

Consider the storage system handles M data streams,denoted by S1,S2,...,SM from M VMs and the estimationinterval size is n, the goal of the temporal locality estimationis to collect k fingerprint samples from the last n writerequests of each stream and compute the LDSS values forthese streams based on fingerprint samples.

Specifically, after sampling, we denote the number ofsampled fingerprints coming from stream i by Ni. By usingunseen estimation algorithm, we can accurately estimate thenumber of unique writes (denoted by ui) in stream Si amonglast n write requests in the mixed stream. Then the estimatedLDSS for stream i can be denoted by LDSSi as below:

LDSSi = Ni − ui

whereas Ni is the total number of write requests for streami in the estimation interval.

The estimation of ui as well as the calculation of LDSSi

is shown in Algorithm 1. Before discussing the algorithm,we introduce a concept named Fingerprint Frequency His-togram (FFH). A FFH of a set of fingerprints F is ahistogram f = {f1, f2...} where fj is the number of distinctfingerprints that appear exactly j times in F .

We derive the FFH from the sampling buffer to estimatethe LDSSi of data stream i. We use Hs to denote the FFHof the samples and H to denote the FFH of the wholeestimation interval for stream i. According to the unseenestimation algorithm, we then compute the transformationmatrix T by a combination of binomial probabilities aboutthe chances an item is drawn a certain times. The expectedhistogram H

′

s for sampled data blocks can be computed byH

′

s = T ·H . To solve the equation and get H , we minimizethe distance between Hs and H

′

s which are the observedhistogram and expected histogram of sampled data blocks,respectively. Once obtained H , we are able to compute theLDSSi for the data stream.

For some streams which have few write requests duringthe estimation interval, it is not necessary to run unseen

Algorithm 1: Temporal Locality Estimation AlgorithmInput : Hs – The FFH for samples in stream i;

Ni – the number of write requests ofstream i in the estimation interval.

Output: Estimated LDSSi of data stream i1 Compute matrix T by binomial probabilities.2 H

′

s for samples is computed by H′

s = T ·H3 Linear programming:4 Objective function: min(∆(Hs, H

′

s)), in which∆(Hs, H

′

s) = Σi1√

Hs[i]+1|Hs[i]− (T ·H)[i]|.

5 under constraints:6 ΣiH[i] = N7 ∀i H[i] > 08 return LDSSi = Ni − ΣiH[i]

estimation algorithm to estimate the LDSS. The LDSS ofthese streams are set to a small value for simplicity.

B. LDSS Estimation Based Fingerprint Cache Management

We use the LDSS of different streams to guide the cacheallocation for these streams. The fingerprints from a datastream with higher predicted LDSS is more likely to be keptin the cache than those from a data stream with lower LDSS.As mentioned earlier, we use the historical LDSS valueswhich are accurately estimated by the unseen algorithmto predict the LDSS of streams. We use self-tuned doubleexponential smoothing method to predict the LDSS values.Using the estimated LDSS(w − 2), LDSS(w − 1), ..., wecan predict LDSS(w) where w is the next estimationinterval. The predicted LDSS is used to guide the fingerprintcache management as follows.

Firstly, we propose a cache admission policy that thefingerprints from streams with very low LDSS would not becached if there exists streams with much higher LDSS. Thisstrategy can avoid caching the fingerprints of data streamscontaining compressed data or other forms of compact data.The cache of each stream can be managed by any cachereplacement policies.

Secondly, for the fingerprints already cached, we assignan evict priority value pi to data stream i, denoted by:

pi =1

LDSSi(w)

The evict priorities are mapped to adjacent non-overlappingsegments in a segment tree. Specifically, stream i is rep-resented by the segment [Σi−1

k=0pk, Σi−1k=0pk + pi). When

evicting a fingerprint from the cache, we generate a randomnumber r and find the segment I to which r belongs. Wethen evict one cache entry from the cache corresponding tointerval I .

As the fingerprint cache management of HPDedup relieson the accurate LDSS estimation of using unseen estimation

algorithm, one may also think about directly estimating theLDSS of data streams by the number of duplicate fingerprintsamples sampled by reservoir sampling. Figure 4 comparesthe effectiveness of using RS-only (Reservoir sampling only,dash lines) or RS + Unseen (Reservoir sampling with unseenalgorithm, solid lines) during the LDSS estimation. It isclear that RS + Unseen based LDSS estimation is able toprovide much higher inline deduplication ratio with a smallerestimation interval compared with RS-only method. Thisshows the effectiveness of temporal locality estimation usingthe unseen algorithm.

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0Estimation Interval Factor

35

40

45

50

55

60

65

70

Inlin

e D

ed

up

lica

tio

n R

ati

o (%)

A

B

C

Workload A - RS+Unseen

Workload B - RS+Unseen

Workload C - RS+Unseen

Workload A - RS only

Workload B - RS only

Workload C - RS only

Figure 4: Inline Deduplication ratio vs. Estimation IntervalFactor for three different workloads.

Moreover, a proper value of estimation interval is im-portant for achieving the good cache efficiency. Too largeinterval may include some out-dated information whichcannot reflect the current temporal locality of duplicates forthe workloads. Too small interval, on the other hand, cannotaccurately capture the temporal locality. Since LDSS is usedto estimate the number of duplicates which can be detectedin the fingerprint cache, the estimation interval can be setto a factor of the number of fingerprint cache entries. Thesolid lines in Figure 4 shows the inline deduplication ratio ofHPDedup for three different workloads (details are shownin Section V) while choosing different estimation intervalfactor. The cache size is set to 160MB. From the workloadA to C, the overall temporal locality for the workloaddecreases. The estimation interval factor needs to be set tolarger values for the workloads with worse temporal locality.Correspondingly, we can see that the optimal estimationinterval factor for workload A, B and C are 0.3, 0.4 and0.6, respectively. In practice, a good approximation is to setthe estimation interval factor to 1 - d where d is the historicalinline deduplication ratio for the mixed streams.

The temporal locality estimation is triggered by the fol-lowing three events: 1. the finish of an estimation interval;2. a significant drop of inline deduplication ratio; 3. the joinor quit of virtual machines/applications.

2 4 6 8 10 12 14 16duplicate sequence threshold

0

20

40

60

80

100

deduplic

ation ratio (%

)

FIU-mail

FIU-web

Cloud-FTP

FIU-home

Figure 5: Deduplication ratio vs. Threshold. Deduplicationratio versus threshold for different threshold of duplicatesequence length.

C. Spatial Locality Aware Threshold for Deduplication

To alleviate disk fragmentation problem of deduplicationon data read, some primary storage deduplication techniquesonly eliminate duplicate block sequences with length greaterthan a given threshold. However, as pointed by [4], forthe applications with many random I/Os, doing so maynot find any duplicates. In the deduplication of primarystorage systems in clouds, the spatial locality of data streamsfor different applications/services varies significantly. Toexplore the relationship between deduplication ratio andthreshold, we analyze both the FIU traces and the trace wecollect. As shown in Figure 5, different workloads showdifferent trends. When the threshold increases from 1 to 16,the inline deduplication ratio for FIU-mail and Cloud-FTPreduces by only 4.3% and 9.1%, respectively. The inlinededuplication ratio for FIU-web drops by around 38.1%when the threshold increases from 1 to 2. When the thresholdis 16, the inline deduplication ratio is 43.1% of that under athreshold of 1. For FIU-home trace, the inline deduplicationratio keeps dropping. When the threshold is set to 16, theinline deduplication ratio is only 32.0% of that under athreshold of 1 .

Figure 5 indicates that the threshold value should beadaptive to data stream characteristics. It is noteworthy thatthere exists a tradeoff between the write latency and readlatency while choosing a proper threshold. Write operationsprefer shorter threshold while read operations prefer longerthreshold. For writes, long sequence indicates more com-parisons before writing data blocks to disks. For reads, longthreshold can avoid many random I/Os thus reducing theread latency.

HPDedup maintains two vectors Vw and Vr for eachstream. Vw is used to store the occurrence number for thelargest length values of sequential duplicates. Vr is usedto store the occurrence number for the length values for

sequential read. Both Vw and Vr have 64 items. For instance,if Vw[3] = 100, there are 100 sequential duplicates withlength 3 since Vw is reset. If Vr[3] = 100, there are 100sequential reads with length 3 since Vr is reset.

Initially, the threshold is 16. The two vectors collect datawhen requests come. When the threshold update is triggered,given the histogram vector Vw and Vr, the threshold T iscomputed by

T = (1− r) · Lend + r · Lenr

where Lend and Lenr are the average length of duplicateblock sequence and average read length, respectively. T ,therefore, is the balance point of the read and write latency.r is the read ratio among all requests. Lend and Lenr arecomputed according to the data collected in Vw and Vr,respectively. To cope with the changes of duplicate patternfor each stream, the two vectors is reset to all 0s when thetotal deduplication ratio decreases by over 50% since thelast threshold update.

V. EVALUATION

The prototype of HPDedup is implemented in C. To eval-uate the performance of HPDedup, we use real-world tracesto feed into HPDedup. We compare the performance ofHPDedup with the following deduplication methods: localitybased inline deduplication (iDedup [4]), post-processingdeduplication schema (e.g., [2] [14]) and hybrid inline-offline deduplication schema DIODE [20].

A. Configuration

The experiments are carried out in a workstation withIntel Core i7-4790 CPU , 32GB RAM and 128GB SSD +1TB HDD. We use the FIU-home, FIU-web, FIU-mail [16]traces in our evaluation. These traces are from three differentapplications, namely remote desktop, web server and mailserver respectively in FIU. Moreover, we also collect a tracefrom a cloud FTP server (Cloud-FTP) used by our researchgroup. The trace is obtained by mounting a network blockdevice (NBD) as the working device for the file server, fromwhich we capture read/write requests through a customizedNBD-server.

To the best of our knowledge, there is no available largerscale I/O traces containing both the fingerprint of datablocks and timestamps. We therefore use the four traces astemplates to synthetically generate VM traces representingmultiple VMs. Table III shows the statistics of the fourworkloads. The arrival order of requests in these workloadsare sorted and merged based on timestamps. The generatedtrace has the same I/O pattern with the original traces. Forthe traces generated from the same template, the contentoverlap is randomly set to 0% - 40% which is the typicaldata redundancies among users [21].

In our experiments, we simulate a cloud host running 32virtual machines. As shown in Figure 1, FIU traces show

Table III: Workload Statistics

Trace Num of requests Write request ratio Duplicate ratio

Cloud-FTP 21974156 83.94% 20.77%FIU-mail 22711277 91.42% 90.98%FIU-web 676138 73.27% 54.98%

FIU-home 2020127 90.44% 30.48%

better temporal locality compared with Cloud-FTP trace. Wemix traces to form three workloads with different overalltemporal locality. Workload A contains 15 mail server traces,5 FTP server traces and 8 remote desktop traces and 4 webserver traces. Workload B contains 10 mail server traces, 10FTP server traces, 6 remote desktop server traces and 6 webserver traces. Workload C contains 5 mail server traces, 15FTP server traces, 6 web server traces and 6 remote desktopserver traces. The ratios of data size between the good-locality (L) traces and bad-locality (NL) traces are around3:1, 1:1 and 1:3 for these three mixed workloads.

The estimation interval factor is set to 0.5 at the beginningand is adjusted by the historical inline deduplication ratiodynamically for each workload.

B. Cache Efficiency in Inline Deduplication

We compare HPDedup with iDedup, a well-known inlinededuplication system that makes use of temporal localityof primary workload. We replay the mixed workloads tosimulate the scenario where multiple applications/servicesrunning on the same physical machine. Each I/O for thethree workloads is a 4KB block and MD5 is used as thehash function to calculate the fingerprints.

Each entry of the deduplication metadata is about 64bytes and contains the fingerprint and the block address.According to the size and footprint of the traces, the totalmemory size for fingerprint cache is set from 20MB to320MB in the experiments. The deduplication threshold isset to 4 for both iDedup and HPDedup.

Figure 6 shows the inline deduplication ratio versus cachesize for iDedup and HPDedup. Here, inline deduplicationratio is defined as the percentage of duplicate data blocksthat can be identified by inline caching. For the cachereplacement policy of each stream, LRU, LFU and ARCcache replacement policies are supported by HPDedup. LRUwas claimed to be the best cache replacement policy byiDedup [4].

When the portion of NL workload increases, the gapbetween iDedup and HPDedup becomes larger. HPDedup-LRU, HPDedup-LFU and HPDedup-ARC improve the inlinededuplication ratio significantly compared with iDedup. Forworkload A, HPDedup-ARC improves the inline deduplica-tion ratio by 10.58% - 27.72%. HPDedup-LRU improves theinline deduplication ratio of iDedup which also uses LRUcache by 8.04% - 23.52%. HPDedup-LFU shows less im-provement (3.56%-13.90%) compared with HPDedup-ARC

0 50 100 150 200 250 300 350Cache Size (MB)

0

10

20

30

40

50

60

70

80

Inlin

e D

ed

up

lica

tio

n R

ati

o (

%)

iDedup

HPDedupe-LRU

HPDedupe-LFU

HPDedupe-ARC

(a) Workload A (L:NL = 3:1)

0 50 100 150 200 250 300Cache Size (MB)

0

10

20

30

40

50

60

70

80

Inlin

e D

ed

up

lica

tio

n R

ati

o (

%)

iDedup

HPDedupe-LRU

HPDedupe-LFU

HPDedupe-ARC

(b) Workload B (L:NL = 2:2)

0 50 100 150 200 250 300Cache Size (MB)

0

10

20

30

40

50

60

70

80

Inlin

e D

ed

up

lica

tio

n R

ati

o (

%)

iDedup

HPDedupe-LRU

HPDedupe-LFU

HPDedupe-ARC

(c) Workload C (L:NL = 1:3)

Figure 6: Inline deduplication ratio vs. Cache size for iDedup and HPDedup with different cache replacement policies.

and HPDedup-LRU. Similarly, for workload B, HPDedup-ARC, HPDedup-LRU and HPDedup-LFU achieve 8.09%-30.19%, 6.36%-22.02% and 3.77%- 16.02% improvementof inline deduplication ratio, respectively. For workloadC, HPDedup-ARC improves the inline deduplication ra-tio by 15.38% - 39.70%. HPDedup-LRU and HPDedup-LFU achieve 13.86% - 37.75% and 12.97% - 28.37%improvement, respectively. The improvement is larger whenthe cache size is small due to cache resource contention.Moreover, HPDedup-ARC outperforms HPDedup-LRU andHPDedup-LFU because ARC cache replacement policymakes the size of T1 (LRU) cache and T2 (LFU) cacheadaptive to the recency and frequency of the workloads. Itis also noteworthy that with the increasing of non-localityworkload share (from Workload A to C), the inline dedupli-cation ratio improvement achieved by HPDedup becomeslarger. The reason is that non-locality workloads providemore space for the optimization of HPDedup.

Note that for HPDedup-ARC, extra memory overheadis introduced by ARC caching replacement policy itselfto track the evicted fingerprints and their metadata. Theoverhead is non-trivial for fingerprint cache. The analysis ofoverhead for obtaining statistics information about evictedfingerprints in existing cache replacement policies is out ofthe scope of this paper. We leave the discussion to the futurework. In the following experiments, HPDedup-LRU is usedby default.

Overall, the weak locality in workloads results in lowinline deduplication ratio. With the locality estimationmethod in HPDedup, the allocation of inline fingerprintcache dynamically gives the streams with better localityhigher priority. Hence, HPDedup improves the overallcache efficiency for multiple VMs/applications runningon the same physical machine in the cloud.

C. Disk Capacity Requirement

In this subsection, we compare the size of the data beforeperforming post-processing deduplication for HPDedup and

workload A workload B workload CWorkload

0

200

400

600

800

1000

1200

1400

1600

Disk

Capaci

ty R

equirem

ent (G

B)

HPDedup Post-processing

Figure 7: The disk capacity requirements for HPDedup andpost-processing deduplication schemas.

pure post-processing deduplication (e.g., [2]). The size isalso the maximum required disk size for the deduplicationmechanisms. Figure 7 shows the result of comparisons.

For HPDedup, the inline fingerprint size is set to 160MBand LRU cache replacement policy is used for simplicity. Asshown in Figure 7, HPDedup significantly reduces the diskcapacity requirements for storage space. The data size hasbeen reduced by 45.08%, 28.29% and 12.78% for workloadA, B and C, respectively. The better the locality is in theworkload, the more duplicates can be detected in the inlinephase and the more duplicate data writes can be eliminated.This clearly shows the benefit of a hybrid deduplicationarchitecture of HPDedup as hundred GBs of data writescan be reduced by only maintaining a 160MB inlinefingerprint cache.

D. Average Hits of Cached Fingerprints

Inline deduplication of HPDedup does not require diskaccess to identify duplicates therefore is faster than thepost-processing based deduplication. We use average hits ofcached fingerprints as an indicator of inline deduplicationperformance. The indicator is obtained by monitoring thenumber of fingerprint entering the fingerprint cache. With a

high average hits value, the inline deduplication is able todetect a large portion of duplicates and reduces the load ofthe more expensive post-processing deduplication.

In the following, we compare HPDedup with DIODE [20]on this metric. DIODE uses file extensions to decide whetherto perform inline deduplication on these files. Files areclassified roughly into three different types. The inline dedu-plication process skips the type of files containing audio,video, encrypted and other compressed data (called P-Typein DIODE). We use a full inline deduplication method [4]as the baseline.

DIODE works at the file system level so that the informa-tion like file extensions are passed to the hypervisor layerin the form of hints, like the method used by [22]. The filetype information of the Cloud-FTP trace is known so that thetrace can be used to test DIODE. The files that are classifiedas P-Type are around 14.2% of the whole trace in size. TheFIU traces are classified into U-Type (unpredictable type),which will be processed by the inline deduplication, just likethat in the evaluation setting of DIODE in [20].

Table IV: The Average Hits of Cached Fingerprints forbaseline, DIODE and HPDedup.

Schema Cache Size Workload A Workload B Workload C

Baseline 320MB 1.301 0.665 0.204160MB 0.781 0.551 0.148

DIODE 320MB 1.437 0.812 0.247160MB 0.805 0.598 0.181

HPDedup 320MB 1.812 4.024 5.918160MB 1.409 3.101 5.002

As shown in Table IV, HPDedup clearly outperformsthe baseline and DIODE on the average hits of cachedfingerprints. HPDedup outperforms DIODE by identifyingdata streams that have weak locality but do not belong to P-Type. It then avoids allocating cache space to these streamsin order to make room for data streams with good locality.

This approach is effective. To show this, we comparethe locality of the following two files in the Cloud-FTPworkload: a Linux kernel 4.6 source code tar file and a VMimage of CentOS 5.8 downloaded from OSBoxes. The twofiles are similar in size (2.7GB and 2.6GB). They are writtento the primary storage after an inline deduplication process.The fingerprint cache size is set to 1% of the data size(27MB and 26MB). Figure 8 shows the number of duplicateblocks found in the two files. Note that for the VM image,nearly all duplicate blocks can be found through the inlinededuplication process. DIODE treats both files as highly-deduplicatable (H-Type in DIODE). However, the numberof duplicate blocks in the VM image file is significantlyhigher than that in the source code tar file. Moreover, DIODEignores all P-Type files during inline deduplication by lettingthem to be processed during inline deduplication. However,

0 5 10 15 20

Time (seconds)

0

5000

10000

15000

20000

25000

Duplicate Block

s Fo

und

Offline Duplicate Block Number

Inline Dedupped Block Number

(a) CentOS VM image

0 5 10 15 20

Time (seconds)

0

5

10

15

20

25

30

35

Duplicate Blocks Fo

und

Offline Duplicate Block Number

Inline Dedupped Block Number

(b) Linux Source Code

Figure 8: The duplicate blocks found in the VM image andLinux Kernel Source Code tar file. The x-axis is the timeduring writing the files while the y-axis is the number ofduplicate blocks.

multiple writes of the same P-Type files result in duplicationand cannot be eliminated through differentiating file types.Different from DIODE, HPDedup allocates much less cacheto the Cloud-FTP stream while writing the Linux SourceCode tar file but allocates more when writing the VM imagefile.

The result shows that simply using file type to guidecache allocation is insufficient. HPDedup classifies dataat finer-grained (stream temporal locality level) so thatthe efficiency of inline deduplication can be furtherimproved.

E. Locality Estimation Accuracy

HPDedup improves the efficiency of inline deduplicationphase by allocating the fingerprint cache based on thetemporal locality of each stream. Since LDSS is an indicatorwhich describes the temporal locality of data streams, it iscritical to achieve accurate LDSS estimation.

Figure 9 shows the observed LDSS for workload B.Here, the cache size is set to 160MB. To make the figuresconcise, the traces generated from the same template areaggregated. Figure 9a shows the observed LDSS over time.The values of LDSS are normalized. FIU-mail streams show

the largest LDSS thus indicating it has the best temporallocality. Nevertheless, as shown in Figure 9b, very littlecache resource is occupied by FIU-mail streams when LDSSestimation is not used. Cloud-FTP streams whose LDSS isnot high occupy the majority of cache resources. As shownin Figure 9c, with the guidance of LDSS estimation, cacheresource is cleverly allocated to streams based on theirtemporal locality. LDSS estimation allocates fingerprintcache resources according to the temporal locality ofstreams and improves the inline deduplication ratio by12.53% (see Figure 6b). The improvement clearly showsthe effectiveness of LDSS estimation in HPDedup.

F. Fragmentation

In this subsection, we evaluate the fragmentation of files instorage system caused by deduplication. The length thresh-old of duplicate block sequence controls the fragmentationin both HPDedup and DIODE.

Both DIODE and HPDedup are able to adjust the thresh-old dynamically. Figure 10 shows the threshold change alongtime for workload A in DIODE and HPDedup.

The inline deduplication ratio for DIODE and HPDedupare 57.62% and 68.96%, respectively. As one may see,HPDedup is able to adjust the threshold for each streamwhile DIODE uses a global threshold. The FIU-mail andCloud-FTP have a higher threshold than FIU-home and FIU-web. Since larger threshold leads to less disk fragmentation,this result shows that HPDedup introduces less fragmen-tation while achieving higher inline deduplication ratiothan DIODE.

G. Overhead Analysis

While HPDedup improves the efficiency of primary stor-age deduplication significantly, it inevitably incurs overhead.We analyze the overhead in this subsection. The overheadcan be classified into computational overhead and memoryoverhead.

1) Computational Overhead: The computational over-head of HPDedup contains the following two parts: thehistogram calculation time and the estimation algorithmexecution time.

To calculate the histogram, we only need to scan thesample buffer and add the count of the fingerprints tocorresponding bins of the histogram. Therefore, the timecomplexity is O(n) where n is the number of samples.Figure 11a shows the time used for generating the histogramin our current implementation. Here, the sampling rate is15%. We can see that the histogram calculation of anestimation interval with 1 million blocks takes less than 7ms.

Estimating the temporal locality of streams is achieved byusing the method described in Section IV-A. The core of theestimation is to solve a linear programming problem. Thelinear programming problem can be solved in O(n) [23] andeven constant time [24] when the number of variables d is

0 200 400 600 800 1000 1200 1400Time (Minute)

0.0

0.2

0.4

0.6

0.8

1.0

Norm

aliz

ed L

DSS

FIU-home Streams

FIU-mail Streams

FIU-web Streams

Cloud-FTP Streams

(a) Normalized Observed LDSS

0 200 400 600 800 1000 1200 1400Time (Minute)

0

50

100

150

200

Cach

e Size (MB)

FIU-home Streams

FIU-mail Streams

FIU-web Streams

Cloud-FTP Streams

(b) Cache Size Distribution (without LDSS estimation)

0 200 400 600 800 1000 1200 1400Time (Minute)

0

50

100

150

200

Cach

e Size (MB)

FIU-home Streams

FIU-mail Streams

FIU-web Streams

Cloud-FTP Streams

(c) Cache Size Distribution (with LDSS estimation)

Figure 9: LDSS estimation accuracy.

fixed and the number of constraints is fixed. In our context,the condition is satisfied because too frequent duplicates inthe sampling buffer will be used in a straightforward wayduring the estimation and will not be put into the linearprogramming.

Note that the linear programming needs to be done foreach stream. For every estimation interval, the temporal lo-cality estimation takes about 26ms for each stream regardlessthe estimation interval size (see Figure 11b). The comput-ing overhead is acceptable as the process is performed in

0 200 400 600 800 1000 1200 1400Time (Minute)

0

2

4

6

8

10

12

14

16Thresh

old

(a) DIODE

0 200 400 600 800 1000 1200 1400Time (Minute)

0

10

20

30

40

50

Threshold

FIU-home

FIU-mail

FIU-web

Cloud-FTP

(b) HPDedup

Figure 10: Threshold vs. Time for HPDedup and DIODE(cache size: 160MB).

0 200000 400000 600000 800000 1000000

Interval Size

0

1

2

3

4

5

6

7

Tim

e (ms)

(a) generating histogram for samples

0 200000 400000 600000 800000 1000000

Interval Size

0

5

10

15

20

25

30

Tim

e (ms)

(b) temporal locality estimation

Figure 11: The time cost for HPDedup.

background and does not affect the data write performance.2) Memory Overhead: The primary memory overhead

of HPDedup comes from the sampling buffer. With anestimation interval of size EI and a sampling rate p, thememory cost for tracking the histogram of samples is asbelow:

EI · p · (fpSize + counterSize)

where fpSize and counterSize are the memory cost forstoring fingerprints and the occurrence count, respectively.For instance, when the cache size is 160MB, there wouldbe approximately 2.62M cache entries. For the sampling rateof 15%, the memory overhead is only 4.49MB (2.81% ofcache size) even if we choose a large estimation intervalfactor (e.g., Workload C, 0.6). In practice, for the mixed

streams with better temporal locality, the memory overheadis much less (e.g., 2.25MB for Workload A and 2.99MBfor Workload B) as the estimation interval factor can beset to smaller values. Compared with the improvement(19.80% - 25.81%) of inline deduplication ratio for thethree workloads with 160MB cache size), the memoryoverhead of HPDedup is acceptable.

VI. RELATED WORK

A. Primary Storage Deduplication Mechanisms

Data deduplication achieves a great success in backupstorage systems. Recent research exploit various waysto apply deduplication in primary storage systems forboth reducing data size in storage devices and improv-ing I/O performance. Existing work can be classified intothree categories: Inline primary storage deduplication, Post-processing/Offline primary storage deduplication and Hybridinline and post-processing deduplication.

Inline primary storage deduplication. Most inline pri-mary deduplication exploits the locality in primary work-loads to perform non-exact deduplication. iDedup [4] ex-ploits the temporal locality by only maintaining an in-memory cache to store the fingerprints of data blocks. Toexploit the spatial locality, iDedup only eliminates duplicatesin long sequences of data blocks. POD [5] aims at improvingthe I/O performance in primary storage systems and mainlyperforms deduplication on small I/O requests. HANDS [6]uses working set prediction to improve the locality offingerprints. Koller et al. [16] uses content-aware cache toimprove the efficiency of I/O by avoiding the influence ofduplicated data. PDFS [15] argues that the locality may notcommonly exist in primary workloads. To avoid the diskbottleneck of storing fingerprint table, a similarity basedpartial lookup solution is proposed. Leach [25] exploits thetemporal locality of workloads by a splay tree. These workdo not consider scenarios involving VMs and containers inthe cloud where workloads for the primary storage containa mix of data streams with different access patterns.

Post-processing/Offline primary storage deduplication.Post-processing deduplication performs deduplication duringthe idle time of primary storage systems. Ahmed El-Shimiet al. [2] propose a post-processing deduplication methodbuilt in Windows Server operating systems. Similar withHPDedup, DEDIS [14] is built in the Xen hypervisor toprovide data deduplication functionality to multiple virtualmachines. The main purpose of post-processing primarystorage deduplication is to avoid the high I/O latency intro-duced by inline on-disk dedupe metadata lookup. However,even though the locality does not always exist in primaryworkloads, it is much more efficient to use inline cachingrather than post-processing to eliminate duplicates in theportion of workloads with decent locality. The contributionof HPDedup is to differentiate the deduplication procedure

for primary workloads according to the temporal locality ofworkloads.

Hybrid inline and post-processing deduplication. Com-bining inline and post-processing deduplication together hasbeen exploited by RevDedup [26] in backup storage dedupli-cation to improve the space efficiency. For primary storagededuplication, DIODE [20] also proposes a dynamic archi-tecture of inline-offline deduplication. Like ALG-Dedupe[27], DIODE is an application-aware deduplication mecha-nism. File extensions are classified into three types accordingto their potential deduplication ratio. Moreover, whetherperforming inline deduplication on a file is determined bythe extension of the files. However, our experiments showthat file types are not sufficient for achieving good inlinededuplication performance and there is a lot of room to im-prove. The key difference between HPDedup and DIODE isthat HPDedup gives a dynamic locality estimation method toimprove inline deduplication performance, therefore reducesthe load of the more expensive post-processing deduplicationprocess.

B. Unseen Distribution Estimation

Estimating the number of duplicates in a time frame fora data stream is similar to estimating the distinct elementsin a large set, for which various statistics based methods(e.g., [28], [29]) have been investigated. Fisher et al. [30] de-scribe a method to estimate the number of unknown speciesgiven a histogram of randomly sampled species. Theoreticalcomputer science community has been trying to addresshow to perform the estimation with less samples [31]–[34].Recently, this line of work has been extended by Valiant andValiant [35] to characterize unobserved distributions. Theyprove that only O( n

log(n) ) samples are sufficient to providean accurate estimation of the whole dataset, in which n isthe size of the whole dataset. Harnik et al. [36] utilizes thetheory to estimate the duplicates in storage systems.

C. Dynamic Flash Cache Management

Using prediction or historical information of workloadsto improve the cache efficiency has been explored in flashcache management [37]–[41]. These work studied cacheadmission policies and dynamic cache allocation to reducethe flash wear-out. To the best our knowledge, HPDedupis the first work to use locality estimation to deal with thecache contention problem in inline deduplication for primarystorage.

VII. CONCLUSION

In scenarios where multiple virtual machines or containersrunning in the cloud, many applications are placed in thesame physical machine. Removing duplicate I/Os from theprimary storage in these scenarios is useful to both improvethe capacity efficiency and I/O performance. We proposedHPDedup, a hybrid prioritized deduplication method for

primary storage in the cloud. HPDedup used a dynamictemporal locality estimation algorithm to achieve high inlinecache efficiency and left the relatively small number ofduplicates that were not in the cache to the post-processingdeduplication phase to handle. By doing so, HPDedupwas able to achieve exact deduplication in primary storagesystems. Comparing to the state-of-art inline deduplicationmethods, HPDedup significantly improved the inline cacheefficiency therefore achieved high inline deduplication ra-tio. HPDedup improves the inline deduplication ratio byup to 39.70% compared with iDedup in our experiments.Meanwhile, the improved cache efficiency made the post-processing deduplication process less a burden for the per-formance of an inline primary deduplication system. Forexample, HPDedup reduces up to 45.08% disk capacityrequirement compared with the state-of-art post-processingdeduplication mechanism in our evaluation.

ACKNOWLEDGMENT

We would like to thank Gregory Valiant from Stan-ford University for helping us to further understand theunseen entropy estimation algorithm. This work is par-tially supported by the The National Key Research andDevelopment Program of China (2016YFB0200401), byprogram for New Century Excellent Talents in Universityby National Science Foundation (NSF) China 61402492,61402486, 61379146, by the laboratory pre-research fund(9140C810106150C81001).

REFERENCES

[1] B. Zhu, K. Li, and R. H. Patterson, “Avoiding the diskbottleneck in the data domain deduplication file system.” inFAST, 2008.

[2] A. El-Shimi, R. Kalach, A. Kumar, A. Ottean, J. Li, andS. Sengupta, “Primary data deduplicationlarge scale study andsystem design,” in Presented as part of the 2012 USENIXAnnual Technical Conference (USENIX ATC 12), 2012, pp.285–296.

[3] D. T. Meyer and W. J. Bolosky, “A study of practicaldeduplication,” ACM Transactions on Storage (TOS), vol. 7,no. 4, p. 14, 2012.

[4] K. Srinivasan, T. Bisson, G. R. Goodson, and K. Voruganti,“iDedup: latency-aware, inline data deduplication for primarystorage.” in FAST, 2012.

[5] B. Mao, H. Jiang, S. Wu, and L. Tian, “Pod: Performanceoriented i/o deduplication for primary storage systems in thecloud,” in IEEE 28th International Parallel and DistributedProcessing Symposium (IPDPS), 2014.

[6] A. Wildani, E. L. Miller, and O. Rodeh, “Hands: A heuris-tically arranged non-backup in-line deduplication system,” inIEEE 29th International Conference on Data Engineering(ICDE), 2013.

[7] J. An and D. Shin, “Offline deduplication-aware block sep-aration for solid state disk,” in 11th USENIX Conference onFile and Storage Technologies (FAST 13), 2013.

[8] C. Constantinescu, J. Glider, and D. Chambliss, “Mixingdeduplication and compression on active data sets,” in DataCompression Conference (DCC), 2011. IEEE, 2011, pp.393–402.

[9] V. Tarasov, D. Jain, G. Kuenning, S. Mandal, K. Palanisami,P. Shilane, S. Trehan, and E. Zadok, “Dmdedup: Devicemapper target for data deduplication,” in 2014 Ottawa LinuxSymposium, 2014.

[10] H. Yu, X. Zhang, W. Huang, and W. Zheng, “Pdfs: Partiallydedupped file system for primary workloads,” IEEE Transac-tions on Parallel and Distributed Systems.

[11] J. Kaiser, T. Suß, L. Nagel, and A. Brinkmann, “Sorteddeduplication: How to process thousands of backup streams,”in Mass Storage Systems and Technologies (MSST), 2016 32thSymposium on. IEEE, 2016.

[12] D. Merkel, “Docker: lightweight linux containers for consis-tent development and deployment,” Linux Journal, vol. 2014,no. 239, p. 2, 2014.

[13] J. Rhoton, de Jan Clercq, and F. Novak, OpenStack CloudComputing. Recursive Press, 2014.

[14] J. Paulo and J. Pereira, “Efficient deduplication in a dis-tributed primary storage infrastructure,” ACM Transactionson Storage (TOS), vol. 12, no. 4, p. 20, 2016.

[15] H. Yu, X. Zhang, W. Huang, and W. Zheng, “Pdfs: Partiallydedupped file system for primary workloads,” IEEE Transac-tions on Parallel and Distributed Systems, 2016.

[16] R. Koller and R. Rangaswami, “I/O deduplication: Utilizingcontent similarity to improve i/o performance,” ACM Trans-actions on Storage (TOS), vol. 6, no. 3, 2010.

[17] W. Li, G. Jean-Baptise, J. Riveros, G. Narasimhan, T. Zhang,and M. Zhao, “Cachededup: in-line deduplication for flashcaching,” in 14th USENIX Conference on File and StorageTechnologies (FAST 16), 2016, pp. 301–314.

[18] J. S. Vitter, “Random sampling with a reservoir,” ACMTransactions on Mathematical Software (TOMS), vol. 11,no. 1, pp. 37–57, 1985.

[19] P. Valiant and G. Valiant, “Estimating the unseen: improvedestimators for entropy and other properties,” in Advances inNeural Information Processing Systems (NIPS), 2013.

[20] Y. Tang, J. Yin, S. Deng, and Y. Li, “Diode: Dynamicinline-offline de duplication providing efficient space-savingand read/write performance for primary storage systems,”in Modeling, Analysis and Simulation of Computer andTelecommunication Systems (MASCOTS), 2016 IEEE 24thInternational Symposium on. IEEE, 2016, pp. 481–486.

[21] Z. Sun, G. Kuenning, S. Mandal, P. Shilane, V. Tarasov,N. Xiao, and E. Zadok, “A long-term user-centric analysisof deduplication patterns,” in International Conference onMassive Storage Systems and Technology (MSST), 2016.

[22] S. Mandal, G. Kuenning, D. Ok, V. Shastry, P. Shilane,S. Zhen, V. Tarasov, and E. Zadok, “Using hints to improveinline block-layer deduplication,” in 14th USENIX Conferenceon File and Storage Technologies (FAST 16), 2016, pp. 315–322.

[23] N. Megiddo, “Linear programming in linear time when thedimension is fixed,” Journal of the ACM, vol. 31, no. 1, pp.114–127, 1984.

[24] N. Alon and N. Megiddo, “Parallel linear programming infixed dimension almost surely in constant time,” in The 31stAnnual Symposium on Foundations of Computer Science,1990.

[25] B. Lin, S. Li, X. Liao, J. Zhang, and X. Liu, “Leach: anautomatic learning cache for inline primary deduplicationsystem,” Frontiers of Computer Science, vol. 8, no. 2, pp.175–183, 2014.

[26] Y.-K. Li, M. Xu, C.-H. Ng, and P. P. Lee, “Efficient hybridinline and out-of-line deduplication for backup storage,” ACMTransactions on Storage (TOS), vol. 11, no. 1, p. 2, 2015.

[27] Y. Fu, H. Jiang, N. Xiao, L. Tian, F. Liu, and L. Xu,“Application-aware local-global source deduplication forcloud backup services of personal storage,” IEEE Transac-tions on Parallel and Distributed Systems, vol. 5, no. 25, pp.1155–1165, 2014.

[28] P. J. Haas, J. F. Naughton, S. Seshadri, and L. Stokes,“Sampling-based estimation of the number of distinct valuesof an attribute,” in VLDB, 1995.

[29] C. X. Mao and B. G. Lindsay, “Estimating the number ofclasses,” the Annals of Statistics, pp. 917–930, 2007.

[30] R. A. Fisher, A. S. Corbet, and C. B. Williams, “The relationbetween the number of species and the number of individualsin a random sample of an animal population,” The Journal ofAnimal Ecology, pp. 42–58, 1943.

[31] Z. Bar-Yossef, R. Kumar, and D. Sivakumar, “Samplingalgorithms: lower bounds and applications,” in The Thirty-Third Annual ACM Symposium on Theory of computing,2001.

[32] T. Batu, S. Dasgupta, R. Kumar, and R. Rubinfeld, “Thecomplexity of approximating the entropy,” SIAM Journal onComputing, vol. 35, no. 1, pp. 132–150, 2005.

[33] P. Valiant, “Testing symmetric properties of distributions,”SIAM Journal on Computing, vol. 40, no. 6, pp. 1927–1968,2011.

[34] S. Guha, A. McGregor, and S. Venkatasubramanian, “Stream-ing and sublinear approximation of entropy and informationdistances,” in The Seventeenth Annual ACM-SIAM Symposiumon Discrete algorithm, 2006.

[35] G. Valiant and P. Valiant, “Estimating the unseen: an n/log(n)-sample estimator for entropy and support size, shownoptimal via new clts,” in The Forty-Third Annual ACMSymposium on Theory of computing, 2011.

[36] D. Harnik, E. Khaitzin, and D. Sotnikov, “Estimating unseendeduplication-from theory to practice.” in FAST, 2016, pp.277–290.

[37] D. Arteaga, J. Cabrera, J. Xu, S. Sundararaman, and M. Zhao,“Cloudcache: On-demand flash cache management for cloudcomputing.” in 14th USENIX Conference on File and StorageTechnologies (FAST 16), 2016.

[38] J. Yang, N. Plasson, G. Gillis, and N. Talagala, “Hec: im-proving endurance of high performance flash-based cachedevices,” in Proceedings of the 6th International Systems andStorage Conference. ACM, 2013, p. 10.

[39] S. Huang, Q. Wei, D. Feng, J. Chen, and C. Chen, “Improvingflash-based disk cache with lazy adaptive replacement,” ACMTransactions on Storage (TOS), vol. 12, no. 2, p. 8, 2016.

[40] J. Liu, Y. Chai, X. Qin, and Y. Xiao, “Plc-cache: Endurablessd cache for deduplication-based primary storage,” in MassStorage Systems and Technologies (MSST), 2014 30th Sym-posium on. IEEE, 2014, pp. 1–12.

[41] R. Koller, A. J. Mashtizadeh, and R. Rangaswami, “Centaur:Host-side ssd caching for storage performance control,” inAutonomic Computing (ICAC), 2015 IEEE International Con-ference on. IEEE, 2015, pp. 51–60.