Heavy-Hi�er Detection Entirely in the Data PlaneVibhaalakshmi

SivaramanPrinceton University

Srinivas NarayanaMIT CSAIL

Ori RottenstreichPrinceton University

S. MuthukrishnanRutgers University

Jennifer RexfordPrinceton University

ABSTRACTIdentifying the “heavy hitter” �ows or �ows with largetra�c volumes in the data plane is important for severalapplications e.g., �ow-size aware routing, DoS detec-tion, and tra�c engineering. However, measurementin the data plane is constrained by the need for line-rate processing (at 10-100Gb/s) and limited memory inswitching hardware. We propose HashPipe, a heavy hit-ter detection algorithm using emerging programmabledata planes. HashPipe implements a pipeline of hashtables which retain counters for heavy �ows while evict-ing lighter �ows over time. We prototype HashPipe in P4and evaluate it with packet traces from an ISP backbonelink and a data center. On the ISP trace (which containsover 400,000 �ows), we �nd that HashPipe identi�es95% of the 300 heaviest �ows with less than 80KB ofmemory.

KEYWORDSSoftware-De�ned Networks; Network Monitoring; Pro-grammable Networks; Network Algorithms

1 INTRODUCTIONMany network management applications can bene�tfrom �nding the set of �ows contributing signi�cantamounts of tra�c to a link: for example, to relieve linkcongestion [5], to plan network capacity [18], to de-tect network anomalies and attacks [23], or to cacheforwarding table entries [36]. Further, identifying such“heavy hitters” at small time scales (comparable to tra�c

Permission to make digital or hard copies of all or part of this workfor personal or classroom use is granted without fee provided thatcopies are not made or distributed for pro�t or commercial advan-tage and that copies bear this notice and the full citation on the �rstpage. Copyrights for components of this work owned by others thanACM must be honored. Abstracting with credit is permitted. To copyotherwise, or republish, to post on servers or to redistribute to lists,requires prior speci�c permission and/or a fee. Request permissionsfrom permissions@acm.org.SOSR ’17, Santa Clara, CA© 2017 ACM. 978-1-4503-4947-5/17/04. . . $$15.00DOI: http://dx.doi.org/10.1145/3050220.3050239

variations [1, 5]) can enable dynamic routing of heavy�ows [16, 35] and dynamic �ow scheduling [41].

It is desirable to run heavy-hitter monitoring at allswitches in the network all the time, to respond quicklyto short-term tra�c variations. Can packets belonging toheavy �ows be identi�ed as the packets are processed inthe switch, so that switches may treat them specially?

Existing approaches to monitoring heavy items makeit hard to achieve reasonable accuracy at acceptableoverheads (§2.2). While packet sampling in the form ofNetFlow [12] is widely deployed, the CPU and band-width overheads of processing sampled packets in soft-ware make it infeasible to sample at su�ciently highrates (sampling just 1 in 1000 packets is common inpractice [34]). An alternative is to use sketches, e.g.,[14, 24, 25, 45] that hash and count all packets in switchhardware. However, these systems incur a large memoryoverhead to retrieve the heavy hitters — ideally, we wishto use memory proportional to the number of the heavy�ows (say the top hundred). There may be tens of thou-sands of active �ows any minute on an ISP backbonelink (§5) or a data center top-of-rack switch [4].

Emerging programmable switches [3, 7, 9] allow usto do more than sample, hash, and count, which sug-gests opportunities to run novel algorithms on switches.While running at line rates of 10-100 Gbps per portover 10-100 ports, these switches can be programmed tomaintain state over multiple packets, e.g., to keep �owidenti�ers as keys along with counts. Stateful manipu-lations can also be pipelined over multiple stages, withthe results carried by the packet from one stage to thenext and compared to stored state in subsequent stages.

However, switches are also constrained by the needto maintain high packet-processing throughput, having:

• a deterministic, small time budget (1 ns [7]) tomanipulate state and process packets at eachstage;

• a limited number of accesses to memory stor-ing state at each stage (typically just one read-modify-write);

• a limited amount of memory available per stage(e.g., 1.4MB shared across forwarding and moni-toring [7]);

• a need to move most packets just once throughthe pipeline, to avoid stalls and reduced through-put (“feed-forward” [19]).

We present HashPipe, an algorithm to track the kheaviest �ows with high accuracy (§3) within the fea-tures and constraints of programmable switches. Hash-Pipe maintains both the �ow identi�ers (“keys”) andcounts of heavy �ows in the switch, in a pipeline of hashtables. When a packet hashes to a location in the �rststage of the pipeline, its counter is updated (or initial-ized) if there is a hit (or an empty slot). If there is a miss,the new key is inserted at the expense of the existingkey. In all downstream stages, the key and count of theitem just evicted are carried along with the packet. Thecarried key is looked up in the current stage’s hash table.Between the key looked up in the hash table and the onecarried, the key with the larger count is retained in thehash table, while the other is either carried along withthe packet to the next stage, or totally removed from theswitch if the packet is at the last stage. Hence, HashPipe“smokes out” the heavy keys within the limited availablememory, using pipelined operation to sample multiplelocations in the hash tables, and evicting lighter keysin favor of heavier keys, with updates to exactly onelocation per stage.

We prototype HashPipe in P4 [6] (§4) and test it onthe public-domain behavioral switch model [32]. Weevaluate HashPipe with packet traces obtained from anISP backbone link (from CAIDA) and a data center, to-gether containing over 500 million packets. We showthat HashPipe can provide high accuracy (§5). In thebackbone link trace, HashPipe incurs less than 5% falsenegatives and 0.001% false positives when reporting 300heavy hitters (keyed by transport �ve-tuple) with just4500 counters (less than 80KB) overall, while the traceitself contains 400,000 �ows. The errors both in false neg-atives and �ow count estimations are lower among theheavier �ows in the topk . At 80KB of memory, HashPipehas 15% lower false negatives with respect to sampleand hold [17], and 3-4% with respect to an enhancedversion of the count-min sketch [14].

2 BACKGROUND ON HEAVY-HITTERDETECTION

2.1 Problem FormulationHeavy hitters. “Heavy hitters” can refer to all �owsthat are larger (in number of packets or bytes) thana fraction t of the total packets seen on the link (the“threshold-t” problem). Alternatively, the heavy hitterscan be the top k �ows by size (the “top-k” problem).Through the rest of this paper, we use the “top-k” de�-nition.

Flow granularity. Flows can be de�ned at various lev-els of granularity, such as IP address (i.e., host), transportport number (i.e., application), or �ve-tuple (i.e., trans-port connection). With a �ner-grained notion of �ows,the size and number of keys grows, requiring more bitsto represent the key and more entries in the data struc-ture to track the heavy �ows accurately.Accuracy. A technique for detecting heavy hitters mayhave false positives (i.e., reporting a non-heavy �ow asheavy), false negatives (i.e., not reporting a heavy �ow),or an error in estimating the sizes of heavy �ows. Theimpact of errors depends on the application. For exam-ple, if the switch performs load balancing in the dataplane, a few false negatives may be acceptable, espe-cially if those heavy �ows can be detected in the nexttime interval. As another example, if the network treatsheavy �ows as suspected denial-of-service attacks, falsepositives could lead to unnecessary alarms that over-whelm network operators. When comparing approachesin our evaluations, we consider all three metrics.Overhead. The overhead on the switch includes thetotal amount of memory for the data structure, the num-ber of matching stages used in the switch pipeline (con-strained by a switch-speci�c maximum, say 16 [7]). Thealgorithms are constrained by the nature and amount ofmemory access and computation per match stage (e.g.,computing hash functions). On high-speed links, thenumber of active �ve-tuple �ows per minute can easilybe in the tens of thousands, if not more. Our central goalis to maintain data-plane state that is proportional to thetarget number k of heavy hitters (e.g., 5k or 10k), ratherthan the number of active �ows. In addition, we wouldlike to use as few pipeline stages as possible, since theswitch also needs to support other functionality relatedto packet forwarding and access control.

2.2 Existing SolutionsThe problem of �nding heavy �ows in a network isan instance of the “frequent items” problem, which isextremely well studied in the streaming algorithms lit-erature [13]. While high accuracy and low overhead areessential to any heavy hitter detection approach, we arealso speci�cally interested in implementing these algo-rithms within the constraints of emerging programmableswitches (§1). We classify the approaches into two broadcategories, sampling and streaming, discussed below.Packet sampling using NetFlow [12] and sFlow [38]is commonly implemented in routers today. These tech-nologies record a subset of network packets by sampling,and send the sampled records to collectors for analysis.To keep packet processing overheads and data collec-tion bandwidth low, NetFlow con�gurations in practiceuse aggressively low sampling probabilities, e.g., 1% or

2

even 0.01% [30, 34]. Such undersampling can impact theestimation accuracy.

Sample and hold [17] enhances the accuracy of packetsampling by keeping counters for all packets of a �owin a �ow table, once a packet from that �ow is sam-pled. Designing large �ow tables1 for fast packet lookupis well-understood: hash table implementations are al-ready common in switches, for example in router for-warding tables and NetFlow [24]. However, it is chal-lenging to handle hash collisions when adding �owsto the �ow table, when a packet from a new �ow issampled. Some custom hardware solutions combine thehash table with a “stash” that contains the over�ow, i.e.,colliding entries, from the hash table [22]. But this intro-duces complexity in the switch pipeline, and typicallyinvolves the control plane to add entries into the stashmemory. Ignoring such complexities, we evaluate sam-ple and hold in §5 by liberally allowing the �ow table tolookup packets anywhere in a list of a bounded size.Streaming algorithms implement data structures withbounded memory size and processing time per packet,while processing every packet in a large stream of pack-ets in one pass. The algorithms are designed with prov-able accuracy-memory tradeo�s for speci�c statistics ofinterest over the packets. While these features make thealgorithms attractive for network monitoring, the spe-ci�c algorithmic operations on each packet determinefeasibility on switch hardware.

Sketching algorithms like count-min sketch [14] andcount sketch [10] use per-packet operations such ashashing on packet headers, incrementing counters athashed locations, and determining the minimum or me-dian among a small number of the counters that werehashed to. These operations can all be e�ciently im-plemented on switch hardware [45]. However, thesealgorithms do not track the �ow identi�ers of packets,and hash collisions make it challenging to “invert” thesketch into the constituent �ows and counters. Simpleworkarounds like collecting packets that have high �owcount estimates in the sketch could result in signi�cantbandwidth overheads, since most packets from heavy�ows will have high estimates.

Techniques like group testing [15], reversible sketches[37], and FlowRadar [24] can decode keys from hash-based sketches. However, it is challenging to read o�an accurate counter value for a packet in the switchpipeline itself since the decoding happens o� the fastpacket-processing path.

Counter-based algorithms [27, 29] focus on measur-ing the heavy items, maintaining a table of �ows and cor-responding counts. They employ per-counter increment1Large exact-match lookups are typically built with SRAM, as largeTCAMs are expensive.

and subtraction operations, but potentially all countersin the table are updated during some �ow insertions. Up-dating multiple counters in a single stage is challengingwithin the deterministic time budget for each packet.

Space saving is a counter-based algorithm that usesonly O (k ) counters to track k heavy �ows, achievingthe lowest memory usage possible for a �xed accuracyamong deterministic heavy-hitter algorithms—both the-oretically [28] and empirically [13]. Space saving onlyupdates one counter per packet, but requires �ndingthe item with the minimum counter value in the table.Unfortunately, scanning the entire table on each packet,or �nding the minimum in a table e�ciently, is not di-rectly supported on emerging programmable hardware(§1). Further, maintaining data structures like sortedlinked lists [28] or priority queues [41] requires multi-ple memory accesses within the per-packet time budget.However, as we show in §3, we are able to adapt thekey ideas of the space saving algorithm and combine itwith the functionality of emerging switch hardware todevelop an e�ective heavy hitter algorithm.

3 HASHPIPE ALGORITHMAs described in §2.2, HashPipe is heavily inspired bythe space saving algorithm [28], which we describe now(§3.1). In the later subsections (§3.2-§3.4), we describeour modi�cations to the algorithm to make it amenableto switch implementation.

3.1 Space Saving AlgorithmTo track the k heaviest items, space saving uses a tablewithm (which is O (k )) slots, each of which identi�es adistinct �ow key and its counter. All slots are initiallyempty, i.e., keys and counters are set to zero.

Algorithm 1: Space Saving algorithm [28]1 . Table T hasm slots, either containing (keyj ,valj )

at slot j ∈ {1, . . . ,m}, or empty. Incoming packethas key iKey.

2 if ∃ slot j in T with iKey = keyj then3 valj ← valj + 14 else5 if ∃ empty slot j in T then6 (keyj ,valj ) ← (iKey, 1)7 else8 r ← arдminj ∈{1, ...,m } (valj )

9 (keyr ,valr ) ← (iKey,valr + 1)10 end11 end

3

The algorithm is summarized in Algorithm 1.2 Whena packet arrives, if its corresponding �ow isn’t alreadyin the table, and there is space left in the table, spacesaving inserts the new �ow with a count of 1. If the�ow is present in the table, the algorithm updates thecorresponding �ow counter. However, if the table isfull, and the �ow isn’t found in the table, the algorithmreplaces the �ow entry that has the minimum countervalue in the table with the incoming packet’s �ow, andincrements this minimum-valued counter.

This algorithm has three useful accuracy properties [28]that we list here. Suppose the true count of �ow keyj isc j , and its count in the table isvalj . First, no �ow counterin the table is ever underestimated, i.e., valj ≥ c j . Sec-ond, the minimum value in the table valr is an upperbound on the overestimation error of any counter, i.e.,valj ≤ c j +valr . Finally, any �ow with true count higherthan the average table count, i.e., c j > C/m, will alwaysbe present in the table (here C is the total packet countadded into the table). This last error guarantee is par-ticularly useful for the threshold-heavy-hitter problem(§2.1), since by using 1/t counters, we can extract all�ows whose true count exceeds a threshold t of the totalcount. However, this guarantee cannot be extended di-rectly to the top-k problem, since due to the heavy-tailednature of tra�c [17], the kth heaviest �ow contributesnowhere close to 1/k of the total tra�c.

The operation of �nding the minimum counter inthe table (line 8)—possibly for each packet—is di�cultwithin switch hardware constraints (§2.2). In the fol-lowing subsections, we discuss how we incrementallymodify the space saving algorithm to run on switches.

3.2 Sampling for the Minimum ValueThe �rst simpli�cation we make is to look at the mini-mum of a small, constant number d of randomly chosencounters, instead of the entire table (in line 8, Algo-rithm 1). This restricts the worst-case number of mem-ory accesses per packet to a small �xed constant d . Themodi�ed version, HashParallel, is shown in Algorithm 2.The main change from Algorithm 1 is in the set of tableslots examined (while looking up or updating any key),which is now a set of d slots obtained by hashing theincoming key using d independent hash functions.

In e�ect, we sample the table to estimate a minimumusing a few locations. However, the minimum of just dslots can be far from the minimum of the entire table ofm slots. An in�ated value of the minimum could impactthe useful error guarantees of space saving (§3.1). Ourevaluations in §5.4 show that the distributions of the

2We show the algorithm for packet counting; it easily generalizes tobyte counts.

minimum of the entire table and of the subsample arecomparable.

However, Algorithm 2 still requires the switch to readd locations at once to determine their minimum, andthen write back the updated value. This necessitates aread-modify-write operation, involving d reads and onewrite anywhere in table memory, within the per-packettime budget. However, multiple reads to the same table(d > 1) are supported neither in switch programminglanguages today [6] nor in emerging programmableswitching chips [7]. Further, supporting multiple readsfor every packet at line rate requires multiported mem-ories with strict access-latency guarantees, which canbe quite expensive in terms of area and power [20, 44].

3.3 Multiple Stages of Hash TablesThe next step is to reduce the number of reads to mem-ory to facilitate operation at line rate. We split the countertable T into d disjoint tables, and we read exactly oneslot per table. The algorithm is exactly the same as Algo-rithm 2; except that now hash function hi returns onlyslots in the ith stage.

This design enables pipelining the memory accessesto the d tables, since di�erent packets can access di�er-ent tables at the same time. However, packets may needto make two passes through this pipeline: once to de-termine the counter with the minimum value among dslots, and a second time to update that counter. The sec-ond pass is possible through “recirculation” of packetsthrough the switching pipeline [2, 39, 42] with addi-tional metadata on the packet, allowing the switch toincrement the minimum-valued counter in the secondpass. However, the second pass is potentially needed forevery packet, and recirculating every packet can halvethe pipeline bandwidth available to process packets.

Algorithm 2: HashParallel: Sample d slots at once1 . Hash functions hi (iKey) → slots, i ∈ {1, . . . ,d }2 H = {h1 (iKey), . . . ,hd (iKey)}

3 if ∃ slot j ∈ H with iKey = keyj then4 valj ← valj + 15 else6 if ∃ empty slot j ∈ H then7 (keyj ,valj ) ← (iKey, 1)8 else9 r ← arдminj ∈H (valj )

10 (keyr ,valr ) ← (iKey,valr + 1)11 end12 end

4

3.4 Feed-Forward Packet ProcessingWe now alleviate the need to process a packet more thanonce through the switch pipeline, using two key ideas.Track a rolling minimum. We track the minimumcounter value seen so far (and its key) as the packettraverses the pipeline, by moving the counter and keythrough the pipeline as packet metadata. Emerging pro-grammable switches allow the use of such metadata tocommunicate results of packet processing between dif-ferent stages, and such metadata can be written to at anystage, and used for packet matching at a later stage [42].

Algorithm 3: HashPipe: Pipeline of d hash tables1 . Insert in the �rst stage2 l1 ← h1 (iKey)

3 if keyl1 = iKey then4 vall1 ← vall1 + 15 end processing6 end7 else if l1 is an empty slot then8 (keyl1 ,vall1 ) ← (iKey, 1)9 end processing

10 end11 else12 (cKey, cVal ) ← (keyl1 ,vall1 )

13 (keyl1 ,vall1 ) ← (iKey, 1)14 end15 . Track a rolling minimum16 for i ← 2 to d do17 l ← hi (cKey)

18 if keyl = cKey then19 vall ← vall + cVal

20 end processing21 end22 else if l is an empty slot then23 (keyl ,vall ) ← (cKey,CVal )

24 end processing25 end26 else if vall < cVal then27 swap (cKey, cVal ) with (keyl ,vall )

28 end29 end

As the packet moves through the pipeline, the switchhashes into each stage on the carried key, instead of hash-ing on the key corresponding to the incoming packet.If the keys match in the table, or the slot is empty, thecounter is updated in the usual way, and the key needsno longer to be carried forward with the packet. Other-wise, the keys and counts corresponding to the larger

of the counters that is carried and the one in the slotis written back into the table, and the smaller of thetwo is carried on the packet. We leverage arithmeticand logical action operations available in the match-action tables in emerging switches [7] to implement thecounter comparison. The key may be carried to the nextstage, or evicted completely from the tables when thepacket reaches the last (i.e., dth) stage.Always insert in the �rst stage. If the incoming keyisn’t found in the �rst stage in the pipeline, there is noassociated counter value to compare with the key thatis in that table. Here, we choose to always insert thenew �ow in the �rst stage, and evict the existing key andcounter into packet metadata. After this stage, the packetcan track the rolling minimum of the subsequent stagesin the usual way described above. The �nal algorithm,HashPipe, is shown in Algorithm 3.

One consequence of always inserting an incomingkey in the �rst stage is the possibility of duplicate keysacross di�erent tables in the pipeline, since the key canexist at a later stage in the pipeline. Note that this isunavoidable when packets only move once through thepipeline. It is possible that such duplicates may occupyspace in the table, leaving fewer slots for heavy �ows,and causing evictions of heavy �ows whose counts maybe split across the duplicate entries.

However, many of these duplicates are easily mergedthrough the algorithm itself, i.e., the minimum trackingmerges counters when the carried key has a “hit” inthe table. Further, switches can easily estimate the �owcount corresponding to any packet in the data planeitself by summing all the matching �ow counters; so cana data collector, after reading the tables out of the switch.We also show in our evaluations (§5.1) that duplicatesonly occupy a small portion of the table memory.

Fig. 1 illustrates an example of processing a packetusing HashPipe. A packet with a keyK enters the switchpipeline (a), and since it isn’t found in the �rst table, it isinserted there (b). Key B (that was in the slot currentlyoccupied by K) is carried with the packet to the nextstage, where it hashes to the slot containing key E. Butsince the count of B is larger than that of E, B is writtento the table and E is carried out on the packet instead (c).Finally, since the count of L (that E hashes to) is largerthan that of E, L stays in the table (d). The net e�ectis that the new key K is inserted in the table, and theminimum of the three keys B, E, and L—namely E—isevicted in its favor.

4 HASHPIPE PROTOTYPE IN P4We built a prototype of HashPipe using P4 version 1.1 [42].We veri�ed that our prototype produced the same resultsas our HashPipe simulator by running a small number of

5

(a) Initial state of table (b) New �ow is placed with value 1 in �rst stage

(c) B being larger evicts E (d) L being larger is retained in the tableFigure 1: An illustration of HashPipe.

1 action doStage1() {2 mKeyCarried = ipv4.srcAddr;3 mCountCarried = 0;4 modify_�eld_with_hash_based_o�set (mIndex, 0,

stage1Hash, 32) ;56 // read the key and value at that location7 mKeyTable = �owTracker[mIndex];8 mCountTable = packetCount[mIndex];9 mValid = validBit [mIndex];

1011 // check for empty location or di�erent key12 mKeyTable = (mValid == 0) ? mKeyCarried : mKeyTable;13 mDif = (mValid == 0) ? 0 : mKeyTable − mKeyCarried;1415 // update hash table16 �owTracker[mIndex] = ipv4 . srcAddr;17 packetCount[mIndex] = (mDif == 0) ? mCountTable+1: 1;18 validBit [mIndex] = 1;1920 // update metadata carried to the next table stage21 mKeyCarried = (mDif == 0) ? 0 : mKeyTable;22 mCountCarried = (mDif == 0) ? 0 : mCountTable;23 }24

Listing 1:HashPipe stagewith insertion of new�ow.Fields pre�xed with m are metadata �elds.

arti�cially generated packets on the switch behavioralmodel [32] as well as our simulator, and ensuring thatthe hash table is identical in both cases at the end of themeasurement interval.

At a high level, HashPipe uses a match-action stage inthe switch pipeline for each hash table. In our algorithm,each match-action stage has a single default action—thealgorithm execution—that applies to every packet. Everystage uses its own P4 register arrays—stateful memory

that persists across successive packets—for the hash ta-ble. The register arrays maintain the �ow identi�ersand associated counts. The P4 action blocks for the �rsttwo stages are presented in Listings 1 and 2; actionsfor further stages are identical to that of stage 2. Theremainder of this section walks through the P4 languageconstructs with speci�c references to their usage in ourHashPipe prototype.

Hashing to sample locations: The �rst step of the ac-tion is to hash on the �ow identi�er (source IP address inthe listing) with a custom hash function, as indicated inline 4 of Listing 1. The result is used to pick the locationwhere we check for the key. The P4 behavioral model[32] allows customized hash function de�nitions. Weuse hash functions of the type hi = (ai ·x +bi )%p wherethe chosen ai ,bi are co-prime to ensure independenceof the hash functions across stages. Hash functions ofthis sort are implementable on hardware and have beenused in prior work [24, 25].Registers to read andwrite�ow statistics:The �owsare tracked and updated using three registers: one totrack the �ow identi�ers, one for the packet count, andone to test validity of each table index. The result ofthe hash function is used to index into the registers forreads and writes. Register reads occur in lines 6-9 andregister writes occur in lines 15-18 of Listing 1. Whena �ow identi�er is read from the register, it is checkedagainst the �ow identi�er currently carried. Dependingon whether there is a match, either the value 1 is writtenback or the current value is incremented by 1.Packet metadata for tracking current minimum:The values read from the registers are placed in packetmetadata since we cannot test conditions directly onthe register values in P4. This enables us to compute

6

1 action doStage2{2 · · ·

3 mKeyToWrite = (mCountInTable < mCountCarried) ?mKeyCarried : mKeyTable));

4 �owTracker[mIndex] = (mDif == 0) ? mKeyTable :mKeyToWrite;

56 mCountToWrite = (mCountTable < mCountCarried) ?

mCountCarried : mCountTable;7 packetCount[mIndex] = (mDif == 0) ? (mCountTable +

mCountCarried) : mCountToWrite;89 mBitToWrite = (mKeyCarried == 0) ? 0 : 1) ;

10 validBit [mIndex] = (mValid == 0) ? mBitToWrite : 1) ;11 · · ·

12 }

Listing 2: HashPipe stage with rolling minimum.Fields pre�xed with m are metadata �elds.

the minimum of the carried key and the key in the tablebefore writing back into the register (lines 11-13 of List-ing 1 and lines 3, 6, and 9 of Listing 2). Packet metadataalso plays a crucial role in conveying state (the currentminimum �ow id and count, in this case) from one stageto another. The metadata is later used to compute thesample minimum. The updates that set these metadataacross stages are similar to lines 20-22 of Listing 1.Conditional state updates to retain heavier �ows:The �rst pipeline stage involves a conditional update toa register to distinguish a hit and a miss for the incomingpacket key in the table (line 17, Listing 1). Subsequentstages must also write back di�erent values to the tablekey and count registers, depending on the result of thelookup (hit/miss) and a comparison of the �ow counts.Accordingly, we perform a conditional write into the�ow id and count registers (lines 4 and 7, Listing 2). Suchconditional state updates are feasible at line rate [19, 40].

5 EVALUATIONWe now evaluate HashPipe through trace-driven sim-ulations. We tune the main parameter of HashPipe—the number of table stages d—in §5.1. We evaluate theperformance of HashPipe in isolation in §5.2, and thencompare it to prior sampling and sketching solutions in§5.3. Then, we examine the performance of HashPipein context of the idealized algorithms it is derived from(§3) in §5.4.Experiment setup. We compute the k heaviest �owsusing two sets of traces. The �rst trace is from a 10Gb/sISP backbone link, recorded in 2016 and available fromCAIDA [8]. We measure heavy hitters aggregated bytransport 5-tuple. The tra�c trace is 17 minutes long,and contains 400 million packets. We split this trace into50 chunks, each being about 20 seconds long, with 10

million packets. The chunks on average contain about400,000 5-tuple �ows each. Each chunk is one trial, andeach data point in the graphs for the ISP trace reportsthe average across 50 trials. We assume that the switchzeroes out its tables at the end of each trial, which cor-responds to a 20 second “table �ush” period.

The second trace, recorded in 2010, is from a datacenter [4] and consists of about 100 million packets intotal. We measure heavy hitters aggregated by sourceand destination IPs. We split the trace into 1 secondintervals corresponding to the time scale at which datacenter tra�c exhibits stability [5].

The data center trace is two and a half hours long,with roughly 10K packets (300 �ows) per second. We ad-ditionally replay the trace at two higher packet rates totest whether HashPipe can provide good accuracy overthe 1 second time scale. We assume a “typical” averagepacket size of 850 bytes and a 30% network utilization asreported in prior tra�c studies [4]. For a switch clockedat 1GHz with 48 ports of 10Gb/s each, this correspondsto roughly 20 million packets per second through theentire switch, and 410K packets per second through asingle link. At these packet rates, the trace contains20,000 and 3200 �ows per second respectively. For eachpacket rate, we report results averaged from multipletrials of 1 second each.Metrics. As discussed in §2.1, we evaluate schemes onfalse negatives (% heavy �ows that are not reported),false positives (% non-heavy �ows that are reported),and the count estimation error (% error for heavy �ows).Note that for the top-k problem, the false positive erroris just a scaled version of the false negative.

5.1 Tuning HashPipeGiven a total memory sizem, HashPipe’s only tunableparameter is the number of table stages d that it uses.Once d is �xed, we simply partition the available mem-ory equally into d hash tables. As d increases, the num-ber of table slots over which a minimum is computedincreases, leading to increased retention of heavier keys.However, with a �xed total memory, an increase in thevalue ofd decreases the per-stage hash table size, increas-ing the likelihood of hash collisions, and also of dupli-cates (§3.4). The switch hardware constrains the numberof table stages to a small number, e.g., 6-16 [7, 31].

Fig. 2 shows the impact of changing d on the falsenegatives. For the ISP trace, we plot false negatives fordi�erent sizes of memory m and di�erent number ofdesired heavy hitters k . As expected, the false negativesreduce as d increases starting at d = 2, but the decreasequickly tapers o� in the range between 5–8, across thedi�erent (m,k ) curves. For the data center trace, weshow false negatives for k = 350 with a memory of 840

7

0

5

10

15

20

25

30

2 3 4 5 6 7 8

Fals

e N

egative %

Number of table stages (d)

k = 140, m = 3360

k = 210, m = 5040

k = 350, m = 6720

k = 420, m = 8400

0

5

10

15

20

25

30

35

40

2 3 4 5 6 7 8

Fals

e N

egative %

Number of table stages (d)

Single Switch

Single Link

Actual Packet Rate

(a) ISP backbone (b) Data center

Figure 2: Impact of table stages (d) on false negatives.Error decreases as d grows, and then �attens out.

counters (15KB), and we see a similar trend across allthree packet rates. The false positives, elided here, alsofollow the same trend.

To understand whether duplicates impact the falsenegative rates, we also show the prevalence of duplicatesin HashPipe’s hash tables in Fig. 3. Overall, duplicatesonly take up between 5-10% of the available table sizein the ISP case and between 5-14% in the data centercase. As expected, in going from d = 2 and d = 8, theprevalence of duplicates in HashPipe’s table increases.

Fig. 4 shows the count estimation error (as a percent-age of actual �ow size) for �ows in HashPipe’s tables atthe end of each measurement interval, with a memorysize of 640 counters (11.2KB). In general, the error re-duces as d increases, but the reduction from d = 4 tod = 8 is less signi�cant than the reduction from d = 2to d = 4. In the ISP trace, the estimation error is stableacross �ow sizes since most �ows recorded by HashPipehave sizes of at least 1000. In the data center trace wherethere are fewer total �ows, there is a more apparentdecrease in error with true �ow size, with �ows of sizex > 1000 having near-perfect count estimations.Choosing d = 6 table stages. To summarize, we �ndthat (i) as the number of table stages increases aboved = 4, all the accuracy metrics improve; (ii) however, theimprovement dimishes at d = 8 and beyond, due to theincreasing prevalence of duplicates and hash collisions.These trends hold across both the ISP and data centerscenarios, for a variety of measured heavy hitters k andmemory sizes. Hence, we choose d = 6 table stages forall further experiments with HashPipe.

5.2 Accuracy of HashPipeWe plot error vs. memory tradeo� curves for HashPipe,run with d = 6 table stages. Henceforth, unless men-tioned otherwise, we run the data center trace at thesingle link packet rate.False negatives. Fig. 5 shows the false negatives asmemory increases, with curves corresponding to di�er-ent numbers of reported heavy hitters k . We �nd thaterror decreases with allocated memory across a range ofk values, and settles to under 10% in all cases on the ISP

0

1

2

3

4

5

6

7

0 10 20 30 40 50 60 70

Duplic

ate

Entr

ies %

Memory (in KB)

d = 8d = 4d = 2

0

2

4

6

8

10

12

14

1 2 3 4 5 6 7

Duplic

ate

Entr

ies %

Memory (in KB)

d = 8d = 4d = 2

(a) ISP backbone (b) Data center (link)

Figure 3: Prevalence of duplicate keys in tables. For dif-ferent d values under the memory range tested, the pro-portion of duplicates is between 5-10% for the ISP traceand 5-15% for the data center trace (link packet rate).

10

15

20

25

30

35

40

45

50

1 10 100 1000 10000

Avg e

stim

ation e

rror

(%)

in flo

ws o

f siz

e >

x

Actual Size of Flow (x)

d = 2d = 4d = 8

0

5

10

15

20

25

1 10 100 1000 10000

Avg e

stim

ation e

rror

(%)

in flo

ws o

f siz

e >

x

Actual Size of Flow (x)

d = 2d = 4d = 8

(a) ISP backbone (b) Data center (link)

Figure 4: Average estimation error (%) of �ows whosetrue counts are higher than the x-value in the graph. Er-ror decreases asd grows but the bene�t diminishes withincreasing d . Error decreases with actual �ow size.

trace at 80KB of memory, which corresponds to 4500counters. In any one trial with the ISP trace, there are onaverage 400,000 �ows, which is two orders of magnitudehigher than the number of counters we use. In the datacenter trace, the error settles to under 10% at just 9KB ofmemory (520 counters) for all the k values we tested.3

These results also enable us to understand the inter-play between k and the memory size required for a spe-ci�c accuracy. For a 5% false negative rate in the ISP trace,the memory required for k = 60 is 60KB (3375 ≈ 55kcounters), whereas the memory required for k = 300is 110KB (6200 ≈ 20k counters). In general, the factorof k required in the number of counters to achieve aparticular accuracy reduces as k increases.Which �ows are likely to be missed? It is natural toask which �ows are more likely missed by HashPipe.Fig. 6 shows how the false negative rate changes as thenumber of desired heavy hitters k is varied, for threedi�erent total memory sizes. We �nd that the heaviest�ows are least likely to bemissed, e.g.,with false negativesin the 1-2% range for the top 20 �ows, when using 3000counters in the ISP trace. This trend is intuitive, sinceHashPipe prioritizes the retention of larger �ows in thetable (§3.1). Most of the higher values of false negativeerrors are at larger values of k , meaning that the smaller

3The minimum memory required at k = 300 is 6KB (≈ 300 counters).8

0

5

10

15

20

25

0 20 40 60 80 100 120

Fals

e N

egative %

Memory(in KB)

k = 60k = 150k = 240k = 300

0

5

10

15

20

25

6 7 8 9 10 11 12

Fals

e N

egative %

Memory(in KB)

k = 60k = 150k = 210k = 300

(a) ISP backbone (b) Data center

Figure 5: False negatives of HashPipe with increasingmemory. Each trial trace from the ISP backbone con-tains an average of 400,000�ows, yetHashPipe achieves5-10% false negatives for the top 60-300 heavy hitterswith just 4500 �ow counters.

0

2

4

6

8

10

0 20 40 60 80 100 120 140 160

Fals

e N

egative %

Number of heavy hitters (k)

m = 1500m = 3000m = 4500

0

1

2

3

4

5

6

7

0 20 40 60 80 100 120 140 160

Fals

e N

egative %

Number of heavy hitters (k)

m = 300m = 720

m = 1080

(a) ISP backbone (b) Data center

Figure 6: False negatives against di�erent numbers ofreported heavy �ows k . The heavier a �ow is, the lesslikely that it is missed.

0

0.005

0.01

0.015

0.02

0.025

0.03

0 20 40 60 80 100 120

Fals

e P

ositiv

e %

Memory (in KB)

k = 300k = 240k = 150k = 60

0

0.5

1

1.5

2

2.5

6 7 8 9 10 11 12

Fals

e P

ositiv

e %

Memory (in KB)

k = 300k = 210k = 150k = 60

(a) ISP backbone (b) Data center

Figure 7: HashPipe has false positive rates of 0.01%-0.1%across a range of table sizes and reported heavy hittersin the ISP trace, and under 3% in the data center trace.

of the top k �ows are more likely to be missed amongthe reported �ows.False positives. Fig. 7 shows the false positives of Hash-Pipe against varying memory sizes, exhibiting the natu-ral trend of decreasing error with increasing memory. Inparticular, we �nd that with the ISP trace, false positiverates are very low, partly owing to the large numberof small �ows. On all curves, the false positive rate issmaller than 0.1%, dropping to lower than 0.01% at atable size of 80KB. In the data center trace, we �nd thatthe false positive rate hovers under 3% over a range ofmemory sizes and heavy hitters k .

In summary, HashPipe performs well both with theISP backbone and data center traces, recognizing heavy-hitter �ows in a timely manner, i.e., within 20 secondsand 1 second respectively, directly in the data plane.

5.3 HashPipe vs. Existing SolutionsComparison baselines.We compare HashPipe againsttwo schemes—representative of sampling and sketching—which are able to estimate counts in the switch directly(§2.2). We use the same total memory as earlier, andmeasure the top k = 150 �ows. We use the ISP backbonetrace henceforth (unless mentioned otherwise), and com-pare HashPipe with the following baseline schemes.(1) Sample and Hold: We simulate sample and hold [17]with a �ow table that is implemented as a list. As de-scribed in §2.2, we liberally allow incoming packets tolook up �ow entries anywhere in the list, and add newentries up to the total memory size. The sampling prob-ability is set according to the available �ow table size,following recommendations from [17].(2) Count-min sketch augmented with a ‘heavy �ow’cache: We simulate the count-min sketch [14], but use a�ow cache to keep �ow keys and exact counts start-ing from the packet where a �ow is estimated to beheavy from the sketch.4 We liberally allow the count-min sketch to use the o�ine-estimated exact count ofthe kth heaviest �ow in the trace, as a threshold to iden-tify heavy �ows. The �ow cache is implemented as ahash table that retains only the pre-existing �ow key ona hash collision. We allocate half the available memoryeach to the sketch and the �ow cache.5

0

10

20

30

40

50

0 20 40 60 80 100 120

Fals

e N

egative %

Memory (in KB)

Sample and Hold

Count-Min + Cache

HashPipe

Figure 8: False negatives of HashPipe and other base-lines. HashPipe outperforms sample and hold andcount-min sketch over the entire memory range.

4A simpler alternative—mirroring packets with high size estimatesfrom the count-min sketch—requires collecting ≈ 40% of the packetsin the data center trace played at the link rate.

5We follow guidance from [17, page 288] to split the memory.9

False negatives. Fig. 8 shows false negatives againstvarying memory sizes, for the three schemes compared.We see that HashPipe outperforms sample and hold aswell as the augmented count-min sketch over the entirememory range. (All schemes have smaller errors as thememory increases.) Notably, at 100KB memory, Hash-Pipe has 15% smaller false negative rate than sampleand hold. The count-min sketch tracks the error rateof HashPipe more closely from above, staying within a3-4% error di�erence. Next, we understand where theerrors occur in the baseline schemes.Where are the errors in the other baselines? Fig. 9shows the count estimation error (%) averaged across�ows whose true counts are higher than the x-value,when running all the schemes with 26KB memory.

Sample and hold can make two kinds of estimationerrors. It can miss the �rst set of packets from a heavy�ow because of not sampling the �ow early enough, or(less likely) miss the �ow entirely due to not samplingor the �ow table becoming full. Fig. 9 shows that sam-ple and hold makes the former kind of error even for�ows with very high true counts. For example, there arerelative errors of about 10% even for �ows of size morethan 80,000 packets. As Fig. 8 shows, the errors becomesless prominent as the memory size increases, since thesampling rate increases too.

The count-min sketch makes errors because of itsinability to discriminate between heavy and light �owsduring hash collisions in the sketch. This means that alight �ow colliding with heavy �ows may occupy a �owcache entry, preventing a heavier �ow later on fromentering the �ow cache. For instance in Fig. 9, even�ows as large as 50,000 packets can have estimationerrors close to 20%. However, as Fig. 8 shows, the e�ectof hash collisions becomes less signi�cant as memorysize increases.

On the other hand, HashPipe’s average error on �owslarger than 20,000 packets—which is 0.2% of the totalpackets in the interval—is negligible (Fig. 9). HashPipehas 100% accuracy in estimating the count of �ows largerthan 30,000 packets.

5.4 HashPipe vs. Idealized SchemesWe now compare HashPipe against the idealized algo-rithms it is derived from (§3), namely space saving [28]and HashParallel.

There are two reasons why HashPipe may do badlyrelative to space saving: (i) it may evict a key whosecount is much higher than the table minimum, hencemissing heavy items from the table, and (ii) it may allowtoo many duplicate �ow keys in the table (§3.4), reduc-ing the memory available for heavy �ows, and evictheavy �ows whose counts are underestimated due to

0

10

20

30

40

50

60

70

0 20000 40000 60000 80000 100000 120000

Avera

ge e

stim

ation e

rror(

%)

in flo

ws o

f siz

e >

x

Actual Size of Flow (x)

Sample and Hold

Count-Min + Cache

HashPipe

Figure 9: Comparison of average estimation error (%) of�ows whose true counts are higher than the x-value inthe graph. Sample and hold underestimates heavy�owsdue to sampling, and count-min sketch misses heavy�ows due to lighter �ows colliding in the �ow cache.HashPipe has no estimation errors for �ows larger than30,000 packets.

the duplicates. We showed that duplicate keys are notvery prevalent in §5.1; in what follows, we understandtheir e�ects on false negatives in the reported �ows.Howfar is the subsampledminimumfrom the truetable minimum? Fig. 10 shows the complementaryCDF of the minimum count that was chosen by Hash-Pipe, obtained by sampling the algorithm’s choice ofminimum at every 100th packet in a 10 million packettrace. We don’t show the corresponding CCDF for theabsolute minimum in the table, which only takes ontwo values—0 and 1—with the minimum being 1 morethan 99% likely.6 We see that the chance that HashPipechooses a certain minimum value decreases rapidly asthat value grows, judging from the straight-line plot onlog scales in Fig. 10. For example, the minimum counterhas a value higher than 5 less than 5% of the time. Thereare a few larger minimum values (e.g., 100), but they arerarer (less than 0.01% of the time). This suggests that itis unlikely that HashPipe experiences large errors dueto the choice of a larger minimum from the table, whena smaller counter exists.Comparison against space saving and HashParal-lel. We compare the false negatives of the idealizedschemes and HashPipe against varying memory sizes inFig. 11 and Fig. 12, when reporting two di�erent num-ber of heavy hitters, k=150 and k=60 respectively. Twofeatures stand out from the graph for both k values.First, wherever the schemes operate with low false neg-ative error (say less than 20%), the performance of thethree schemes is comparable (i.e., within 2-3% of each

6Note that this is di�erent from the minimum of space saving, whosedistribution contains much larger values.

10

1e-06

1e-05

0.0001

0.001

0.01

0.1

1

1 10 100 1000

CC

DF

(P

r(m

inim

um

>=

x))

Value of minimum (x)

HashPipe Minimum

Figure 10: Complementary CDF of the minimum cho-sen by HashPipe on 26KB of memory, sampled every100th packet from a trace of 10 million packets.

0

5

10

15

20

25

0 20 40 60 80 100 120

Fals

e N

egative %

Memory (in KB)

Space Saving

HashParallel

HashPipe

Figure 11: Comparison of false negatives of HashPipeto idealized schemes when detecting k=150 heavy hit-ters. In somememory regimes, HashPipemay even out-perform space saving,whileHashParallel closely tracksspace saving.

0

5

10

15

20

25

0 10 20 30 40 50 60 70 80

Fals

e N

egative %

Memory (in KB)

Space Saving

HashParallel

HashPipe

Figure 12: Comparison of false negatives ofHashPipe toidealized schemes when detecting k=60 heavy hitters.

other). Second, there are small values of memory whereHashPipe outperforms space saving.

Why is HashPipe outperforming space saving? Spacesaving only guarantees that the kth heaviest item is inthe table when the kth item is larger than the averagetable count (§3.1). In our trace, the 60th item and 150thitem contribute to roughly 6000 and 4000 packets outof 10 million (resp.), which means that they requireat least7 1700 counters and 2500 counters in the table(resp.). These correspond to memory sizes of 30KB and45KB (resp.). At those values of memory, we see thatspace saving starts outperforming HashPipe on falsenegatives.

We also show why space saving fails to capture theheavier �ows when it is allocated a number of coun-ters smaller than the minimum number of countersmentioned above. Note that HashPipe attributes everypacket to its �ow entry correctly (but may miss somepackets entirely), since it always starts a new �ow atcounter value 1. However, space saving increments somecounter for every incoming packet (Algorithm 1). In con-texts where the number of active �ows is much largerthan the number of available counters (e.g., 400,000 �owswith 1200 counters), this can lead to some �ows havingenormously large (and grossly overestimated) counters.In contrast, HashPipe keeps the counter values small forsmall �ows by evicting the �ows (and counts) entirelyfrom the table.

At memory sizes smaller than the thresholds men-tioned above, incrementing a counter for each packetmay result in several small �ows catching up to a heavy�ow, leading to signi�cant false positives, and higherlikelihood of evicting truly heavy �ows from the table.We show this e�ect on space saving in Fig. 13 for k=150and m=1200 counters, where in fact m = 2500 coun-ters are required as described earlier. The distribution ofthe number of keys contributing to a false positive �owcounter in the table is clearly shifted to the right relativeto the corresponding distribution for a true positive.Impact of duplicate keys in the table. Finally, weinvestigate how duplicate keys and consequent underes-timation of �ow counts in the table may a�ect the errorsof HashPipe. In Fig. 14, we show the bene�ts of reportingmore than k counters on false negatives, when the topk=300 �ows are requested with a memory size ofm=2400counters. While the false negative rates of space savingand HashParallel improve signi�cantly with overreport-ing, the errors for HashPipe remains �at throughoutthe interval, dropping only around 1800 reported �ows.We infer that most heavy �ows are retained somewherein the table for space saving and HashParallel, whileHashPipe underestimates keys su�ciently often that

710 million / 6000 ≈ 1700; 10 million / 4000 ≈ 2500.11

0

0.2

0.4

0.6

0.8

1

0 500 1000 1500 2000 2500 3000 3500 4000

CD

F

Number of keys contibuting to a bucket

Neither True Positive Nor False Positive

True Positive

False Positive

Figure 13: CDF of the number of distinct �ows con-tributing to a counter in space saving’s table, when iden-tifying k=150 heavy �ows with m=1200 counters. Weshow three distributions according to the �ow’s labelafter the experiment.

they are completely evicted—to the point where overre-porting does not lower the false negative errors. We �ndthat overreporting �ows only increases the false posi-tive errors slightly for all schemes, with values stayingbetween 0.1-0.5% throughout.

0

5

10

15

20

25

200 400 600 800 1000 1200 1400 1600 1800 2000 2200

Fals

e N

egative %

Number of flows reported

HashPipe

HashParallel

Space Saving

Figure 14: Bene�ts of overreporting �ows on false nega-tive errorswithk=300 heavy�ows andm=2400 counters.While space saving and HashParallel improve signi�-cantly by reporting even 2k �ows, HashPipe does not,because of evictions due to duplicate entries.

6 RELATEDWORKApplications that use heavy hitters. Several applica-tions use information about heavy �ows to do bettertra�c management or monitoring. DevoFlow [16] andPlanck [35] propose exposing heavy �ows with lowoverhead as ways to provide better visibility and reducecongestion in the network. UnivMon [25] uses a top-kdetection sketch internally as a subroutine in its “univer-sal” sketch, to determine more general statistics about

the network tra�c. There is even a P4 tutorial applica-tion on switch programming, that performs heavy hitterdetection using a count-min-sketch-like algorithm [11].Measuring per-�ow counters. Prior works such asFlowRadar [24] and CounterBraids [26] have proposedschemes to measure accurate per-�ow tra�c counters.Along similar lines, hashing schemes like cuckoo hash-ing [33] and d-left hashing [43] can keep per-�ow statememory-e�ciently, while providing fast lookups on thestate. Our goal is not to measure or keep all �ows; justthe heavy ones. We show (§5) that HashPipe uses 2-3orders of magnitude smaller memory relative to havingper-�ow state for all active �ows, while catching morethan 90% of the heavy �ows.Other heavy-hitter detection approaches.The multi-resolution tiling technique in ProgME [46], and the hier-archical heavy-hitter algorithm of Jose et al. [21] solve asimilar problem as ours. They estimate heavy hitters intra�c online, by iteratively “zooming in” to the portionsof tra�c which are more likely to contain heavy �ows.However, these algorithms involve the control plane inrunning their �ow-space-partitioning algorithms, whileHashPipe works completely within the switch pipeline.Further, both prior approaches require temporal stabil-ity of the heavy-hitter �ows to detect them over multi-ple intervals; HashPipe determines heavy hitters usingcounters maintained in the same interval.

7 CONCLUSIONIn this paper, we proposed an algorithm to detect heavytra�c �ows within the constraints of emerging pro-grammable switches, and making this information avail-able within the switch itself, as packets are processed.Our solution, HashPipe, uses a pipeline of hash tablesto track heavy �ows preferentially, by evicting lighter�ows from switch memory over time. We prototypeHashPipe with P4, walking through the switch pro-gramming features used to implement our algorithm.Through simulations on a real tra�c trace, we showedthat HashPipe achieves high accuracy in �nding heavy�ows within the memory constraints of switches today.

ACKNOWLEDGMENTSWe thank the anonymous SOSR reviewers, AnirudhSivaraman, and Bharath Balasubramanian for their feed-back. This work was supported partly by NSF grant CCF-1535948, DARPA grant HR0011-15-2-0047, and gifts fromIntel and Huawei. We also thank the industrial membersof the MIT Center for Wireless Networks and MobileComputing (Wireless@MIT) for their support.

12

REFERENCES[1] M. Al-Fares, S. Radhakrishnan, B. Raghavan, N. Huang, and

A. Vahdat. Hedera: Dynamic �ow scheduling for data centernetworks. In USENIX NSDI, 2010.

[2] Arista Networks. Arista 7050x switch architecture, 2014.https://www.corporatearmor.com/documents/Arista_7050X_Switch_Architecture_Datasheet.pdf.

[3] Barefoot Networks. Barefoot To�no. https://www.barefootnetworks.com/technology/.

[4] T. Benson, A. Akella, and D. A. Maltz. Network tra�c character-istics of data centers in the wild. In ACM IMC, 2010.

[5] T. Benson, A. Anand, A. Akella, and M. Zhang. MicroTE: Finegrained tra�c engineering for data centers. In ACM CoNEXT,2011.

[6] P. Bosshart, D. Daly, G. Gibb, M. Izzard, N. McKeown, J. Rexford,C. Schlesinger, D. Talayco, A. Vahdat, G. Varghese, and D. Walker.P4: Programming protocol-independent packet processors. ACMSIGCOMM Computer Communication Review, 44(3):87–95, 2014.

[7] P. Bosshart, G. Gibb, H.-S. Kim, G. Varghese, N. McKeown, M. Iz-zard, F. Mujica, and M. Horowitz. Forwarding metamorphosis:Fast programmable match-action processing in hardware forSDN. In ACM SIGCOMM, 2013.

[8] CAIDA. The CAIDA UCSD Anonymized Internet Traces2016 - March. http://www.caida.org/data/passive/passive_2016_dataset.xml.

[9] Cavium. Cavium and XPliant Introduce a Fully ProgrammableSwitch Silicon Family Scaling to 3.2 Terabits per Second. http://cavium.com/newsevents- Cavium-and-XPliant-Introduce-a-Fully-Programmable-Switch-Silicon-Family.html.

[10] M. Charikar, K. Chen, and M. Farach-Colton. Finding frequentitems in data streams. In Springer ICALP, 2002.

[11] S. Choi. Implementing Heavy-Hitter Detection, 2016.https://github.com/p4lang/tutorials/tree/master/SIGCOMM_2016/heavy_hitter.

[12] Cisco Networks. Net�ow. http://www.cisco.com/c/en/us/products/ios-nx-os-software/ios-net�ow/index.html.

[13] G. Cormode and M. Hadjieleftheriou. Finding frequent items indata streams. In VLDB Endowment, 2008.

[14] G. Cormode and S. Muthukrishnan. An improved data streamsummary: The count-min sketch and its applications. Journal ofAlgorithms, 55(1):58–75, 2005.

[15] G. Cormode and S. Muthukrishnan. What’s hot and what’s not:Tracking most frequent items dynamically. ACM Trans. DatabaseSyst., 30(1), 2005.

[16] A. R. Curtis, J. C. Mogul, J. Tourrilhes, P. Yalagandula, P. Sharma,and S. Banerjee. DevoFlow: Scaling �ow management for high-performance networks. In ACM SIGCOMM, 2011.

[17] C. Estan and G. Varghese. New directions in tra�c measurementand accounting. ACM Trans. Computer Systems, 21(3), 2003.

[18] A. Feldmann, A. Greenberg, C. Lund, N. Reingold, J. Rexford, andF. True. Deriving tra�c demands for operational IP networks:Methodology and experience. In ACM SIGCOMM, 2000.

[19] V. Jeyakumar, M. Alizadeh, Y. Geng, C. Kim, and D. Mazières.Millions of little minions: Using packets for low latency networkprogramming and visibility. In ACM SIGCOMM, 2014.

[20] John Wawrzynek and Krste Asanovic and John Lazzaro andYunsup Lee. Memory (lecture), 2010. https://inst.eecs.berkeley.edu/~cs250/fa10/lectures/lec08.pdf.

[21] L. Jose, M. Yu, and J. Rexford. Online measurement of largetra�c aggregates on commodity switches. In USENIX Hot-ICE,2011.

[22] M. Kumar and K. Prasad. Auto-learning of MAC addresses andlexicographic lookup of hardware database. US Patent App.

10/747,332.[23] A. Lakhina, M. Crovella, and C. Diot. Characterization of

network-wide anomalies in tra�c �ows. In ACM IMC, 2004.[24] Y. Li, R. Miao, C. Kim, and M. Yu. FlowRadar: A better NetFlow

for data centers. In USENIX NSDI, 2016.[25] Z. Liu, A. Manousis, G. Vorsanger, V. Sekar, and V. Braverman.

One sketch to rule them all: Rethinking network �ow monitoringwith UnivMon. In ACM SIGCOMM, 2016.

[26] Y. Lu, A. Montanari, B. Prabhakar, S. Dharmapurikar, and A. Kab-bani. Counter braids: A novel counter architecture for per-�owmeasurement. In ACM SIGMETRICS, 2008.

[27] G. S. Manku and R. Motwani. Approximate frequency countsover data streams. In VLDB Endowment, 2002.

[28] A. Metwally, D. Agrawal, and A. El Abbadi. E�cient computationof frequent and top-k elements in data streams. In InternationalConference on Database Theory. Springer, 2005.

[29] J. Misra and D. Gries. Finding repeated elements. Science ofcomputer programming, 2(2):143–152, 1982.

[30] NANOG mailing list. SFlow vs NetFlow/IPFIX.https://mailman.nanog.org/pipermail/nanog/2016-February/thread.html#84418.

[31] R. Ozdag. Intel Ethernet Switch FM6000 Series - Soft-ware De�ned Networking. http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/ethernet-switch-fm6000-sdn-paper.pdf.

[32] P4 Language Consortium. P4 Switch Behavioral Model. https://github.com/p4lang/behavioral-model.

[33] R. Pagh and F. F. Rodler. Cuckoo hashing. J. Algorithms,51(2):122–144, May 2004.

[34] P. Phaal. SFlow sampling rates. http://blog.s�ow.com/2009/06/sampling-rates.html.

[35] J. Rasley, B. Stephens, C. Dixon, E. Rozner, W. Felter, K. Agarwal,J. Carter, and R. Fonseca. Planck: Millisecond-scale monitoringand control for commodity networks. In ACM SIGCOMM, 2014.

[36] O. Rottenstreich and J. Tapolcai. Optimal rule caching and lossycompression for longest pre�x matching. IEEE/ACM Trans. Netw.,2017.

[37] R. Schweller, A. Gupta, E. Parsons, and Y. Chen. Reversiblesketches for e�cient and accurate change detection over net-work data streams. In ACM IMC, 2004.

[38] SFlow. http://s�ow.org/.[39] Simon Horman. Packet recirculation, 2013. https://lwn.net/

Articles/546476/.[40] A. Sivaraman, A. Cheung, M. Budiu, C. Kim, M. Alizadeh, H. Bal-

akrishnan, G. Varghese, N. McKeown, and S. Licking. Packettransactions: High-level programming for line-rate switches. InACM SIGCOMM, 2016.

[41] A. Sivaraman, S. Subramanian, M. Alizadeh, S. Chole, S.-T.Chuang, A. Agrawal, H. Balakrishnan, T. Edsall, S. Katti, andN. McKeown. Programmable packet scheduling at line rate. InACM SIGCOMM, 2016.

[42] The P4 Language Consortium. The P4 Language Speci�cation,Version 1.1.0 - Release Candidate, January 2016. http://p4.org/wp-content/uploads/2016/03/p4_v1.1.pdf.

[43] B. Vöcking. How asymmetry helps load balancing. Journal ofthe ACM (JACM), 50(4):568–589, 2003.

[44] N. Weste and D. Harris. CMOS VLSI Design: A Circuits andSystems Perspective. Addison-Wesley, 2010.

[45] M. Yu, L. Jose, and R. Miao. Software de�ned tra�c measurementwith OpenSketch. In USENIX NSDI, 2013.

[46] L. Yuan, C.-N. Chuah, and P. Mohapatra. ProgME: Towardsprogrammable network measurement. In ACM SIGCOMM, 2007.

13