Kenji SHIMIZUNTT Network Innovation Labs.

This work was partially supported by the National Institute of Information and Communications Technology.

1

The 1

6th A

sia-Pa

cific C

onfe

rence

on C

om

munica

tions,

Nov. 1

st, 20

10

Available Bandwidth Estimation for Gb/s-Class Applications Using 10-

Gb/s Network Interface Cards’ Hardware Assistance

OUR MOTIVATION: NETWORK MEASUREMENT

2

We have engaged for several years in R&D of high-quality video sharing-over-IP-network technologies including 1.5-/6.4-Gbps uncompressed HDTV/4K video streaming system.

Video transmitting PC

Video transmitting PC

Video Receiving PC

Video Receiving PC

Video archiving storage

Video archiving storage

Packet droppings because of traffic bustiness

Traffic congestion at switches

Degraded traffic characteristics because of misconfiguration of routers

Trouble!!

Trouble!!

Trouble!!

There are a lot of troubles when and after starting the high-quality streaming service. Therefore, we have to…

1. make sure the network quality whether it meet the application’s requirements.

2. keep on observing the network quality during the services to avoid troubles.

OUR ONGOING PROJECT: PRESTA 10G

PRESTA 10G offers high-speed packet processing and traffic monitoring functions for 10-Gb/s IP networks, based on PRESTA 10G network interface card (NIC) and an off-the-shelf PC with Linux OS. Less expensive but high-performance systems Flexible software development while taking advantage of the hardware

assists. We can deploy lots of high-performance monitoring systems among the

network.

3

PRESTA: PRotocol Engine for Streaming Acceleration

PRESTA 10G NIC

Our key device: PRESTA 10G NIC 10-Gbps wire-rate packet capturing/generating. Well-controlled packet generation capable of sending up to 10

Gbps.• Inter-packet gaps can be configured in packet-by-packet manner in

nano-second resolution. Appends high-resolution and precise timestamps to all sent and

received packets generated in the NIC. Globally synchronized using GPS, and They have fine resolution of

100-nsec

GOAL

4

Implementing an available bandwidth estimation (ABE) system1. to make sure whether the network can satisfy the application requirement

before starting the service. 2. using an off-the-shelf PC because of it less-expensive advantage3. for 10-Gbps high-speed networks where Gb/s-class streams flows.

Available bandwidth estimation technique

Available bandwidth (Ba) is :Ba = Bl - Bc

Bl: Network physical bottleneck speedBc: Competing traffic bandwidth

Competing traffic

Traffic with pre-defined attributes (probe packets)

Resulting multiplexed traffic

When the incoming traffic bandwidth exceeds the available bandwidth, the pre-defined attributes are effected. Therefore, by detecting the effects while changing the probe packet’s attributes, we can determine the available bandwidth.

CHALLENGES

5

• Our scope does not include development of new ABE algorithm but include development of how to apply existing ABE algorithm to 10-Gb/s networks for Gb/s-class applications.

Points:It increases Inter-packet gaps in packet trains gradually.The algorithm is terminated when observed inter-packet gaps go through the network under test without any inter-packet gap changes.

Competing traffic bandwidth

The ABE results

Observation of inter-packet gaps of packet trains.

Network under test

An ABE system (TX)

An ABE system (RX)

Packet train

IGI algorithm : Initial Gap Increasing

：Network bottleneck speed：Inter-packet gaps when IGI was terminated：Inter-packet gaps corresponding to the

bottleneck speed：Inter-packet gaps satisfying

Difficulties:1. How to control the inter-packet gaps2. How to observe the inter-packet gaps

accurately

6

How to control the inter-packet gaps to generate probe packets with arbitrary IPGs

HARDWARE ASSISTANCE FOR OUR ABE SYSTEM(1)

We have two difficulties to accomplish it based on an off-the-shelf PC.1. Generating minimum inter-packet gaps corresponding to 10-Gb/s wire rate.

• The maximum probe-packet generating bit rate degrades because of various reasons.• CPU performance shortage, bus bandwidth limitation and software scheduling which are worsened

when the probe packet sizes are relatively short.

2. Controlling inter-packet gaps at the resolution of some nano-second.• General-purpose operating systems schedule all execution at the resolution of some milli-second.

1. Software only passes the packet header (hatched) part to reduce the amount of data which software processes.

2. Our NIC appends padding data to generate probe packets in an arbitrary length.

1. Software just passes all the packet data stored in a single buffer to our NIC all at once.

2. Our NIC splits each packet, then send them while controlling the inter-packet gaps at 5-nano second resolution.

7

HARDWARE ASSISTANCE FOR OUR ABE SYSTEM(2)How to observe inter-packet gaps accurately to detect the inter-packet gaps changes.

Two difficulties1. The timestamps on an off-the-shelf PC are not so accurate.

Generating timestamps based on on-board hardware clocks gives accurate measurement results.2. Packet droppings in the receiver side makes inter-packet gap calculation inaccurate.

Three steps to capture packets with their arrival timestamps1. Our NIC extracts only the header parts of the packets discarding the remains.2. Our NIC appends hardware-generated timestamps to the extracted headers.3. After collecting multiple pairs of headers and timestamps, our NIC makes single

data blocks containing multiple pairs for efficient transfer from NIC’s memory to the PC’s memory.

Series of captured packets in PRESTA 10G NIC

3. Our NIC makes single data block

1.Our NIC discards the unnecessary parts for ABE analysis.

2. Our NIC appends hardware-generated timestamps.

EVALUATION

8

Using PRESTA 10G hardware-assisted ABE systems and software ABE systems, we calculated available bandwidth using IGI (Initial Gap Increasing) algorithm.

ConfigurationNetwork Switches

Competing traffic

ABE systems

Shared 10-Gb/s link

Competing traffic

Constant bit rate traffic

Elapsed time

Ba

nd

wid

th

Type 1 Type 2 Uncompressed HDTV stream (bursty)

RESULTS (1/3)

9

Actual available bandwidth [Mb/s]

Est

ima

tion

re

sults

[M

b/s]Actual

available bandwidth

Software

Hardware-assisted

Comparison between newly developed hardware-assisted ABE and software-based ABE

1. System based on software implementation only gives ABE results not exceeding 1-Gb/s available bandwidth.

2. Our ABE systems gives further accurate results even if the actual available bandwidth are up-to 10-Gb/s bandwidth.

RESULTS (2/3)

10

Degraded estimation accuracy when the competing traffic (uncompressed HDTV stream) was bursty.

1. When the competing traffic was a realistic HDTV streaming traffic, a larger number of probe packets were required for getting nearly accurate ABE results.

2. This is because the probe packets can fall in the instant bandwidth drops with small amount of probe packets.

RESULTS (3/3)

11

Packet droppings of uncompressed HDTV stream due to the incoming probe packets.

Probe packets are generated with small inter-packet gaps corresponding from 5 Gb/s to 9 Gb/s, which caused network switch’s buffer over flow.

FUTURE WORKS

12

In this work, we clarified1. what is required to accomplish an accurate ABE system based on

an off-the-shelf PC with NIC’s hardware assistance.2. how the ABE results was improved by the implementation.3. problems when the ABE system was applied to the network with

actual competing HDTV streaming application.• Packet droppings and degraded ABE accuracy

In our future work, we will try to1. avoid from packet droppings and degraded ABE accuracy by

utilizing the application’s packets (for example, video signal packets) as probe packets.

2. integrate our high-accuracy ABE into the transmission layer, for example, into a transmission control layer for congestion control.d

13

Intentionally blank

14

Multi-layer traffic monitoring system executes multiple network monitoring software in a single platform taking advantage of our PRESTA 10G platform.

Layer-7: SAA (streaming analyzing agent), collects streaming headers such as RTP and i-Visto (http://www.i-visto.com) in our case, and reports stability of video playbacks from multiple sites.

GUI shows whether video can be played back stably and whether there are any dropped packets, packet reorders, and delays in each measurement site. Also saves history data.

Layer-3/4: Open-source Netflow probes to show amount of traffic and protocol distribution of each flow. We used Softflowd (compliant with Netflow v9 protocol).

NFprobe

SAAPS-HRA

QoSmon

TX

RX

Layer-1: High-resolution perf SONAR-HRA introduced in Winter Joint Techs in Feb. 2010. Shows traffic bit-rate in micro-second resolution using modified perfSONAR-PS and HRA daemon. In-service QoS monitor shows real-time behavior of delays and jitters in micro-second resolution.

perfSONAR-HRA viewer in-service QoS monitor

All monitoring software can run in single PRESTA box without performance degradation due to hardware assistance.

Video streaming

Video streaming

NFprobe

SAAPS-HRA

QoSmon

tcpdump

NFprobe

SAAPS-HRA

QoSmon

tcpdump

NFprobe

SAAPS-HRA

QoSmon

tcpdump

tcpdump

Our ongoing projects of multi-layer traffic monitoring system including passive and active

measurements

15

Visualization example

One-way delay distribution measurement results at closest site to video sender.

Layer-1

One-way delay at closest site to video receiver.

Traffic burstiness calculated for 100-ms time slots.

Layer-1

Availability of traffic playback in three measurement sites shown.

Layer-7

around 11.30 ms

around 17.45 ms

16

17.35ms

16.70ms

Dynamic configuration of layer-1 path

Initial configuration

Stable: all green.

In trouble: packet reorders between site B and site C (not losses).

A B C

A B C

A B C

A B C

After dynamic configuration

Stable: all green again.

No delay jitters observed. Therefore, layer-7 status might become stable.

Use case of high-resolution perfSONAR in Snow festa 2010, Japan

RESULTS

17