Optimised Memory Transfer & Flow Control for High Speed Networks

- Codito Technologies Pvt. Ltd.

- D Y Patil College of Engineering

Current Trends in High Speed Networks

Total network bandwidth triples every 12 months

CPU Processing power doubles every 18 months

Memory performance increases by 10% every 12 months

Software Challenges in High Speed Networks

Data MovementReasons For Copying.

Copying as a bottleneck.

Checksum

Flow Control AlgorithmLarge Bandwidth Delay Product

Software Requirements Specification

Product FunctionsFramework for Data Copy Elimination

Flow Control Algorithm for Optimal Bandwidth Utilization

User Characteristics

ConstraintsApplications

Max Memory Limit per Process

CPU Speed/Network Speed

Characterizing Network I/O

Zbuf Framework

Assumptions: Memory is not limited. Hardware Checksumming support would further

enhance our implementation. Application program may reuse the buffer but not

the contents of the buffer. Hence we perform explicit exchange of buffers.

Zbuf Framework

Modules:

Zbuf Allocator

API Calls

Send Module

Receive Module

Zbuf Allocator

Zbuf Allocator

Process Aaddress space

zalloczbuf

User Domain

Kernel Domain

Memory

Zbuf Zone

zbuf

User/Kernel sharing of memory !!

Zbuf Zone

Zbuf Allocator

User Domain

Kernel DomainMemory

Maintaining Zbufs In User Space

Zbuf Table

zd(Zbuf Descriptor)

zbuf

Size < MTU

Zbuf Table Zbuf Indirect Table

zbuf

Size > MTU

API Calls

int zalloc(size_t size)

int z_send(int sockfd,int zd,size_t len, int flags )

int z_recv(int sockfd,int zd,size_t len, int flags )

size_t zfread(int zd,size_t size, size_t nmemb, FILE *fp)

size_t zfwrite(int zd, size_t size, size_t nmemb, FILE *fp)

Send Module

User Domain

Kernel Domain

Process Aaddress space

Socket Queue

zbuf

Explicit exchange of buffers !!

zbufz_send

Zbufs are allocated inMTU-sized units.

Check-summing ?

Device Driver Interface

Receive Module

Zbuf Allocator

Zbuf Port Registration !!

Socket Queue

z_recv

User Domain

Kernel Domain

Process Aaddress space

Device Driver Interface

zbufs are grouped together.

zbuf

Flow Control Algorithms for High-Speed Networks

BW-delay product: 1Gb/s WAN * 100 msec RTT = 100Mb

TCP Reno:

Re-convergence to optimal b/w takes a long time !!

Packet-Pair Flow Control

Feedback Based Adaptive Rate Control Scheme.

Estimates n/w state by sending data packets in pairs.

Adapts to changes in n/w state within one RTT.

Assumptions: Network consists of FQ servers.

Routing table updates are infrequent.

Packets are of same size.

Packet Pair

Server 1Source Bottleneck Sink

Rate estimate

RTT

Bottleneck rate

Packet Pair Probes

Inter ack spacing estimates bottleneck service rate.

Pipeline depth is determined within one RTT.

Sending rate is modified according to inter ack spacing.

Design Strategy

State MachineStartup

Queue Priming

Normal Transmission

Buffer Management Strategy

Retransmission Strategy

State Chart Diagram for Packet Pair

Testing Strategy

Testing Strategy – II

Bottom up Integration for Zbuf Framework Smoke Testing for Packet Pair

Zbuf Framework Statistics

0 1024 2048 5120 8192 102400

510

15

2025

30

3540

45

5055

60

6570

TCP STATISTICS

Zbuf

Normal

Buffer Size(bytes)

Time (m

icrose

c)

0 1024 2048 5120 8192 10240051015202530354045505560657075

UDP STATISTICS

Zbuf

Normal

Buffer Size(bytes)

Tim

e (m

icro

sec)

Conclusion

The Zbuf framework is scalable across MTU sizes.

The Zbuf framework is scalable across multiprocessing environs.

Use of Packet Pair algorithm on high speed network would result in optimal utilization of bandwidth.