Effect of TCP SACK on Throughput

10
Effect of TCP SACK on throughput On this Saturday evening, I have finally completed my work with TCP SACK analysis. This post was in my mind for sometime but now I have done it after building my big local Internet at home. You will also find some stuff about receive segmentation offload, wireshark tips etc. Here is the topology used for TCP SACK tests. F irst of all, this big setup isn’t really necessary but I am using this setup for my BGP tests and have found it suitable for a real world scenario. What are we testing?

description

junos 01

Transcript of Effect of TCP SACK on Throughput

Page 1: Effect of TCP SACK on Throughput

Effect of TCP SACK on throughputOn this Saturday evening, I have finally completed my work with TCP SACK analysis. This post was in my mind for sometime but now I have done it after building my big local Internet at home. You will also find some stuff about receive segmentation offload, wireshark tips etc. Here is the topology used for TCP SACK tests.

First of all, this big setup isn’t really necessary but I am using this setup for my BGP tests and have found it suitable for a real world scenario. What are we testing?

We are downloading a wireshark.exe file around ~30MB from debian1 (144.2.3.2) webserver to client PC hostE(60.60.60.60) simply by wget ;

Page 2: Effect of TCP SACK on Throughput

1 root@hostE:~# wget http://144.2.3.2/wireshark.exe

under three different states.

when there is no packet loss, SACK enabled When there is packet loss, TCP SACK option is enabled on hostE When there is packet loss, TCP SACK option is disabled on hostE

For the completness of the post, I would like to write a few bits about ACKnowledgement and SACK in TCP as well.An ACK segment in TCP means that the party sending the segment has received all bytes up to the number set in ACK field but not including. Here is an example wireshark output:

On this segment, we have received all bytes up to 102809 (note: relative numbers used for simplicity) This is a cumulative number which means once remote peer receives this ACK, it will remove segments below this byte waiting in its retransmission queue. Hang on a second which retransmission queue? When a host sends a segment, it puts a copy of this segment in its retransmission queue in case it is lost. Once it is acknowledged, copy is removed from the queue.So far everything seems to be good. Let’s assume that Debian1 host sends 3 TCP segments and 2nd one is lost.

root@hostE:~# w get http://144.

Page 3: Effect of TCP SACK on Throughput

What will the hostE do? It can definitely ACK 1st segment but what about 3rd? If we ACK 3rd, Debian1 will assume all segments i.e 1,2,3 have been received and remove them from the retransmission queue. That is why hostE can’t ACK 3rd segment. Here TCP SACK option comes into play. When hostE acknowledges 1st segment, it piggybacks the SACK information in ACK segment i.e it informs Debian1 that “Hey peer, I have received 1st segment but I acknowledge 3rd segment in SACK option so not cumulatively, hence I have missing bytes” Debian1 tags the segments for this SACK byte range in retransmission queue so that I doesn’t have to send them again as they already arrived.Now what do we do. HostE keeps asking for the 2nd segment and after I think 3-4th duplicate ACK, Debian1 realizes that segment has been lost and without waiting for the retransmission timeout, fast retransmission triggers and 2nd segment is sent. After this, hostE can now finally ACK 3rd segment cumulatively, after which Debian1 removes the segment in its retranmission queue as it doesn’t have to send it.  So far I have attached my sample drawings. Let’s see a wireshark capture what happens when there is packet loss

Now, focus on packet number 168 Wireshark says previous segment was not captured. Actually

Page 4: Effect of TCP SACK on Throughput

it wasn’t captured, as it got lost in transmission. That previous segment never arrived to 60.60.60.60 host. Because the packet has been lost, we keep asking for the missing segment on packet 169,171,173 again and again and informing the 144.2.3.2 server that we have got everything up to 188241 bytes. However at this point, 144.2.3.2 doesn’t really care about this much as it still has window to send more bytes because of which it keeps sending. If we zoom in the packet no 170

we see that we have received bytes bigger than missing one 188241, as Debian1 is still transmitting. Now we are zooming in packet 171 duplicate ACK which contains SACK byte range.

Heyyy, here it is. We inform Debian1 web server that we have received this range of bytes.

Now after this SACK troubleshooting, I would like to go back to my test cases I have listed above but for wireshark not to confuse us with oversized, gigantic IP packet sizes, we need to turn off “Generic Receive Offload” on HostE. For more details you can check my other post to see why we really need this.

Page 5: Effect of TCP SACK on Throughput

1 root@hostE:~# ethtool -K eth1 gro off

1) No packet loss, SACK enabled

On the topology I have drawn above, AS8500 transit carrier consists of 8 routers and there is asymmetric routing involved as it can be seen below. I did this in purpose but not exactly related to this test.

12345678

root@debian1:~# traceroute -n 60.60.60.60traceroute to 60.60.60.60 (60.60.60.60), 30 hops max, 60 byte packets1  144.2.3.1  9.950 ms  9.943 ms  9.936 ms2  212.6.1.2  14.878 ms  14.885 ms  14.889 ms3  212.6.1.10  24.865 ms  24.872 ms  24.868 ms4  172.40.1.2  24.808 ms  24.817 ms  24.810 ms5  31.1.1.3  44.659 ms  44.667 ms  44.662 ms6  60.60.60.60  29.771 ms  19.669 ms  24.457 ms

1234567

root@hostE:~# traceroute -n 144.2.3.2traceroute to 144.2.3.2 (144.2.3.2), 30 hops max, 60 byte packets1  60.60.60.1  2.588 ms  2.583 ms  2.574 ms2  98.1.1.1  17.284 ms  17.278 ms 31.1.1.1  17.245 ms3  24.1.1.1  17.167 ms  17.214 ms 172.40.1.1  17.199 ms4  87.1.1.1  32.287 ms  32.282 ms  32.346 ms5  144.2.3.2  32.078 ms 192.168.196.1  31.871 ms 144.2.3.2  32.098 ms

When there is no packet loss, wireshark.exe file is fetched at around 4-5 seconds.

root@hostE:~# ethtool -K eth1 g

root@debian1:~# traceroute -n traceroute to 60.60.60.60 (60.60 1 144.2.3.1 9.950 ms 9.943 m 2 212.6.1.2 14.878 ms 14.885

root@hostE:~# traceroute -n 14traceroute to 144.2.3.2 (144.2.3 1 60.60.60.1 2.588 ms 2.583 2 98.1.1.1 17.284 ms 17.278

Page 6: Effect of TCP SACK on Throughput

12345678910

root@hostE:~# wget http://144.2.3.2/wireshark.exe--2015-01-31 20:07:01--  http://144.2.3.2/wireshark.exeConnecting to 144.2.3.2:80... connected.HTTP request sent, awaiting response... 200 OKLength: 29826488 (28M) [application/x-msdos-program]Saving to: `wireshark.exe.4'

100%[================================================================================>] 29,826,488  7.75M/s   in 4.6s

2015-01-31 20:07:06 (6.21 MB/s) - `wireshark.exe.4' saved [29826488/29826488]

On this test, presence of SACK doesn’t make any difference anyway:

12

root@hostE:~# sysctl net.ipv4.tcp_sacknet.ipv4.tcp_sack = 1

2) Packet loss, SACK enabledNow we need to introduce packet loss. This is very easy with Linux’s advanced net emulation so we add %1 of packet loss on Debian1

1 root@debian1:~# tc qdisc add dev eth1.956 root netem loss 1%

Packet loss is activated, now fetch the file!

root@hostE:~# w get http://144.--2015-01-31 20:07:01-- http://1Connecting to 144.2.3.2:80... coHTTP request sent, aw aiting res

root@hostE:~# sysctl net.ipv4.tnet.ipv4.tcp_sack = 1

root@debian1:~# tc qdisc add d

Page 7: Effect of TCP SACK on Throughput

12345678

root@hostE:~# wget http://144.2.3.2/wireshark.exe--2015-01-31 20:48:50--  http://144.2.3.2/wireshark.exeConnecting to 144.2.3.2:80... connected.HTTP request sent, awaiting response... 200 OKLength: 29826488 (28M) [application/x-msdos-program]Saving to: `wireshark.exe.8'

100%[========================================================================================================================>] 29,826,488   232K/s   in 2m 13s

The same file download duration jumped from 5 seconds to 2mins and13secs with only %1 packet loss. Fair enough but this isn’t what we are trying to find. I have tested the same download again and again and on average all of them are above 2mins.

3) Packet loss, SACK disabledNow we are disabling SACK feature

12

root@hostE:~# sysctl net.ipv4.tcp_sack=0net.ipv4.tcp_sack = 0

12345678

root@hostE:~# wget http://144.2.3.2/wireshark.exe--2015-01-31 21:08:15--  http://144.2.3.2/wireshark.exeConnecting to 144.2.3.2:80... connected.HTTP request sent, awaiting response... 200 OKLength: 29826488 (28M) [application/x-msdos-program]Saving to: `wireshark.exe.15'

100%

root@hostE:~# w get http://144.2.3.2/w ireshark--2015-01-31 20:48:50-- http://1Connecting to 144.2.3.2:80... co

root@hostE:~# sysctl net.ipv4.tnet.ipv4.tcp_sack = 0

root@hostE:~# w get http://144.2.3.2/w ireshark--2015-01-31 21:08:15-- http://1Connecting to 144.2.3.2:80... co

Page 8: Effect of TCP SACK on Throughput

910

[========================================================================================================================>] 29,826,488   370K/s   in 1m 43s

2015-01-31 21:09:58 (282 KB/s) - `wireshark.exe.15' saved [29826488/29826488]

Now download duration is 1 min, 43 secs. I have downloaded the same file as I did for the previous test again and again. On average, it is below 2mins.

Still these results don’t prove much but apparently SACK doesn’t bring much considering a topology like I have. If you have a satellite connection in which you have small MTU and high latency constraints, it may be beneficial though as you don’t resend the packets on a link latencies are above 500ms.

By the way, during investigation wonderful wireshark filter “tcp.analysis.lost_segment” helped me a lot! Look what it does:

By using this filter, I could analyse the packet loss amount and could compare how many packets I lost in each test.

Finally I have reached the end of this post. It was a simple test to show how SACK works practically. Please do let me know if you have any corrections/feedback.