Application Performance Analysis using Wireshark - Sharkfest
SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal...
Transcript of SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal...
![Page 1: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/1.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
SharkFest ’19 Europe
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Session #14 - TCP Split Brain – Part II
John Pittle
Using Wireshark to
Compare & Contrast
behavior and TCP state of
client vs. server
Riverbed TechnologiesPerformance Management
![Page 2: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/2.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Premise: TCP Split Brain
• When troubleshooting TCP, you often have to consider both the sender’s unique perspective and the receiver’s unique perspective
• Both endpoints are independent, but at the same time, they do react to packets from the other end
• The joint behavior gets even more interesting when there’s “high” latency in the path
![Page 3: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/3.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Welcome Back!
• Continuation of Part I … after the break
![Page 4: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/4.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Session Goals
• Compare and contrast TCP end point behavior
• Drill down into the “what is it doing?” and “why is it doing that?”
• Promote Wireshark Profiles Feature
• Share experience and ideas
• Expose you to visualizations that help reinforce the end point behavior we will be discussing
![Page 5: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/5.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Part I Comparisons Recap
• 3-way handshake
• Latency
• Expert Info
• Fragment Overlaps / OOS / Retransmissions
![Page 6: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/6.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Summary Slide from Part I
![Page 7: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/7.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Part II Agenda
• Finish Fragmentation Topics
• Bytes in Flight Comparison
• Bonus – If Time Available
![Page 8: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/8.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
About me?
• Performance Engineering since 1980
• Protocol Analysis since 1991• Professional Services with
OPNET / Riverbed since 2005• Love the mystery of a
complicated performance issue • Shaved off beard in 2003…
![Page 9: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/9.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
My Ask of You
• Engage
• Participate
• We have a lot of detailed material
• We will explore conflicting, contradictory, and possibly confusing details
• Ask Questions
• Question Answers
![Page 10: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/10.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Application Scenario
• HTTPS Web Application
• Private key is not available
• Host based captures on web server and my laptop
![Page 11: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/11.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Symptoms to Analyze
• Downloading files take *forever*
• 16 seconds to download a 1.4MB file
• One TCP connection has been isolated as the connection of interest – TCP/52942-443
![Page 12: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/12.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
![Page 13: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/13.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Where we left off in Part I
![Page 14: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/14.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Dropped or OOS?
• If we freeze time right here, we can’t be sure if it’s just OOS or really a dropped packet
• We have to examine what comes next…
521543
![Page 15: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/15.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Dropped or OOS?
• Our missing segment is not showing up yet…
521543
![Page 16: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/16.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Anchor on SEQ==521543
521543 524112
2,569 Stream Bytes 3,876 Stream Bytes
Server capture
![Page 17: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/17.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Anchor on SEQ==521543
521543
1,292
524112 525404
1,292 1,292
522834
1,277 bytes missing
Client capture
#1000 #1001 #1003
![Page 18: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/18.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Anchor on SEQ==521543
521543 524112
2,569 Stream Bytes
521543
1,292
524112 525404
1,292 1,292
3,876 Stream Bytes
522834
1,277 bytes missing
Server capture
Client capture
#1000 #1001 #1003
![Page 19: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/19.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Other Clues
• Next seq after #1000 should have been 522835
• Wait! Isn’t this the segment that was retransmitted by server? (yes)
![Page 20: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/20.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
(Slide From Part I)
FRAME #714
![Page 21: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/21.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Other Clues
• Notice the change in ACK behavior
• Client ACKs every other packet then starts to ACK every packet
• Why is this?
![Page 22: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/22.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Profile Power
• Let’s flip the view a little so we can quickly see SACK fields in the decode summary
![Page 23: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/23.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Profile: SB-SACK
• Client “SACKs” the new segments, but continues to report - I’m missing 522835
![Page 24: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/24.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Discussion
• We can see Client is reporting a missing segment
• Yet, why does server continue to send segments other than the one requested?
• Visualization really helps with this…
![Page 25: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/25.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
End Point State Review
This right most red bar is selected (see black dots on ends),
which populates the summary decode panel at the bottom.
Packet #1002 is our packet of interest for current discussion.
![Page 26: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/26.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Discuss state of each end point
![Page 27: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/27.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Discuss state of each end point
![Page 28: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/28.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Zoom-in a little…
![Page 29: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/29.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Discuss state of each end point
![Page 30: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/30.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Discuss state of each end point
![Page 31: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/31.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Zoom-in
![Page 32: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/32.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Discuss state of each end point
![Page 33: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/33.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Discuss state of each end point
![Page 34: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/34.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Discuss state of each end point
![Page 35: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/35.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Missing segment finally arrives
• Let’s examine a few more details…
![Page 36: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/36.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
107ms Time Delta
• Why 107ms from previous packet?
![Page 37: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/37.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Wireshark Feature Parade
![Page 38: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/38.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
With time ref set to DupACK #3
![Page 39: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/39.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Why “Out of Order”?
• Wouldn’t “Retrans” or “Fast Retrans” be more accurate?
![Page 40: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/40.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Why ACK with SACK Fields?
• We’re all caught up, nothing is missing…so why send SACK?
• “Oh, btw Sender – I received these 15 bytes of TCP stream again unexpectedly. Just letting you know”
![Page 41: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/41.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Why 119ms delay here?
• What does this time delta suggest about bytes in flight and the congestion window?
![Page 42: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/42.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Here’s the delay – 1 x RTT
![Page 43: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/43.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Comparison Summary
![Page 44: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/44.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
5th Leg Completed
![Page 45: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/45.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Discussion
• Did you find this deep dive interesting?
• Anything new you’ve not seen before?
• Do you find visualization helpful?
• Other Comments?
![Page 46: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/46.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Split Brain Comparisons
• Bytes in Flight
![Page 47: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/47.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Review
• What is the meaning of Bytes in Flight?
• How is this metric related to performance?
• BIF for sender and receiver captures look very different, let’s compare a few packet exchanges
![Page 48: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/48.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Dry Cleaners Conveyor Belt
![Page 49: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/49.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Bytes in Flight - Decode
![Page 50: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/50.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Right Click, Apply as Column
![Page 51: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/51.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Server Side - Incrementing
![Page 52: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/52.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Sample - Math Drill Down
1761 5636
BIF == 3876
1761 5636
BIF == 1292
ACK 4345
(5636-4344)
4344
![Page 53: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/53.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Send Segment #2
1761 5637
ACK 4345
10804
BIF == 6460
1761 5636
BIF == 1292
ACK 4345
5636-4344
4344
(10804-4344)
![Page 54: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/54.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Receive ACK
1761 5637
ACK 6929
10804
BIF == 6460
1761 5637 10804
BIF == 3876
![Page 55: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/55.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Server Side – Decreasing?
![Page 56: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/56.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Server Side - Timing
![Page 57: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/57.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Let’s jump to Receiver now…
![Page 58: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/58.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Client Side
![Page 59: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/59.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
What else can we learn from BIF?
• So far, we’ve seen how BIF can inform about network health & congestion window
• We can also “get a sense” of Send Buffer sizing and the application’s TCP API options
![Page 60: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/60.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Inferring Send Buffer Size
• Bytes In-flight can give you some insight into how the Send Buffer size might be limiting the Congestion Window
• How can we find the maximum observed Bytes In-Flight?
![Page 61: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/61.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Throughput Throttling
Congestion Control Window
Receive Buffer
In-Flight Data
Notice how green
line never gets
above red line
![Page 62: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/62.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Throughput Throttling
Congestion Control Window
Receive Buffer
In-Flight Data
What’s holding us
back now?
![Page 63: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/63.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Inferring Send Buffer Size
• Bytes In-flight can give you some insight into how the Send Buffer size might be limiting Throughput
• How can we find the maximum observed Bytes In-Flight?
![Page 64: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/64.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Server Capture – Sort by BIF
![Page 65: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/65.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Discussion
• Assume that network is healthy + TCP Congestion Window is “happy” + receive window is higher than BIF
• What does this infer about Send Buffer size and implementation?
![Page 66: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/66.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Btw, something was odd…
• Did you notice anything odd in the last decode summary screen?
• Let’s look again…
![Page 67: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/67.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Server Capture – Out of order?
![Page 68: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/68.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Quick Detour via OOS…
![Page 69: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/69.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
How can sender segments be OOO?
• Is this even possible?
• Why would this happen?
• Let’s re-sort and have another look…
![Page 70: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/70.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Zoom in…sort by frame #
![Page 71: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/71.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Interpretation vs. Decode
![Page 72: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/72.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Why this is important…
• Heuristics, though generally pretty good, but they aren’t perfect
• Corner cases within corner cases…
• If you relied solely on the Expert Info summary you might incorrectly conclude the network is messing with packet order (more than it really is…)
• Human interpretation is key
![Page 73: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/73.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Discussion
![Page 74: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/74.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
OK, back to BIF…
![Page 75: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/75.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
BIF - Receiver Side
![Page 76: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/76.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
In-Flight & OOS / Loss
• Let’s look at how Wireshark calculates Bytes in-flight when there is packet loss or OOS
![Page 77: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/77.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Client – Receive #1275 OOS
• BIF did not increment for #1275
![Page 78: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/78.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Comparison Storyboard #1
![Page 79: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/79.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Comparison Storyboard #2
![Page 80: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/80.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Comparison Storyboard #3
![Page 81: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/81.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Comparison Storyboard #4
![Page 82: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/82.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Comparison Storyboard #5
![Page 83: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/83.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Why doesn’t BIF Increment
![Page 84: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/84.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Discussion
• We’ve compared based the stream seq #
• We’ve looked at LEN, ACK, BIF, and packet order differences
• We tried to compare the “when”, but this is a real Brain Bender
![Page 85: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/85.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
We need to visualize this…
• Now that we’ve seen how Wireshark counts bytes in flight…
• …and the challenges of side by side comparisons…
• …let’s look at how we can gain better insight from using visualization
![Page 86: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/86.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
First, a word about Merging
• What if we could merge captures?
• High Fidelity merge client and service side captures
• Fine tune “Send Time” and “Recv Time”
• Explicitly identify drops
• Accurately measure congestion
• Accurately measure Server Response Time
• Increases Accuracy of Advanced Analytics
![Page 87: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/87.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Merge Melds Split Brain
![Page 88: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/88.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Circa 2019…
![Page 89: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/89.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Merged Packet Exchange with BIF Overlay
![Page 90: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/90.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Quick Orientation
![Page 91: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/91.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
In-flight Bytes
![Page 92: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/92.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Symptom Pop-up Labels
![Page 93: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/93.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
What does this upward progression tell us about CWND?
![Page 94: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/94.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Why In-flight drops to 0?
![Page 95: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/95.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
What does this delta tell us?
18KB
9KB
![Page 96: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/96.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Wireshark BIF Chart
![Page 97: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/97.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Looking closer @ timing…
![Page 98: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/98.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
When does Server hear the news?
![Page 99: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/99.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
#998 is dropped
BTW, could this be a “tail drop”?
![Page 100: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/100.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
2 of the DupACKs
![Page 101: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/101.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
3 of 3 DupACKs
![Page 102: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/102.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Delay Cost of Drop: # of RTTs?
![Page 103: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/103.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Delay Cost of Drop: # of RTTs?
3 RTT x iRTT == 387 ms
![Page 104: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/104.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Discussion
![Page 105: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/105.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Let’s Visualize BIF for the Entire TCP Connection
![Page 106: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/106.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
…drill into the 1.4MB Xfer
![Page 107: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/107.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Delay Analysis during Xfer
![Page 108: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/108.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Review
• BIF for receiver side traffic doesn’t tell you much about actual congestion window
• Segmentation offload can create misleading conclusions on sender capture
• Packet loss on high latency path can have severe impact on performance
![Page 109: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/109.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Comparison Summary
![Page 110: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/110.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
6th Leg Completed
![Page 111: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/111.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Bonus Section
• Time Permitting
![Page 112: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/112.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Split Brain Comparisons
• Congestion
![Page 113: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/113.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Remember this from Part I
• Client Capture – 121ms
• Server Capture – 129ms
![Page 114: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/114.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Group Discussion
• What’s the definition of latency?
• What’s the definition of congestion?
• How can you tell latency from congestion?
![Page 115: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/115.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Congestion is not…
• …protocol delay
• …serialization (bandwidth delay)
• …server delay (well maybe…)
• …others?
![Page 116: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/116.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Introducing…1W-TTT
• One Way - Total Transfer Time
![Page 117: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/117.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
1-Way Total Transfer Time
• Probably other names for this…
• Time required for bits to leave sender and arrive at receiver…
• TTT == Total Transfer Time
![Page 118: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/118.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
One Possible Formula
• 1-Way Total Transfer time (TTT) ………minus
• (Bandwidth Delay + Latency Delay)
• TTT – (BD + LD) == Congestion
![Page 119: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/119.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
15 seconds
5 seconds
30 seconds
Congestion delay
Latency delay
Bandwidth delay
Exaggerated TTT Example
• Bandwidth = 1,000 bps
• Latency = 5 seconds
Client
Server
T = 0
50
30,000 bits
10 20 30 40 50
![Page 120: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/120.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
RTT vs. 1-Way TTT
• RTT is not the same as 1-Way TTT
• What can Wireshark tell us about 1-Way TTT?
• Answer: not a lot…but we can infer a few things
![Page 121: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/121.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
WireShark RTT include PD
• 1-Way Total Transfer time (TTT) ………minus
• (Bandwidth Delay + Latency Delay + Protocol Delay)
• TTT – (BD + LD + PD) == Congestion
![Page 122: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/122.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Finding congestion with Wireshark
• How can we find examples of congestion in a capture with Wireshark?
• For TCP we can use RTT2ACK, and infer congestion based on our knowledge about how Wireshark calculates RTT2ACK
![Page 123: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/123.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
It’s not Perfect…
• …but you’ll get a decent approximation
• …however, be aware…
• … it could be overstated due to protocol delay
![Page 124: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/124.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Sender or Receiver Capture
• Which capture would be most interesting to understand congestion?
• Answer: each one is unique and will tell part of the story (split brain)
![Page 125: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/125.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
RTT2ACK Decode
![Page 126: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/126.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Apply as Column
![Page 127: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/127.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Column added to our view…
![Page 128: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/128.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Sort by RTT2ACK
![Page 129: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/129.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Zoom in a little…
![Page 130: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/130.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
How else could we do this?
![Page 131: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/131.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
How else could we do this?
• Display filter…
![Page 132: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/132.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Server Cap Shows 342ms RTT
• Could this be right? We just established RTT is 120-ish ms
• What is interesting about this ACK?
![Page 133: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/133.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Client shows 209ms RTT?
![Page 134: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/134.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Recap
• Server shows 342ms RTT2ACK
• 1TTT + Delayed ACK Timer + 1TTT
• Delayed ACK Timer 209ms (from client capture)
• RTT 342ms – 209ms == 133ms (2 x 1TTT)
• Is this in the ball park?
• What’s the smallest latency we’ve seen so far in these captures?
![Page 135: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/135.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Let’s Visualize – Server
342 ms
![Page 136: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/136.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Let’s Visualize – Client
209 ms
![Page 137: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/137.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Discussion
![Page 138: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/138.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
But Wait!
I thought we were here to talk about congestion? We are, but it
turns out we really need to understand latency first…
![Page 139: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/139.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Can we Predict RTT?
• Forget the terms “Client” vs. “Server” momentarily
• Consider instead the terms “Sender” vs. “Receiver”
• What would you expect the possible time components for RTT to be for a Receiver capture?
• …and for a Sender side capture?
• (assume there’s “some” latency between hosts)
![Page 140: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/140.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Receiver Capture
• Should be….
• Either super fast, or….
• Influenced by Delayed ACK Timer, and/or…
• Influenced by OOS and retransmissions
![Page 141: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/141.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Receiver Capture
![Page 142: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/142.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Scroll down a little…
![Page 143: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/143.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
A little more…
![Page 144: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/144.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
~Speed of Light RTT2ACK
![Page 145: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/145.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Sender Capture
• Never less than 2 x TTT (…right?)
• Add in Receiver’s Delayed ACK Timer and/or
• Add in Receiver’s time to wait for sender to fix OOS and Packet Loss
![Page 146: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/146.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Sender Capture – Lowest
![Page 147: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/147.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Enough about Latency…
![Page 148: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/148.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
More Visualizations
• Congestion is really hard to analyze with lists of packets and time deltas
• Let’s use visualizations starting at the 10K foot view and then drill down to the details
• We’ll use merged captures…
![Page 149: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/149.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
10,000 ft/m View
Merge of both captures
Congestion reported for each direction
• Entire capture visualized
• You can see bursts of packet exchanges
• You can see the 10 sec keep alive pattern
• You can see call outs for packet loss
![Page 150: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/150.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
10,000 ft/m View
Overlay chart of the Congestion Metric
Congestion reported for each direction
![Page 151: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/151.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Let’s Zoom-In
![Page 152: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/152.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Zoom-in to File Download
![Page 153: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/153.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Advanced Analytics: ~ 20% of overall delay (blue)
![Page 154: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/154.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Why so much congestion?
ISPInternet
NYC VPN
ISP
ISP
Transit
SFC VPN
GatewaySFC Web
Server
Home
Office
Orlando• Multiple Tunnels & Protocols
• Multiple ISPs
• ESX Host Oversubscribed?
• VPN Client Overhead?
![Page 155: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/155.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Congestion Recap
• Any surprises about the amount of congestion?
• Congestion occurs in both directions independently
• Does this view help to explain how congestion impacts performance?
![Page 156: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/156.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Summary Results
![Page 157: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/157.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Marathon Completed !
![Page 158: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/158.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Summary & Wrap-Up
• Interpreting TCP behavior can be confusing and complicated, especially when there is “high” latency in the path
• Captures from both end points can be beneficial
• You need to split your brain into “server perspective” and “client perspective”
![Page 159: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/159.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Summary & Wrap-Up
• Wireshark features are extremely helpful, but can only take you so far
• Visualization can help you understand behavior and quickly interpret root cause
• Advanced analytics are icing on the cake…
![Page 160: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/160.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Recap Part I Topics
• Background on app, topology, and symptoms
• Compare and Contrast (aka Split Brain)• 3-way Handshake
• Latency
• Expert Info
• Fragment Overlaps ,OOS, Retransmissions
![Page 161: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/161.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Recap Part II Topics
• Continue where we left off
• Compare and contrast…• Bytes in Flight
• Bonus Topic
• Session Wrap-Up
![Page 162: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/162.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Full Summary Results
![Page 163: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/163.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
Final Questions / Comments
![Page 164: SharkFest ’19 Europe · 2019. 11. 11. · #sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8 SharkFest ’19 Europe #sf19eu • Palacio Estoril Hotel, Estoril, Portugal](https://reader036.fdocuments.net/reader036/viewer/2022071212/6023d9911348516b2e71c65f/html5/thumbnails/164.jpg)
#sf19eu • Palacio Estoril Hotel, Estoril, Portugal • Nov 4 - 8
End of Session