High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j...
Transcript of High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j...
![Page 1: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/1.jpg)
C U R A T E D B Y
IP SHOWCASE THEATER AT NAB – APRIL 8-11, 2019
High-Precision Capturing and Measuring of 2110 Streams
Using Commodity IT Equipment Willem Vermost, Ievgen Kostiukevych
IP Media Technology Architects
European Broadcasting Union
![Page 2: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/2.jpg)
Essent ial measurements and compliance
evaluat ion in SMPTE ST 2110 based faci l i t ies using
commodity of f- the-shelf IT hardware
• Willem Vermost, EBU
• Pedro Ferreira, Bisect
• Ievgen Kostiukevych, EBU
![Page 3: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/3.jpg)
EBU – Live IP Software Toolkit offline tool - open source project
![Page 4: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/4.jpg)
Waveform Monitor Packet Analyser
Broadcast Engineer IT Engineer
SKILLSET – BRIDGING THE GAP
![Page 5: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/5.jpg)
PCAP STORAGE
INFLUX DB
MONGO DB
STREAM ANALYZER
STREAM PREPROCESSOR
MIDDLEWARE GUI
HTTP SERVER
PCAP files are stored
time-series data: information about every
packet on the stream
The analyses of the data are totally based on the LIST
Core libraries, which are developed in C++
The middleware coordinates the workflows
(based on node.js)
ARCHITECTURE – OFFLINE non-time related data: user data static stream
information, …
The workflow can be accelerated if the PCAP
capture process is included in LIST
PCAP FILE
The UI is developed in JavaScript using the React
framework
![Page 6: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/6.jpg)
Critical success factor for the analysis:
• Accuracy of IP Packet Timestamp
• Clock of capture device synchronized with PTP
PREREQUISITE FOR VALID MEASUREMENT / CAPTURE
![Page 7: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/7.jpg)
SMPTE ST 2110-21 MEASURING THE BEHAVIOR OF A SENDER
In order to measure the behavior of a sender,
Join the multicast stream as close to the sender
Port mirror
Use a network TAP, if not available use a switch port mirror
Record the packet stream with a high precision capturing device
Make sure PTP is available
NIC
OS Kernel
Application
NIC
OS Kernel
Application
IP switch
• Accurate timestamps
• Everything locked to PTP
![Page 8: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/8.jpg)
LAB EXPERIMENT
![Page 9: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/9.jpg)
LAB EXPERIMENT: OUR TEST SETUP
![Page 10: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/10.jpg)
NIC PIPELINE: STREAM JOIN, PTP INBAND
Application
ens5f0
ens5f1
/dev/ptp6
/dev/ptp7
CPU OS Kernel
PCIe lanes
Driver
SYS clock
tcpdump Stream, PTP
-j host • - j host
![Page 11: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/11.jpg)
NIC PIPELINE: STREAM JOIN, PTP INBAND
Application
ens5f0
ens5f1
/dev/ptp6
/dev/ptp7
CPU OS Kernel
PCIe lanes
Driver
SYS clock
tcpdump Stream, PTP
-j host • CINST analysis
• - j host
![Page 12: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/12.jpg)
NIC PIPELINE: PORT MIRRORING, PTP OUTBAND
Application
ens5f0
ens5f1
/dev/ptp6
/dev/ptp7
CPU OS Kernel
PCIe lanes
Driver
SYS clock
tcpdump Sender port mirror
PTP
-j adapter_unsynced • - j adapter_unsynced
![Page 13: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/13.jpg)
NIC PIPELINE: PORT MIRRORING, PTP OUTBAND
Application
ens5f0
ens5f1
/dev/ptp6
/dev/ptp7
CPU OS Kernel
PCIe lanes
Driver
SYS clock
tcpdump Sender port mirror
PTP
-j adapter_unsynced • CINST analysis
• - j adapter_unsynced
![Page 14: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/14.jpg)
PTP: CLOCK CORRECTION OVER TIME (ns)
-250
-200
-150
-100
-50
0
50
100
150
200
250
Co
rrec
tio
n (
ns)
Clocks correction over time (s)
PHC2SYS
PTP4L
• PTP4L drives PHC2SYS
![Page 15: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/15.jpg)
LOOKING AT THE FIRST RESULTS
• Something is wrong!
• Don’t forget to lock your SDI source to
the reference
![Page 16: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/16.jpg)
NIC PIPELINE: PORT MIRRORING, PTP OUTBAND + KERNEL OFFLOAD
Application
ens5f0
ens5f1
/dev/ptp6
/dev/ptp7
CPU OS Kernel
PCIe lanes
Driver
SYS clock
tcpdump Sender port mirror
PTP
VMA
• PTP4L drives PHC2SYS
![Page 17: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/17.jpg)
WE STARTED WITH…
• Our first implementation
• DL 380 G10
![Page 18: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/18.jpg)
WE MINIATURIZED …
• Mini PC
• Thunderbolt-PCIe Enclosure
+
![Page 19: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/19.jpg)
SMPTE ST 2110-21 Traffic Shaping and Delivery Timing for Video
![Page 20: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/20.jpg)
• Describes a virtual test for the network (CMAX) • Constrains the packet delay variation of a sender
• Narrow senders
• Wide senders
• Describes a virtual test for the receive buffer (VRXFULL) • Narrow receivers (not capable of receiving wide senders)
• Wide receivers (universal)
• Describes Packet Read Schedules
SMPTE ST 2110-21 – TRAFFIC SHAPING AND DELIVERY TIMING FOR VIDEO
![Page 21: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/21.jpg)
SMPTE ST 2110-21 LEAKY BUCKET I: THE NETWORK COMPATIBILITY MODEL
• Tested at the output of the sender
• Measures PDV introduced by a sender
• CMAX is the maximum amount of packets allowed in the
“leaky bucket”
• If CINST > CMAX -> packets might get dropped in the
network
ST 2110-20 Sender
CMAX
CINST
Drain Rate = 1/TDRAIN (packets / second)
We just said, don’t drop any packet??
![Page 22: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/22.jpg)
ST 2110-20 Sender
VRXFULL
VRXINST
Packet J drains at Time TPRJ
SMPTE ST 2110-21 LEAKY BUCKET II: THE VIRTUAL RECEIVE BUFFER
• Tested at the output of the sender
• The VRX buffer is drained as a function of the packet
read schedule
• VRXFULL is the maximum amount of packets allowed in
the “leaky bucket”
• If VRXINST > VRXFULL -> packet might get dropped at
receiving end
We just said, don’t drop any packet??
![Page 23: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/23.jpg)
SMPTE ST 2110-21 PACKET READ SCHEDULES
Gapped
Packet Read Schedule
Linear
Packet Read Schedule
t
t
N ∗ TFRAME
Since Epoch
(N + 1) ∗ TFRAME
Since Epoch
TRS(gapped)
TRS(linear)
TVD
TPR0 TPR0
TPR0 TPRj
TROFFSET
TROFFSET
(𝑁𝑃𝐴𝐶𝐾𝐸𝑇𝑆 − 1) ∗ TRS(gapped)
(𝑁𝑃𝐴𝐶𝐾𝐸𝑇𝑆 − 1) ∗ TRS(linear)
![Page 24: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/24.jpg)
SMPTE ST 2110-21 GAPPED PACKET ARRIVAL
Gapped Packet Read Schedule
t
𝐍 ∗ 𝐓𝐅𝐑𝐀𝐌𝐄 Since Epoch
(N + 1) ∗ TFRAME Since Epoch
TRS(gapped)
TVD
𝐓𝐏𝐑𝟎 TPRj
TROFFSET
TROFFSET
𝑁𝑃𝐴𝐶𝐾𝐸𝑇𝑆 ∗ TRS(gapped)
Packet Arrival
t
Tarrival gap
𝐓𝐏𝑨𝟎 TP𝐴j
First Packet Offset
TP𝐴n
Marker Bit Set
RTPTimestamp = x
𝐓𝐏𝑨𝟎
TVD
RTPTimestamp = y
First Packet Offset
• TRS (gap) = 4,4 us (1080p50)
• TPA0 = Arrival time first packet
![Page 25: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/25.jpg)
TEST RESULTS Practical Test Results with EBU - LIST
![Page 26: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/26.jpg)
![Page 27: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/27.jpg)
![Page 28: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/28.jpg)
![Page 29: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/29.jpg)
TIMESTAMP COMPARISON OF TWO DIFFERENT CAPTURE DEVICES
-1400
-1200
-1000
-800
-600
-400
-200
0
283869560 283869760 283869960 283870160 283870360 283870560
Tim
esta
mp
dif
fere
nce
(n
s)
RTP Sequence number
Timestamp comparison of two different devices
Diff.
Linear (Diff.)
![Page 30: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/30.jpg)
• Avoid any bottlenecks in the capturing pipeline: • Use a NIC capable of handling the bandwidth
• Use kernel bypass to avoid packet drops by the OS
• Check storage performance - enough to store the capture in real time
• Frequency and Phase lock • Use a NIC with hardware clocking / timestamp support
• Synchronize the NIC to PTP
• It is possible to perform accurate captures with COTS equipment
CONCLUSION
![Page 31: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/31.jpg)
Want to know more tech.ebu.ch/list
Github github.com/ebu/pi-list
Docker hub.docker.com/r/ebutech/pi-list
Dematerialized list.ebu.io
JT-NM Tested Program jt-nm.org/jt-nm_tested/
![Page 32: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/32.jpg)
Thank you!
Any Questions?
![Page 33: High-Precision Capturing and Measuring of 2110 Streams ......tcpdump Sender port mirror PTP -j adapter_unsynced • C INST analysis • - adapter_unsynced. PTP: CLOCK CORRECTION OVER](https://reader034.fdocuments.net/reader034/viewer/2022052615/6081d9154490175e86638ddd/html5/thumbnails/33.jpg)