White Rabbit Tutorial IEEE
of 124
Transcript of White Rabbit Tutorial IEEE
-
7/21/2019 White Rabbit Tutorial IEEE
1/124
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
White RabbitEthernet-based solution for sub-ns synchronization and
deterministic, reliable data delivery
Maciej Lipinski
on behalf of White Rabbit Team
Hardware and Timing Section @ CERNInstitute of Electronic Systems @ Warsaw University of Technology
15 July 2013
IEEE Plenary Meeting Geneve
Maciej Lipinski White Rabbit 1/59
-
7/21/2019 White Rabbit Tutorial IEEE
2/124
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Outline
1 Introduction
2 CERN Control & Timing
3 WR Network
4 Time Distribution
Timing demo
5 Data Distribution
Redundancy demo
6 WR @ CERN
7 Summary
Maciej Lipinski White Rabbit 2/59
-
7/21/2019 White Rabbit Tutorial IEEE
3/124
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Outline
1 Introduction
2 CERN Control & Timing
3 WR Network
4 Time Distribution
Timing demo
5 Data Distribution
Redundancy demo
6 WR @ CERN
7 Summary
Maciej Lipinski White Rabbit 3/59
-
7/21/2019 White Rabbit Tutorial IEEE
4/124
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
CERN
Maciej Lipinski White Rabbit 4/59
-
7/21/2019 White Rabbit Tutorial IEEE
5/124
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
CERN
Maciej Lipinski White Rabbit 4/59
C C @ C S
-
7/21/2019 White Rabbit Tutorial IEEE
6/124
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
CERN
Maciej Lipinski White Rabbit 4/59
I t d ti CERN C t l & Ti i WR N t k Ti Di t ib ti D t Di t ib ti WR @ CERN S
-
7/21/2019 White Rabbit Tutorial IEEE
7/124
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
CERN Accelerator Complex
Maciej Lipinski White Rabbit 5/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
-
7/21/2019 White Rabbit Tutorial IEEE
8/124
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
CERN Accelerator Complex
Maciej Lipinski White Rabbit 5/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
-
7/21/2019 White Rabbit Tutorial IEEE
9/124
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
CERN Accelerator Complex
Maciej Lipinski White Rabbit 5/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
-
7/21/2019 White Rabbit Tutorial IEEE
10/124
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
CERN Accelerator Complex
Maciej Lipinski White Rabbit 5/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
-
7/21/2019 White Rabbit Tutorial IEEE
11/124
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Beams Controls Hardware & Timing
Maciej Lipinski White Rabbit 6/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
-
7/21/2019 White Rabbit Tutorial IEEE
12/124
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Beams Controls Hardware & Timing
Maciej Lipinski White Rabbit 6/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
-
7/21/2019 White Rabbit Tutorial IEEE
13/124
g @ y
Beams Controls Hardware & Timing
Maciej Lipinski White Rabbit 6/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
-
7/21/2019 White Rabbit Tutorial IEEE
14/124
g y
Beams Controls Hardware & Timing
Maciej Lipinski White Rabbit 6/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
-
7/21/2019 White Rabbit Tutorial IEEE
15/124
Beams Controls Hardware & Timing
Maciej Lipinski White Rabbit 6/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
-
7/21/2019 White Rabbit Tutorial IEEE
16/124
What is White Rabbit?
Renovation of accelerators control and timing
Based on well-known technologies
Open Hardware and Open Software
International collaboration
Maciej Lipinski White Rabbit 7/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
-
7/21/2019 White Rabbit Tutorial IEEE
17/124
White Rabbit features
standard-compatible
sub-ns accuracy
tens-ps precision
upper-bound low-latency
white-box simulation &
analysis
high reliability
tens-km span
thousands-nodes systems
Maciej Lipinski White Rabbit 8/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
-
7/21/2019 White Rabbit Tutorial IEEE
18/124
White Rabbit applications
Deployed for time distribution:
CERN Neutrinos to Gran Sasso
Maciej Lipinski White Rabbit 9/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
-
7/21/2019 White Rabbit Tutorial IEEE
19/124
White Rabbit applications
Deployed for time distribution:CERN Neutrinos to Gran Sasso
Future applications:
CERN and GSI
Maciej Lipinski White Rabbit 9/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
-
7/21/2019 White Rabbit Tutorial IEEE
20/124
White Rabbit applications
Deployed for time distribution:CERN Neutrinos to Gran Sasso
Future applications:
CERN and GSI
HiSCORE: Gamma&Cosmic-Rayexperiment (Tunka, Siberia)
Maciej Lipinski White Rabbit 9/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
-
7/21/2019 White Rabbit Tutorial IEEE
21/124
White Rabbit applications
Deployed for time distribution:CERN Neutrinos to Gran Sasso
Future applications:
CERN and GSI
HiSCORE: Gamma&Cosmic-Rayexperiment (Tunka, Siberia)The Large High Altitude Air ShowerObservatory (China)
Maciej Lipinski White Rabbit 9/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
-
7/21/2019 White Rabbit Tutorial IEEE
22/124
White Rabbit applications
Deployed for time distribution:CERN Neutrinos to Gran Sasso
Future applications:
CERN and GSI
HiSCORE: Gamma&Cosmic-Rayexperiment (Tunka, Siberia)The Large High Altitude Air ShowerObservatory (China)
Potential applications:
Cherenkov Telescope Array
Maciej Lipinski White Rabbit 9/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
-
7/21/2019 White Rabbit Tutorial IEEE
23/124
White Rabbit applications
Deployed for time distribution:CERN Neutrinos to Gran Sasso
Future applications:
CERN and GSIHiSCORE: Gamma&Cosmic-Rayexperiment (Tunka, Siberia)The Large High Altitude Air ShowerObservatory (China)
Potential applications:
Cherenkov Telescope ArrayInternational ThermonuclearExperimental Reactor (ITER)
Maciej Lipinski White Rabbit 9/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
-
7/21/2019 White Rabbit Tutorial IEEE
24/124
White Rabbit applications
Deployed for time distribution:CERN Neutrinos to Gran Sasso
Future applications:
CERN and GSIHiSCORE: Gamma&Cosmic-Rayexperiment (Tunka, Siberia)The Large High Altitude Air ShowerObservatory (China)
Potential applications:
Cherenkov Telescope ArrayInternational ThermonuclearExperimental Reactor (ITER)European deep-sea research
infrastructure (KM3NET)
Maciej Lipinski White Rabbit 9/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
-
7/21/2019 White Rabbit Tutorial IEEE
25/124
Outline
1 Introduction
2 CERN Control & Timing
3 WR Network
4 Time DistributionTiming demo
5 Data Distribution
Redundancy demo
6 WR @ CERN
7 Summary
Maciej Lipinski White Rabbit 10/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
-
7/21/2019 White Rabbit Tutorial IEEE
26/124
CERN Control and Timing System
6 accelerators
including LHC: 27km
A huge real-time
distributed system
Thousands of devices
Maciej Lipinski White Rabbit 11/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
-
7/21/2019 White Rabbit Tutorial IEEE
27/124
Controlling accelerators
Maciej Lipinski White Rabbit 12/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
-
7/21/2019 White Rabbit Tutorial IEEE
28/124
Controlling accelerators
Maciej Lipinski White Rabbit 12/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
-
7/21/2019 White Rabbit Tutorial IEEE
29/124
CERN Control System event distribution (1)
Events messages which trigger actions
Each event is identified by anID
Maciej Lipinski White Rabbit 13/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
-
7/21/2019 White Rabbit Tutorial IEEE
30/124
CERN Control System event distribution (2)
Devices are subscribed to events
Each device knows what to do on a particular event
Maciej Lipinski White Rabbit 14/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
-
7/21/2019 White Rabbit Tutorial IEEE
31/124
CERN Control System event distribution (3)
Each event (ID) has a trigger time associated
A set of events is sent as a single Control Message (CM)
CM is broadcast to all the end devices (nodes)
CM is sent in advance (upper-bound message latency)
Maciej Lipinski White Rabbit 15/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
-
7/21/2019 White Rabbit Tutorial IEEE
32/124
CERN Control & Timing Network requirements
Maciej Lipinski White Rabbit 16/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
-
7/21/2019 White Rabbit Tutorial IEEE
33/124
White Rabbit Network Ethernet-based
Standard Ethernet network
Few thousands nodesBandwidth: 1 Gbps
WR Switch: 18 ports
Non-WR Devices
Ethernet features (VLAN)& protocols (SNMP)
Maciej Lipinski White Rabbit 17/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
-
7/21/2019 White Rabbit Tutorial IEEE
34/124
White Rabbit Network Ethernet-based
High accuracy/precisionsynchronization
Deterministic, reliable and
low-latency Control Data
delivery
Maciej Lipinski White Rabbit 18/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
-
7/21/2019 White Rabbit Tutorial IEEE
35/124
White Rabbit Switch
Central element of WR networkOriginal design optimized for timing, designed from scratch
18 ports
1000BASE-BX10 SFPs: up to 10 km, single-mode fiber
Open design (H/W and S/W)
Maciej Lipinski White Rabbit 19/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
-
7/21/2019 White Rabbit Tutorial IEEE
36/124
White Rabbit Node
Ethernet MAC with White Rabbit
Open IP CoreEasily integrated into custom FPGA-based designs
Maciej Lipinski White Rabbit 20/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Whi R bbi N d
-
7/21/2019 White Rabbit Tutorial IEEE
37/124
White Rabbit Node
Ethernet MAC with White Rabbit
Open IP CoreEasily integrated into custom FPGA-based designs
WR Node: universal carrier board
Maciej Lipinski White Rabbit 20/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
O li
-
7/21/2019 White Rabbit Tutorial IEEE
38/124
Outline
1 Introduction
2 CERN Control & Timing
3 WR Network
4 Time DistributionTiming demo
5 Data Distribution
Redundancy demo
6 WR @ CERN
7 Summary
Maciej Lipinski White Rabbit 21/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Ti Di t ib ti i Whit R bbit N t k
-
7/21/2019 White Rabbit Tutorial IEEE
39/124
Time Distribution in White Rabbit Network
Synchronization withsub-nsaccuracytens-psprecision
Combination of
Precision Time Protocol (IEEE1588) synchronizationLayer 1 syntonizationDigital Dual-Mixer Time Difference (DDMTD) phasedetection
Maciej Lipinski White Rabbit 22/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
P i i Ti P t l (IEEE1588)
-
7/21/2019 White Rabbit Tutorial IEEE
40/124
Precision Time Protocol (IEEE1588)
Simple calculations:
linkdelayms: ms= (t4t1)(t3t2)
2
clockoffsetms= t2
t1 + msAssumes medium symmetry
Maciej Lipinski White Rabbit 23/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Precision Time Protocol (IEEE1588)
-
7/21/2019 White Rabbit Tutorial IEEE
41/124
Precision Time Protocol (IEEE1588)
Simple calculations:
linkdelayms: ms= (t4t1)(t3t2)
2
clockoffsetms= t2
t1 + msAssumes medium symmetry
Disadvantages
all nodes have free-running oscillatorsfrequency drift compensation vs.
message exchange traffic
Maciej Lipinski White Rabbit 23/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Layer 1 Syntonization
-
7/21/2019 White Rabbit Tutorial IEEE
42/124
Layer 1 Syntonization
Maciej Lipinski White Rabbit 24/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Phase Tracking (DDMTD)
-
7/21/2019 White Rabbit Tutorial IEEE
43/124
Phase Tracking (DDMTD)
Monitor phase of bounced-back clock
Enhance PTP timestamps with phase measurement
Phase-locked loop in the slave follows the phase changes
Maciej Lipinski White Rabbit 25/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Link Delay Model
-
7/21/2019 White Rabbit Tutorial IEEE
44/124
Link Delay Model
Maciej Lipinski White Rabbit 26/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Link Delay Model
-
7/21/2019 White Rabbit Tutorial IEEE
45/124
Link Delay Model
Maciej Lipinski White Rabbit 26/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Link Delay Model
-
7/21/2019 White Rabbit Tutorial IEEE
46/124
Link Delay Model
delayms= txm+ ms+rxs
delaysm= txs+ sm+rxm
masterfixed delays
txm
rxm sm
ms
variable delays fixed delays
rxs
txs
slave
t1
t4 t3
t2
Maciej Lipinski White Rabbit 27/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Link Delay Model
-
7/21/2019 White Rabbit Tutorial IEEE
47/124
Link Delay Model
delayms= txm+ ms+rxs
delaysm= txs+ sm+rxm
masterfixed delays
txm
rxm sm
ms
variable delays fixed delays
rxs
txs
slave
t1
t4 t3
t2
Relative Delay Coefficient ()
for 1000base-X over a Single-mode
Optical Fibre
ms= (1 + ) sm
Maciej Lipinski White Rabbit 27/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Link Delay Model: fiber optic solution
-
7/21/2019 White Rabbit Tutorial IEEE
48/124
Link Delay Model: fiber optic solution
Solution for Ethernet
over a Single-mode
Optical Fiberasymmetry= txm+rxs
+
2 +
Maciej Lipinski White Rabbit 28/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
White Rabbit extension to PTP
-
7/21/2019 White Rabbit Tutorial IEEE
49/124
White Rabbit extension to PTP
White Rabbit requires:WR-specific statesExchange of WR-specific information
Maciej Lipinski White Rabbit 29/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
White Rabbit extension to PTP
-
7/21/2019 White Rabbit Tutorial IEEE
50/124
White Rabbit extension to PTP
White Rabbit requires:WR-specific statesExchange of WR-specific information
White Rabbit estimates link asymmetry
Maciej Lipinski White Rabbit 29/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
White Rabbit extension to PTP
-
7/21/2019 White Rabbit Tutorial IEEE
51/124
White Rabbit extension to PTP
White Rabbit requires:WR-specific statesExchange of WR-specific information
White Rabbit estimates link asymmetry
WR PTP
Maciej Lipinski White Rabbit 29/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
White Rabbit extension to PTP
-
7/21/2019 White Rabbit Tutorial IEEE
52/124
White Rabbit extension to PTP
White Rabbit requires:WR-specific statesExchange of WR-specific information
White Rabbit estimates link asymmetry
WR PTP
PTP extensions mechanismsEnhanced precisiont1,t2,t3,t4Correction for asymmetryInteroperability with PTP gear
Maciej Lipinski White Rabbit 29/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
White Rabbit extension to PTP
-
7/21/2019 White Rabbit Tutorial IEEE
53/124
White Rabbit requires:WR-specific statesExchange of WR-specific information
White Rabbit estimates link asymmetry
WR PTP
PTP extensions mechanismsEnhanced precisiont1,t2,t3,t4Correction for asymmetryInteroperability with PTP gear
ISPCS Plug Fest
WR: most accurate PTP implementation
in the world!
Maciej Lipinski White Rabbit 29/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
WR Standardization under IEEE1588
-
7/21/2019 White Rabbit Tutorial IEEE
54/124
We want to standardize!
Maciej Lipinski White Rabbit 30/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
WR Standardization under IEEE1588
-
7/21/2019 White Rabbit Tutorial IEEE
55/124
We want to standardize!
Intention by p1588 SGexpressed in PAR
Maciej Lipinski White Rabbit 30/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
WR Standardization under IEEE1588
-
7/21/2019 White Rabbit Tutorial IEEE
56/124
We want to standardize!
Intention by p1588 SGexpressed in PAR
Enhanced Accuracy
Options / Profile
Maciej Lipinski White Rabbit 30/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
WR synchronization performance
-
7/21/2019 White Rabbit Tutorial IEEE
57/124
Maciej Lipinski White Rabbit 31/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
WR synchronization performance
-
7/21/2019 White Rabbit Tutorial IEEE
58/124
Maciej Lipinski White Rabbit 31/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
WR synchronization performance
-
7/21/2019 White Rabbit Tutorial IEEE
59/124
Maciej Lipinski White Rabbit 31/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
WR synchronization performance
-
7/21/2019 White Rabbit Tutorial IEEE
60/124
Maciej Lipinski White Rabbit 31/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
WR synchronization performance
-
7/21/2019 White Rabbit Tutorial IEEE
61/124
Maciej Lipinski White Rabbit 31/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
WR synchronization performance
-
7/21/2019 White Rabbit Tutorial IEEE
62/124
Maciej Lipinski White Rabbit 31/59
-
7/21/2019 White Rabbit Tutorial IEEE
63/124
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
WR synchronization performance
-
7/21/2019 White Rabbit Tutorial IEEE
64/124
Maciej Lipinski White Rabbit 31/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
WR synchronization performance
-
7/21/2019 White Rabbit Tutorial IEEE
65/124
Maciej Lipinski White Rabbit 31/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Timing demo
-
7/21/2019 White Rabbit Tutorial IEEE
66/124
Maciej Lipinski White Rabbit 32/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Outline
-
7/21/2019 White Rabbit Tutorial IEEE
67/124
1 Introduction
2 CERN Control & Timing
3 WR Network
4 Time DistributionTiming demo
5 Data Distribution
Redundancy demo
6 WR @ CERN
7 Summary
Maciej Lipinski White Rabbit 33/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Data Distribution in a White Rabbit Network
-
7/21/2019 White Rabbit Tutorial IEEE
68/124
Maciej Lipinski White Rabbit 34/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Determinism and Latency (Switch)
-
7/21/2019 White Rabbit Tutorial IEEE
69/124
Maciej Lipinski White Rabbit 35/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
High Priority
-
7/21/2019 White Rabbit Tutorial IEEE
70/124
Types of data distinguished by802.1Q tag:
High Priority(strict priority)Standard Data (Best Effort)
High Prioritycharacteristics:
Broadcast/MulticastLow-latencyDeterministicUni-directionalRe-transmission excluded
Failure ofHigh Priority:Medium imperfectionNetwork element failureExceeded latency
Maciej Lipinski White Rabbit 36/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Determinism and Latency
-
7/21/2019 White Rabbit Tutorial IEEE
71/124
Deterministic Latency of High Priority
By design:
-
7/21/2019 White Rabbit Tutorial IEEE
72/124
Maciej Lipinski White Rabbit 38/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Data Redundancy
-
7/21/2019 White Rabbit Tutorial IEEE
73/124
Forward Error Correction (FEC) transparent layer:
One message encoded into N Ethernet framesRecovery of message from any M (M
-
7/21/2019 White Rabbit Tutorial IEEE
74/124
Forward Error Correction (FEC) transparent layer:
One message encoded into N Ethernet framesRecovery of message from any M (M
-
7/21/2019 White Rabbit Tutorial IEEE
75/124
Forward Error Correction (FEC) transparent layer:
One message encoded into N Ethernet framesRecovery of message from any M (M
-
7/21/2019 White Rabbit Tutorial IEEE
76/124
Forward Error Correction (FEC) transparent layer:
One message encoded into N Ethernet framesRecovery of message from any M (M
-
7/21/2019 White Rabbit Tutorial IEEE
77/124
Maciej Lipinski White Rabbit 40/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Topology Redundancy (Switch)
-
7/21/2019 White Rabbit Tutorial IEEE
78/124
Ideas:
Enhanced Link Aggregation Control Protocol (eLACP)WR Rapid Spanning Tree Protocol (WR RSTP)WR Shortest Path Bridging (WR SPB)
Seamless redundancy = FEC + WR RSTP/SPB/eLACP
Redundant data received in end stations
Take advantage ofbroadcast/multicastcharacteristic of
Control Data traffic (within VLAN)
Maciej Lipinski White Rabbit 41/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Topology Redundancy: eLACP (short explanation)
-
7/21/2019 White Rabbit Tutorial IEEE
79/124
Control Message encoded into 4 Ethernet Frames
(F1,F2,F3,F4). Reception of any two enables to recover ControlMessage (Cesar Prados, GSI).
Courtesy of Cesar Prados
Maciej Lipinski White Rabbit 42/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Topology Redundancy: WR RSTP
-
7/21/2019 White Rabbit Tutorial IEEE
80/124
Speed up RSTP max 2 frames lost on
re-configuration
H/W switch-over to the backup linkRSTPs a priori information
(alternate/backup) used
Limited number of allowed topologies
Drop only on reception within VLAN
Maciej Lipinski White Rabbit 43/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Topology Redundancy: WR SPB
-
7/21/2019 White Rabbit Tutorial IEEE
81/124
Shortest Path Bridging VID (SPBV)
Backup ports blocking on reception
Single port forwarding from sourceH/W switch-over to path equally or
more distant to the root
Maciej Lipinski White Rabbit 44/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Topology Redundancy: WR SPB
-
7/21/2019 White Rabbit Tutorial IEEE
82/124
Shortest Path Bridging VID (SPBV)
Backup ports blocking on reception
Single port forwarding from sourceH/W switch-over to path equally or
more distant to the root
Not fully congruent
Maciej Lipinski White Rabbit 44/59 Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Topology Redundancy: WR SPB
-
7/21/2019 White Rabbit Tutorial IEEE
83/124
Shortest Path Bridging VID (SPBV)
Backup ports blocking on reception
Single port forwarding from sourceH/W switch-over to path equally or
more distant to the root
Not fully congruent
New link metrics: link delay
Maciej Lipinski White Rabbit 44/59 Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Topology Resolution Unit (TRU)
-
7/21/2019 White Rabbit Tutorial IEEE
84/124
Configurable module to support various software protocols
Accepts active and backup port masks (ingress and egress)
Monitors and controls ports state
Takes actions on HW-filtered frames and link-down
Triggers hardware generation of frames
Maciej Lipinski White Rabbit 45/59 Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Topology Resolution Unit (TRU)
-
7/21/2019 White Rabbit Tutorial IEEE
85/124
Marker-based hardware-switch-over
Hardware-generated priority-based PAUSEHardware-generated BPDUs
Hardware-detection of BPDUs to open
blocking (pre-configured) port
Maciej Lipinski White Rabbit 46/59 Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Other features/ideas
-
7/21/2019 White Rabbit Tutorial IEEE
86/124
Semi-automatic reconfiguration
Maciej Lipinski White Rabbit 47/59 Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Other features/ideas
-
7/21/2019 White Rabbit Tutorial IEEE
87/124
Semi-automatic reconfiguration
Time-triggered reconfiguration
Maciej Lipinski White Rabbit 47/59 Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Other features/ideas
-
7/21/2019 White Rabbit Tutorial IEEE
88/124
Semi-automatic reconfiguration
Time-triggered reconfiguration
Time-aware shaper
Maciej Lipinski White Rabbit 47/59 Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Other features/ideas
-
7/21/2019 White Rabbit Tutorial IEEE
89/124
Semi-automatic reconfiguration
Time-triggered reconfiguration
Time-aware shaper
Drop non-High Priority frames
whenHigh Priorityarrives
Maciej Lipinski White Rabbit 47/59 Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
White Rabbit and IEEE 802
-
7/21/2019 White Rabbit Tutorial IEEE
90/124
We want to be standard-compatible!
Maciej Lipinski White Rabbit 48/59 Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
White Rabbit and IEEE 802
-
7/21/2019 White Rabbit Tutorial IEEE
91/124
We want to be standard-compatible!
Ideas in line with Time Sensitive Networks
Maciej Lipinski White Rabbit 48/59 Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
White Rabbit and IEEE 802
-
7/21/2019 White Rabbit Tutorial IEEE
92/124
We want to be standard-compatible!
Ideas in line with Time Sensitive Networks
Great potential for collaboration between CERN and IEEE
Maciej Lipinski White Rabbit 48/59 Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
White Rabbit and IEEE 802
-
7/21/2019 White Rabbit Tutorial IEEE
93/124
We want to be standard-compatible!
Ideas in line with Time Sensitive Networks
Great potential for collaboration between CERN and IEEE
Perfect platform for prototyping
Maciej Lipinski White Rabbit 48/59 Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Topology reconfiguration performance
-
7/21/2019 White Rabbit Tutorial IEEE
94/124
Maciej Lipinski White Rabbit 49/59 Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Topology reconfiguration performance
-
7/21/2019 White Rabbit Tutorial IEEE
95/124
Maciej Lipinski White Rabbit 49/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Redundancy demo
-
7/21/2019 White Rabbit Tutorial IEEE
96/124
Maciej Lipinski White Rabbit 50/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Outline
1 I t d ti
-
7/21/2019 White Rabbit Tutorial IEEE
97/124
1 Introduction
2 CERN Control & Timing
3 WR Network
4 Time Distribution
Timing demo
5 Data Distribution
Redundancy demo
6 WR @ CERN
7 Summary
Maciej Lipinski White Rabbit 51/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
WR-based Control and Timing System (concept)
-
7/21/2019 White Rabbit Tutorial IEEE
98/124
4 accelerator networks
SeparateData Master (DM)for each network
LIC Data Mastercommunicates with other DMs andcontrol devices in their networks
Broadcast/multicast ofControl Messages
Maciej Lipinski White Rabbit 52/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
WR-based Control and Timing System (concept)
-
7/21/2019 White Rabbit Tutorial IEEE
99/124
Maciej Lipinski White Rabbit 53/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
WR-based Control and Timing System (concept)
-
7/21/2019 White Rabbit Tutorial IEEE
100/124
Maciej Lipinski White Rabbit 53/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Accelerator Networks
-
7/21/2019 White Rabbit Tutorial IEEE
101/124
Maciej Lipinski White Rabbit 54/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Traffic distribution: VLANs + multicast
-
7/21/2019 White Rabbit Tutorial IEEE
102/124
Maciej Lipinski White Rabbit 55/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Outline
1 Introduction
-
7/21/2019 White Rabbit Tutorial IEEE
103/124
1 Introduction
2 CERN Control & Timing
3 WR Network
4 Time Distribution
Timing demo
5 Data Distribution
Redundancy demo
6 WR @ CERN
7 Summary
Maciej Lipinski White Rabbit 56/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
White Rabbit Family
Successful international collaboration of institutes, universities
-
7/21/2019 White Rabbit Tutorial IEEE
104/124
and companies
WR Users:
http://www.ohwr.org/projects/white-rabbit/wiki/WRUsers
Maciej Lipinski White Rabbit 57/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
White Rabbit Family
Successful international collaboration of institutes, universities
-
7/21/2019 White Rabbit Tutorial IEEE
105/124
and companies
WR Users:
http://www.ohwr.org/projects/white-rabbit/wiki/WRUsers
Maciej Lipinski White Rabbit 57/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Pushing frontiers
-
7/21/2019 White Rabbit Tutorial IEEE
106/124
Scientific, open (H/W & S/W), with companies
Maciej Lipinski White Rabbit 58/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Pushing frontiers
-
7/21/2019 White Rabbit Tutorial IEEE
107/124
Scientific, open (H/W & S/W), with companies
More applications than ever expected
Maciej Lipinski White Rabbit 58/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Pushing frontiers
-
7/21/2019 White Rabbit Tutorial IEEE
108/124
Scientific, open (H/W & S/W), with companies
More applications than ever expected
A versatile solution for general control and data acquisition
Maciej Lipinski White Rabbit 58/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Pushing frontiers
-
7/21/2019 White Rabbit Tutorial IEEE
109/124
Scientific, open (H/W & S/W), with companies
More applications than ever expected
A versatile solution for general control and data acquisition
Fulfilling all our needs in synchronization and determinism
Maciej Lipinski White Rabbit 58/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Pushing frontiers
-
7/21/2019 White Rabbit Tutorial IEEE
110/124
Scientific, open (H/W & S/W), with companies
More applications than ever expected
A versatile solution for general control and data acquisition
Fulfilling all our needs in synchronization and determinismStandard-compatible and standard-extending
Maciej Lipinski White Rabbit 58/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Pushing frontiers
-
7/21/2019 White Rabbit Tutorial IEEE
111/124
Scientific, open (H/W & S/W), with companies
More applications than ever expected
A versatile solution for general control and data acquisition
Fulfilling all our needs in synchronization and determinismStandard-compatible and standard-extending
Active participation in IEEE1588 revision process
Maciej Lipinski White Rabbit 58/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Pushing frontiers
-
7/21/2019 White Rabbit Tutorial IEEE
112/124
Scientific, open (H/W & S/W), with companies
More applications than ever expected
A versatile solution for general control and data acquisition
Fulfilling all our needs in synchronization and determinismStandard-compatible and standard-extending
Active participation in IEEE1588 revision process
Eager to collaborate with IEEE802
Maciej Lipinski White Rabbit 58/59
Introduction CERN Control & Timing WR Network Time Distribution Data Distribution WR @ CERN Summary
Thank you
-
7/21/2019 White Rabbit Tutorial IEEE
113/124
More information:http://www.ohwr.org/projects/white-rabbit/wiki
Maciej Lipinski White Rabbit 59/59
Backup slides
Fixed Delays Measurement
-
7/21/2019 White Rabbit Tutorial IEEE
114/124
PHYRx RxCLK
TxCLKTx
00 00 01 11 11 11 11 10 00 01 0 0 01
Phase
detector
Phase
detectorBuffer
Buffer
125 MHz
rx
tx
Maciej Lipinski White Rabbit 60/59
Backup slides
WR RSTP: adding new network element
-
7/21/2019 White Rabbit Tutorial IEEE
115/124
Maciej Lipinski White Rabbit 61/59
Backup slides
Topology Resolution Unit (TRU)
-
7/21/2019 White Rabbit Tutorial IEEE
116/124
Maciej Lipinski White Rabbit 62/59
Backup slides
WR RSTP + FEC
-
7/21/2019 White Rabbit Tutorial IEEE
117/124
Maciej Lipinski White Rabbit 63/59
Backup slides
Digital Dual Mixer Time Domain (DMTD) phase
detector
-
7/21/2019 White Rabbit Tutorial IEEE
118/124
Fully digital, so fully linearCan handle multiple channels without need for extra
hardware
Maciej Lipinski White Rabbit 64/59
Backup slides
New time transfer with WR for CNGS
-
7/21/2019 White Rabbit Tutorial IEEE
119/124
Maciej Lipinski White Rabbit 65/59
Backup slides
WR installation for CNGS
-
7/21/2019 White Rabbit Tutorial IEEE
120/124
Grandmaster WR Switch
8 km of fiber between switches
Boundary Clock WR Switch
WR Node includes Time-to-DigitalConverter (TDC):
55 ps precision (std. dev)300 ps accuracy
Performance monitoring
Maciej Lipinski White Rabbit 66/59
Backup slides
Temperature tests setup (1)
-
7/21/2019 White Rabbit Tutorial IEEE
121/124
Measurement of WR Timebase (clock)Skew measurement with oscilloscope
Maciej Lipinski White Rabbit 67/59
Backup slides
Temperature tests setup (2)
-
7/21/2019 White Rabbit Tutorial IEEE
122/124
Maciej Lipinski White Rabbit 68/59
Backup slides
Temperature tests results (1)
-
7/21/2019 White Rabbit Tutorial IEEE
123/124
Maciej Lipinski White Rabbit 69/59
Backup slides
Temperature tests results (2)
-
7/21/2019 White Rabbit Tutorial IEEE
124/124
The change of time offsetdue to temperature changes
4 ps per 1C
Maciej Lipinski White Rabbit 70/59
