CERN IT Department CH-1211 Genève 23 Switzerland t Status and Plans TERENA 2010 Vilnius, Lithuania...
-
Upload
olivia-mason -
Category
Documents
-
view
217 -
download
0
Transcript of CERN IT Department CH-1211 Genève 23 Switzerland t Status and Plans TERENA 2010 Vilnius, Lithuania...
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
Status and Plans
TERENA 2010
Vilnius, Lithuania
John Shade /CERN
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
Large Hadron Collider
LHCOPN Presentation, TERENA 2010 – Slide 2
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
CERN Accelerator Complex
LHCOPN Presentation, TERENA 2010 – Slide 3
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
26659m in Circumference
5000 SC Magnets pre‑cooled to -193.2°C (80 K)using 10 080 tonnes of liquid nitrogen
60 tonnes of liquid helium bring them down to -271.3°C (1.9 K).
In 1 second, a proton will circulate the LHC 11245 times
The internal pressure of the LHC is 10-13 atm, ten times less than the pressure on the Moon
600 Million Proton Collisions/second
LHC (Some) Facts and Figures
LHCOPN Presentation, TERENA 2010 – Slide 4
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
CERN’s LHC Detectors
LHCOPN Presentation, TERENA 2010 – Slide 5
ALICE
26 m long 16 m high 16 m wide10 000 tonnes
CMS
21 m long 15 m high 15 m wide12 500 tonnes
LHCb
21m long 10m high 13m wide5600 tonnes
46 m long 25 m high 25 m wide7000 tonnes
ATLAS
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
30-MAR-2010 – collisions at 7TeV
LHCOPN Presentation, TERENA 2010 – Slide 6
On 30 March 2010, beams collided in the LHC at 7 TeV, the highest energy ever achieved in a particle accelerator, marking a new world record and the start of the LHC research programme.
More than half a billion collisions observed to date. Physicists from all over the world are analysing the new data and retracing the particles discovered in past experiments (e.g. W particle and the B-meson).
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
ALICE Events
LHCOPN Presentation, TERENA 2010 – Slide 7
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
ATLAS Collisions
LHCOPN Presentation, TERENA 2010 – Slide 8
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
CMS Collisions
LHCOPN Presentation, TERENA 2010 – Slide 9
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
LHCb Events
LHCOPN Presentation, TERENA 2010 – Slide 10
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
Detectors ~150 million electronic channels40 MHz collision rate
Fast response electronics,FPGA, embedded processors,very close to the detector
N x 10 Gbit links to the Computer Center
LHC
Accelerator and 4 Experiments
Level 1 Filter and Selection
Level 2 Filter and Selection
1 Pbytes/s
150 Gbytes/s
0.7 Gbytes/s
CERN Computer Center
Event selection is based on the physics model,(“prejudice”, expectations)Will change over time,“limits” the physics
LHCOPN Presentation, TERENA 2010 – Slide 11
Raw Data Rates
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
4 Experiments
3 PBytes/s
We are looking for 1 ‘good’ snapshot in 10 000 000 000 000 ‘photos’ We are looking for 1 ‘good’ snapshot in 10 000 000 000 000 ‘photos’
2 GBytes/sto the CERN computer center
Store on disk and tape
World-Wide Analysis
Export copies
Create sub-samples
col2f
2f
3Z
ff2Z
ffee2Z
0
ff
2z
2Z
222Z
2Z0
ffff
N)av(26
m and
m
12
withm/)m-(
_
__
FG
ss
s
PhysicsExplanation of nature
PhysicsExplanation of nature
10 GBytes/s 4 GBytes/s
1 TByte/s ?Distributed + local
The DataflowThe Dataflow
Filter and first selection
The Data Flow
LHCOPN Presentation, TERENA 2010 – Slide 12
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
WLCG Project
• The Worldwide LHC Computing Grid Project, WLCG, (http://lcg.web.cern.ch/LCG/) is a global collaboration of more than 140 computing centres in 34 countries.
• The mission of the WLCG project is to build and maintain a data storage and analysis infrastructure for the entire high energy physics community that will use the Large Hadron Collider at CERN.
LHCOPN Presentation, TERENA 2010 – Slide 13
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
Tier0 center, CERN very large center; tape storage; 1st-level processing and meta-data
storage; quality service (24*7)
• Tier1 center, 11 world-wide large capacity; tape storage; quality service
• Tier2 center, 129 world-wide medium size, some large; no 24*7 service; no custodial storage
• Tier3 center very small to medium size; no availability guarantee; focus on end-user analysis activity
Tier Structure
LHCOPN Presentation, TERENA 2010 – Slide 14
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
The Beginning ...
• Essential for Grid functioning to distribute data out to the T1’s.– Capacity had to be large enough to deal with most situations
including “Catch up”
• LHCOPN proposed in 2004 by D. Foster (CERN) as a “Community Network”– Renamed as “Optical Private Network” as a more descriptive
name after the initial meeting of stakeholders in Amsterdam.– Based on 10G as the best choice for affordable adequate
connectivity by 2008.• Considered by some as too conservative - can fill a 10G pipe with just (a
few) PC’s!• 10G is commodity now!
• Simple end-end model– This was not a research project, but, an evolving production
network relying on emerging facilities.
LHCOPN Presentation, TERENA 2010 – Slide 15
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
Issues, Risks, Mitigation
• It is a complex multi-domain network relying on infrastructure provided by:– (links) NREN’s, Dante and commercial providers– (L3) T1’s and CERN– (operations) T1’s, CERN, EGI and USLHCNet
• Managed by the community– “Closed Club” of participants– Simple L3 model, routers at the end points – Federated operational model
• Design separated from implementation• Need to combine innovation and operation
LHCOPN Presentation, TERENA 2010 – Slide 16
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
LHCOPN L2 Network Map
LHCOPN Presentation, TERENA 2010 – Slide 17
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
Current Situation
• T0-T1 Network is operational and stable.• Several areas of focus:
– Physical Path Routing– Operational Support– Monitoring
LHCOPN Presentation, TERENA 2010 – Slide 18
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
Operational Model
• GGUS used as ticket & tracking system• Twiki used for collaboration documents
• Incident Management procedures• Change Management procedures
• Conference calls used for Operations and Monitoring, with F2F meetings on a regular basis
LHCOPN Presentation, TERENA 2010 – Slide 19
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
T0-T1 links’ availability
LHCOPN Presentation, TERENA 2010 – Slide 20
ExplanationLink availability calculated on the uptime of the BGP routing protocol relationship with each neighbor. It says for how long the T0's routers were able to route traffic to/from every T1 over the LHCOPN link. It doesn't say anything about Data-Centre to Data-Centre connectivity.
FNAL-Sec and BNL-sec are the additional 10G links recently deployed, thus the high “Time Undertermined”.
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
Basic Link Layer Monitoring
• perfSONAR - Integrated into the “End to End Coordination Unit” (E2ECU) run by DANTE– Provides simple indications of “hard” faults.– Insufficient to understand the quality of the
service
• Dante MDM deployed (but is it useful?)– Configuration audit underway.
• Specifications for visualisation dashboard(s) being developed– Access to the MDM data is needed
LHCOPN Presentation, TERENA 2010 – Slide 21
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
LHCOPN Traffic
LHCOPN Presentation, TERENA 2010 – Slide 22
http://network-statistics.web.cern.ch/network-statistics/ext/?p=sc
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
LHCOPN Core Weekly Traffic
LHCOPN Presentation, TERENA 2010 – Slide 23
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
LHCOPN Yearly Traffic Volume
LHCOPN Presentation, TERENA 2010 – Slide 24
CA-TRIUMF
DE-KIT
ES-PIC
FR-IN2P3
IT-INFN-CNAF
NDGF
NL-T1
TW-ASGC
UK-T1-RAL
US-FNAL-CMS
US-T1-BNL
0 1000 2000 3000 4000 5000 6000
Volume From T1s (TB)Volume To Tier1s (TB)
Tera Bytes
Traffic exchanged among the T0 and the Tier1s and transited through the CERN routers. Period: 05-2009 to 05-2010(source: http://network-statistics.web.cern.ch/network-statistics/ext/?p=sc&q=LHCOPN%20Total%20Traffic&m=LHCOPN-Total)
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
Link capacity
LHCOPN Presentation, TERENA 2010 – Slide 25
• LHCOPN uses 10Gb/s links between Tier0 and all Tier1s• For BNL, FERMI and RAL, two 10Gbps
links are configured in round-robin, thus providing 20Gbps
• Renater dark fibre to Lyon would allow IN2P3 to upgrade
• We await (next year?) to see the prices of 40Gb interfaces. Will they be cheaper than 4*10?
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
Future Plans
LHCOPN Presentation, TERENA 2010 – Slide 26
• LHCOPN “core” (Tier0, Tier1s):• Extend capacity for Tier1-Tier1?• Extended deployment of Cross-Border
Fibre?• Dedicated Tier1 Exchange Point (T1XP)?
• Use LHCOPN operational model for Tier2s?• Hard to impose a central model on Tier2s
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
Conclusion
LHCOPN Presentation, TERENA 2010 – Slide 27
• Current LHCOPN (core) does what it was designed for - shouldn’t be modified (extend – not redesign)
• We should care about the “broader picture”- data distribution and movement does not
stop at Tier1s and is important for LHC experiments’ operation
• Tier1-Tier1 and Tier1-Tier2 data movements will need to be addressed, probably sooner rather than later
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
Conclusions (continued)
LHCOPN Presentation, TERENA 2010 – Slide 28
• Experiments’ use of the OPN is exceeding initial estimates
• We need to foresee growth of connectivity to 40G and 100G and explore the paradigms of dynamic circuit provisioning • Transatlantic connectivity will be a challenge!