LHC enhanced Quench Protection System project

Fabio Formenti - Feb. 25th, 2009 1

LHC enhanced

Quench Protection System

project

Review committee meeting

February 24th–26th 2009


Presentation outline

Upgrade motivations

Main requirements for the project

Previous internal review of new electronics for QPS

Project organization

Project planning

Outsourcing

Project management: main follow up meetings

Quality assurance: main points

Conclusions


Upgrade motivations Top priorities before next machine startup

improved coverage of quench detection due to:

- Abnormal splice joint resistance

(from Sept. 19th 2008 task force incident analysis)

- Heat, beam loss, …etc…, induced symmetric quench

(decision at Chamonix meeting, Feb. 2nd-6th, to run from day 1 at high energy)

improved fault reliability of 230V UPS system

(decision at Chamonix meeting, Feb. 2nd-6th)

Optional need for later machine startup:

capability to monitor magnet ground voltage variations

(this case is not treated now)


Main goals for the project

1. Avoid changing the present magnet quench protection system• It is proven to perform as expected• Avoid re-commissioning

2. Add more electronics and infrastructure for bus bar joints and symmetric quench protections:

• ~500 DQAMG type S controller boards• ~2500 DQQBS bus detection boards• ~2000 DQQDS symmetric quench detection boards• ~450 DQLPUS power supply units• ~450 Crates with backplanes• ~4400 Cables to dipoles and quadrupoles, total length ~230 km• ~1600 Interlock patch panel

3. Duplicate UPS power distribution system• ~900 Power packs• ~1600 Distribution patch panels (may be two different types)• ~1600 New power cables to QPS racks• ~1600 QPS racks to update cable layout• ~80 Protection units for the insertion region magnets (triplets)

Total

~5900 boards

Another ~900 boards

5Fabio Formenti - Feb. 25th, 2009

Internal review of new electronics for QPS

• Held on Jan. 13th

• Mainly aimed at production readiness

• Report available on INDICO(http://indico.cern.ch/getFile.py/access?resId=0&materialId=minutes&confId=51077

Note: it needs NICE login)

• Main recommendations concerned:

– Management, planning and resources

– Radiation Tolerance

– Technical issues for electronics

(EMC, cabling, power, electronics reliability, electronics implementation, documentation)

– Production and its follow up

– Quality assurance

• 22 recommendations in total that are being applied according to project progression

=> a meeting with referees will be organized for providing status <=

http://indico.cern.ch/getFile.py/access?resId=0&materialId=minutes&confId=51077


• Project breakdown structure has been defined (started beginning January)

• Project leader: Knud Dahlerup-Petersen

• Project technical coordinator: Fabio Formenti

• 17 upgrade Work Packages

it reflects today’s understanding, however trend seems pointing to a growth

e.g. protection of insertion regions, more routine activities, other new priorities (maybe this review recommendations?)

• 10 different responsibles of Work Packages

• Go to working document =>

Project organization

WP Task I tem DescriptionQPS EUP 1 DQQBS detection boards R&D and procurement

1.1 Board design define design, analyse/simulate the board architecture, define performance for specifications, estimate performance boundary conditions

1.2 Final board specifications write board functional specifications, provide board manufacturing documentation, define test system for mass production

1.3 Call for tender CERN simplified call for tender, negotiate and place the contract

1.4 Board production follow-up the production of the boards, perform acceptance test and quality assurance, provide tester for mass production to Contractor

1.5 Board reception perform receptions at CERN, repair faulty boards, manage storage for installation, dowload final embedded s/w or f/w

ResponsibleReiner Denz

WP Task I tem DescriptionQPS EUP 2 DQAMG type S Controller board R&D and procurement

ResponsibleReiner Denz

WP Task I tem DescriptionQPS EUP 18 DQQDS Symmetric quench detector board R&D and procurement

ResponsibleVincent Froidbise

WP Task I tem DescriptionQPS EUP 3 DQLPU type S Crate including motherboard and connector board R&D and procurement

ResponsibleJ oaquim Mourao

WP Task I tem DescriptionQPS EUP 4 DQLPUS Power Supply R&D and procurement

ResponsibleVincent Froidbise


Project planning (PRELIMINARY ESTIMATE!)• Chosen MS Project as project management tool to implement resource planning (as part of QA)

• Detailed dates, milestones and resource assignment (ongoing process)

-> The document is now being revised because of new defined priorities (symm. quench & UPS)

• Included: all upgrade WPs and most urgent M&O tasks impacting on resource sharing

• Timescale of QPS Electronics Upgrade Project: started Sept. 1st 08 – ending milestone Aug. 31st 09

• Planning based on: 40h/w working time

not included (yet?) other holidays than week-ends and official holidays

contingency cannot simply apply -> rescheduling and adding resources

• Go to working document =>ID Task Number Task Name Duration Start Finish

1 QPS EUP Proj ectQPS Electronics Upgrade Project (release date: February 12nd, 2009) 262 days? Fri 15-08-08 Fri 04-09-09

2 QPS EUP 1 DQQBS detection boards R&D and procurement 217 days Mon 29-09-08 Fri 14-08-09

3 1.1 Board design 70 days Mon 29-09-08 Fri 16-01-09

6 1.2 Final board specifications 25 days Mon 05-01-09 Fri 06-02-09

10 1.3 Call for tender 25 days Mon 05-01-09 Fri 06-02-09

11 1.3.1 definiti on of techni cal matters, final technical speci fications 5 days Mon 05-01-09 Fri 09-01-09

12 1.3.2 definiti on of commercial matters 5 days Mon 05-01-09 Fri 09-01-09

13 1.3.3 decision and suppli er selection 15 days Mon 12-01-09 Fri 30-01-09

14 1.3.4 visi t Contractor and placing order 5 days Mon 02-02-09 Fri 06-02-09

15 1.4 Board production 165 days Mon 20-10-08 Wed 24-06-09

16 1.4.1 contract fol low up 90 days Mon 16-02-09 Wed 24-06-09

17 1.4.2 parts procurement 75 days Mon 20-10-08 Fri 13-02-09

18 1.4.3 quali fication of the preseries producti on 5 days Mon 16-02-09 Fri 20-02-09

19 1.4.4 tester for mass production (h/w) 15 days Mon 26-01-09 Fri 13-02-09

20 1.4.5 tester for mass production (s/w) 15 days Mon 26-01-09 Fri 13-02-09

21 1.4.6 production, test and QA 85 days Mon 23-02-09 Wed 24-06-09

22 1.5 Board reception 141.88 days Mon 26-01-09 Fri 14-08-09

31 QPS EUP 2 DQAMG type S Controller board R&D and procurement 202 days Mon 20-10-08 Fri 14-08-09

61 QPS EUP 3 DQLPU type S Crate including motherboard and connector board R&D and procurement182 days Mon 17-11-08 Fri 14-08-09

85 QPS EUP 4 DQLPUS Power Supply R&D and procurement 182 days Mon 17-11-08 Fri 14-08-09

108 QPS EUP 5 Signal cables and their connectiv ity 146.13 days Mon 27-10-08 Fri 05-06-09

129 QPS EUP 6 Controls and interlocks signals upgrade 113.13 days Mon 24-11-08 Fri 15-05-09

139 QPS EUP 7 System test setup and IST in QPS lab 63 days Mon 02-03-09 Fri 29-05-09

150 QPS EUP 8 Radiation tolerance tests 21 days Mon 16-03-09 Tue 14-04-09

158 QPS EUP 9 System installation and IST in LHC tunnel 55 days Mon 06-04-09 Fri 26-06-09

166 QPS EUP 13 Implementation of frontend s/w for specific measurement campaign 86.13 days Mon 23-02-09 Fri 26-06-09

170 QPS EUP 14 Infrastructure and connectiv ity 82 days Thu 18-12-08 Fri 24-04-09

179 QPS EUP 15 Remote reset facilities 43 days Fri 24-04-09 Fri 26-06-09

184 QPS EUP 17 Hardware commissioning of new protection system 45 days Mon 29-06-09 Fri 28-08-09

201 QPS EUP 18 DQQDS Symmetric quench detector board R&D and procurement 262 days? Fri 15-08-08 Fri 04-09-09

230 QPS EUP 19 DQQDE v oltage to ground feeler board R&D and procurement 1 day? Thu 29-01-09 Thu 29-01-09

237 QPS EUP 20 Installation of already existing arc detectors in DQS (option) 0 days Fri 26-06-09 Fri 26-06-09

241 QPS EUP 21 UPS upgrade 15 days? Mon 19-01-09 Fri 06-02-09

QPS Electronics Upgrade Project (release date: February 12nd, 2009) Knud Dahlerup[10%],Fabio Formenti[10%]

DQQBS detection boards R&D and procurement Reiner Denz[10%]

Board design

Final board specifications

Call for tender

Knud Dahlerup[2%],Arend Dinius[10%]

Laszlo Abel

Knud Dahlerup[1%],Laszlo Abel,Reiner Denz[2%],Fabio Formenti[2%],Arend Dinius[10%]

Arend Dinius[0%],Reiner Denz[10%]

Board production

Reiner Denz[10%],Knud Dahlerup[2%],Fabio Formenti[2%],Arend Dinius[15%],Vincent Froidbise[5%]

Vincent Froidbise[5%],Contractor

Vincent Froidbise[2%],Mathieu Favre[15%]

Vincent Froidbise[2%],Grzegorz Seweryn[10%]

Reiner Denz[2%]

Arend Dinius[20%],Contractor

Board reception

DQAMG type S Controller board R&D and procurement Reiner Denz[10%]

DQLPU type S Crate including motherboard and connector board R&D and procurement Joaquim Mourao[5%],Arend Dinius[5%],Fabio Formenti[5%]

DQLPUS Power Supply R&D and procurement Vincent Froidbise[5%],Knud Dahlerup[5%],Arend Dinius[5%]

Signal cables and their connectiv ity Gert-Jan Coelingh[10%]

Controls and interlocks signals upgrade Herv e Milcent[10%]

System test setup and IST in QPS lab Reiner Denz[10%]

Radiation tolerance tests Fabio Formenti[10%],Reiner Denz[5%]

System installation and IST in LHC tunnel Knud Dahlerup[10%]

Implementation of frontend s/w for specific measurement campaign Zinur Charifoulline[10%]

Infrastructure and connectiv ity Noel Fournier[10%]

Remote reset facilities Reiner Denz[10%]

Hardware commissioning of new protection system Rudiger Schmidt[10%]

DQQDS Symmetric quench detector board R&D and procurement Vincent Froidbise[10%]

DQQDE v oltage to ground feeler board R&D and procurement Joaquim Mourao[10%]

Installation of already existing arc detectors in DQS (option) 26 Jun

UPS upgrade Knud Dahlerup[10%]

Jul Sep Nov Jan Mar May Jul Sep3rd Quarter 1st Quarter 3rd Quarter


Outsourcing

Sharing of responsibilities:

specifications, supervision, follow up and commissioning -> MPE group

specific competences -> support from TE, BE, EN, PH and external institutes

production -> industry

Detailed Work Package definition for activities to be outsourced is in final editing

Outsourcing is the only flexibility for this project planning

To be effective, large WPs should be released to well selected competent teams

Go to working document =>


Project management: follow up meetings

Observation: many meetings and discussions are organized…

… but this also takes time to the ongoing activities

=> Carefully attendance to most relevant meetings

and when agenda is highly pertaining <=

• Project status report meetings (Siemko)

When: every Friday, since January 1st

Who: MPE management and engineering staff

What: major technical issues, outsourcing coordination, planning & project management status

• Technical report meetings (Dahlerup Petersen)

When: every Monday, since 13/02/09

Who: all MPE-CP section people and collaborators involved in the QPS project

What: detailed technical issues, activity status, incoming week work planning, supplier follow up, component procurement, logistics

• Hardware commissioning meeting (Schmidt)Starting soon (March 19th)


Project management: contribution to other meetings

• CCC machine re-commissioning meetings

When: every Monday

Who: management, project responsibles, coordinators

What: activity status of magnet repair and new protection system

(basically information meeting)

• Shut down planning meeting

When: every Thursday

Who: CERN activity responsibles, external work contractors

What: global upgrade planning coordination

… this is not surely fully exhaustive list …


Quality assurance

Not trying to compromise on this issue

Main points being implemented:

1. Maximum availability of technical documentation (drawings, procedures, testers, s/w, etc.) on public domain (e.g. EDMS, DFS/public)

2. Thorough prototype verification in house by designers

3. Radiation tests of all key electronics components

4. Careful industry selection to conform to international quality standards (ISO9000)

5. Maximum of system component traceability, use of manufacturing and installation databases

6. Rigorous pre-production qualification and production acceptance tests with standardized procedures

7. Full system pre-commissioning in QPS lab and test result database

8. Installation with trained operators and verification tests by different team

9. Final commissioning supervised by QPS team engineers


Conclusions

Bus bar quench detection

Symmetric quench detection (new priority) Motivate upgrade of QPS system

UPS improved redundancy (new priority)

=> Some new 6800 electronics cards to build and commission in tight time schedule <=

=> Some 1600 QPS racks to update <=

Work breakdown structure has been defined for all main upgrade activities

=> Routine activities, e.g. related to s34 reparation, compete for resource sharing <=

Schedule follow up and resource management have been introduced in planning tool

=> It is under revision for complying with new added priorities <=

Outsourcing becomes a necessity and is being implemented

=> We are editing tasks to define precise competences <=


Extras

Example: cable installation test


1. HV test (500V)


The automated HV tester system

Power taken from QPS crate or tunnel blue power boxes Safe ground connection


Analog cables (blue) and interlock cables (white) are only connected to the tester at its near end (the other end is left open)


Automatic testing (it takes about 8 minutes)

Testing …

… and storing test report

Temperature and humidity shall be verified


2. Continuity test


Coming in front of QPS crate (below B dipole)

Identification of right location in cable database

Connecting tester to power (taken from QPS crate)


Connecting cables to tester

Analog cables (blue) and interlock cables (white) have to be connected to good tester port following the database

First check done with other cable side open

(tester lights must be OFF)


Putting diode taps at other ends of cables

HV check mark must already be present


The two cable ends could be up to a few 100 meters away


Testing and labelling

Tester lights ON if diode tap is connected to a good cable end Put barcode labels at two cable ends

and annotate cable database

LHC enhanced Quench Protection System project

Documents

Transcript of LHC enhanced Quench Protection System project