what were the issues in 2011, minimizing impact and improving recovery

what were the issues in 2011, minimizing impact and improving

recovery M.Solfa

roli

Technical

Stops:Acknowledgments: S.Claudet, K.Dahlerup, F.Duval, K.Foraz, M.Nonis, M.Poyer, H.Thiesen, J.Uythoven, W.Venturini

Evian 2011

M.Solfaroli - Technical Stops 2Evian - 12/12/11

Outline

General view

Methodology

The 5 TSs one by one

Pulling the numbers together

Conclusions


General OverviewStart

dateEnd date Length Time before

next TS (days)

TS#1 28/03 31/03 4 days 38 (5.4 wk)

TS#2 09/05 12/05 4 days 52 (7.4 wk)

TS#3 04/07 08/07 5 days 51 (7.3 wk)

TS#4 29/08 02/09 5 days 65 (9.3 wk)

TS#5 07/11 11/11 5 days 26 (3.7 wk)

Time allocated for recovery: 1 day

23 % of global time = TUNNEL

INTERVENTIONS(including Xmas break)


Methodology

HOW IT HAS BEEN DONE: All TS interventions screened and all recovery issues examined Some beam checks need to be done repetitively (i.e. loss

maps)...should they be considered as part of recovery time? YES! What does represent the limit between recovery and operation

(back in SB, back on program, first beam in,...)?

GOALS: Spot out possible correlations between

recovery time and TS interventions Identify major problems Investigate strategies for improvement

BEGINNING END

TS Mon @7am First HW re-commissioning test

MACHINE RECOVERY First HW re-commissioning test First beam commissioning test

BEAM COMMISSIONING First beam commissioning test GOAL of the week is accomplished


WHERE WERE WE?

Slot of 1.38 TeV operation Last 3.5 TeV physics fill (1645):• 200 b (24 bpinj) - (ready for 296 b)• ~1.22E11 p per bunch• Peak lumi: 2.5E32 cm-2 s-1

TS#128 – 31 March – 4 days + 1 recovery day

60%

30%10%

3243 keys given

GOAL of the weekRecovery from TS and start

preparation for high intensity

M.Solfaroli - Technical Stops 6

Sat 2nd ~2pm

Recovery

Evian - 12/12/11

TS#1 - Recovery

tThu 31st 6.25pm

CRYO

Fri 1st 10.29am

Fri 1st 9.59pm

Beam comm

Start of HWC

Global CRYO start First

pilot

Inj region aperture measurements for

higher intensity

Activity Duration [h]Tunnel activities (TS) 84

Recovery 31

Beam commissioning 12

TOT 127

MKB.B2

Sat 2nd 01.23am

Dump @450GeV

66%

10%24%

TOTAL NOT related HW SWTIME lost 14.5 h 48% 62% 38%

Mon 28th

7am

TS


TS#2

WHERE WERE WE?

Last 3.5 TeV physics fill (1755):• 768 b (72 bpinj)• ~1.25E11 p per bunch• Peak lumi: 9E32 cm-2 s-1

o Quench propagation test on RB.A56o FPA on RQD/F.A67

2831 keys given

60%

16% 24%

09 - 12 May – 4 days + 1 recovery day

GOAL of the weekTS, re-establish physics conditions,

alignment of RPs, VdMs


CRYOCRYO

Evian - 12/12/11

TS#2 - Recovery

tThu 12th 2.10pm

RecoverySat 14th 2.10pm

Beam commSun 14th

9am

Start of HWC


pilotPreparation

for VdM scan


Recovery 55(27h cryo stop not counted)


TOT 146

Thu 12th 6.31pm

Fri 13th 11am

Lost P8(communication)

Fri 13th 08.42am

Sat 14th 9pm

Pilots cycle

54%38%

8%


Mon 9th

7am

TS

~27 h


04 - 08 July – 5 days + 1 recovery day

Evian - 12/12/11

TS#3 - The longest one

WHERE WERE WE?

Last 3.5 TeV physics fill (1901):• 1380 b (144 bpinj)• ~1.16 E11 p per bunch• Peak lumi: 1.26E33 cm-2 s-1

o Quench propagation test on RB.A56o Test of DQQBS boards in S81

3062 keys given

65%

26%9%

GOAL of the weekContinue with the satellite

collision scheme(264b and higher)


CRYO

Evian - 12/12/11

TS#3 – Recovery

tFri 8th 7.19pm

RecoveryMon 11th 7.22pm

BeamThu 14th

7am

Start of HWC


pilotEnd of loss

maps


Recovery 22

Beam commissioning 22(18 + 4)

TOT 148

Sat 9th 8.05am

Sun 10th 11.11am

Sat 9th

5.27pmTue 12th 3.15pm

InjectionCRYOPower cut

When the power cut arrived, beam commissioning was almost completed

(a TCDQ-TCSG cross check was ongoing)

Recovery Beam comm


14.5%14.5%

71%

Mon 04th

7am

TS


TS#4

WHERE WERE WE?

Last 3.5 TeV physics fill (2040):• 1380 b (144 bpinj)• ~1.29E11 p per bunch• Peak lumi: 2.1E33 cm-2 s-1

3645 keys given

69%

24%7%

o Quench propagation test on RQD/F.A56

29 August - 02 September – 5 days + 1 recovery day

GOAL of the weekTS, recovery from TS, 1 m

beta*, Alice polarity change


CRYO

Evian - 12/12/11

TS#4 - Recovery

tMon 29th

7amFri 2nd

2.10pm

Beam commSat 3rd

10.59am

Start of HWC


pilot

Start of fill that reached

beta* 1m


Recovery 9


TOT 124

Fri 2nd

5.14pmFri 2nd

10.33pm

RecoverySat 3rd

1.41am

First pilots ramp

86%

7% 7%

~3 h without beam from injectors


TS


TS#5

WHERE WERE WE?

Last 3.5 TeV physics fill (2267):• 1380 b (144 bpinj)• ~1.5E11 p per bunch• Peak lumi: 3.5E33 cm-2 s-1

o Quench propagation test on RQD/F.A56

3404 keys given

70%

24%6%

07 - 11 November – 5 days + 1 recovery day

GOAL of the weekTS, recovery, ions (stable

beams) over the weekend


CRYO

Evian - 12/12/11

TS#5 - Recovery

tFri 11th

4.58pm

Beam commSat 12th

06.40am

Start of HWC


pilotPhysics

conditions


Recovery 7


TOT 120

Fri 11th 5.48pm

Sat 12th

00.13am

RecoverySat 12th

00.32am

Asynch BD @ 450 GeV

Pb commissioning included

Sat 12th

02.31am

SB

89%

6% 5%

TOTAL NOT related HW SWTIME lost 4 h 50% 62% 38%

Mon 7th

7am

TS


Keys Maintenance Improving Problem fixing

TS#1 3243 60% 30% 10%

TS#2 2831 60% 24% 16%

TS#3 3062 65% 26% 9%

TS#4 3645 69% 24% 7%

TS#5 3404 70% 24% 6%

Tunnel activities vs TSs

Some numbers…


Time lost [h] NOT related HW SW

TS#1 14.5 48% 62% 38%

TS#2 15.5 94% 90% 10%

TS#3 19.5 46% 85% 15%

TS#4 6.5 15% 8% 92%

TS#5 4 50% 62% 38%

Pretty low statistics to have meaningful conclusions......in general HW issues require more time to be fixed

…more…


…and more!Recovery + Beam

commissioningTOT TS time

(x-1)*24 + 12 + 24Recovery coefficient

(theoretical)Recovery

coefficient (real)

TS#1 43 h 108 h 0.22 0.4

TS#2 40 h(67 h including cryo stop) 108 h 0.22 0.37

TS#3 44 h(130 h considering the power cut) 132 h 0.18 0.33

TS#4 18 h 132 h 0.18 0.13

TS#5 13 h 132 h 0.18 0.09

X = number of days allocated Allocated time for recovery = 24 h

Recovery time vs TSs Recovery coefficient


• Cryo: no systematic maintenance anymore, it would be OK with 4 (or even 3) TSs provided a minimum of 5 days each (this year was good!)

• QPS: minimum of 3 stops (ideally in May, August and October). Minimum duration 4 ½ days as in 2011.

• EPC: not particular systematic intervention needed, but at least 1-2 days every 9 – 10 weeks

• EL: TSs needed to repair/reset/maintain some systems that are not in good shape. Nevertheless, till the machines work…

• CV: global time already at the limit. It would be difficult to keep for more than 8-9 weeks

• Experiments? (CMS: “we would be ready to go to 4/5d every 10 weeks if needed”)

• Injectors?

Client requests


Conclusions Need to improve fault details recording

Most of activities is maintenance, can it be reduced?

No systematic source of trouble over the 5 TSs !!

It seems clear that we are improving in recovery… (“After TS, an increment in faults was observed. Effect

is decreasing along the run” Walter @Chamonix 2011) Need to apply a control for SW changes (through a

meeting to coordinate and create a list?) which could: Improve changes, by coordinating them Increase operational efficiency, by making easier

the identification of the source of problems Reduce impact of changes on other systems

4 TSs foreseen for 2012...can we push forward some maintenance and have 3 TSs of 5 days?

2012


Back-up slides


TS#1 – Issues 1/2

Issue Duration Reason Connected?

BLM sanity check failure ~12 h but temporary solution adopted Update of configuration DB Y

RB.A12 multiple trip 3 h Card to be changed Likely (active filter manipulation)

Vacuum valve control 4 h not continuously RBAC issue Likely (RBAC modification?)

RCS.A78B2 trip 2 h Power module to be changed N

QPS (B16.R1 opening quench loop) 1.5 h Noise coming from SPS quad N

Beam process regeneration failure 0.5 h Timeout in DB connection to

be increased N

Loading setting on PCs

0.5 h

Ref mismatch Likely (connected to DB

configuration update?)

...on COLL Position error

...on PCs Java exception

RCBXH3.L1 trip 1 h Manipulation on PC N


TS#1 – Issues 2/2


Dump kicker fault 1 h Local intervention Likely

QPS on RQD/F.A81 0.5 h Expert reset N

RQTD.A45B1 trip 1.5 h Short circuit on the current lead Y

ROD./F.A34B1 trip 0.5 h Fault reset by EPC N

XPOC 1 h Expert check N

Orbit instability (not blocking) ~2 h Not fully understood but solved by

taking out RPLA.30L4.RCBV30.L4B1 N

RF cavity 6B2 trip 0.5 h Spurious interlock N

XPOC/IPOC failure on dump 0.5 h Expert check N

COLL dump 0.5 h TCDQ position interlock enabled during TS Y


TS#2 – Issues 1/2


RB.A23 quench loop cannot be closed 3 h Broken relay to be changed N

RD3.R4 quench3.5 h was OK, but

further investigation done the day after

Reset solved the problem N

RD34 Few min Acc value was incorrect Possibly

DIP communication glitch Few min unknown N

HS failure Masked in SIS, but solution the day after Parameter manipulation on SIS Y

RF desynchronized 0.5 h “We do not know why (aging, temp, humidity...?)” N

CRYO loss 26.5 h Communication N

VAC valves cannot be opened 0.5 h Unknown N


TS#2 – Issues 2/2


XPOC latch and IQC failure 0.5 h DB manipulation done during TS Y

Interlock on TCLIB.6R2B1 2 h Broken sensor N

S12 & S23 trip 4 h AUG TI2 N

DFB level manipulation in S78 1.5 h Level instable N

LBDS self-triggering 3.5 h Power supply of MKD generator M broke N

XPOC analysis repetitive faults n.a. Missing VAC data and wrong

dump patter Y


TS#3 – Issues 1/2


Cannot drive settings on TDI in P8 1.5 h Left in expert mode Y

RQD/F.A78 switches cannot be closed 5h Fuse to be changed N

Beam dump faulty for both beams 0.5 h Not clear N

MCS checks on MKD2 and SMP E limit failure 2.5 h HW and DB inconsistent Y

RCO.A56B2 QPS not OK 0.5 h + 0.5 h Expert reset needed N

MKI1 fails during softstart 2 h Possibly

LBDS cannot be armed 0.5 h Possibly

XPOC expert signature needed 0.5 h Server to be reboot Probably

RQ10.L1B2 trip 3 h Water fault Y


TS#3 – Issues 2/2

Issue Duration Reason Connected?RQ6.L8B1/B2 trip 0 h Water fault Y

Patrol lost 2 h PAD problem N

Problem with actual trim

0.5 h Re-deployment needed LikelyADT setting cannot be charged

Actual trim not working for lumi-knob

XPOC manual reset needed during test

Test was not needed for safe beam...initially delayed, then postponed to next TS

Trying to solve a bad timing problem in inject&dump.Not solved, rolled-back!

Y, but problem was already

present

RQ4.R6B2 trip 0.5 h FGC fault N


TS#4 - Issues


COLL error 2 h CO modification to be rolled-back Y

DC BCT calibration failure Can live with Likely

BPM calibration failure 2 h Crate to be re-started Likely

HS not working properly Still working Not clear Possibly

XPOC EIC reset does not work 1 h Had to be rolled-back Y

Problem with FIDEL corrections

QFB switching OFF + chroma measurement problem

0.5 h Tune fitter settings re-deployed Likely

4 RQDFs trip 0.5 h QFB correction too high N

XPOC reset failure 0.5 h Not clear N

OP vistar not working Not blocking Not clear possibly


TS#5 - Issues


Some settings don’t match DB Y

Some VAC valves don’t open 0.5 h VAC still not good Y

DIP and vistar problem 0 h

RF interlock cannot be reset 2 h

ITL1 wire position not OK Not blocking SIS masked Y

XPOC latched 1 h TSU failure Y

BSRA publishing not valid results 0 h “check mode” left (masked in

the XPOC Y

XPOC failure BLM values out of threshold


CRYO

Evian - 12/12/11

TS#3 - Recovery

tMon 04th Fri 8th 7.19pm

TS RecoveryMon 11th 7.22pm

BeamThu 14th

7am

Start of HWC


pilotEnd of loss

maps


Recovery 92 (32h cryo stop)

Beam commissioning 38 (2 h SB not considered)

TOT 238

Sat 9th 8.05am

Sun 10th 11.11am

Sat 9th

5.27pmTue 12th 3.15pm

Injection

2 h SB with 2 bunches done meanwhile

CRYOPower cut

When the power cut arrived, beam commissioning was almost completed

(a TCDQ-TCSG cross check was ongoing)

Recovery Beam comm

16%45%

39%


General Overview

MD,

tech

nica

l sto

p

MD,

tech

nica

l sto

p

MD,

tech

nica

l sto

p, S

QU

EEZE

MD,

tech

nica

l sto

p, s

crub

bing

75 ns50 ns

Smaller emittance

Increased n. bunches Increased bunch

intensityBeta* = 1m

28 % of global time = TUNNEL INTERVENTIONS

what were the issues in 2011, minimizing impact and improving recovery

Documents

Transcript of what were the issues in 2011, minimizing impact and improving recovery