An Operations Model of the Tevatron Complex

An Operations ModelAn Operations Modelof theof the

Tevatron ComplexTevatron Complex

Elliott McCroryFermilab/Accelerator Division

13 October 2005

Elliott McCrory, Fermilab/AD 213 Oct 2005

OutlineOutline Fermilab Overview of the Operations Model Data input to the Model

SDA: Database indexed by store number

Optimization Strategies Optimizations with Future

ImprovementsQuick transfers24 mA/hrElectron Cooling (3E12 pbars)


Fermilab OverviewFermilab Overview

Main Injector& Recycler

TevatronLinac

Booster

Pbar Source


Fermilab TerminologyFermilab Terminology Stack

The antiprotons in the Accumulator

Stash The antiprotons in the Recycler

Shot The process of transferring antiprotons to the

Tevatron Done during a “Shot Setup”

Store Proton/antiproton collisions in the Tevatron Begins at the end of the Shot Setup

Transfer AccumulatorRecycler antiproton transfer and its

associated setup time


Fermilab OperationsFermilab Operations Stacking

Antiproton production in DebuncherAccumulator• Every 2 to 3 seconds• 15E10 per hour

AccRecycler transfers• Three or four time per store, today

– Depends on stacking rate

Shot Setup • Each step is 10 to 60 minutes

Tuning Transfer protons into Tevatron Transfer antiprotons into Tevatron Accelerate Squeeze/scrape

Collisions 20 to 40 hours

Between stores …

The Operations The Operations ModelModel


One Week of OperationOne Week of Operation

Recycler Stash

Luminosity

AccumulatorStack


One Simulated Week of OpsOne Simulated Week of Ops

Hours

Blu

e:

recy

cler

Sta

sh [

E1

0]

Red

: Lu

min

osi

ty [

1/(

cm2 s

ec)

]G

reen

: A

ccu

mu

lato

r S

tack

[E

10

]

Recycler Stash

Luminosity

AccumulatorStack


Operations ModelOperations Model Phenomenological representation of the

Tevatron Complex Mostly non-analytic & randomized

Monte Carlo (randomizations) Complexity is replaced by randomizations

Online and offline data are used to match model to reality

This model’s genesis: To develop intuition and provide guidance for

optimizing luminosity

Now: Extrapolations/”What If”, based on today’s

performance• The effect of Recycler improvements


Complexity Complexity Randomness Randomness Downtime

For the Tevatron, stacking and the PBar Source

Variations in all realistic parameters For example

• Transmissions during a shot, • Luminosity lifetimes, • Extraction efficiency from antiproton sources, • Shot setup time, • Downtime for each sub-system,• Etc…


Model AssumptionsModel Assumptions Performance does not improve

Random fluctuations around a specific set of parameters

Performance determined largely by these parameters

No shutdown periods “Shot data” & Ops Summaries are

accurate Supplemented and supported by other sources


LuminosityLuminosity One average proton and 36 antiprotons are

tracked Proton bunches are all the same Recycler & Accumulator antiprotons are different

L i(t=0) =

K H Np(0) NPBar, i(0)

[єp (0) + єPBar, i

(0)]

L (t) = L (0) e -t/τ(t)

τ(t) = τ(0) + C1 t C2

• τ(0) depends on L (0) and is adjusted to fit Real Data

• C1 = 3 ± 2

• C2 = f(C1)


The Other 200 ParametersThe Other 200 ParametersAccum AccumExtrFractionOffset 1.14 General DebugLevelValue 0 Shot AccumExtractionFraction 0.88 Tev betastar 40 Tev TevNoFailInSS2 0.9Accum AccumExtrFractionPerE10 -0.0013 General NumberOfWeeks 1 Shot PBarEmittanceGrowths[0] 1.2 Tev betastarMax 45 Tev TevNoFailRecovery 1Accum MinUsefulAccumulatorSize 80 General RandomNumberSeed 2E+09 Shot PBarEmittanceGrowths[1] 1.2 Tev betastarMin 35 Tev TevNoFailureInAccess 1Accum PbarCoalEfficIntensity -0.0005 General TimeStepSize 0.1 Shot PBarEmittanceGrowths[2] 2.8 Tev energy 980 Tev TevNoFailureInStudies 1Accum PbarCoalEfficIntensityOffset 0.9 Proton ProtonDowntimeOff 0.1 Shot PBarEmittanceGrowths[3] 1.5 Tev ExpNeedAccess 0 Tev TevNoFailWithPbars 0.9Accum PbarCoalEfficZEmit -0.0938 Proton ProtonDowntimeOffMax 1 Shot PBarEmittanceGrowths[4] 1.6 Tev FinalLumLifetimeVary 1 Tev TevRampMax 0.2Accum PbarCoalEfficZEmitOffset 1.1789 Proton ProtonDowntimeOffMin 0 Shot PBarEmittanceGrowthsMax[0] 2.2 Tev FinalLumLifetimeVarySpread 0.3 Tev TevRampMin 0Accum PbarEmitGrowthVsStackOffset 0.2 Proton ProtonUpTime 0.9 Shot PBarEmittanceGrowthsMax[1] 1.7 Tev kLong 0.8 Tev TevRampNoFailure 0.88Accum PbarEmitGrowthVsStackPerE10 0.03 Recyc A2REfficMax 1 Shot PBarEmittanceGrowthsMax[2] 3.3 Tev KStarBeta 1.45 Tev TevRcvNotAccess 0.8Accum PbarTransferNotFail 0.1 Recyc A2REfficMin 0.9 Shot PBarEmittanceGrowthsMax[3] 2.2 Tev LumFinalLife 28 Tev TevReadyToPBarsMax 0.4Accum PbarTransmissionVsStack 0.00055 Recyc AccumToRecycMinutesFromStacking 1 Shot PBarEmittanceGrowthsMax[4] 2.1 Tev LumInitLife 9.5 Tev TevReadyToPBarsMin 0.1Accum PbarZEmitOffset 2.26 Recyc AccumToRecycTranferFail 0.01 Shot PBarEmittanceGrowthsMin[0] 0.6 Tev LumLifetimeC1 3 Tev TevRecoveryTimeMax 4Accum PbarZEmitPerE10 0.0013 Recyc AccumToRecycTransferWaitTime 0.3 Shot PBarEmittanceGrowthsMin[1] 1E-06 Tev LumLifetimeC1Max 8 Tev TevRecoveryTimeMin 0Accum PbarZEmitVary 0.05 Recyc MaxAccumToRecXferWait 0.54 Shot PBarEmittanceGrowthsMin[2] 1.3 Tev LumLifetimeC1Min 0.1 Tev TevRecovNotStudies 1Accum RemoveVary 0.93 Recyc MaxNumTransfersPerStore 3 Shot PBarEmittanceGrowthsMin[3] 0.9 Tev LumLifetimeVary 0.6 Tev TevScrapeMax 0.5Accum RemoveVaryMax 1 Recyc MinAccumToRecXferWait 0.015 Shot PBarEmittanceGrowthsMin[4] 0.7 Tev LumLifetimeVarySpread 0.45 Tev TevScrapeMin 0Accum RemoveVaryMin 0.8 Recyc MinimumStackForRecyclerStudies 120 Shot PBarTransmissions[0] 0.975 Tev Tev150PBarsMax 0.3 Tev TevScrapeNoFailure 0.88Accum ShotLostPBarFraction 0.01 Recyc MinTimeTransfersToRecycler 72 Shot PBarTransmissions[1] 0.975 Tev Tev150PBarsMin 0.1 Tev TevShortDownTimeMax 1Accum StackBadDowntime 0.1 Recyc MinUsefulRecyclerSize 20 Shot PBarTransmissions[2] 0.96 Tev TevAccessTimeMax 14 Tev TevShortDownTimeMin 0Accum StackBadRateLower 0 Recyc MixedExtractionRatio 2 Shot PBarTransmissions[3] 0.975 Tev TevAccessTimeMin 1 Tev TevSqueezeMax 0.2Accum StackBadRateUpper 0.77 Recyc RapidTransferMaxStack 100 Shot PBarTransmissions[4] 0.975 Tev TevBunchesAtLowBeta 36 Tev TevSqueezeMin 0Accum StackDowntimeOff 0.4 Recyc RapidTransferMinLum 5 Shot PBarTransmissions[5] 0.999 Tev TevExtraDown 2.2 Tev TevSqueezeNoFailure 0.88Accum StackDowntimeOffMax 2 Recyc RapidTransferMinStack 10 Shot PBarTransmissionsMax[0] 0.99 Tev TevExtraDownOffset 0.2 Tev TevSSFirstStepMax 1Accum StackDowntimeOffMin 0.2 Recyc RecyclerCoolingTransferInMax 0.8 Shot PBarTransmissionsMax[1] 1 Tev TevFailureNotAccess 0.95 Tev TevSSFirstStepMin 0.4Accum StackEmitPerE10 0.02 Recyc RecyclerCoolingTransferInMin 0.4 Shot PBarTransmissionsMax[2] 0.99 Tev TevHourGlassMax 0.6 Tev TevStbyNotAccess 1Accum StackEmitQuadratic 0.00007 Recyc RecyclerCoolingTransferOutMax 0.4 Shot PBarTransmissionsMax[3] 1 Tev TevHourGlassMin 0.5 Tev TevStbyNotFailure 0.92Accum StackEmitQuadraticMax 0.0001 Recyc RecyclerCoolingTransferOutMin 0.2 Shot PBarTransmissionsMax[4] 1 Tev TevInfantFailure 0.01 Tev TevStbyNotStudies 1Accum StackEmitQuadraticMin 0.00001 Recyc RecyclerCoreEmitOffset 4 Shot PBarTransmissionsMax[5] 1 Tev TevInitLumMax 0.4 Tev TevStudiesNotAccess 0.95Accum StackingSucks 0.05 Recyc RecyclerCoreEmitPerE10 0.02 Shot PBarTransmissionsMin[0] 0.9 Tev TevInitLumMin 0 Tev TevStudNotFailure 0.97Accum StackingUpTime 0.85 Recyc RecyclerExtrFractionOffset 1.06 Shot PBarTransmissionsMin[1] 0.95 Tev TevInitLumNoFailure 0.985 Tev TevStudyTimeMax 24Accum StackNotLost 0.9965 Recyc RecyclerExtrFractionPerE10 0.0006 Shot PBarTransmissionsMin[2] 0.9 Tev TevInjPBarTimeMax 0.4 Tev TevStudyTimeMin 6Accum StackRateVary 0.95 Recyc RecyclerLikelyLifetime 800 Shot PBarTransmissionsMin[3] 0.93 Tev TevInjPBarTimeMin 0.1 Tev TevTermLumMax 0.4Accum StackRateVaryMax 1.03 Recyc RecyclerLongDowntime 5 Shot PBarTransmissionsMin[4] 0.93 Tev TevMediumDownTimeMax 1.5 Tev TevTermLumMin 0Accum StackRateVaryMin 0.7 Recyc RecyclerMaximumStashSize 110 Shot PBarTransmissionsMin[5] 0.97 Tev TevMediumDownTimeMin 0.5 Tev TevTurnArNotFailure 0.98Accum StopStackOnTevFailure 0.3 Recyc RecyclerMaxLifetime 1000 Shot ProtonEmitPercentRange 0.1 Tev TevNeedsESStudies 0 Tev TevTurnAroundTime 0.1Accum ZeroRateStack 300 Recyc RecyclerMIBestCurrent 20 Shot ProtonEmitPerE9 0 Tev TevNeedsStudy 0 Tev TevTurnAroundTimeMax 0.5Accum ZeroStackRate 13.4 Recyc RecyclerMIMaxCurrent 35 Shot ProtonEmitPerE9Offset 18 Tev TevNoFail4PBars 0.9 Tev TevTurnAroundTimeMin 0EndSto ECTLumiEnd 70 Recyc RecyclerMinLifetime 600 Shot ProtonILower 230 Tev TevNoFailInSS 0.98 Tev TevUpTime 0.988EndSto ECTLumiMid 50 Recyc RecyclerNotFailed 0.99 Shot ProtonIntensity 250EndSto ECTLumiZero 0 Recyc RecyclerPrepareTransferInMax 0.5 Shot ProtonIUpper 270EndSto ECTStackEnd 400 Recyc RecyclerPrepareTransferInMin 0 Shot ProtonLengthPercentRange 0.1EndSto ECTStackMid 200 Recyc RecyclerPrepareTransferOutMax 0.5 Shot ProtonLengthPerE9 0.0013EndSto ECTStackZero 0 Recyc RecyclerPrepareTransferOutMin 0 Shot ProtonLengthPerE9Offset 1.4727EndSto LowerVFactor 2 Recyc RecyclerShortDowntime 1EndSto LumDifference 30 Recyc RecyclerStudyTime 8EndSto MaximumStoreLength 24 Recyc RecyclerTax 0EndSto MinimumLuminosity 10 Recyc RecycMaxParcels 1EndSto RatioFactor 2 Recyc SimpleTransferMaxStack 100EndSto ReasonableStackSize 100 Recyc SimpleTransferMaxStash 100EndSto TargetIntegratedLum 5000 Recyc SimpleTransferMinLum 30EndSto TargetStackSize 150 Recyc SimpleTransferMinStack 40

Recyc StashLostDuringDowntime 0.02


Match Model to RealityMatch Model to Reality

Data SourcesSDA (Shot Data Acquisition)

• The “Supertable”• Other data tables

Data loggersWeekly summaries from operations

GoalAppropriate range of values for important

parametersCorrelations among the parameters

SDASDA

Sequenced Data Acquisitionor

Shot Data Analysis


SDA: DefinedSDA: Defined Overloaded definitions

Sequenced Data Acquisition• Defines alternate “clock” for recording data• Extends definition of what can be stored

Shot Data Analysis• Look at Sequenced Data Acquisition database • Look at conventional data loggers• Create summaries• Do certain types of calculations

– More complicated (transmission efficiencies)– Time dependent (Emittances)

• Observe/alert


Sequenced Data AcquisitionSequenced Data Acquisition More relevant “clock”

Shot/store number• Today: store # 4410

Case• Collider shot: 15 main

cases1. Proton Injection Porch2. Proton Injection tune up3. Eject Protons4. Inject Protons5. Pbar Injection Porch6. Inject Pbars7. …8. Before Ramp9. Acceleration10. Flattop11. Squeeze 12. Initiate Collisions13. Remove Halo14. HEP15. Pause HEP

Set• Each case may have one

or more sets

For example: “What happened at 4401,

Inject Protons, second bunch injection [a.k.a. Set 2]?”

Other common processes use this clock abstraction AccumulatorRecycler

transfers Pbar Transfers to Tevatron


Sequenced Data Acquisition Sequenced Data Acquisition (2)(2)

Data collection abstraction All types of data can be acquired Implemented as a Java interface

SDA Database Detailed information Often, 36 bunch data Often, raw data from front ends Indices:

• Store Number (**Most widely used**)• Accumulator to Recycler Transfer Number

30 GB today Data Loggers

Not strictly part of this, but very relevant Store <timestamp, value> pairs in relational DB

• Essentially Unix + milliseconds timestamp 70+ instances at Fermilab

• O(100 GB)


Shot Data AnalysisShot Data Analysis Data mining applications

Example Sequenced Data Acquisition cross-

checks Summary tables on the web The Supertable

A summary of key information, mostly from SDA database

Excel, HTML, AIDA/JAS One row = one store Over 200 columns for each store http://www-bd.fnal.gov/sda/supertable


SDA Database ExampleSDA Database Example


http://www-bd.fnal.gov/sda/supertablehttp://www-bd.fnal.gov/sda/supertable


Supertable ExampleSupertable Example


SDA Examples Relevant to SDA Examples Relevant to ModelModel

Initial Luminosity versus Number of Antiprotons

Initial Luminosity versus Initial Luminosity Lifetime

Antiproton Emittances Uncertainty at the IP

Beta-star changing??


Initial Luminosity vs. # PBarsInitial Luminosity vs. # PBars # 45 pbars at Remove Halo (1E09)

0

200

400

600

800

1000

1200

1400

1600

1800

2000

0 1000 2000 3000 4000 5000

Sequential Store Number

An

tip

roto

ns

at

Lo

w B

eta

[E

9]

# 10 MCR CDF initial lum - default (1E30)

0

20

40

60

80

100

120

140

160

0 500 1000 1500 2000

Antiproton Intensity [E9]

Init

ial

Lu

min

os

ity

at

CD

F [

E3

0]


Initl Lum Vs. Init Lum LifetimeInitl Lum Vs. Init Lum Lifetime # 23 CDF lum lifetime (hours)

0

2

4

6

8

10

12

14

16

18

20

0 20 40 60 80 100 120 140 160

Initial Luminosity at CDF [E10]

Av

era

ge

life

tim

e o

f C

DF

Lu

m o

ve

r 1

st

2

ho

urs

[h

rs]


PBar Emittance at ExtractionPBar Emittance at Extraction

0

2

4

6

8

10

12

14

16

0 50 100 150 200 250

Number of Antiprotons removed [E10]

Em

itta

nce

at

Ext

ract

ion

[95

% p

i mm

mr]

# 61 pbar H RR extracted emitt (pi-mm-mrad) # 62 pbar H Acc core emitt (pi-mm-mrad)

Accumulator

Recycler


PBar Emittance at ExtractionPBar Emittance at Extraction

Model generated Emittances

Real Emittances from Recycler


0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1800 2300 2800 3300 3800 4300

Store Number

Klu

dg

e F

acto

r to

get

CD

F L

um

ino

sity

Uncertainty at the IPUncertainty at the IP

Luminosity / (all known factors)

β* = 28 cm

Better emittance measurements Better lattice understanding Better instrumentation


= 0.975 / hour

Tevatron Failure RateTevatron Failure Rate

f(t) = e - t

σ = < t > = 1/

Time Between Tevatron Failures; Real Data

R ≈ 1 - ΔtΔt = 42 hours

e - t

Model data for Tevatron Failures


Failure Rate: InterpretationFailure Rate: Interpretation is “Tevatron Up Time” is measured directly from real data

< t > = σ = 1/ Probability of having stores of:

1 hour: 0.9752 hours: (0.975)2 = 0.95110 hours: (0.975)10 = 0.77620 hours: 0.60330 hours: 0.459

Failures are Independent of Time This is a random process!!


Reliability of Tevatron TodayReliability of Tevatron Today Low Beta: ~0.988

20 hours: 0.78530 hours: 0.696

Preparation for store: ~0.95Ramping & squeezing: 0.88!

Recovery time from unexpected failure is severe for superconducting machine

∴ Longer stores Tevatron is more reliable in collisions

Details on Details on ModelModel

ImplementationImplementation


State MachineryState Machinery All machines are implemented as Finite

State Machines Vary in complexity

Proton source: 5 states• Ready, Down, Sick, Studies, Access

Accumulator/Debuncher: 7 states• ReadyStacking, ReadyShot, ReadyRecTransfer, Down,

Recovery, Sick, Studies Tevatron: 17 states

• Ready, 7 shot-setup, 4 luminosity, Failure, Studies, Access, Recovery, Turn-Around

Recycler: 12 states• Ready, 4 transfers (2 in, 2 out), 2 down, recovery, 2

studies, access, cooling, turn-around.


Program StructureProgram Structure How does this work?

Step size = 0.1 hours “Listeners” provide connections among State

Machines Main program guides time progression & venue for

main decisions• Stack

– Do transfer to Recycler?

• “End-store” criterion satisfied? Start shot setup. Repeat for N weeks, dumping lots of relevant data.

Input parameters Output handler

Lots of data files can be dumped

C++/Linux 800 weeks/minute

• On 1.8 GHz Celeron

220+ parameters


Random NumbersRandom NumbersR

an

dom

Like

ly(-

2,

12

, 8

)

Product of these two distributions

RandomLikely(0, 5, 2)

Linux drand48( )


DecisionsDecisions Same as reality Store

When to end the storeWhen to begin a store after a failure

• Answer: Wait for accumulation of antiprotons

AntiprotonsWhen and how much to transfer from

Accumulator to Recycler

CombinationHow many antiprotons to get from two

sources• Recycler only, Accumulator only, Combined Source


Some End-Store CriteriaSome End-Store Criteria Store Duration Number of Antiprotons we have

available How low L can the experiments use Best: Combination of last two:

Np expected luminosity R = Expected Luminosity / Actual Luminosity

This criterion works very well algorithmically, but there are other considerations in Real Life Nowadays, the Run Coordinator ends a store based on this factor and many

other factors, e.g., time of day.

If Model is believable Can change the performance See how the End-Store criteria respond Find the Best criterion for ending stores for lots of

parameters


Best End-Store Criterion?Best End-Store Criterion? How to decide which is the “Best”

criterion? It integrates lots of luminosity It insensitive to natural fluctuations in

parameters• Some of these changes may be unnoticed• Random fluctuations or improvements?!

It is simple• Everyone can understand it!

– Some effective but complex schema have been rejected


Minimum Luminosity Minimum Luminosity CriterionCriterion

Show typical week


Show optimization plot


Show duration plot


Target Ratio CriterionTarget Ratio Criterion Show typical week


Show optimization plot


Store DurationStore Duration

Store Duration [hours]

dN

/dt

[sto

res/

1 h

our

bin

]

End store when Ratio=4

5

6

8

Remake this?


Ratio: End at R>4; R(t)Ratio: End at R>4; R(t)


Decisions Involving RecyclerDecisions Involving Recycler More degrees of freedom/choice

with RecyclerWhen to shoot from Acc to Recycler?How?How much to take from each?

Emittances from Recycler are smaller than from AccumulatorBetter coalescing efficiency

Optional slides??


Optimizations: AccOptimizations: AccRec Rec TransfersTransfers

Number of transfers per store0, 1, 2, 3, ...

Number of parcels per transferTime: NumParcels * 3 minutes

Min stack from which a transfer can occur10, 20, 30, 40, ... E10

“Bottom Line” ≡ Integrated Luminosity per week!


Ongoing Studies on Optimum Ongoing Studies on Optimum RecyclerRecycler

Some crucial dependenciesTime required to transfer into Recycler

• Currently, 0.75 to 2 hours, most likely=1 hour.• Plan: 15 minutes or less

Transmission efficiency• Now ~90%

Emittance from Recycler is ~4pi less than Accumulator

• Improved L (0), but diminished initial lifetime; – ∫L dt ?


# Pbars Avail vs. Transfer # Pbars Avail vs. Transfer TimeTime

Analytic calculation by D. McGinnis,BeamDocs # 1948


Accum: 24E10/hr; Recy:6E12Accum: 24E10/hr; Recy:6E12N

um

ber

of

Anti

pro

tons

[E10

]O

r Lu

min

osi

ty [

E30/c

m2/s

ec]

Hours from start of simulated “Run”

Stack Size

Stash Size

Luminosity


Optimization of 24 mA/hr: 6E12Optimization of 24 mA/hr: 6E12

3 transfers per store,4 parcels per transfer

Avera

ge Inte

gra

ted L

um

/week

[1/n

b]

End-of-store Ratio

RE-MAKE THIS PLOT:Show 24+6E12 & “today” on same plot

Summary & Summary & ConclusionsConclusions


ConclusionsConclusions

Tevatron Operations Model has

helped us understand the Complex

SDA has been a crucial element to

understanding the Tevatron and

making this model work


Lessons for LHC?Lessons for LHC? Reliable, redundant, easily

accessible performance data are crucial to understanding how you are operating

Shot/Case/Set organization of data is more user-friendly

Complexity of LHC Loading may be better to model

Reliability matters (duh!)

An Operations Model of the Tevatron Complex

Documents

Transcript of An Operations Model of the Tevatron Complex