Operations Planning

Greg SullivanUniversity of Maryland

Operations Planning

SAC/PAP Review UW-Madison March 1-2, 2007

March 1, 2007SAC – Operations Planning

Operations Concept

• Analysis– Coordinated by the Collaboration/Analysis

Coordinator– Collaborating institutions supported by

their respective funding agencies

• Operations– Central organization to coordinate and

manage in partnership with the collaboration

– Cost sharing determined by the funding agencies

– Staffing plans include core full-time staff and fractions of staff covered by both research grants and operations support


USA: Bartol Research Institute, Delaware Pennsylvania State University UC Berkeley UC Irvine Clark-Atlanta University University of Maryland IAS, Princeton University of Wisconsin-Madison University of Wisconsin-River Falls Lawrence Berkeley National Lab. University of Kansas Southern University and A&M

College, Baton RougeUniversity of Alaska, Anchorage

Sweden: Uppsala Universitet Stockholm Universitet

UK: Imperial College,

London Oxford University

Netherlands: Utrecht University Belgium:

Université Libre de Bruxelles

Vrije Universiteit Brussel Universiteit Gent Université de Mons-Hainaut

Germany: Universität Mainz DESY-Zeuthen Universität Dortmund Universität Wuppertal Universität Berlin MPI Heidelberg RWTH Aachen

Japan: Chiba university

New Zealand: University of

Canterbury

THE ICECUBE COLLABORATION

33 institutions, ~250 members http://icecube.wisc.edu


Operations Scope

• Detector Maintenance & Operations (M&O) - Maintain and operate a properly functioning detector.

• Data Systems - Enable the collaboration to access data.

• Data Integrity - Confirm that high quality data can be used for analysis.

• R&D - Modest improvements and evaluation of external threats such as EMI, and technical interfaces with ongoing R&D efforts supported separately.

• Education and Outreach

• Program Management


Planning Background

• Operations tasks and budgets are informed by construction (and operations!) experience.

• Organization plans are maturing and during the concurrent construction and operations phase will rely heavily on the ongoing construction project.

• Funding agencies need a clear sense of the level of support required in order to advance discussions on the respective funding responsibilities.


Operations Planning Issues

• Host Laboratory (NSF)– Responsibilities (NSF, UW, Collaboration)– Influence of resource constraints (power, population, etc.)

• Boundary Conditions– MREFC and Research and Related Activities Boundary– Cost sharing algorithms defined by the funding agencies– Data rates, quality and timeliness of analysis– Actual cost of maintenance and operations activities

• Coherency of the IceCube Operations Program– Construction, Operations, Analysis Grants, R&D, E&O, …

• Emphasis on operations and advanced R&D

IceCube OperationsDRAFT

International Oversight and Finance GroupNSF, Executive Agent

IceCube CollaborationCollaboration Spokesperson

Science Advisory Committee

A3RI & Project Support

IceCube Neutrino ObservatoryPrincipal InvestigatorDirector of Operations

Software CoordinationEducation & Outreach

Operations Support

R&D

AdministrativeSupport

Tech Support

Logistics

RPSC

Science Operations

Computing

Data CenterDESY

MC ProductionEurope

Northern DataCenter

Metadata & Database Support

Data Distribution

Data Storage

Software & Data Processing

Data Production

Filtering &Software Systems

ReconstructionSoftware

Online FilteringSoftware

SimulationSoftware

Data Integrity

Detector Calibration

Data Verification

Offline Filtering Software

Tier 2 Support

Monte Carlo

Offline Data

Collaboration Board

Executive Committee

Detector Operations

Trigger Filter Transmit Board

Run Coordination

Detector Monitoring

Online Processing

DAQ

Detector Hardware

SPS Data Handling

AMANDA

Winter Overs

SPTS


Operating Strings

Operations Begin

(Planned going into 06-07 season)


Deployments

AMANDA 01/ 2000

IceCube string and IceTop station 01/06

IceTop station only 2006

21

3029

40

50

3938

4748

49

595857

6667

74

65

73

78

46

56

72



2005: 12006: 82007: 13 (12 planned)2008: 142009: 142010: 142011: 7+


0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

2005 2006 2007 2008 2009 2010 2011 2012

Date

km3·yr

0

10

20

30

40

50

60

70

80

# Deployed Strings

km3·yr Strings

• Graph shows cumulative km3·yr of exposure × volume

• 1 km3·yr reached 2 years before detector is completed

• Close to 4 km3·yr at the beginning of 2nd year of full array operation.

IceCube Integrated Volume (Projected)

3 yr M&O


Already “Operating”

• Last Year – IC-9 and Amanda array taking data– Physics analysis ongoing

• Continuing Amanda analysis• Publication of results

– First IC-9 analysis within 6 months of data• Publication soon! (paper in pub committee)

• This year 22 strings– People working on Physics filters for Pole

– Active working groups and coordination• See later talks


Atmospheric neutrinos

• Dominant background– Point source

searches– Diffuse neutrinos

• Important calibration “beam”– Standard candle to

measure sensitivity to neutrinos

– Established IceCube as a neutrino detector!


Atmospheric neutrinos in 2006

John Pretz, Ph.D. thesis

Data selection done online at S. Pole and transferred by satellite NorthRatio of data to simulation:

At selected cut strength 10:156 up-going muon eventsPurity of neutrino sample > 95%


First Neutrinos from IceCube-9

Physics paper prepared for publication within 6 months of initial operation!


IceCube-22 calibrations

LED Flashers Standard Candle


IceCube-22 Muon

Plan for Physics data taking with filters in place at Pole (15-25 GB/day satellite rate) to begin before May 1


Operations Planning - History

NSF Baseline Review Feb. 2004 Supported plan to operate during construction phase

International Oversight & Finance Group (IOFG)

Oct. 2004 1st meeting w/discussion of plan for cost sharing of operations

NSF Annual Review May 2005 Recommendations on the scope of an operations program

Collaboration Scrutiny Group

Aug. 2005 Critiqued scope and budgets and encouraged cost reductions and more in-kind contributions

Project Advisory Panel/Science Advisory Committee

Mar. 2006 More recommendations on preparing a comprehensive program and organization

IOFG Apr. 2006 Effort to understand cost and possibilities for cost sharing arrangements

NSF Annual Review

M&O Proposal to NSF

NSF Operations Review

IOFG

Revised M&O budget Review

May 2006

June 2006

Aug. 2006

Sep. 2006

Mar. 2007

Encouragement to proceed with plans for operations

Sent to reviewers & organized review committee

Positively reviewed some suggestions for savings, redistribute M&O and Grants Discussion of funding realities


Revised M&O Budget

FY07 (M$) FY08 (M$) FY09 (M$)

Non-US in-kind 3.3 3.4 3.5

Original M&O Request to NSF/IOFG 6.5 7.2 8.4

Revised M&O Request to NSF/IOFG 3.6 5.1 7

Revised M&O cost delta -2.9 -2.1 -1.4

Non-US Cash in-kind 0.4 0.4 0.4

Revised M&O Request from the NSF 3.2 4.7 (6.6)


Revised M&O details (Non in-kind)

FTE Capital Labor Total FTE

Diff

Capital

Diff

Labor

Diff

Total

Diff FTE Capital Labor Total

FTE

Diff

Capital

Diff

Labor

Diff

Total


FTE

Diff

Capital

Diff

Labor

Diff

Total

Diff

T ier 0 Coordination 0.8 0 102 102 0.0 0 0 0.8 0 102 102 0.0 0 0 0.8 0 102 102 0.0 0 0

Winter Over Personnel 3.0 0 441 441 -0.4 -54 -54 2.0 0 294 294 -1.4 -201 -201 2.0 0 294 294 -1.4 -201 -201

SPS & SPT S at UW 2.0 0 272 272 0.0 -200 0 -200 2.0 100 272 372 0.0 -100 0 -100 2.0 200 272 472 0.0 0 0 0

M onitoring 1.5 0 125 125 -0.5 -36 -36 1.5 0 125 125 -0.5 -37 -37 2.0 0 161 161 0.0 -1 -1

Instr. M aintenance (non-DAQ) 0.2 0 31 31 0.0 -45 0 -45 0.5 0 65 65 -0.2 -45 -33 -78 1.0 50 125 175 -0.3 5 -71 -66

IceT op M aintenance 1.0 0 140 140 -0.5 -66 -66 1.5 0 210 210 0.0 -12 -12 1.5 0 200 200 0.0 -22 -22

Run Coordinator 0.5 0 80 80 0.1 21 21 0.5 0 80 80 0.1 21 21 0.5 0 80 80 0.1 21 21

DAQ M aintenance 0.0 0 0 0 -1.5 -57 -240 -297 2.1 24 336 360 -0.7 -33 -126 -159 2.6 30 411 441 -0.4 -27 -82 -109

Su b -T o tal 9.0 0 1,191 1,191 -2.7 -302 -374 -676 10.9 124 1,484 1,608 -2.6 -178 -387 -565 12.4 280 1,645 1,925 -1.9 -22 -355 -377

UW Data Center 3.0 200 368 568 -1.0 -230 -122 -352 3.0 200 368 568 -1.0 -230 -122 -352 3.0 430 368 798 -1.0 0 -122 -122

F iltering & Software 2.0 0 250 250 -1.0 -19 -54 -73 3.0 0 350 350 0.0 -19 42 23 3.0 20 350 370 0.0 1 42 43

Sim ulation Production 2.0 0 250 250 -1.0 -34 -41 -75 3.0 0 350 350 -1.0 -34 70 36 4.0 20 500 520 -0.5 -14 209 195

Detector Calibration 2.0 0 200 200 -2.2 -211 -211 2.5 0 250 250 -0.9 -124 -124 3.0 0 312 312 -0.3 -50 -50

Detector Verification 2.0 0 200 200 -0.9 -100 -100 2.0 0 200 200 -0.9 -131 -131 2.5 0 250 250 -0.2 -56 -56

Data Warehouse 1.0 0 136 136 0.0 -320 0 -320 1.0 100 136 236 0.0 -220 0 -220 1.0 320 136 456 0.0 0 0 0

South Pole & UW Networking 0.5 0 61 61 0.0 -55 0 -55 0.5 0 61 61 0.0 -55 0 -55 0.5 55 61 116 0.0 0 0 0

US M onteCarlo 0.0 25 0 25 0.0 -47 0 -47 0.5 50 50 100 0.5 -22 50 28 0.0 72 0 72 0.0 0 0 0

T ier 2: Infrastructure Support 0.8 100 75 175 -0.2 0 -31 -31 1.0 100 100 200 0.0 50 -6 44 1.0 100 110 210 0.0 0 4 4

Reconstruction 0.0 0 0 0 -0.4 -39 -39 0.0 0 0 0 -0.3 -31 -31 0.0 0 0 0 -0.3 -31 -31

Su b -T o tal 13.2 325 1,540 1,865 -6.7 -705 -598 -1,303 16.5 450 1,865 2,315 -3.6 -530 -252 -782 18.0 1,017 2,087 3,104 -2.3 -13 -4 -17

Instrum entation R&D 1.0 50 110 160 -1.0 0 -136 -136 2.0 50 220 270 0.0 0 -64 -64 3.0 75 350 425 0.0 0 -76 -76

Science & T echnical Support 0.5 0 70 70 -0.5 -145 -145 1.0 0 140 140 0.0 -83 -83 2.0 0 280 280 0.5 26 26

M anagem ent/Adm in 0.5 0 130 130 -0.8 -85 -85 1.0 0 225 225 -0.5 -44 -44 2.0 0 400 400 -0.5 -70 -70

Education & Outreach 0.0 0 0 0 -1.0 -157 -157 0.0 0 0 0 -1.0 -157 -157 1.0 0 157 157 0.0 0 0

Su b -T o tal 2.0 50 310 360 -3.3 0 -523 -523 4.0 50 585 635 -1.5 0 -348 -348 8.0 75 1,187 1,262 0.0 0 -120 -120

24.2 375 3,042 3,417 -12.7 -1,007 -1,495 -2,502 31.3 624 3,934 4,558 -7.8 -708 -988 -1,696 38.3 1,372 4,919 6,291 -4.3 -35 -480 -515

75 0 175 -91 225 -135

Perso n n el & Cap ital Reserve 150 -454 350 -471 500 -606

Escalatio n 0 0 0 0 0 0

M &O T o tal 3,642 -2,956 5,083 -2,258 7,016 -1,256

1030 Proposal totals 6,568 7,272 8,441

Subtotal

FY07 (April 2007 - March 2008) K$ FY08 (April 2008 - March 2009) K$ FY09 (April 2009 - March 2010) K$

Travel, Materials & Supplies

Ops Support

U.S. M&O Budget Revised

(1/30/07)

Detector Operations

Science Operations


Revised M&O Details (In-kind)

FTE Capital Labor Total FTE

Diff

Capital

Diff

Labor

Diff

Total


FTE

Diff

Capital

Diff

Labor

Diff

Total


FTE

Diff

Capital

Diff

Labor

Diff

Total

Diff

T ier 0 Coordination 0 0.0 0 0 0 0 0.0 0 0 0 0 0.0 0 0 0

Winter Over Personnel 1.5 136 136 -0.2 -112 -112 1.5 136 136 -0.2 -112 -112 1.5 136 136 -0.2 -112 -112

SPS & SPT S at UW 0 0.0 0 0 0 0 0.0 0 0 0 0 0.0 0 0 0

M onitoring 0.8 82 82 0.0 0 0 0 0.8 82 82 0.0 0 0 0 1.0 117 117 0.0 0 0 0

Instr. M aintenance (non-DAQ) 0.1 15 15 0.0 0 0 0 0.2 31 31 0.0 0 0 0 0.3 47 47 0.0 0 0 0

IceT op M aintenance 0.3 25 25 0.0 0 0 0 0.4 39 39 0.0 0 0 0 0.4 39 39 0.0 0 0 0

Run Coordinator 0.5 140 140 0.1 81 81 0.5 140 140 0.1 81 81 0.5 140 140 0.1 81 81

DAQ M aintenance 0.4 34 34 0.0 0 0 0 0.5 44 44 0.0 0 0 0 0.5 49 49 0.0 0 0 0

Su b -T o tal 3.6 0 432 432 -0.1 0 -31 -31 3.9 0 472 472 -0.1 0 -31 -31 4.2 0 528 528 -0.1 0 -31 -31

UW Data Center 0 0.0 0 0 0 0 0.0 0 0 0 0 0.0 0 0 0

F iltering & Software 1.5 204 204 0.0 0 0 0 1.5 204 204 0.0 0 0 0 1.5 204 204 0.0 0 0 0

Sim ulation Production 2.5 280 280 0.0 0 0 0 2.5 280 280 0.0 0 0 0 2.5 280 280 0.0 0 0 0

Detector Calibration 0.5 26 26 0.0 0 0 0 0.5 26 26 0.0 0 0 0 0.5 26 26 0.0 0 0 0

Detector Verification 1.3 109 109 0.0 0 0 0 1.5 126 126 0.0 0 0 0 1.7 143 143 0.0 0 0 0

Data Warehouse 0 0.0 0 0 0 0 0.0 0 0 0 0 0.0 0 0 0

South Pole & UW Networking 0 0.0 0 0 0 0 0.0 0 0 0 0 0.0 0 0 0

US M onteCarlo 0.5 150 61 211 0.0 0 0 0 0.5 150 61 211 0.0 0 0 0 0.5 150 61 211 0.0 0 0 0

DESY Data Center 2.0 392 230 622 0.0 0 0 0 2.0 392 230 622 0.0 0 0 0 2.0 392 230 622 0.0 0 0 0

Wuppertal CPU F arm 1.5 200 184 384 0.0 0 0 0 1.5 200 184 384 0.0 0 0 0 1.5 200 184 384 0.0 0 0 0

Belgium CPU F arm 0.5 78 58 136 0.0 0 0 0 0.5 78 58 136 0.0 0 0 0 0.5 78 58 136 0.0 0 0 0

Sweden SPU F arm 0.5 78 58 136 0.0 0 0 0 0.5 78 58 136 0.0 0 0 0 0.5 78 58 136 0.0 0 0 0

Shared CPU F arm s 0 0.0 0 0 0 0 0.0 0 0 0 0 0.0 0 0 0

T ier 2: Infrastructure Support 0 0.0 0 0 0 0 0.0 0 0 0 0 0.0 0 0 0

Reconstruction 0.2 17 17 0.0 0 0 0 0.4 39 39 0.0 0 0 0 0.5 49 49 0.0 0 0 0

Su b -T o tal 11.0 898 1,227 2,125 0.0 0 0 0 11.4 898 1,266 2,164 0.0 0 0 0 11.7 898 1,292 2,190 0.0 0 0 0

Instrum entation R&D 0.5 42 42 0.0 0 0 0 0.5 42 42 0.0 0 0 0 0.8 73 73 0.0 0 0 0

Science & T echnical Support 1.0 196 196 -0.4 0 -59 -59 1.0 196 196 -0.4 0 -59 -59 1.0 196 196 -0.4 0 -59 -59

M anagem ent/Adm in 0.5 42 42 0.0 0 0 0 0.5 42 42 0.0 0 0 0 0.8 63 63 0.0 0 0 0

Education & Outreach 0.8 118 118 0.0 0 0 0 0.8 118 118 0.0 0 0 0 0.8 118 118 0.0 0 0 0

Su b -T o tal 2.8 0 398 398 -0.4 0 -59 -59 2.8 0 398 398 -0.4 0 -59 -59 3.3 0 450 450 -0.4 0 -59 -59

17.4 898 2,056 2,954 -0.5 0 -90 -90 18.1 898 2,136 3,034 -0.5 0 -90 -90 19.2 898 2,270 3,168 -0.5 0 -90 -90

51 0 57 0 61 0

Perso n n el & Cap ital Reserve 301 -2 309 -9 323 -9

Escalatio n 0 0 0 0 0 0

M &O T o tal 3,306 -93 3,400 -99 3,551 -280

T ravel, M aterials & Su p p lies

Detecto r Op eratio n s

Scien ce Op eratio n s

Op s Su p p o rt

Su b to tal

FY08 (April 2008 - March 2009) K$ FY09 (April 2009 - March 2010) K$In-Kind M&O Budget

Revised (1/30/07)

FY07 (April 2007 - March 2008) K$


Where are we now?

• NSF funding reality plus $400k common fund fall short in first two years of operation– Prioritization of budget elements & cost cutting– Developed the revised budget

• Plan to support core operations areas for IceCube in FY07 and FY08– Goal to Maintain low impact on MRE contingency!– Increased risk to operations

• Need to make case to NSF for better support in FY09 and later years– FY09 (~$6.6M funding needed)– Steady state FY11 on (~$10M)


Operations Planning - Next Steps

• SAC/PAP Review – March 1-2, 2007– collect operations material for reviews– get feedback from you

• NSF Review of Revised Proposal - March 15 2005– At NSF with same review committee from last year

• No direct IceCube participants – Review Operations/Construction transition

• Same committee gave mostly positive review of full funding request

– Revised Budget already drastically cut (~50% in FY07)– Evaluate pending and existing base grants in “big picture” context

• IceCube is operating with 22 strings in the ice–Planning for operations funding to start April, 2007 at reduced levels to support core operations.


Summary

• IceCube is Physics Capable Now!– Competition & Synergy with Other Exps– Students & Postdocs …

• IceCube M&O funding begins this year (April)!!– Planned from initial baseline review in 2004– Budget estimates well grounded in MREFC &

operating experience

• Have “Replanning” numbers for first two years– Ramp up in M&O funding slower then planned– Prioritized tasks to ensure core operations– Try to minimize impact on MRE

• Need to get full requested amount for Year 3 (FY09) funding in NSF budget plans now


• Backup Slides


U.S. Funding Transition Plans

0

10

20

30

40

50

60

FY03 FY04 FY05 FY06 FY07 FY08 FY09 FY10 FY11

(M$)

0

10

20

30

40

50

60

70

80

Strings

MRE

OPS

Strings

US M&O onlyNon-US in-kind ~

$3.5M


SP Systems (Tier 0) Data Flow

5120 DOMs

Event Builder

Online monitor & alert system

All Data

Filtered Data

~30 GB/day 10 TB/year

TWR DAQ

AMANDA

677 OMs

ICECUBE

JEB/PnF

1 MB/s

Filter Reconstruction Processor Farm

5-6 MB/s >400 GB/day

Tape Archive

Tier 0

Tier 1

Tier 2

South Pole System (SPS)

17 Racks, 150 computers, real time DAQ, PnF system, networking, security, data handling 300GB/day tape, 28 GB/day satellite

South Pole Test System (SPTS)

8 Racks, ~60 computers, cable, DOM, DAQ, PnF

Winter Over Operations (WO)

2 Winter Overs, training 3FTE/yr, 300GB/day tape

Data Warehouse

SP&NH Data movement, 30TB/yr Data&MC disk, Tape archive, Database

Data Center US Data Center DESYMC Production UW

MC Production USMC Production UMDMC Production Sweden/Belgium

MC Production GermainyOffline data formatting &

merging, L1 Filtering, >800 GHz-CPU dedicated

Offline data formatting & merging, re-Filtering, >800 GHz-CPU Dedicated

Manage MC Production, MC production, >800 GHz-CPU dedicated + shared CPU resources

MC production, >800 GHz-CPU dedicated + shared CPU resources

MC production, using shared CPU resources

MC production, >400 GHz-CPU dedicated

MC production, >400 Ghz-CPU using shared CPU resources

ALL ICECUBE COLLABORATING INSTITUTIONS

Core IceCube Resources

Distributed Resources

UW Madison Data Center

Data Warehouse (Tier 0) & Data Processing Centers (Tier 1) Data Flow

Data Warehouse (Tier 0)

Monte Carlo Production

US, Germany, Sweden, Belgium (Tier 1) (including GRID resources)

~4 GB/day now growing to ~30GB/day

Level 0 process Calibrate, Detector Monitor Offline data formatting

Monitor data

L1/L2 Production Reconstruction/Filter Muon, Cascade, …

US Tier 1 US, Germany Tier 1

IceCube shift Detector Monitoring

Tier 2 Collaboration Physics Analysis

Operations Planning

Documents

Transcript of Operations Planning