OFFICE OF SCIENCE Dark Energy Survey (DES) Project at the Fermi National Accelerator Laboratory June...

OFFICE OF

SCIENCE

Dark Energy Survey (DES) Projectat the Fermi National Accelerator Laboratory

June 22-23, 2010

Department of Energy/National Science Foundation

Review Committeefor the

Daniel R. Lehman, Chair

DOE/SC Review Committee

Office of Science, U.S. Department of Energy

http://www.science.doe.gov/opa/

http://www.science.doe.gov/opa/

OFFICE OF SCIENCE

2

Hexapod

Optical Lenses

CCDReadout

Filters

Barrel

Prime Focus Cage

Imager(includingfocal plane)

Review Committee Participants

OFFICE OF

SCIENCE

Daniel R. Lehman, DOE/SC, Review Chairperson

Observers Kathy Turner, DOE/SC Michael Salamon, DOE/SC Paul Philp, DOE/CH Nigel Sharp, NSF

Review Committee SC 1—DECam Optics, Opto-Mech, CFIP *Matt Johns, Carnegie Inst. Ian Dell’antonio, Brown SC 2—DECam CCDs, SISPI, Electronics *Roger Smith, CalTech Paul Padley, Rice U. SC 3—DESDM, Simulations, and Calibrations *Julian Borrill, LBNL Rob Cameron, SLAC SC 4—Cost and Schedule, Management, and ES&H * Mark Reichanadter, SLAC Kurt Fisher, DOE/SC Henry Heetderks, LBNL Dennie Parzyck, DOE/FSO *Lead

4

Charge MemorandumCharge Questions

1. Technical: Are the accomplishments to date and planned future activities consistent with the baseline objectives and is the project progressing adequately?

2. Cost and Schedule: Are the current cost, schedule and contingency projections consistent with the approved baseline for all the projects?

3. Management: Is the management structure adequate and appropriate to successfully execute the projects within specifications, budget, and schedule? Are major risks being addressed effectively?

4. ES&H: Are ES&H aspects being properly addressed and are Integrated Safety Management Principles being followed?

5. Operations: Are the preliminary plans for the operations phase appropriate for this stage of the project?

6. Previous Review: Has the project responded satisfactorily to recommendations from the previous review?

OFFICE OF

SCIENCE

5

Report Outline/Writing Assignments

OFFICE OF

SCIENCE

Operations Planning and Calibration(

Executive Summary .................................................................................................................Fisher

1. Introduction ....................................................................................................................... Turner

2. Technical (Charge Questions 1, 6)

2.1 DECam and CFIP

2.1.1 DECam Optics and Opti-Mech, CFIP ...........................Johns*/Dell’antonio (SC 1)

2.1.2 DECam CCDs, SISPI, Electronics......................................... Smith*/Padley (SC 2)

2.1.2 DECam Integration & Test ............................................Dell’antonio*/Johns (SC 1)

2.2 DESDM ......................................................................................Borrill*/Cameron (SC 3)

3. CQ 5,6) ...............................

Dell’antonio*/Johns (SC 1)

4. Cost (DECam, DESDM, CFIP) (CQ 2, 6)…………... ................. Fisher*/Reichanadter (SC 4)

5. Schedule and Funding (DECam, DESDM, CFIP) (CQ 2, 6) ........ Fisher*/Reichanadter (SC 4)

6. Management (CQ 3, 6). ..........................................................Reichanadter */Heetderks (SC 4)

7. Environment, Safety and Health (CQ 4, 6). ...................................................... Parzyck* (SC 4)

* Lead SC Subcommittee

6

2.1.1 DECam Optics and Opti-Mech, CFIP

Matt Johns, Carnegie Institute/ Ian Dell’antonio, Brown University

OFFICE OF

SCIENCE

1.Are the accomplishments to date and planned future activities consistent with the baseline objectives and is the project progressing adequately?

Yes, through the assembly and hand-off of the DECam to CTIO. the major components of the Opto-mechanical system are either finished and ready for assembly or are progressing satisfactorily.

2. Has the project responded satisfactorily to recommendations from the previous reviews?

Yes. The recommendation regarding oversight of the optics polishing, installation and delivery continues to be valid until commissioning. Recommendations regarding management issues were reviewed by SC4.

• FindingsThe major opto-mechanical subsystems have been procured or are in the late

stages of procurement. These includes DECam itself, the Telescope Simulator, the F/8 Changer, and other handling fixtures. The effort is now shifting to the assembly and installation of DECam in the Telecope Simulator and testing of the assembly under computer control with dummy optics.

Testing of the F/8 Changer at FNAL is nearing completion and the assembly will be ready to ship to CTIO in the near future. Shipping of other elements not required for TS testing will follow shortly.

The Telescope Simulator allows both the DECam project and the CTIO personnel who will be working on the telescope installation to gain experience with the assembly of the final DECam system. This will greatly speed up acceptance testing.

7



OFFICE OF

SCIENCE

• Findings (continued)

The corrector lenses have nearly finished polishing at SESO and are due to arrive at UCL in a staggered fashion over the next three months. The first two elements have been received.

CTIO appears to be nearly ready for the arrival of DECam. The major work items remaining are the completion of the Telescope Control System upgrade (scheduled for completion in the fall of 2010) and the installation of the cooling system cabling.

8



OFFICE OF

SCIENCE

• CommentsAlthough several risks have been retired, significant risks to the project will

unavoidably remain in this section of the project. In particular, there are no spare corrector lenses or filters, and the damage to any one element means schedule delays and potential cost overruns on for the project.

This is particularly important for optical element 5, which also serves as the Dewar window. Because this element is destined to be handled much more during integration and testing, it is the most at risk for damage.

No full-scale filters have yet been made available to DES from Asahi, although their delivery date is scheduled for about three months. More troubling is the potential effect of breaking a filter, given its importance in DES observations and the lack of any spares. On a positive note, the vendor has several spare blanks, which would reduce the time of recovery for a broken filter.

9



OFFICE OF

SCIENCE

• Comments (continued)

UCL has developed the concept for an end-to-end on-axis optical test of the corrector assembly using a test sphere and computer generated hologram (CGH). The are both benefits and risks associated with implementing such a test. Funds (~$5K) are available.

SESO has notified UCL that the coating lab may shut down for renovation before the full complement of corrector lenses are coated. SESO is considering out-sourcing the coatings. This represents potential delays and added technical risk.

10



OFFICE OF

SCIENCE

• Recommendations

2.1.1.1 Investigate the purchase of an additional z’ filter if contingency allows, to mitigate the risk to the Dark Energy Survey due to loss of a filter.

2.1.1.2 Consult with UCL regarding the advisability as a risk reduction measure of a full-aperture on-axis null test for the corrector assembly in the next ~2 months.

2.1.1.3 Discuss with UCL the risks associated with coating DECam lenses by third-party vendor and develop a recommendation on whether such a plan is acceptable to the project immediately.

11



OFFICE OF

SCIENCE

12

2.1.2 DECam CCDs, SISPI and Electronics

Roger Smith, Caltech / Paul Padley, Rice University

OFFICE OF

SCIENCE

1. Are the accomplishments to date and planned future activities consistent with the baseline objectives and is the project progressing adequately?

Yes


Yes, … clearly and explicitly.

13



OFFICE OF

SCIENCE

Findings:• Production of DECam Electronics is well advanced

• Boards produced in Barcelona are behind schedule, but this should not impact overall schedule

• SISPI software is also well advanced and is actively being used providing useful user feedback to the developers.

• 91 science grade CCDs have been produced, significantly more than the 62+10(spares) needed for DECam. This is consistent with yield projections.

• There have been two separate “full well degradation” events, occurring only in the MCCDTV, affecting different groups of engineering grade CCDs.• The reason for this degradation is not understood.• There have been modifications to the electronics to mitigate undesirable

transients to address potential risks, though stress testing does not support a causal relationship.

• None of the 91 science grade CCDs have been exposed to risk.• Funding for the early commissioning phase (in FY 2011) is uncertain.

14



OFFICE OF

SCIENCE

Comments:• The reviewers are impressed by the excellent team of scientists and

engineers and the outstanding progress, and consequent low technical risk.• The project should have no difficulty reaching its CD4 milestone

• The degradation of CCD full well is a source of serious concern:• The reviewers are confident that the DECam CCD team has pursued

every conceivable avenue to understand this problem but to date neither the physics of the CCD behavior nor the cause are understood.

• Performance before and after the “events” has been well documented and shown to be stable, suggesting that risk exposure is intermittent.

• While the uncertain funding of the FY11 commissioning activities is not a threat to the CD4 milestone, it is a serious risk to the timely transition to scientific operations.• Lack of funding for travel for the university groups is a related issue.

15



OFFICE OF

SCIENCE

Recommendations:

• Continue to monitor engineering grade CCDs for degradation events, to investigate the cause, and to identify the full well degradation mechanism within the CCDs.

• Identify the key DECam personnel that should be available for travel to CTIO, or at CTIO for the commissioning phase.

• Work with the participating institutions to secure funding for the FY11 commissioning activities.

• We urge NOAO to provide commensurate travel support for technical staff at CTIO to participate in system testing on the telescope simulator at SiDet.

• Evaluate simple illumination options to support *daytime* testing of DECam during commissioning, on the telescope.

16

2.1.3 DECam Integration & TestMatt Johns, Carnegie Institute/

Ian Dell’antonio, Brown University

OFFICE OF

SCIENCE


Yes, for the part of the work related to integration and testing on the telescope simulator at Fermilab. For the part relating to the integration in Chile, the plans are not yet complete.


Yes, although the CTIO installation plan that was created in response to our recommendations is now obsolete and should be redone very soon.

FindingsThe integration of optical elements at Fermilab is proceeding smoothly. The

telescope simulator has been working, and testing of components beginning with the F8 assembly is underway.

The lens integration at UCL is slightly behind schedule due to fabrication issues with the mirror cells. At the moment, this is not critical to the schedule.

There is currently no detailed plan describing the installation procedures for DECam at CTIO, only drafts. At the moment, this is still acceptable because the experience with the telescope simulator will modify these plans. The current schedule has a detailed plan in place by early 2011.

17



CommentsThe planning and funding commitments for DECam installation and

commissioning at CTIO need to be in place very soon, as installation is currently planned in ~15 months. The Fermilab DECam team and CTIO need to jointly develop a plan that encompasses the scope of work, management structure, schedule and funding requirements for submission to their respective administrations. Key decision points should be identified.

The timing of the installation faces critical scheduling decisions that have to be taken at about the same time as the planned arrival of DECam at CTIO in June 2011.

Removal of the current Blanco top-end marks a “point of no return” for the telescope, when no other science programs can be scheduled at least until the end of the first phase of commissioning (when the F8 secondary system is commissioned.) At the moment, this is scheduled to begin as soon as DECam has successfully been powered on in the Coude’ room at CTIO. This phase of the project must be managed carefully.

18

2.1.3 DECam Integration & TestMatt Johns, Carnegie Institution


Recommendations2.1.3.1 Complete the plan for installation of DECam on the telescope

jointly with CTIO and prepare a funding proposal to support work by DECam staff to assist with installation and commissioning as per the management section recommendation.

19



20

2.2 DESDMJulian Borrill, LBNL / Rob Cameron, SLAC

OFFICE OF

SCIENCE


Yes, to meet the minimal objectives; meeting the full objectives will be challenging in the remaining time.


For the most part, yes, although restructuring the project leadership is still in progress.

21


OFFICE OF

SCIENCE

Findings• Project leadership has been restructured, with a new PI, project manager, and data

challenge coordinator.• DC5 has been partially completed, demonstrating reasonable data quality and

effective use of node reservation.• DC5a/b also exposed human errors, software bugs, algorithmic limitations, task

management challenges, computation, IO and database bottlenecks, and hardware reliability issues; these are being addressed.

• Analysis of MOSAIC data illustrated real data issues not present in simulations.• DC6a (Community Pipeline testing), DC6b (DESDM System acceptance testing)

and DC7 (stress testing) are being planned for the next 13 months.• Data processing has moved to NCSA’s Abe system, augmented by smaller clusters

in Louisiana and Texas; NCSA’s Blue Waters system, scheduled to come online at the end of the DESDM development phase, is central to operating phase plans.

• CP requirements have been drafted but are yet to be agreed.• Scientific analysis computing requirements are being gathered from SWGs and

institutions.• The current budget only extends through FY11, ending before commissioning.

22


OFFICE OF

SCIENCE

Comments• The combination of the Project Scientist’s relocation to Europe, the new PI

only having an 8% time-commitment, and the delay in appointing a Project Manager, has been detrimental but the situation should now improve.

• The number and complexity of the remaining data challenges has increased, resulting in a very tight, success-oriented, schedule.

• Although a formal deliverable, no commissioning plan exists for the CP; in the current schedule, CP will be the first element exercised in spring 2012.

• The need to reprocess the data coupled with the frequency of Abe’s node failures made DC5 unexpectedly time- and energy-consuming. Some DC5a error flags were either unset or unnoticed, and slow write-intensive jobs have required throttling, introducing a serial bottleneck.

• Analysis of real data alongside simulations is very valuable; preCAM is an obvious candidate.

• Stress-testing is not currently included in the DESDM acceptance tests, but only occurs immediately before DECam deployment.

• Feedback from the SWGs, as DESDM customers, would be extremely useful in future reviews.

23


OFFICE OF

SCIENCE

Recommendations

• Revise the risk management plan in the light of DC5 issues, particularly including extra-project hardware problems and transition costs.

• Include stress-testing in the DESDM acceptance plan.

• Develop a timely commissioning plan for the Community Pipeline upon completion of its specification.

• Consider including requests for additional resources where those are found to be necessary to meet project objectives on time when submitting the supplementary proposal to NSF to fund DESDM activities through at least the DECam/CP commissioning phase.

• Schedule a post-DC7, pre-commissioning, external review of the readiness of both the DESDM System and the Community Pipeline.

24

3. Operations Planning, CalibrationMatt Johns, Carnegie Institute/


OFFICE OF

SCIENCE

1. Are the preliminary plans for the operations phase appropriate for this stage of the project?

The planning for commissioning and operations is adequate for now, but key decisions need to be made in the next 6-9 months.


Yes, a commissioning plan has been produced.

FindingsThere is now a detailed plan from CTIO describing the activities to be carried

out during commissioning.

The current schedule has commissioning of DECam at CTIO completing in February, 2012. This is right at the end of the observing season for the DES. Any schedule slip would mean that observations of the DES would begin in September 2012. It is important that even if the schedule slips to commissioning beyond March 2012 the DES team obtain some data during the science verification stage to test the DESDM pipeline.

The survey operations plan is adequate at the moment, although no definite funding to support operations has yet been approved.

The DES calibration plan is at a relatively advanced stage, with the first phase of the calibration, the PreCam survey, beginning observations in September 2010.

25



CommentsThere are still concerns about control of the camera during the commissioning

phase. While the MOU states that ownership of the commissioning process rests with the CTIO director, there will likely be circumstances that require work by DECam staff to resolve problems.

Funding the work related to commissioning the camera in FY 2011 and FY 2012 is a crucial need for the DES project.

PreCam, the precursor calibration survey to be carried out at the Schmidt telescope at CTIO, has gone from conception to a critical part of the calibration plan for the DES in less than a year. This is significant progress.

The spectrophotometric calibration system has been successfully demonstrated at Las Campanas. The calibration hardware is on track for arrival at CTIO well prior to DECam.

26



Recommendations:

3.1 Develop a detailed plan including budget requirements for DES operations to submit to the relevant funding agencies as soon as possible.

27



28

4. Cost Kurt Fisher, DOE/SC

Mark Reichanadter, SLAC

OFFICE OF

SCIENCE

2. Are the current cost, schedule and contingency projections consistent with the approved baseline for all the projects?

DECAM available cost contingency ($1.7M) is low. CFIP is probably OK. Significant uncertainty in DES DM schedule projection.


Yes, it appears that the project has been responsive to the cost recommendations.

29



OFFICE OF

SCIENCE

• Findings The DECam Total Project Cost (TPC) is $35.15M

As of May 31, 2010, project is 78% complete ~$26.0M progress is complete. $7.5M of work remains Existing DECam contingency/remaining work has dropped from

~30% to ~22% since last review. Dollar value reserve is $1.7M. Installation/commissioning are not included in project. A bottoms-up cost to complete has been completed since the last

review. The project is not carrying any scope contingency Currently the project has 11 medium to high level risks identified The risks identified in the risk register are not correlated to dollars

30



OFFICE OF

SCIENCE

• Comments

• DECam has made substantial progress since the last review. Most procurements are in hand or under contract. Remaining work is primarily testing and assembly activities (mostly labor).

• After evaluating the remaining risks to DECam, cost contingency appears low.

• DECam should work to identify scope items that could be deferred should contingency demand remain high.

31



OFFICE OF

SCIENCE

• Recommendations

1. Prepare (for DECam) a potential list of scope reductions to adequately address lean contingency reserves. Present to the JOG by August 1, 2010.

2. Monitor contingency on a monthly basis and report usage in the DECam Monthly Report.

32

5. Schedule and Funding Kurt Fisher, DOE/SC


OFFICE OF

SCIENCE

2. Are the current cost, schedule and contingency projections consistent with the approved baseline for all the projects?

DECAM available cost contingency ($1.7M) is low. CFIP is probably OK. Significant uncertainty in DES DM schedule projection.


Mostly, the integrated schedule between the three subprojects has not fully developed. With the hire of the DESDM Project Manager on July 1, 2010, the project will have responded to a key committee recommendation.

33



OFFICE OF

SCIENCE

• Findings

• Critical-Decision 4 (CD-4) for DECAM is scheduled for September 2012.

• The current schedule has 12 months schedule contingency.

• A chronological list of major activities between the 3 subprojects has been developed.

• The property transfer MOU between DOE and NSF has not been ratified.

34



OFFICE OF

SCIENCE• Comments

• Today, the committee finds that it is likely that the DECam will be ready in about a year, CTIO will be prepared to accept the DECam, but DES DM will be late.

• The lack of an MOU between DOE and NSF could impact the first shipment of materials to CTIO scheduled for July, 2010.

• Obtain input from all subproject and develop and maintain an integrated project schedule between the three DES subprojects.

• Recommendations

• Execute the property transfer MOU immediately so as not to impact the first shipment of materials to CTIO.

• Obtain input from all subproject and develop and maintain an integrated project schedule between the three DES subprojects .

35

6. ManagementMark Reichanadter, SLACHenry Heetderks, LBNL

OFFICE OF

SCIENCE

3. Is the management structure adequate and appropriate to successfully execute the projects within specifications, budget, and schedule? Are major risks being addressed effectively?

Yes, for DECAM. Probably for the CFIP. Uncertain for the DESDM.


All recommendations have been adequately addressed with the exception of a DES DM Project Manager, who is planned to start on July 1, 2010.

36


OFFICE OF

SCIENCE

• Findings

• The DES PEP includes a detailed list of deliverables required to achieve CD-4 for the DECam. The DECam relies on ~$6M of in-kind contributions and is tracking their progress. Lens polishing is the most critical outstanding issue.

• A set of tests have been developed and agreed upon by DECam and CTIO to formally accept the DECam hardware as complete and begin the final installation process.

• A new Project Director has assumed the leadership of the DES Project in March 2010.

• Local PI for DESDM has been identified and is working at 8% time.

• A full time DESDM PM has been hired and will start on 1 July, 2010

37


OFFICE OF

SCIENCE

• Findings

• DECam PM indicated that projected FY11 funding is inadequate to support cost of DECam installation at CTIO. In addition, DOE-supported operations activities in FY12 are not adequately funded.

• MOU between DOE and NSF necessary to get favorable tax treatment in Chile for the DECam is not yet signed.

38


OFFICE OF

SCIENCE

• Comments

DECam – Substantial progress, 78% complete. Team has added critical engineering resources. Configuration control / risk processes functioning adequately. Remaining cost contingency is a concern.

FY11 funding to support DECam installation and commissioning has not yet been identified. This should be addressed very soon to avoid a work interruption and delay to DES science.

DES DM – Experiencing significant technical difficulties and schedule delays, related to insufficient resources in DES DM leadership. DES DM PM starts on July 1. Will need substantial support by the JOG, DES Council, Fermilab and NCSA to avoid delays to DES ‘ready for commissioning’.

CFIP – Some schedule delays but adequate float. Expected to be ready for the DECam next year.

39


OFFICE OF

SCIENCE

• Comments

• Summarizing, the committee found challenges in DES DM leadership, FY11/12 funding needs, and property transfer require immediate attention. A sufficiently detailed action plan for each of the three elements should be prepared and presented to the DES Council and JOG for approval (see recommendation).

• DOE and Fermilab would be well served by developing a DES “End Game Plan” to fully develop the strategy and plans for completing project activities and transition into operations and science.

• Fermilab should give due consideration to its role in DES operations and the science program. Include the JOG as appropriate.

40


OFFICE OF

SCIENCE

• Recommendations

1. The JOG, in collaboration with NCSA, Fermilab and the DES team, should work to address the following issues:

a) Resolve any remaining issues with the DOE / NSF property transfer by August 1.

b) Develop a plan and supporting budget for FY11 (installation & commissioning) and FY12 (operations) for resolution by August 1, 2010.

c) Develop a DES DM Action Plan that addresses the tight schedule and emerging technical challenges by August 1, 2010. This should include a credible set milestones and adequate resources.

7. ES&HDennie Parzyck, DOE/FSO

Are ESH aspects being properly addressed and are Integrated Safety Management (ISM) principles being followed? Yes

– DECam has maintained a strong ISM ESH program. – Technical Design Report fully describes the ISM Program. – An effective DOE/Fermilab ESH oversight program has been maintained.– Hazard Assessment (13) and Readiness Clearance ( 4 ) programs have been

implemented. – The DECam Safety Assessment Document is in draft stage. – Communication with Chris Smith demonstrates a transfer of ISM culture to CTIO.

Has the project responded satisfactorily to recommendations from the previous reviews? Yes

– 2009 No previous recommendations. – 2008 Previous recommendation on safety documentation has been addressed.

Recommendations:– Completion of remaining safety documentation.– Continued coordination with CTIO safety program plans.

41

OFFICE OF

SCIENCE

OFFICE OF SCIENCE Dark Energy Survey (DES) Project at the Fermi National Accelerator Laboratory June...

Documents

Transcript of OFFICE OF SCIENCE Dark Energy Survey (DES) Project at the Fermi National Accelerator Laboratory June...