Review of Scientific User Facilities Division

US Particle Accelerator School

Review of

Scientific User Facilities Division

Committee of Visitors

Presented by W. A. Barletta, Chair


Top level charge

Assess processes used during FY2010, 2011, & 20121. to solicit, review, recommend, & document proposal actions 2. to monitor active projects and programs.

Program elements to assess1. Light Sources including the Accelerator & Detector Research

Program2. Neutron Sources3. Nanoscale Science Research Centers & Electron Beam Micro-

Characterization Centers4. Construction Projects & MIEs

Report to the Summer 2013 BESAC meeting


Charge specifics

(1) For scientific user facilities including the accelerator & detector program, Assess the efficacy Assess the quality of processes used to:

• (a) Solicit, review, recommend, & document proposal actions (b) Monitor active projects, programs & facilities

(2) Within boundaries set by DOE missions & funding, Comment on how the award process has affected:

• (a) Breadth & depth of portfolio elements• (b) National & international standing of portfolio elements


BES SUFD Committee of VisitorsDr. William Barletta (MIT/USPAS), Chair

Construction Projects James Krupnick (LBNL, Ret.) Angus Bampton (PNNL) Jeff Hoy (Trident Service LLC) Maria Dikeakos (DOE PPPL)

Nano-Science & E-Beams Prof. Donald Tennant (Cornell) Prof. Beatriz Roldan Cuenya

(UCF) Dr. Ernie Hall (GE) Dr. James Liddle (NIST)

Light Sources, Accelerator & Detector Research

Dr. Simon Bare (UOP) Prof. Nora Berrah (WMU) Dr. Gene Ice (ORNL) Dr. Joel Ullom (NIST) Dr. David Robin (LBNL)

Neutron Facilities Prof. Sunil Sinha (UCSD) Dr. Robert Dimeo (NIST) Prof. Thomas Russell (UMass) Dr. John Tranquada (BNL)

Chairman in RedBESAC Member in Italics


Overarching conclusions

We commend SUFD for effective use of its available funding to construct & operate a system of facilities which deliver world-leading science

The efficacy of SUFD’s processes to review, recommend & document proposal actions are excellent

The efficacy of the processes to monitor & review active projects, programs & facilities are also excellent

SUFD staff members are to be commended for rigorous & effective program management

Comment: BES growth over the decade is impressive


Overarching conclusions

Within the scope of DOE missions & available funding, SUFD’s award processes continue to enhance the breadth & depth of portfolio elements as well as their national and international standing

International competition in scientific user facilities is fierce; maintaining scientific leadership will require increased investments for the user facilities for user support


SUFD-wide findings & recommendations - 1

A. Implementation of previous COV recommendations FINDING: SUFD has effectively addressed the majority of the

recommendations made in the previous COV report FINDING: SUFD travel budget continues to be incommensurate

with most effective oversight of its program

RECOMMENDATION: Enhance the effectiveness of program oversight by increasing the flexibility of SUFD managers

• to interact with facility managers • to communicate with the facilities staff via increased on-site

presence



B. Assessment of COV process effectiveness FINDING: Providing all data on computers to each reviewer is

highly responsive to previous recommendations & greatly improves the COV process.

• The electronic documentation was thorough and well organized

RECOMMENDATION: The move toward a full, searchable database is commendable. It should be available to the next COV



C. Facility review process description & effectiveness FINDING: The 3-year reviews of the facilities are well organized

and well executed with appropriate review teams FINDING: A uniform definition of high impact publications has

been established for light sources & neutron sources. RECOMMENDATION: Finalize the set of uniform definitions

of high impact publications for the nanoscience centers



D. General Issues FINDING: The quality of the scientists at the facilities is the

critical asset that ensures excellence & success. RECOMMENDATION: Place added emphasis on career

development as well as on maintaining state-of-the-art experimental apparatus, sample environment & software at all facilities to maximize scientific productivity

FINDING: Different types of facilities serve different scientific communities. They are all needed & important.

RECOMMENDATION: New metrics that account for scientific impact should apply to all the types of scientific user facilities



Recommendations: If an increased fraction of industrial users is desired, this fact should be clearly communicated to the management of facilities Proposal evaluation should include criteria that value factors such

as economic & technological impact Thorough review of user agreements should be undertaken with a

view to removing barriers to industry users


Light sources and

Accelerator & Detector Research


Light source findings - 1

The review process is comprehensive, balanced, fair, and transparent to all facilities They are all reviewed using the same major criteria.

The number and breadth of the reviewers for each facility was appropriate

Reviews provided important comments & specific actionable recommendations where appropriate.

The reviews were uniformly forthright, detailed, & contained a summary section at the beginning of the review The detail in review comments indicated a deep appreciation of

important issues that went beyond the documentation solicited. This indicates the importance of the on-site review.


Light source findings - 2

Review follow-up was not documented between reviews RECOMMENDATION: A formal yearly follow up of

facility recommendations should be documented each year Ideally this could be a short response saying all issues had been

previously addressed when appropriate

Recommendations from the prior reviews were not sent to the facility reviewers

RECOMMENDATION: Make previous review recommendations & facility response available to the facility reviewers at the outset to allow the review committee to assess how the facility has

responded to the prior recommendations.


Accelerator & Detector Research - Findings

Resources available are so limited ~$10M (~1% of total SUFD) that the breadth is necessarily small (focusing primarily on FELs) This figure is too small to maintain scientific leadership

Projects are generally excellent & matched to SUFD needs At present the portfolio does not include x-ray optics


Accelerator & Detector Research - Recommendations

Increase the ADR portfolio to $20M to $30M per year (2-3% of SUFD budget)

Consider the concept of a HUB to advance ADO technology in support of its scientific mission

Add X-ray optics to the ADR portfolio

As part of increasing the portfolio, formalize the proposal solicitations

Continue using workshop reports to guide research initiatives & to shape investment priorities

Develop topic-specific metrics to assess / characterize US capabilities in accelerators, detectors & optics


Neutron sources


Neutron sources

Finding : Few instruments in ORNL’s Neutron Sciences Directorate operate in the steward-partner model.

RECOMMENDATION: Strongly encourage neutron scattering facilities to explore forming partnerships on instruments with potential partners from other agencies in the cooperative stewardship model Exploit neutron scattering capabilities to benefit the broadest

possible scientific community Finding: It is unreasonable to frontload facilities with

understaffed instrumentation or beamlines Recommendation: When an MIE is being considered, a

the facility should have well-designed plan to ensure its robust, long-term operation for users


Neutron sources & neutron science

Finding: Trends in the science enabled by neutrons have evolved significantly since last studies (1993, 2002) on the future of neutron science in the US

Recommendation: We recommend that BES join with other agencies, such as DOC, NSF, & NIH, to assess current status & future directions for U.S. neutron science Include factors, such as neutron measurement capacity &

capabilities and present & future needs of the U.S. scientific community


Nanoscience Research Centers &

E-beam Microcharacterization Centers


NSRCs & EBCs - General Observations

Finding: NSRCs are entering a post-ramp up phase in which they face issues of high demand for facilities & instruments & over subscribed staff

Recommendation: Provide guidance to NSRCs to plan for expansion of facilities or expanded operating hours Alternatively adopt higher rejection rates of user proposals


Finding: SUFD program managers have done a commendable job at establishing thorough & transparent processes for evaluating ongoing projects The 3-year review cycle might be excessive for timely corrective actions

Recommendation: Increase face-to-face to time between DOE officials & management, scientific staff, & user community of the NSRCs &EBCs, including regular (yearly) on-site visits

Methods of assessing NSRCs have not evolved since the centers were first commissioned

Recommendation: NSRCs (& EBCs) are sufficiently different from light sources to warrant tailored metrics & assessment methods (user surveys)




BES recommended merger of EBMCs & NSRCs in response to a triennial review finding that EBMCs were understaffed & underfunded to carry out their mission. This recommendation focuses on improvements in synergy &

operational efficiency. Recommendation: SUFD should ensure that facility

merger plans focus clearly on these improvements


High Utilization Instruments

Finding: The unique, world-leading instruments associated with the EBMCs are in high demand but are not utilized optimally Staff funding is currently for 40 hour week, yet the labs are

open more than 8 hours per day.

Recommendation: Merger plans should include expanded staffing (>8 hr/day) on select tools


Industry Participation NSRCs have not attracted a significant number of

industry users

Recommendations: If an increase in the fraction of industrial users is desired, this fact should be clearly communicated to the NSRCs Proposal evaluation should include criteria that value

potential technological or economic impact Thorough review of user agreements should be undertaken

with a view to removing legal barriers to industry users


CONSTRUCTION PROJECTS


Findings regarding project management

Comment: PM processes are robust & successful in delivering projects on schedule, within budget & meeting baseline technical performance parameters SUFD met, and often exceeded, BES goal of remaining within +

/ - 10% for cost/schedule performance Project management performance metrics have become even more

visible indicators of DOE performance for external stakeholders

Recommendation: Ensure that CD4 requirements are reasonable, broadly understood by all stakeholders, & fully achievable within the project budget Effort should be made to manage & align expectations for

what constitutes successful initial scientific operations


Impact of travel reduction

Comment: SUFD management should determine the correct level of field presence for Program Managers to provide adequate Federal oversight, operational awareness, & fosters strong and open communication between field and HQ elements On-site presence, graded to project risk, should be appropriately

balanced with use of remote communication tools to live within constrained budgets

Recommendation: Mitigate negative impacts of reduced travel funds Balance field presence with communication technology to

maintain robust communication between program managers & the on-site members of IPTs


The COV thanks Jim Murphy and the entire SUFD staff

for their candor & responsiveness

Special thanks to Linda Cerronefor her help in making this review

easier and more pleasant

Review of Scientific User Facilities Division

Documents

Transcript of Review of Scientific User Facilities Division