HiRes-1 Operations

Presented by: C. Jui

HiRes Review

NSF, Nov. 22, 2002

HiRes-1 Operations

• HiRes-1 operated continuous since 1997 on nights with > 3 hours of darkness EXCEPT:– August 1999: U. of Utah hosted the 26th

International Cosmic Ray Conference (ICRC)– Pentagon Security Review (5/2000-7/2000) – Sensitive tests (6/2001, and 8/2001) +…– Security Alert status (10/2001- 4/2002)

HiRes-1HiRes-1OperationsOperations

1997

moonless night time:

884

monocular observation:

436

El-Nino: Dugway Precip

> 200% of normal

HiRes-1HiRes-1OperationsOperations

1998

moonless night time:

1428

monocular observation:

738

El Nino: Dugway Precip

> 200% of normal

HiRes-1HiRes-1OperationsOperations

1999

moonless night time:

1369

monocular observation:

820

ICRC 1999 in

Salt Lake City

HiRes-1HiRes-1OperationsOperations

2000

moonless night time:

1489

monocular observation:

866

Shutdown for Pentagon Review

HiRes-1HiRes-1OperationsOperations

2001

moonless night time:

1400

monocular observation:

461

Sensitive Tests at Dugway

Post 9/11

HiRes-1HiRes-1OperationsOperations

2002(as of 11/17/02)

moonless night time:

1215

monocular observation:

323

Restart with LANL operators

HiRes-1HiRes-1OperationsOperations

Cumulative(05/28/97-11/17/02)

moonless night time:

7642

monocular observation:

3563

7.45% duty cycle including shutdown (~20%) time

Monday Nov. 18 was Day 2000

1998 1999 2000 2001 2002

Stereo Observations

• Full stereo operation in coincidence with HiRes-2 since November 1999.

• During 2000: added Auto-Pilot system for routine start and shutdown to maximize observation time and to improve security.

• Both sites were achieving target duty cycle of 10% when 9/11 event occurred.

HiRes-1HiRes-1OperationsOperations

Stereo Era(11/15/99 - 10/01/02)

moonless night time:

4164

HiRes-1 observation:

1688

Stereo observation:

968

2000 2001 2002

Since Sep. 25, 2001

• Dugway has been operating under heightened security: only personnel with military SECRET or DOE L-level clearance are allowed to operate detectors.

• Subcontract Los Alamos National Lab staff– 7 staff physicists (4 are HiRes collaborators).– 3 engineer/technicians.

Restart

• Successful restart in May of 2002 with 2 LANL personnel at EACH HiRes site.

• To reduce costs, restrict observations to nights with > 5 hours of darkness

• Remote operation of HiRes-1 site from University of Utah since 6/2002

Remote HiRes-1 Operations

• Remote operations of HiRes-1 was planned well before 9/11 event:– New Army regulation requiring two operators

at each site added significantly to manpower requirement for stereo observations.

• HiRes-1 was logical choice for remote ops:– More convenient for pre- and post-run stops.– Fewer detector units: easier to operate.

HiRes-1 Upgrades before 9/01

• University of Adelaide provided new scanning cloud monitor

• More sensitive than in-situ visual observation

HiRes-1 Upgrades before 9/01

• Failsafe curtains added to all HiRes-1 mirrors:– protects all mirrors against direct sunlight

– photo sensor drops curtains when light level reaches ~30 min. before sunrise or in case of power failure.

• Backup diesel generator for closing doors in case of power failure (use by HiRes-2 runners).

• UPS systems added to all critical electronic equipment in Central Facility.

• Purchased remote power controller system.

HiRes-1 Upgrades since 5/02

• Installed remote power controllers (Baytech RPC and DS-3 systems):– RS-232 interfaced power strips placed between

equipment and respective UPS modules

– Critical equipment can be power-cycled via dial-up.

– Dial-up modem disconnects automatically after 60 min.

– Modem has fail-safe watch-dog timer which will restart the module in case of lock-up.

– High Voltage for entire site enabled/disabled via remotely controlled power switch

HiRes-1 Operating Procedure

• HiRes-1 detector started at 3.5 hours before “run” time (moonset or end of astronomical Twilight) to diagnose readout electronics problems.

• Runners visit HiRes-1 at 3 hours before “run” time to check status of curtains and doors and to swap-out defective readout modules (max. stay: 30 min).

• After shutdown, runners wait for confirmation of door closure at HiRes-1. They visit HiRes-1 on return trip only in the event of door failure.

• HiRes-1 run control uses the VNC remote session client/server system

• VNC sessions stay up in the event of network failure

• Interrupted session can be resumed from HiRes-2 or using modem link

• 3 nights of modem- only running so far

HiRes-1 Remote Run Control

HiRes-1HiRes-1OperationsOperations

Since Restart(05/02-10/02)

moonless night time:

392 ***

HiRes-1 observation:

194 (323)

Stereo observation:

156 (processed.)

*** nights > 3 hours of darkness

Sensitive Tests at DugwayRemote

HiRes-1 Operation

Future HiRes Operation• Will continue to operate with LANL operators, and

for now on nights with > 5 hours of darkness.• Currently contract 1.0 FTE from LANL.• We are applying for clerance for HiRes citizens

through our contact at LLNL.• One member of HiRes has received clearance and is

already taking on a heavy load of shifts• Four more members of HiRes have been “finger-

printed”: clearance expected over the next 6-12 months.

Future HiRes Operation• Those already fingerprinted includes one engineer

and one technician.• Five more applications are in the pipeline:

expected to come through over the next 2 years.• Expect to ramp-down LANL sub-contract:

– 1.0 FTE in year 1

– 0.75 FTE in year 2

– 0.5 FTE during years 3-5.

• Also explore other possibilities to reduce costs.

Future HiRes Operation• As more clearances come through we will return

operating on all nights > 3 hours of darkness. Expect this to be possible before FY2005.

• Results of operations from 2002 are very encouraging.

• We anticipate to be able to achieve 10% duty cycle for the last 4 years of this proposal.

HiRes-1 Operational Issues• Major repairs restricted to pre-arranged “access”

days (1-2 before each run) + emergency access– this will improve as our technical staff get clerance

• Moving parts (garage doors and curtains) :– have installed small (~400 VA) UPS units to prevent

curtains from deploying during power glitch.

– installation of fail-safe curtain position sensors within 30 days: will eliminate need for routine pre-run visits.

– installation of redundant backup door control and sensing system planned for the next 2 months.

HiRes-1 Operational Issues• Remote operations puts more demand on network

reliability.– Current arrangement with Utah Education Network

(UEN) is convoluted and has many points of failure

– Propose dedicated T1 line ($1k/month) from U.of U. to HiRes-1.

• Remote operations make us more vulnerable to power glitches and brown-outs.– These events are occurring more frequently

– Propose to purchase 3kVA UPS units for HiRes-1 mirrors.

HiRes-1 Upgrade• A number of the components used for HiRes-1 read-

out system (designed in 1990) are no longer available from manufacturers.

• HiRes-1 limits the low-energy (< 31018 eV) stereo aperture (elevation limited to < 17 deg.)

• Sample-and-hold read-out limits the precision of reconstruction.

• Upgrade to FADC system + some “ring-2,3” mirrors (between 17 and 45 deg. in evelation) proposed over the next 5 years.

Summary• HiRes-1 reliably operated since 1997.

• Successful restart since 5/2002 with 323 hours of data collected to date.

• Successful and reliable remote operations since 6/2002.

• Return to >3 hour observational regime by FY2005 (year 2) expect to achieve 10% duty cycle.