© Richard Wainscoat/University of Hawaii Institute For ...kibrick/p32-kibrick.pdf · Today’s...

32 i n t e r a c t i o n s . . . m a y + j u n e 1 9 9 8

In the past 2 decades, computing science has revolutionized astronomy. The cartoon image of

an astronomer squinting through an eyepiece, twiddling dials, and taking photographs is as

out of date as a bank of keypunch operators feeding stacks of paper cards into a vacuum-tube

computer. Today’s astronomer observes with giant telescopes and state-of-the-art instrumen-

tation controlled by dedicated computer hardware and software. The light he or she gathers

from the furthest reaches of the universe is transformed into a digital data stream.

Collaborative RemoteObserving With the

W.M. Keck Observatory

Through the Far Looking Glass:Collaborative RemoteObserving With the

W.M. Keck ObservatoryRobert Kibrick, Al Conrad, and Andrew Perala

© Richard Wainscoat/University of Hawaii Institute For Astronomy

Robert Kibrick

UCO/Lick Observatory

University of California

Santa Cruz, California

[email protected]

http://www.ucolick.org/

~ kibrick

+1-408-459-2262

Fax: +1-408-459-5244

Al Conrad and

Andrew Perala

California Association for

Research in Astronomy

W.M. Keck Observatory

Kamuela, Hawaii

[email protected]

[email protected]

33i n t e r a c t i o n s . . . m a y + j u n e 1 9 9 8

a r t i c l e

time. In addition, observing teams whosemembers reside at different sites will be ableto observe jointly and simultaneously fromtheir respective locations.

Observing time on the twin Keck Tele-scopes is shared by astronomers from four dif-ferent institutions (see Figure 2): theCalifornia Institute of Technology (Caltech);the University of California, with nine cam-puses plus three research laboratories; theNational Aeronautics and Space Administra-tion (NASA), which sponsors qualified U.S.astronomers interested in planetary science;and the University of Hawaii.

Although these four institutions have notyet formally agreed to establish a “collaborato-ry,” the software tools for mainland remoteobserving described in this article representthe first step toward the formation of a Keckcollaboratory.

Remote Observing From KeckHeadquartersRemote observing from Keck headquartersallows observers to think more clearly, func-tion more efficiently, and avoid the hazards ofthe summit. At the summit of Mauna Kea, theambient air pressure and oxygen level is only60% of that found at sea level. Blood–oxygen

Atop Mauna Kea on the island of Hawaii,at an altitude of almost 14,000 feet, a dozenworld-class telescopes peer nightly into thecosmos. The largest of these (in fact, thelargest in the world) belongs to the W.M.Keck Observatory (see Figure 1). Home totwin 10-meter optical/infrared telescopes,Keck is pioneering the routine use of remoteobserving—controlling the telescope and itslight-analyzing instruments—from its head-quarters in the cattle-ranching town ofWaimea, 32 kilometers distant.

Currently, virtually all astronomers workfrom a control room located immediately adja-cent to the dome that houses the telescope, butat Keck, as at all other high-altitude observato-ries, this arrangement presents unique andpotentially dangerous physical challenges. AtKeck, these challenges have pushedastronomers toward observing remotely fromcontrol rooms located at observatory head-quarters far down the mountain. Astronomersthus avoid the severe physical and mentalstresses of working long nights at high altitude[2]. Currently, 90% of the observations per-formed with the Keck telescopes are conduct-ed remotely from the headquarters facility.

For many astronomers, the ultimate goal isobserving with the Keck telescopes from themainland United States. “I’vealways wanted to observe frommy kitchen table,” says Universi-ty of California Santa Cruz(UCSC) astronomer and instru-ment builder Dr. Steve Vogt.Unfortunately, remote observingfrom the mainland is not yetroutinely available because of thelimited Internet bandwidthbetween Mauna Kea and Cali-fornia. In anticipation ofimproved bandwidth, however,software tools and methods tosupport mainland remoteobserving have been developedand tested.

When the bandwidthbecomes available to supportthis observing mode, mainlandastronomers will reap significantsavings in travel costs and travel Figure 1. W.M Keck Observatory viewed from the north (Keck II, left; Keck I, right)

© R

ichard Wainscoat/U

niversity of Haw

aii Institute For Astronom

y

observing run standing just 10 feet from theelevator door and was completely unable tofind the elevator. Other mistakes made at thataltitude, however, can be lethal. A misstepinside the dome could send a person tumbling60 feet to a concrete floor.

Remote observing from the Waimea head-quarters has proven hugely popular. From Jan-uary to October 1997, the percentage ofastronomers remotely observing jumped from10% to 80%. Observers who opt to workfrom Waimea use one of two remote opera-tions rooms located at Keck Headquarters (seeFigure 3). The work stations in these roomscommunicate with data-taking and telescope-control computers at the summit via a dedi-cated link. The original link, a 1.5-Mbit/secT1, was upgraded to a 45-Mbit/sec DS3 inOctober 1997.

In addition, because Keck instrument spe-cialists work primarily in Waimea and not atthe summit, astronomers working from

levels at the summit drop precipitously in thefirst few hours and never completely recover,even for observatory employees, some ofwhom have made thousands of trips to thesummit.

High-altitude sickness is not uncommon;pulmonary and cerebral edemas are real risks.Some observers, including pregnant womenor those with chronic heart or lung condi-tions, cannot observe at the summit.

Acclimatization is key to performing wellat altitude. Astronomers who observe at thesummit are required to sleep at the 9,200-footbase camp and must arrive at least 24 hoursbefore the start of observing to begin theacclimatization process; even with acclimati-zation, the dramatic physiological changescause thought processes and mental acuity tosuffer dramatically.

Some of the mistakes made at this altitudeare comical. One of the world’s most promi-nent astronomers was found midpoint in an


Figure 2.


a r t i c l e

X protocol approach when remote observingfrom Waimea.

Video ConferencingTypically, the personnel involved in an observ-ing run include the principal investigator (PI),a support team of graduate students and post-doctoral candidates, and the observing assis-tant (OA). The PI conducts the observation,

and the OA operates the telescope and assiststhe PI. Other astronomers, instrument spe-cialists, engineers, and technicians often assistduring the observation.

Personnel in the summit and remote con-trol rooms communicate with one another viaa dedicated video conferencing system (Pic-tureTel Venue 2000, see Figure 4) runningover a portion of a dedicated T1. In the caseswhere the PI and OA are separated, this sys-tem has proven critical for successful observ-ing. Although the system is seldom used toactually view a diagram or hand-drawn pic-

Waimea are able to speak directly to these spe-cialists regarding instrument performance andsoftware improvements.

“We find it helpful to interact directly withthe astronomers to improve the systems,” saidHilton Lewis, software manager for the W.M.Keck Observatory. “By doing so, you’ll noticelittle things that would never be reported butwhich may have a profound influence onobserving.”

X ProtocolAll observing software runs on summitmachines (just as it would for summit observ-ing), but the X displays for the various soft-ware applications are redirected to Waimeaand displayed on the workstation screens inthe remote operations rooms. The primaryadvantage of the X protocol method is sim-plicity. No configuration changes or redistri-bution of software is needed to switchbetween summit and remote observing, andobservers are presented with the identical userenvironment at both sites.

A major disadvantage of the X protocolmethod is performance. The throttling effectof the 1.5-Mbit T1 link on simple, text-onlydisplays is negligible. For displays that requirelarge bitmap transfers, however, the effect isglaringly noticeable. When observing remotelyon both Keck-1 and Keck-2 over the single T1circuit, a zoom/pan operation on a largebitmap image display may take as long as 5seconds. Although this is marginally accept-able for viewing the result, the delayedresponse often causes impatient users to repeatrequests. The subsequent buffering of theserequests (e.g., mouse clicks and mousemotions) puts the observer out of synch withthe application. The initial 5-second delay ismagnified as the resulting disorientation expe-rienced by the observer results in minutes oflost observing time.

Our experiments with remote observingfrom Waimea with a 45-Mbit link show littleperformance degradation with bitmap dis-plays. This is not surprising, given that the 45-Mbit rate exceeds the 10-Mbit rate achievedon the LAN backbones in place at Waimeaand the summit. Given the recent upgrade toa 45-Mbit link, we plan to continue with the

Figure 3. Paul Butler (foreground) and Geoff Marcy, planet huntingastronomers in the Keck I remote control-room

Figure 4. Picture-Tel Venue 2000 shows OA Ron Quick (left) and electronictechnician Hector Rodriguez (right) at Keck I summit.

© B

arbara Schaefer/ W. M

. Keck O

bservatory©

Barbara Schaefer/ W

. M. K

eck Observatory

for remote observing, currently its inability totransmit data securely makes it inappropriatefor this use.

Remote Observing From the MainlandDespite the inadequate and unpredictablebandwidth of existing network links betweenMauna Kea and the mainland and despite thelack of any formal collaboratory infrastructure,Keck observers have requested a variety ofmainland remote observing capabilities. Inresponse to these requests, researchers at UCSCand Caltech have informally collaborated witheach other and with their counterparts at Keckon several different experiments to developmainland remote observing capabilities. Theexperiments conducted from Caltech werefunded by a NASA grant and investigated theuse of dedicated, high-speed links provided byan experimental NASA geostationary commu-nication satellite [1]. The experiments conduct-ed from UCSC had no external funding andhave explored various techniques to supportmainland remote observing over existing publicInternet links [3].

Mainland Remote Observing Over theExisting InternetInternet bandwidth and reliability present thegreatest challenges for mainland remoteobserving. As most observing teams are allo-cated only a small number of nights on thetelescope each year, they understandably areunwilling to risk a significant loss of observ-ing time caused by Internet outages or con-gestion. Similarly, because the cost ofoperating the Keck Telescopes is several thou-sand dollars an hour, the observatory cannotallow its telescopes to sit idle because aremote observer on the mainland has lost anetwork connection. As a result, until Inter-net bandwidth and reliability issues areresolved (or until a separate, reliable, dedicat-ed link to the mainland becomes available),mainland remote observing via the Internetwill require at least one member of theobserving team to be in Hawaii, either atWaimea or at the Keck summit.

If the network connection to the main-land is interrupted, the on-site observerassumes responsibility for the observing pro-

ture, we have discovered that the visual cues ofbody language and being aware of which oth-er team members are listening are essential fortwo people collaborating on the operation oftelescope and instrument [4].

Before acquiring the dedicated PictureTelsystem, we investigated generic systems thatuse TCP/IP and work-station technology totransmit audio and video over the network.Although such systems are less expensive andmore flexible, we found that PictureTel pro-vided superior performance.

Audio quality stands out as the mostimportant factor favoring the dedicated sys-tem. Other features include better picturequality, preset camera positions, and a largerfield of view. In addition, the dedicatedvideo system isolates the link used for videoconferencing from the link used for summitto Waimea computer network traffic, thusconserving network bandwidth and prevent-ing the loss of audio and video contactshould the computer network fail or becomecongested.

Hogging the NetOne of the greatest challenges for remoteobserving is determining which cooperatingprocesses are consuming disproportionateamounts of the available bandwidth or, moresimply stated, “Who is hogging the net?”

Although a 45-Mbit link accommodatesnormal traffic, we have found that a largeblock transfer, which is network limited (typ-ically a tape backup of a cross-mounted filesystem), can effectively lock out communica-tion between two event-limited processes. Todate, we have not found a method of throt-tling these block transfers and rely on opera-tional rules to prevent this type of transferfrom interfering with observing.

Use of the World Wide WebAlthough we have started using World WideWeb methods such as HTML, CGI, and Javato facilitate observing preparation (i.e., run-ning planning tools, specifying instrumentconfigurations, and scheduling the telescope),to date we have not experimented with thesemethods for real-time observing. AlthoughWeb technology may become the best method

Audioqualitystands outas themostimportantfactorfavoringthe dedicatedsystem.


We neededan applica-tion thatmade minimalassumptionsabout thesystem softwarealreadyinstalled on theobserver’smachine.


a r t i c l e

(FTP) or Web connections to collect imagestaken by their on-site colleagues. Although theon-site team members could initiate outgoingconnections and could push these images tothe mainland sites, they are usually too busyto do this.

Although technical solutions (e.g., dis-tributed secure file systems) exist that addressmany of these security-related concerns, theyrequire a significant level of coordinated sys-tem management between the participatingsites. Although this level of coordination ispossible within a heavily funded and centrallyadministered collaboratory, it was not feasiblein our more diverse environment.

Often, the remote observer’s workstationwas located at some remote mainland institu-tion that had no affiliation with Keck Obser-vatory. Frequently, the machine had littlesystem management support, and theastronomers who used it were not particularlyskilled in installing, configuring, or maintain-ing complex system software. In addition,requests for mainland remote observing sup-port often arose at the last minute, especiallyin cases where an observing team member hadplanned to go to Hawaii but suddenly couldnot because of illness or a family emergency.

For these reasons, we needed an applicationthat made minimal assumptions about thesystem software already installed on theobserver’s machine, could be easily installedby a novice user, required no configurationnor intervention by local system management,and only relied on outbound connectionsfrom Keck to the remote site.

As a result, the “shadowmon” suite wasdeveloped at Lick Observatory/UCSC andwas successfully used to support the automat-ed relaying of compressed Keck images to avariety of California remote observing sites(i.e., Santa Cruz, Berkeley, San Diego, andLivermore), as well as to the Space TelescopeScience Institute (STSci) in Baltimore, Mary-land, and the Harvard Center for Astrophysicsin Cambridge, Massachusetts.

This suite consists of a very small set ofsimple utility programs, some that run at theKeck summit and others that run at the main-land remote observing site. The remoteobserver simply downloads the software and

gram. Except for the Caltech experimentsnoted earlier here, all mainland remoteobserving conducted to date has included atleast one member of the observing teamlocated on-site in Hawaii. Coordinationbetween the on-site observer and the observ-ing team on the mainland is accomplished bykeeping open a telephone line (often con-nected to speaker phones at each end) or byusing digital audio tools to carry conversa-tions over the Internet.

Conventional telephones provide superioraudio quality, greater ease of use, andenhanced conferencing capabilities and arephysically more portable than digital audiotools running on the work stations. The costsof all-night phone calls to Hawaii have provedminimal (less than $50 per night).

Internet network security presented anadditional problem for mainland remoteobserving. Our experiments at UCSCaddressed this problem in different ways,depending on the characteristics of the remoteobserving site. We investigated two separateapproaches to mainland remote observing:✖ “Eavesdropping” to arbitrary, nontrusted

sites at low-to-moderate bandwidth.✖ “Full-up” remote observing to a specific

trusted site at moderate bandwidth.

Eavesdropping to Arbitrary Sites Overthe InternetWhile the on-site member of the observingteam attends to local tasks (e.g., confirmationof target acquisition, instrument setup, andinitiation of exposures), the mainland remoteobservers focus on quick-look data analysisand overall monitoring of the science pro-gram. Accordingly, a mainland remote observ-ing system must at least provide remoteobservers with timely access to the scienceimages from the telescope. We refer to thismode of operation as eavesdropping to distin-guish it from the full-up remote observingcurrently provided at Waimea.

Network security measures in place at theKeck summit made implementation of eaves-dropping difficult. The summit computers areprotected by a firewall that rejects incomingconnection requests. Thus, mainland remoteobservers cannot initiate file transfer protocol

gested. Although the on-site observer who col-lected the data is sleeping, the remote observ-er will be busy analyzing it and can report theresults to the on-site colleague that evening.This multiplexing of activities betweenobservers on different sleep schedules can pro-vide an extremely efficient means of collectingand analyzing data.

Full-Up Mainland Remote ObservingOver the InternetThis mode of observing requires a trustedmachine at an affiliated site and usuallydemands close collaboration between therespective system management staffs at bothsites. The trusted machine is allowed to makeincoming connection requests to the summitcomputers and is thus able to run the fullcomplement of observing software. Encryp-tion mechanisms safeguard the security ofthese connections. Full-up remote observingover the Internet currently requires morebandwidth than is routinely available, and it isnow possible only a few hours each nightwhen the Internet is relatively idle.

We have tested two different models:1. All observing software runs at Keck sum-

mit with X displays routed to the remotemainland site.

2. Components of the observing software arerun both at the Keck summit and at oneor more of the remote sites.The first model parallels the current mode

of remote observing from Waimea and thusshares all of its advantages and disadvantages.Because, in most cases, the bandwidth of thelink between mainland sites and the Kecksummit is significantly lower (and often lesspredictable) than that between the summitand Waimea, disadvantages that are band-width sensitive become more significant. Giv-en the bandwidth of the existing Internet link,this model is extremely difficult to use at bestand results in significantly reduced operatingefficiency.

The second model allows the user inter-faces, the image display, and the image record-ing software to be run locally at the remotesite. This provides several advantages. First,interactive response is significantly improved,as mouse motions and button clicks are han-

starts it running on his or her machine. Thesoftware requires no further action for theremainder of the night. Once correspondingsoftware is started at Keck, any subsequentimages from the specified instrument will beautomatically relayed to the remote observer’smachine and recorded to its disk.

As each new image is received, severalbeeps are issued to announce its arrival. Bar-ring network disruptions or congestion, therelayed compressed images, which are severalmegabytes in size and up to 10 megabytesuncompressed, are available for use at theremote site in under 3 minutes from the timethey were written to disk at Keck.

On the Keck side, the corresponding soft-ware is invoked by Keck observatory staff,who specify as a parameter the location of theremote observer’s computer. Because the net-work connection to that site is initiated fromthe Keck side of the link, there is no conflictwith the firewall at the Keck summit. At theend of the observing session, Keck staff shutdown the remote observing software at bothsites. Remote observers cannot continue toobtain Keck images beyond the end of theirobserving run, even if they restart the remote-side software on their machine.

The shadowmon suite has proved to be avery effective, secure, and easy-to-use tool tosupport eavesdropping over low-to-moderatebandwidth public Internet links. Once startedat the beginning of the night, it transparentlyrelays images from Keck to the remote sitewithout any intervention required from theobservers at either site. Thus, observers at bothsites can focus on their respective tasks of col-lecting and analyzing the data without havingto waste any time on the transport mechanism.

Clever observers have found innovativeuses for this suite that were not anticipatedwhen first implemented several years ago. Forexample, an observing team may elect to leaveshadowmon running even if the mainlandremote observer has to retire early. The relayeddata will accumulate on the remote observer’smachine and will be waiting for analysis thenext morning. Otherwise, the remote observ-er would face a several-hour delay trying to getthe remaining data transferred from Keck dur-ing daytime hours when the Internet is con-

Becausethe networkconnectionto that siteis initiatedfrom theKeck sideof the link,there is noconflictwith thefirewall atthe Kecksummit.


Mainlandobserverswill beable toconductsciencealmost aseffectivelyfrom theirhomeinstitutionsas theynow canfromWaimea.


a r t i c l e

eavesdropping from the mainland clearlydemonstrates that both modes increase observ-ing efficiency and enhance the ability of bothastronomers and support staff to collaborateeffectively in real time over long distances.

Our experiments with full-up remoteobserving from the mainland establish thatobservers will be able to conduct sciencealmost as effectively from their home institu-tions as they now can from Waimea, once thenecessary network infrastructure is in place.

Finally, Internet 2 holds the promise ofsolving most of the bandwidth and bandwidthmanagement concerns that will prove criticalfor mainland remote observing with our nextgeneration of Keck instruments. These newinstruments have larger imaging detectors andfaster readout rates and will require the signif-icantly higher bandwidth that Internet 2would provide. Because Internet 2’s primarygoals include the enhancement of collaborato-ry efforts, including remote operation of andaccess to major research facilities, it is essentialthat a path be found to extend Internet 2 con-nectivity to Keck and the other Hawaii obser-vatories. Once that is done, full-up mainlandremote observing may become commonplace.

References1. Conrad, A. R., Kibrick, R. I., Gathright, J. “Remote

observing with the Keck telescopes”, In Lewis, H. (Ed.),

Proceedings of the SPIE 3112, 1997, pp.99–110. (URL:

http://www.ucolick.org/~kibrick/remoteobs/article.ps).

2. Forster, P. J. C. “Health and work at high altitude: A

study at the Mauna Kea observatories,” Publications of

the Astronomical Society of the Pacific 96 (1984),

478–487.

3. Shopbell, P. L., Cohen, J. C., and Bergman, L.

“Remote observing with the Keck Telescope from Cali-

fornia using NASA’s ACTS Satellite,” In Lewis, H.

(Ed.), Proceedings of the SPIE 3112, 1997, pp. 209–220.

(URL: http://astro.caltech.edu/~pls/papers/spie-97).

4. Storck, J., and Sproull, L. “Through a glass darkly:

What do people learn in videoconferences.,” Human

Communication Research 22 (December, 1995),

197–219.

dled locally. Equally important, it allows thetransmission of image data to be overlappedwith readout of the image detector. This is acrucial point, as our current detectors require1 to 2 minutes to read out each exposure. Ifwe wait for the complete image to be read outbefore we begin transmitting it, we waste sev-eral minutes of network bandwidth that couldhave been used to transmit the image.

By overlapping image transmission to theremote site with image readout, overall uti-lization of the link bandwidth is significantlyincreased. Observing efficiency is alsoimproved because the remote observer beginsto see the displayed image data as soon as thedetector readout commences. He or she canthen contribute to team decisions about whatto do next (e.g., continue observing the cur-rent object or begin moving the telescope tothe next target) based on the data from thefirst part of the image. The remote observer isthus able to participate more actively in theobserving process.

Once the image readout completes, theremote observer has a local copy (identical tothe one recorded at the Keck summit) of theimage data file on disk, allowing more effi-cient data reduction and analysis, as well aseliminating the need for repeated NFS or FTPaccess to the data files recorded at the Kecksummit.

Another advantage of this model is that itallows the distribution of the image inreal–time to multiple sites (e.g., Keck summit,Waimea, and California), protecting against aloss of data caused by hardware failure at oneof the sites. This capability also greatlyenhances the collaborative efforts of observingteams whose members are jointly observingfrom different sites, because all team membershave simultaneous real-time access to the dataas it is read out of the detector.

ConclusionOur operational experience with full-upremote observing from Waimea and with

PERMISSION TO MAKE DIGITAL OR HARD COPIES OF ALL OR PART OF THIS WORK FOR PERSONAL OR CLASSROOM USE IS GRANTED WITHOUT FEE PROVIDED THAT COPIES ARE NOT MADE OR DISTRIBUTED FOR

PROFIT OR COMMERCIAL ADVANTAGE AND THAT COPIES BEAR THIS NOTICE AND THE FULL CITATION ON THE FIRST PAGE. TO COPY OTHERWISE, TO REPUBLISH, TO POST ON SERVERS OR TO REDISTRIBUTE TO

LISTS, REQUIRES PRIOR SPECIFIC PERMISSION AND/OR A FEE. © ACM 1072-5220/98/0500 $5.00

© Richard Wainscoat/University of Hawaii Institute For ...kibrick/p32-kibrick.pdf · Today’s...

Documents

Transcript of © Richard Wainscoat/University of Hawaii Institute For ...kibrick/p32-kibrick.pdf · Today’s...