Workshop in ALMA Logs

Prepared by Juan Pablo Gil – Arturo HoffstadtADC Software GroupALMA Observatory

March, 2015

ALMA Hardware Layout

• 66 Antenna

• ~45 device per antenna

• AOSTiming

• Photonic References

• 64 ant-correlator

• 16-antenna correlator

ALMA Software Layout

• ALMA Software is built upon ACS

• Different Subsystems

• CONTROL• Antennas• Time reference

• SCHEDULING

• DATACAPTURER

• BLCORR (64 ant)

• ACACORR (16 ant)

• TELCAL

• QUICKLOOK

• EXECUTIVE

• ARCHIVE

ACS Deployment

• Component

• Actual code that controls one physical device, or does one specific task

• Eg. IFproc, DTX, ARCHIVE_CORR_BULKRECEIVER_NT, CONTROL_MASTER

• Container

• Logic grouping of components.

• Has PATH/containerName

• Eg. CONTROL/DV01/cppContainer has IFProc, DTX

• Host

• Computer which runs containers.

• Eg. dv01-abm runs CONTROL/DV01/cppContainer

• xx-abm, gasXX, cob-cpn-xx, lo-lmc-x, etc

dv01-abm

CONTROL/DV01/cppContainer

Other containers…

IFProc ADC

Mount WCA

ALMA Logging

• Human-readable description of one action at one specific time

• ~30 GB of every day log

• Container Logs:

• Coarse grained (Info)

• Local to machine

• Stored in plain text files

• XML Logs:

• Fine grained (Debug)

• System wide

• Stored in XML files

Introducing high performance distributed logging service for ACS - Avarias et all, 2010

XML Log Example

• A typical XML log entry

<Debug

TimeStamp="2014-06-07T17:14:50.658" File="WVRImpl.cpp” Line="665” Routine="void WVRImpl::publishWVRData(const Control::WVRData#, ACS::Time)” Host="pm01-abm" Process="CONTROL/PM01/cppContainer" Thread="CONTROL/PM01/WVRMonitorThread" Context="” SourceObject="CONTROL/PM01/WVR" Audience="Developer” >

<![CDATA[Sucessfully published the following WVR event. Antenna: PM01 Mid-point of data: 17:14:49.920 Temperatures: 284.069, 284.029, 284.105, 284.014 deg. K. Sampled every 1.152 seconds. Event is valid. Data monitored at: 17:14:50.496]]>

</Debug>

• Log Levels: TRACE, DEBUG, INFO, NOTICE, WARNING, ERROR, EMERGENCY, …

• 2.5 millions of XML entries in 4 hours of observatory operation.

• Valuable information hidden in Debug and below.

• Main tool: jlog

Container Log Example

• Subset of XML log.

• Bird of Eye view of operation (By default, only stores up to INFO level)

• Generated at container level as text log line.

2015-03-19T11:52:07.546 [CONTROL/DV10/FrontEnd/ACD - void ACDImpl::setCalibrationDeviceBandInternal(CalibrationDeviceMod::CalibrationDevice, ReceiverBandMod::ReceiverBand)] Setting Hot load in front of band 4Vector Size: 1Starttime: 1e4ce2e5e639700Band: 3Freq: 7.68268e+10Caldevice 22015-03-19T11:52:07.631 [CONTROL/DV10/FrontEnd - void FrontEndImpl::lockFrontEndInternal(Control::FrontEnd::SubscanInformation, bool, bool)] Receiver band was already locked at 1.537186e+11ACD Movement: Time: e602f8 To: 3 Set As: 2ACDCSV,e602f8,3,22015-03-19T11:52:15.461 [CONTROL/DV10/cppContainer-GL - void PositionStreamConsumer::processData(Control::MountStatusData)] At 11:52:15.461 got the last needed data with a timestamp of 11:52:15.360. The delay is 0.101 seconds.2015-03-19T11:52:16.253 [CONTROL/DV10/MountController - virtual void Control::MountControllerImpl::abortDataCollection()] Cleaning up past data collection that started at 11:51:54.000 and was due to stop at 11:52:15.360. Time now is 11:52:16.253

What is not an ALMA Log

• It is not observational data. It is behavior description.

• Linux messages: dmesg, /var/log/messages

• Mount circular buffers

• BL CORR internal coding

• Engineering scripts

• HVAC shutdowns

• Utility Modules

Where are the Logs?

• Container logs

• /alma/logs/{hostname}

• XML Logs

• /mnt/gas02/data1/AcsLogs-8.1/

• Don´t be afraid of the name!

• log2015-03-20T19:42:07.557_2015-03-20T20:03:30.984.xml

• Other places to search

• Computing Logs: http://computing-logs.aiv.alma.cl

• Kibana (beta): http://hypatia.osf.alma.cl:5601

Useful locations at AOS

CONTROL

• /alma/logs/gas01/CONTROL/ACC/javaContainer

• /alma/logs/gas01/CONTROL/ACC/pythonContainer

Antennae

• /alma/logs/ANT-abm/CONTROL/ANT/cppContainer

Photonic References

• /alma/logs/lo-lmc-1/CONTROL/LMC1/cppContainer

• /alma/logs/lo-lmc-2/CONTROL/LMC2/cppContainer

BL Corr

• /alma/logs/cob-cdpm-b/CORR/CDPMaster/cppContainer

• /alma/logs/cob-cc/CORR/CCC/cppContainer

ACA Corr

• /alma/logs/coj-cc-1/ACACORR/OBSERVATION_CONTROL/cppContainer

• /alma/logs/coj-cc-1/ACACORR/CMD_TRANSFER/cppContainer

• /alma/logs/coj-cpm-1/ACACORR/CDPM/cppContainer

Thank you!

The Atacama Large Millimeter/submillimeter Array (ALMA), an international astronomy facility, is a partner- ship of Europe, North America and East Asia in cooperation with the Republic of Chile. ALMA is funded in Europe by the European Organization for Astronomical Research in the Southern Hemisphere (ESO), in North America by the U.S. National Science Foundation (NSF) in cooperation with the National Research Council of Canada (NRC) and the National Science Council of Taiwan (NSC) and in East Asia by the National Institutes of Natural Sciences (NINS) of Japan in cooperation with the Academia Sinica (AS) in Taiwan. ALMA construc- tion and operations are led on behalf of Europe by ESO, on behalf of North America by the National Radio Astronomy Observatory (NRAO), which is managed by Associated Universities, Inc. (AUI) and on behalf of East Asia by the National Astronomical Observatory of Japan (NAOJ). The Joint ALMA Observatory (JAO) provides the unified leadership and management of the construction, commissioning and operation of ALMA.