ALMA quality assurance: concepts, procedures, and toolsamchavan/papers/spie2016-paper.pdfe Joint ALM...

ALMA quality assurance: concepts, procedures, and tools

A.M. Chavan∗a, S.L. Tanneb, E. Akiyamac, R. Kurowskia, S. Randalla, B. Vila Vilarob, E. Villardb

a European Southern Observatory, Karl-Schwarzschild-Str. 2, 85748 Garching, Germany; b Atacama Large Millimeter/submillimeter Array (ALMA), Alonso de Córdova 3107, Vitacura,

Santiago, Chile; c National Astronomical Observatory of Japan, 2-21-1 Osawa, Mitaka, Tokyo 181-8588, Japan

ABSTRACT

Data produced by ALMA for the community undergoes a rigorous quality assurance (QA) process, from the initial observation ("QA0") to the final science-ready data products ("QA2"), to the QA feedback given by the Principal Investigators (PIs) when they receive the data products (“QA3”). Calibration data is analyzed to measure the performance of the observatory and predict the trend of its evolution ("QA1").

The procedure develops over different steps and involves several actors across all ALMA locations; it is made possible by the support given by dedicated software tools and a complex database of science data, meta-data and operational parameters. The life-cycle of each involved entity is well-defined, and it prevents for instance that "bad" data (that is, data not meeting the minimum quality standards) is ever processed by the ALMA pipeline.

This paper describes ALMA's quality assurance concepts and procedures, including the main enabling software components.

Keywords: Radio Astronomy, Quality Assurance, Science Operations, Data-flow Software

1. INTRODUCTION The Atacama Large Millimeter/submillimeter Array (ALMA) strives to provide the community with scientific data of the highest possible quality. An extensive Quality Assurance program was devised during the construction phase and is being constantly refined to be more effective and efficient: the product QA begins as data is being taken and follows it as data is processed, packaged and distributed to the Principal Investigators (PIs) — their feedback is also taken into account and incorporated in the process.

The plan includes four distinct tiers, named QA0 to QA3.

QA0 is performed in quasi-real-time by astronomers and operators on duty as observations proceed by examining the QuickLook (QL) display and extracting selected calibration information from the generated data. It is meant to confirm that the observation was performed within the constraints set by the PI and the data can be calibrated correctly.

The QA1 tier monitors the overall system performance by executing special observations defined by the Observatory calibration plan, and collecting and analyzing the data. Long-term trends can be identified and used to perform preventive maintenance and problem avoidance. Not that unlike the other steps QA1 is not directly tied to a specific observing project.

All science observations are reduced (with the ALMA Pipeline or manually) and selected parameters are assessed to evaluate data quality -- that is what ALMA calls QA2. ALMA is committed to deliver science-ready data and this stage checks whether the data meets all scientific quality constraints; observed sensitivity and angular resolution must meet the requirements set by PI. Reduced, validated data is stored in the ALMA Science Archive where it can

∗ [email protected]

Observatory Operations: Strategies, Processes, and Systems VI, edited by Alison B. Peck, Robert L. Seaman, Chris R. Benn, Proc. of SPIE Vol. 9910, 99101H · © 2016 SPIE · CCC code: 0277-786X/16/$18 · doi: 10.1117/12.2232426

Proc. of SPIE Vol. 9910 99101H-1

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 08/23/2016 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx

be accessed breports are de

Any defects dfed back to thDedicated sofALMA QualiALMA's qual

1.1 Structure

ALMA Obseonly with a h

Figure 1.

The highest lgiven by the o

The Observinthe Pipeline rthey include Each MOUS,

QA0 is perfor

Finally, data generated by

1.2 The life-c

ALMA entitibe representeinstance, an ReadyForPro

by the PI, andelivered to the

detected by thhe Joint ALMAftware tools anity Assurance lity assurance

e of an ALMA

rving Projectsandful of elem

An example AL

level in the hieoriginal subm

ng Program inreduction andone or more

, finally, conta

rmed examini

reduction anall SB observ

cycle of ALM

es like SBs aned as state diagOUS is in th

ocessing when

d, after the pe PI as well.

he PI after dataA Office (JAOnd databases stool (AQUA)

e processes and

A Observing

s are assemblements, as show

LMA Observin

erarchy is themitted observin

ncludes one ord QA phases. e Group OUains one or mo

ing calibration

nd QA2-QA3 vations (that is

MA entities

nd OUSs follograms, every he FullyObsen all generate

roprietary per

a delivery (QAO) in Chile forsupport all QA) to many ad-hd the tools and

Project

ed hierarchicawn in the follo

ng Project (simp

e project itselfng proposal; in

r more hierarcThe top-leveSs (GOUSs),ore Scheduling

n data related

are performes, all contained

ow a well-defstate transitiorved state whd raw data ha

riod expires,

A3) are collectr investigationA stages, fromhoc scripts and infrastructur

ally. The actuaowing, simplif

plified view)

f, identified byn this case Rev

chies of ObsUl OUSs corre, in turn cong Blocks (SBs

to an observat

ed at the OUd EBs).

fined life-cyclen being the rehen all its coas been transf

the entire com

ted by the ALn and resolutio

m the QuickLod displays. In re elements su

al structure is fied example.

y a code like 2vealing the ch

UnitSets (OUSspond to the

ntaining one os), the smalles

tion; that is, to

S level, proc

e, from creatioesult of an opeontained SBs ferred to the A

mmunity. The

MA Regionalon. ook display tothe following

upporting them

rather comple

2012.B.00033hemical evolut

Ss), playing anproposal’s Scor more Memst schedulable

o an ExecBloc

essing and ev

on to their eneration perforhave been o

ALMA Archi

e QA0, QA1

l Centers (AR

o the Pipeline tg sections we dm.

ex but QA is c

3.S; the projection...

n important rocience Goals mber OUSs (

unit in the hie

ck (EB).

valuating the

d state. Life-crmed on that eobserved, and ive in Santiag

and QA2

RCs) and

to the detail

concerned

ct’s title is

ole during (SOUSs); MOUSs). erarchy.

raw data

cycles can entity. For

becomes go. If data



was taken iReadyForPLC

Fig. 2. Lif

1.3 Database

A complex nstorage facilitechnologiesdatabases (Ca

QA0 targets aphase fluctuaReceiver andALMA Calib

The QA0 prowhich is thenparameters ufollows:

• Chec

• Chec

• Chec

• Chec

• ChecWV

• A suexec

in a modeCalibration st

fe-cycle of Obs

es

network of daities is beyoninclude relati

assandra, see

atmospheric cations and totad Correlator pbration Device

ocedure is carn stored in theusing the AQU

ck of the integ

ck of the poin

ck of the scien

ck of the Atm

ck of the PhaR)

ummary pagecution fraction

that’s currentate, otherwise

sUnitSets (partia

atabases suppond the scopeional databasewww.planetca

conditions likeal power leveparameters li

e output, delay

rried out by te Archive: boUA Web, vali

grity of the EB

nting and focu

nce target: pos

mosphere quali

ase quality (Se

e is producedn and any warn

ntly supportee to ReadyFor

al view)

ort ALMA ope of this papes, XML storssandra.org/w

2. QUALe weather parals; Antenna isike bandpass,y and output s

the Astronometh binary dataidating the EB

B: metadata, a

s of the antenn

sition, mosaic

ity: PWV, sys

ee Fig. XX):

d, displayingnings about th

d by the PrManualCalib

perations andper; we’ll onage, file repowhat-is-apach

LITY ASSUameters, WVRssues like pos, image rejectignal level.

er on Duty (Aa and metadatB or flagging

all calibration

nas

cing, time inte

tem and recei

one importan

information he execution

ipeline thebration, and so

QA in particnly mention hositories (NGAhe-cassandra)

URANCE 0R measuremensition, pointingtion ratio (sid

AoD). Each eta in XML fog it as “QA0

intent (Bandp

egration

ver temperatu

nt criterion is

about EB c

OUS will tho on.

cular. A comphere that theAS, see [2]) an).

0 nts, system temg, focus, targedeband ratio),

execution of aormat. The Ao

Fail”. The pr

pass, Flux, Pha

ure, according

the phase RM

ontent, estim

hen transition

plete descriptie range of pend so-called “

mperature, skyet tracking; an, receiver tem

as SB produceoD checks a nrocess can be

ase)

to fixed crite

MS (after corr

mated vs. exp

n to the

ion of theersistence“NoSQL”

y opacity,nd finally

mperature,

es an EB, number of e listed as

ria

rection of

ected EB

• Finaally, a QA0 Flag is set to eitther Pass, Faiil, or SemiPasss



Fig. 3. AQpolar

Some of thetool (see belo

2.1 QuickLo

QuickLook isreal-time, tha(TelCal) and

Figure 4.

A ScanView scan complet

QUA plot of therization

parameters foow). AQUA us

ok

s an applicatioat is, as an opresents it in

Some of the inf

pane is refrested. The pane

e phase RMS fo

or the QA0 arses as well ex

on running on observation pr

easy-to-read g

formation prese

shed whenevepresents scan

or a Phase Calib

re in fact chexternal tools (e

the operator’sroceeds. It prgraphical form

ented by QuickL

er a ScanReduns in the conte

brator in one of

cked in real-te.g. aos-check

s console and ocesses data

mat.

Look

ucedEvent isext of their EB

f the basebands;

time during thk.py) called via

dedicated to generated by

received, indiB, and allows

; an outlier is cl

he observationa a Web servic

show calibratiy the Telescop

icating that athe user to di

learly visible in

n, using the Qce infrastructu

ion informatiope Calibration

reduction of isplay plots re

n YY

Quicklook ure.

on in near n module

the latest elated to a



single scan orreceived.

A filter allow

2.2 Scripts

A number of check the consoftware engfrom rapidly

Scripts like A

For instance, phase qualityof the final Q

=== Usab

Phas PWV: Phas Flux Numb Band High QA0 2.3 AQUA, Q

The ALMA Qoperational inadded and wFeatures relat

The left side shown, most

Figure 5.

Selecting an pane. A summto be interrupobserved freqobserved outsindicated.

r a complete E

ws to select res

f ad-hoc scriptnsistency and ineers) and suchanging exte

AOS Check and

AOS Check isy after correctiQA0 status; forQA0 summary f

ble antennas: se rms (Anten 0.81 mm N

se cal: 0522-x: 1353.06 Jber of cycles d observed: 3 hest recommenPASS (no sig

QA0 tab

Quality Assurn Cycle 3 (Ocwill integrateted to the diffe

of the QA0 recent first, pr

The list of EBs

EB in the listmary executiopted, what thquency and whside the corre

EB (that is, al

strict the plots

ts have been wcorrectness o

ubject to chanernal software

d the metadata

s a Python scrion of the WVr instance for id___A00218

na,phaseCal)o online WVR-364 y of science/p

ded: 8-8 nificant prob

rance tool (Act 2015) allowe with the iferent QA tiers

tab lists EBsroducing a kin

s in the QA0 tab

t leads to dispon pane gives e weather wahether any PI-ect window, 1

ll scans of tha

to data specif

written in the of the data itsenge as circume, the scripts ru

a checker typ

ript analyzingVR and the sy

2_Xb21386_X19

: 0.0 deg (=0-corrected da

phaseCal: 99

blems)

AQUA) is a wing a centraliz

imaging Pipes are grouped

s (that is, SB nd of “to-do”

b of AQUA

playing a largeall key inform

as like at the -defined timin1.07 days off”

at EB). EB plo

fic to individu

course of timelf. They are t

mstances requiun in as Web

ically process

g the quality ofystem tempera

95 === 0.1um) ata available

Web-based azed entry poineline when iin separate ta

observations)list:

e informationmation, includi

time (water vng constraints ”). The time s

ots are updated

ual antennas o

me to extract atypically writtire it. To isolservices in a d

s the metadata

f the data prodature, and pro

e: assuming c

application sunt for QA0 infit becomes abs.

); by default,

n set about thaing whether thvapor, wind swere violatedpent on all sc

d automatical

r baselines.

and analyze Aten by staff sclate ALMA’s dedicated envi

a generated for

duced by eachoduces a list o

orrection fac

upporting QAformation. Suavailable in

only those w

at EB being shhe observationspeed, humidid (for instancecience and ca

ly when a new

Archive informcientists (as op“official” ap

ironment.

r an EB.

h antenna. It cof flags and an

ctor of 1.413

A activities. Iupport for QA2

Cycle 4 (Oc

with no QA0

hown in the rn was successity, atm. pres

e, “Time criticalibrator sourc

w event is

mation, or pposed to plications

checks the n estimate

378396845

It became 2 is being ct 2016).

status are

right-hand ful or had sure), the

cal project ces is also



Calibration inPointing calib

Figure 6.

Figure 7. (cloc

That extensivscores to an analysis but dless of what w

nformation isbration; Sourc

Antenna and B

More AQUA Qckwise from top

ve informationobservation: Pdo not reach twas required,

shown in grce (for mosaic

aseline coverag

QA0 calibrationp)

n set includesPass, Fail anthe minimum less than 50%

raphical form s) and Baselin

ge plots in the Q

n plots: Pointing

s all the QA and SemiPass.

requested to % of the requir

in right-handne coverage ar

QA0 tab of AQU

g Offsets, Sourc

astronomer neThe latter appbe processed

red antennas w

d pane tabs dre available as

UA

ce Coverage, Ph

eeds to knowplies to EBs wfor QA2; for

were available

dedicated to As well.

hase and Atmos

w to assign onwhose data cr instance, obs, etc.

Atmosphere, P

sphere calibrati

e of the foresan be used foserving time i

Phase and

on

seen QA0 or science is 20% or



A simple popthe output of

QA1 targets efficiency, su

QA1 tackles measured by measured asperformance hoc software Telescope Castronomers; section than c

The main QAdown by arraprogrammaticThe most crit

QA1 checks ALMA staffNote that all areas of the observation e

3.1 AOS Mo

AOS Monitorobservations status of antperformance

At the momechecking the helping in the

Fig. 8.1. E

p-up window some of the s

Array correurface roughne

array performthe AoD by e

s “Observatordeterioration packages (Pyalibration andtheir output

can be read by

A1 parametersay (12m and 7mcally. tical paramete

are performef. The resultQA0 and QAALMA syst

efficiency, ima

nitor

r is a softwarmade at the Aennas, and hof the ALMA

ent, the monitmodels in the

e issues associ

Example of poin

allows the opcripts describ

ection (baseliness or deforma

mance paramexecuting “stary Tasks” atduring operat

ython scripts, Td/or CASA twill include

y AQUA for fu

s are stored inm). The datab

ers will also be

ed following ts of the an

A1 data is protem (pointingaging quality,

re tool that chALMA Opera

helps in the dA array every d

tor is limited e TMCDB andiated with tiltm

nting analysis w

perator to enteed above.

3. QUALne and delayation); and mo

meters that chandard” SBs ct predefined tions — measTpoint, etc). Ithanks to a jall the inform

further process

n the Telescopbase can be mo

e displayed on

a predefined nalysis are reocessed by Scg, focus, antecalibrators, et

haracterises thations Site (Adetection and day.

to the analysd proposing nmeters, delays

with AOS Moni

er the final sco

LITY ASSUy); Antenna conitoring of C

hange slowly, created as per

periods durisuring those pIn the near futjoint effort omation relevasing.

pe Configuratodified directl

n the main ope

schedule as eported to thience Operatienna positiontc)

he performancAOS), where t

solving of th

sis of the poinew models. Ins, antenna mot

itor

ore and comm

URANCE 1calibration (p

Calibrator flux

over timescathe Calibratioing the monparameters is ture most of thof the AoDs ant to the spe

tion and Monly through a d

erator’s conso

part of scienhe PI as a ions to performn, delay, wea

ce of all ALMthe array is lohe problems.

nting and focun the future, thtions, primary

ment; the latter

1 pointing, beamx (solar system

ales longer thon Plan. QA1 nth, or when

currently perhose packagesand the Dat

ecific measure

nitoring Databdedicated GUI

ole.

ce operationssummary of

m trending anather conditio

MA antennas ocated. This f

The goal is

us calibrationhis tool will iny surfaces and

r is pre-initial

m pattern, mm objects and q

han a week. parameters ar

n detecting sformed by rus will integratta Managemeement and a

base (TMCDBI (TMCDB Exp

s, and analyzef the array nalysis on the

on, antenna e

every day, utfacilitates asse

enable impro

ns in all ALMnclude new ca

d aperture effic

lized with

ain beam quasars).

They are re usually

significant unning ad-ted within nt Group summary

B), broken plorer) or

ed by the behavior.

e different efficiency,

tilising all essing the oving the

MA bands, apabilities ciency.



Fig. 8.2. Example of pointing analysis with AOS Monitor

4. QUALITY ASSURANCE 2 QA2 constitutes the final quality assurance step before the data are released to the PI. It is performed on the fully calibrated and imaged data products available either from the ALMA pipeline or manual processing. In Cycle 3, some 75% of ALMA data are calibrated by the pipeline at JAO and subsequently imaged manually at the ARCs, while the remaining 25% are processed completely manually. In the future, the pipeline is expected to fully reduce at least 80% of the ALMA QA0 Pass data.

Many aspects of QA2 were covered in [1].

While QA0 pertains to the EB level and is designed to check that the data are useful and calibratable, QA2 is performed at the OUS level and assures the data products meet the specifications requested by the PI in their proposal. The checks currently focus on the sensitivity achieved for a given frequency and bandwidth as well as the angular resolution (AR) of the image. The recoverable largest angular scale (LAS) requested by the PI is not specifically checked during QA2, but accommodated from the outset by adding MOUSes containing SBs scheduled for more compact configurations in the same GOUS if necessary. In Cycle 3, the QA2 sensitivity and AR checks are applied only to the MOUS observed with the most extended configuration necessary, while the SBs coming from complimentary more compact arrays (including the Atacama Compact Array, also known as the Morita Array) receive only basic checks such as that the expected time on source has been reached. Once the procedure for making and assessing combined images has been finalised it is expected that a final QA2 evaluation will be carried out at the GOUS as well as the MOUS level.

QA2 targets calibration measures like WVR, Tsys and antenna position, data flagging (atmospheric absorption, bad visibility), bandpass and flux calibration, phase stability; imaging part is reconstructing the image from the calibrated visibilities and measures like deconvolution by CLEAN method, equalization and concatenation among EBs, setting channel ranges for science targets, and image and grid size; and finally products (images and scripts).

QA2 is performed ALMA’s Regional support Centers by ARCs in EA, EU, and NA, as well as in Chile (pipeline reduction).



4.1 WebLog

The WebLogALMA delivrich hierarchifor exploring step (e.g. calithat step. Themany places on multiple hscoring to givthumbnails ar

Figure 9.

4.2 AQUA, Q

Support for Q4. The paradiselecting one ALMA pipelibandpass cali

Once the imaachieved senrequested valand more rou

ALMA recomis called QA3

g is a set of Heries. It can bical repository the pipeline pibration or flaese plots can histograms of

histograms to ve an “at a gre used to prov

Some of the inf

QA2 tab

QA2 is currentigm will be thwill display a

ine, includingibrations, and

aging pipelinensitivity, AR alues. The planutine.

mmends that P3. Any proble

HTML pages be viewed usiny. The WebLoprocessing in agging); users

easily be filtef appropriate select the plo

glance” indicavide an intera

formation prese

tly being builte same as QAall relevant de

g a set of QA sso on. For det

e is fully comand LAS will

n is for the QA

PIs perform a ems that they

giving a sumng most standog aims to prodetail. Most pcan then dril

ered by a commetrics are diots associatedation of any tractive environm

ented by the We

t into AQUA,A0: QA staff wetailed informascores related tailed info, dir

mmissioned, tl also be acce

A2 assessment

5. QUALpost-reductionfind to is repo

mmary of how dard Web browovide a quick pages give an l down to a m

mbination of oisplayed, in m

d with those drouble points.ment allowing

ebLog

with the goalwill be presentation. As indito the Pipelinrect access to

he most relevessible from wt to become in

LITY ASSUn evaluation oorted to Cont

the calibratiowsers, providioverview of tinitial single

more detailed outlier, antenn

modern web bdata points. A. Throughout g intuitive data

l of integratinted with a “to-cated above, Q

ne calibration the Weblog is

vant images awithin the AQncreasingly au

URANCE 3of the data proact Scientist v

on proceeded,ing a portablethe datasets a“representativview of all thna and spectr

browsers it is pAll steps also

the WebLog,a and processi

g that activity-do” list of OUQA2 informatstages: data fls also availabl

and imaging pQUA interfacutomated as op

3 oducts delivervia the ALMA

, and is include familiar inteas well as giveve” view of a he plots associral window crpossible to drundergo an a

, links and ening exploratio

y in the courseUSs awaiting tion is generatlagging, flux, le.

parameters suce together wiperations beco

red to them; thA Helpdesk. (

ded in all erface to a e methods particular iated with riteria. In raw boxes automated nlargeable on.

e of Cycle QA2, and ted by the TSys and

uch as the ith the PI ome more

he process (The QA3



process may be also triggered by contact scientists themselves or other ARC personnel.) Problems with the data products may reflect an underlying problem with the original data, observing procedure or calibration.

Problem resolution begins with retrieval of the data from the archive to evaluate the nature of the problem, including an assessment of whether the problem applies to a specific dataset or data taken under similar set-ups and conditions is also affected. Depending on the nature and gravity of the problem the ARC may request involvement of the Observatory; ultimately, it may require repeating the observation(s).

QA3 is currently a very ad-hoc activity and no dedicated software tools were developed to support it.

6. CONCLUSIONS ALMA’s success with the community is due in large part to the quality of the data it delivers. Ensuring that raw and reduced data maintain ALMA’s rigorous quality standards requires the application of focused policies and instruments at all stages of data generation, processing and distribution. Over the years, ALMA Science Operations have worked in close collaboration with the Engineering and Computing teams to develop and refine those policies and the procedures to implement them, coupled with the software tools and computing infrastructure required to support the entire process.

QA at ALMA is a constantly evolving and still manpower-intensive process; the goal of supporting a wider community by providing science-ready data imposes an additional burden to the staff. The efficiency and productivity of the QA team needs to increase and it’s important that processes get automated and tools take over the most repetitive tasks. While quality assessment is ultimately a human responsibility, the process of forming an opinion about a data set or product can be facilitated greatly by providing scientists with a well-defined set of summary information as well as software tools to distribute work and collect the results.

The challenge is to maintain ALMA’s level of excellence while providing for a sustainable workload and manpower budget.

7. ACKNOWLEDGEMENTS The authors would like to thank Christian Lopez for his contributions to this paper.

REFERENCES

1. Petry, D, Vila-Vilaro, B., Villard, E., Komugi, S., and Schnee, S., "ALMA service data analysis and level 2 quality assurance with CASA", SPIE 9152, Software and Cyberinfrastructure for Astronomy III, 91520J (July 18, 2014); doi:10.1117/12.2054715. 2. J. Knudstrup, A. Wicenec, and S. Johnston, 2002, NG/AMS: The Next Generation Archive Management System. in ASP Conf. Ser. 281, ADASS XI, ed. D. A. Bohlender, D. Durand, & T. H. Handley (San Francisco: ASP)[O-17]



ALMA quality assurance: concepts, procedures, and toolsamchavan/papers/spie2016-paper.pdfe Joint ALM...

Documents

Transcript of ALMA quality assurance: concepts, procedures, and toolsamchavan/papers/spie2016-paper.pdfe Joint ALM...