Atacama Large Millimeter/submillimeter ArrayExpanded Very Large Array

Robert C. Byrd Green Bank TelescopeVery Long Baseline Array

Data ProcessingProgress and PlanningJune 2014

Jeff KernData Processing Group

Lead

CASASummary Status• CASA-4.2.2 brings capabilities desired for cycle 2 data

reduction– Currently under test issues with plotMS, otherwise

looking good– Key deliverables:

• Weight handling (per SPW)• Flag Data Improvement (Tbuff parameter to avoid

quacking)• Strict Flagging in Applycal• Single Dish Interface Update

• Pipeline – Provisional Acceptance in March– Long punch list, mostly cosmetic. Largely addressed at

this point– Doing integration testing against CASA-4.2.2

CASAPlanning: Near Future• June 27: Prerelease of CASA-4.2.2 and Pipeline to PLWG for

testing• July 1: Release of CASA-4.2.2 (Public)

– Both OSX and Linux• ~July 1: Release of ALMA Pipeline to project

– Based on testing from PLWG• ~July 15: CASA-4.2.2-Pipeline release available to public

These dates are all very tight, and are based on not finding issues in the testing. Small slips can have large impact as we

are in vacation season

CASACASA Planning• September 2014 CASA-4.3

– Weight Spectrum– Imaging Refactor V1– Parallelization

(Calibration)

– SetJy Improvements– VO Viewer

• March 2015 CASA-4.4– Imaging Refactor V2

• Support for Pipeline– OTF Cal Application

– Pipeline Parallelization– Enhanced Phase

Connection

CASAPipeline Planning• From ObsMode Meeting

– Support for Cycle 2 Operations• This could stop all other progress!

– Data Flagging Heuristics– Bandwidth Switching Heuristics

• First of low SNR Heuristics– Science Target Imaging

• 1 Cube per Science SPW per Science Target

• In parallel:– Automated trigger and cluster exectuion– This is NOT a magic bullet.

5

CASAWeblog Example

Pipeline Regression Server

6

CASAPLWG: User Test

• Pipeline Working Group will perform a user test to receive feedback on Weblog– Format TBD either special run and request comments or

just survey of Cycle 2 Pis

• Will need to prioritize feedback from this vs. functional testing

7

CASAPipeline Single Dish

• Well developed and tested– Not much knowledge outside of NAOJ so far

• Hoped to be doing parallel testing at this point– Need lots of data (Project wide issue)

• Concerned about connection between manual reduction, script generator, and pipeline

ICT CPM-3 June 17-19 Socorro New Mexico 8

CASA

Prelim

inar

y

Pipeline Releases• November Update

– Bandwidth Switching– Data Flagging– Other Cycle 2 Experience (TBD)

• March 2015– Imaging capabilities V1– Long Baseline capabilities– Low SNR (?)

• Q3 2015– Continuum Subtraction– Self Cal– Other Calibration

9

Infrastructure

&Automation

CASASteps to Pipeline Use in Operations

First Pipeline-assisted deliveries

Jun

e/J

uly

2014

Cycle 2 Phase C testing

Test commissioned observing modes for Cycle 2 data

Test commissioned observing modes for Cycle 2 data

Pipeline documentation review

Send final drafts broad review within Observatory

Submit final drafts to SciOpsIPT and DRM WG for review

PWG full acceptance recommendation Submit report to SciOpsIPT

Submit full acceptance report to Observatory

PWG full acceptance testing

Test full acceptance criteria

Test full acceptance criteria implementation

DRM WG end-to-end testing

Test e-2-e QA2 process, packaging & documentation

Test e-2-e QA2 process, packaging & QA2 documentation

Pipeline-assisted QA2 in operations

For commissioned observing modes, data calibrated by Pipeline and imaged manually

Pipeline released to users

CASA 4.2.2 tarball including Pipeline

CASAPipeline Issues• Staffing:

– Pipeline has exactly 1 full time developer– Currently stepdown another 0.5 FTE at end of year

• Formally 3.0 -> 2.5 but realistically 1.8 -> 1.3 on interferometric

– “Borrowing” expertise from CASA but not the same as dedicated staff

• Maintenance / Support– Tracking down errors for failed SBs could halt progress on new

targets– Need robust QA0 process at observatory

• Balance between Cosmetic and Functional requirements– Weblog has grown very large, suspect it may be overspecified at

this point

11

CASA

www.nrao.eduscience.nrao.edu

The National Radio Astronomy Observatory is a facility of the National Science Foundation

operated under cooperative agreement by Associated Universities, Inc.

CASACASA and CASA Based Pipelines

Pipeline Weblog: Home Page

CASACASA and CASA Based Pipelines

Pipeline Weblog: Observation Summary

CASACASA and CASA Based Pipelines

Pipeline Weblog: Topics Page

CASACASA and CASA Based Pipelines

Pipeline Weblog: Task Summary

CASACASA and CASA Based Pipelines

Pipeline Weblog: Task Detail

CASACASA and CASA Based Pipelines

Pipeline Weblog: Plot Details

CASACASA and CASA Based Pipelines

Pipeline Weblog: Detail Zoom

CASACASA and CASA Based Pipelines

Pipeline Weblog: Imaging Summary

CASACASA and CASA Based Pipelines

Pipeline Weblog: Imaging Details