NHSC/PACS(Web(Tutorials Running(PACS(Photometer(pipelines...

NHSC PACS Web Tutorial

http://nhsc.ipac.caltech.edu/helpdesk/index.php

NHSC/PACS Web TutorialsRunning PACS Photometer pipelines

[PACS-‐401]

Level 2.5 Map-‐Making with MADmap





IntroducMon

• This tutorial provides will walk you through Level 1 to 2.5 processing using the MADmap branch of the PACS photometer pipeline.

• The tutorial follows the ipipe script: L25_scanMapMadMap.py

• At the end of the tutorial, you will have created a PACS map from calibrated bolometer readouts at Level 1 using the opMmal map-‐mapping algorithm MADmap.





What is MADmap?

• MADmap paper: Cantalupo et al. 2010, ApJS, 187, 212.

• It implements a least square soluMon of the projecMon inversion problem.

• It makes use of noise properMes to separate sky signal from 1/f noise.– It assumes the covariance matrix of the noise is diagonal in the Fourier

space.

– Actual noise does not really comply with this, pre-‐processing will be required.





Pipeline secMon covered here

Level 0 container

PACS data, House Keeping

PoinMng

Assign poinMng

Masking: bad/dead pixels, glitches, saturaMon

Data Reorganiza<onLogical blocksMme stampsUnit conversion

Chopper ProcessingAssign offsets on skyFlag “moving” frames

Distor<on correcMon

Level 0.5Minimally processed uncalibrated data cubes

IniMal Processing

Chopped AOTs

Combine & Subtract ON-‐OFF beams

Correct Flat Field

Flux Calibrate

Mapping AOTs

High Pass Filter

Correct Flat Field

Calibrate

Project on sky

Subtract offsets

Correct Flat Field

Calibrate

OpMmal Map Making Algorithm

Combine individual beams

(a) (b)

Level 1 cube

Level 2 images

Level 1 cubes

Level 2 images





DocumentaMon Reference

• PACS data reducMon guide, chapter 7



Processing Overview

Step 1Check script and so2ware pre-‐requisites

Step 2Loading ipipe script L25_scanMapMadMap.py

Step 3Pre-‐amble and script parameters

Step 4IdenFfy the data to process

Step 5Making sense of the main processing loop





Processing (Cont.)

Step 6MADmap pre-‐processing (post Level 1)

Step 7Remove correlated signal dri2s

Step 8Re-‐visit Deglitching

Step 9Select which frames to use for mapping

Step 10 Create MADmap ToD product





Processing (Cont.)

Step 11Create the “naive” and opFmal maps





Step 1Check your software version





Check # 1: HIPE 6.0 build ....

From the “Help” menu, select “About”

Which shows the HIPE version

If your Build number does not start with 6.0, stop and upgrade.





Step 2Load ipipe script

“L25_scanMapMadMap.py”





Step 2: Load ipipe script

From the pipeline menu, make the selections as shown to get to “L25_scanMapMadMap”

If you successfully loaded the script, it’ll appear as a folder tab under the Editor window.





Step 2: Warnings

• You should always save the template ipipe script under a new name before making changes to prevent accidentally overwriMng and destroying the original templates.





Step 3Pre-amble and script parameters





The preambleHighlight and execute the block of import and definition statements with the single green arrow.

The ini?al lines (57 in the case of example shown) are comments and may be skipped.

The import statement define java classes and func?onal blocks to be used later.





Preamble (cont.)

Replace these lines with this one, which selects the red (long-wavelength) PACS channel.

Execute this statement by positioning the marker next to the statement and clicking on the single green arrow.

The next set of lines define the PACS bolometer channel to work on.





Processing Parameters

The next segment sets the run-time parameter values. These are discussed briefly in the next slide.





ParametersParameter Descrip<on Recommend Value

Verbose Print processing step details. True

doplot For signal drii correcMon, show the plot of best-‐fit model.

True

Outdir Where to save the output from ‘doplot’ and final maps and reduced data.

A valid directory on your system.

globalDriiModel OpMon for drii correcMon. See step 6. 1

ModelOrder Polynomial order. See step 6. 2

ignoreFirst Mask and remove this many readouts from the start of the observaMon

460

noDeglitching do not deglitch the data? False

unmaskSources Obsolete parameter. Ignore. False

useIIndLevelDeglitching Which deglitcher to use? MMT or IInd level. True

envSize Envelope size for IInd level deglitcher. 10

nsigma Glitch rejecMon criteria 30

scales Parameter for MMT deglitching 1

mmt_mode Parameter for MMT deglitching ‘mulMply’

Incr_fact Parameter for MMT deglitching. 2

unMaskRad Obsolete parameter. Ignore 20





The parameters aier the recommended edits.

Highlight and execute this stanza after editing the values.

WARNING: The directory specified in ‘outdir’ must already exist on your system.





Check # 2: Preamble okay?

Execute the following line either by typing in console or editing your script.

print verbose doplot ignoreFirst noDeglitching

The output should contain the values set in Step 3.





Step 4Identify the data (scan and cross-scan) for

processing.





Step 4

Add, highlight and execute the following lines in your script.

The scan and cross-scan OBSID pairs are identified in a jython vector. The values are long integers.

pooldir is the full path name to where the data pools are stored. It can be ignored if the data has been placed in your default local store, as defined in the HIPE properties.

The nested getObservation commands in the last line read and populate a vector of observation contexts.This identifies your data to HIPE.

To explore an observation context, try printing it!print obsList[0]





Check # 3: Data is properly loaded

Highlight and execute the following lines in your script.

If there was an error, you (likely) already got a notification. This step ensures that there are no problems with the pools themselves.The output should be the programmed values for the right ascension and declination of the object in the first OBSID.

Shows the intended pointing as found in the metadata.The metadata holds a wealth of information. Try print obsList[0].meta





Step 5Making sense of the main processing loop





The Processing Loop

A. For each observaMon in your scan and cross-‐scan pair, the following processing steps are executed:• Step 6: Post level 1, MADmap pre-‐processing.

• Step 7: Remove correlated signal drii.

• Step 8: Re-‐deglitch the Level 1 product.

• Aier the processing steps, on the first pass through the loop a super-‐frames object is created.

• On the next pass the cross-‐scan data is appended to the super-‐frames object.

• At the end of the loop, we have a frames object that contain all the observaMon data.



The Processing Loop

A. Step 6

A. Step 7

B. First pass

C. Second pass

Is this the first time through?

Next pass is not the first one.

See previous slide for explanation of A. B. C.

A. Step 8





ExecuMng the loop

Highlight the section of the code shown and execute with the single green arrow.

You are encouraged to read about the steps performed in the loop in the next few slides before executing the loop.





Step 6Post level 1 MADmap pre-processing





MADmap preprocessingExecuting the loop will automatically execute these steps for both OBSIDs.

These commands pull level 1 data out of the observation context and the associated calibration files.

Cleanup unstable frames at the start of observation.

Assign pointing to each and every pixel and readout

Apply pixel-to-pixel electronic offset correction.

Which Blue filter?

In the data distributed for the workshop, the calibration data is not present. Replace this line with:calTree = getCalTree()





Your output will likely look slightly different but you should NOT get an error message and the important “ra” and “dec” datasets exist in your “frames” object.

Issue this command in the console window

Look for “dataset”s Ra and Dec

Check # 4: PosiMon cubes exist





Check # 5: Offsets are removed

Display( frames.signal[:,:,100] )

Expected Output:With proper offset removal, the image shows a rela?vely constant signal. Note: extremely bright sources may also be observed on a single image.

Type this command

An example of improper or no pixel-‐to-‐pixel offset correc?on.





Step 7Remove Correlated Signal Drifts





PACS’ correlated signal driT.

Background

This Figure illustrates what is meant by both correlated and driT for PACS signal. The Figure shows the median value of the bolometer array as a func?on of readout index. The monotonic signal showing a decay in intensity is commonly observed in PACS’ image cubes, and is thought to be related to focal plan temperature driTs.





Background

Mi?ga?ng the signal driT.

Divide the array median in bins of N readouts. N is typically 1000 readouts.

Take the minimum value in each N readout bins.

Fit the resul?ng curve with a polynomial.





Background

If the sources are weak (i.e. do not produce significant signal in a single image) it may be sufficient to fit the median values directly. However, for strong sources, the minimum approach becomes necessary.

An observa?on with strong sources. The minimum values s?ll manage to trace the overall driT fairly accurately.





The drift correction is automatically applied to the data when the main loop is executed.

Step 7

The photGlobalDri-Correc0on module allows several op?ons for fi[ng and removing the driT. See: PDRG chapter 7 Or type Print photGlobalDriTCorrec?onIn the HIPE window.





The most important parameters

model=1This is the default and uses the minimum of the bins as discussed above.

polyOrder=3Sets the order of the fiZed polynomial

binSize=1000Sets the size of the bin from which the minimum value is determined.



Step 8Re-visit Deglitching





Re-‐Deglitch

Why redo the glitch removal?

By default, the wavelet based “MMT deglitcher” is used by the standard pipeline. One issue noted for this deglitcher has been that the cores of bright point sources, or indeed any PSF-‐like structure is incorrectly iden?fied as a glitch and (usually) re-‐interpolated by MADmap using surrounding values in the ?me-‐line. This leads to “shadow” ar?facts in the final maps.

Instead, we introduce a 2nd level deglitcher which uses sigma-‐clipping algorithms to remove glitches.





Step 8: details

Was MMTdeglitching done?

By serng noDeglitching to ‘True’, you can unmask all glitches. This parameter in Step 3.

Also idenMfied in Step 3 is whether to use 2nd level sigma-‐clipper deglitcher.

Execute the 2nd level deglitcher. The following condiMons have to be met:nframes < the limit shown. The default limit of 50000 appears here because larger data require very large amount of RAM. Before serng this to a larger value, make sure your HIPE session has adequate memory allocaMon.useIIndLevelDeglitching = True (Step 3)noDeglitching = False (Step 3)





Step 9Select which frames to use for mapping





Step 9

Select and execute one of these options.

Including the turn-‐around frames provides more data at the edges of the map; however, the coverage is also sparse and may result in non-‐op?mal solu?on with MADmap.





Check # 6: On-‐Target flag is

Type this command

PlotXY( Int1d( frames.getStatus(“OnTarget”) ) )

Expected output. Either all values of OnTarget flag are True (for op?on #1), OR

The edges of each scan legs are set to false (op?on #2)





Step 10Create MADmap ToD product





Step 10

Add the parameters shown, Select and execute this block of commands

The ToD stands for Time-‐ordered-‐Data and is the internal format used by MADmap.In fact, makeTodArray will create a binary file in your temporary area that has the rearranged PACS signal in the proper format.• The scale parameter selects the size of the output sky grid rela?ve to the ideal PACS pixel sizes. E.g. scale=0.5 for PACS blue channel will result in final pixel sampling of 1.6”/pixel.• Crota2 is the FITS keyword crota2 of the final map.





Check # 7: TesMng the ToD

Print the WCS associated with this time-ordered-data (ToD) product.

Expected output: You should see some familiar FITS keywords. You can also check to make sure that crval1 and crval2 values are consistent with your object.





Step 11Create Naive and Optimal Maps





Step 11

Documenta?on Reference:PDRG Chapter 7

Both the naive and op?mal maps are created with the same call. The last parameter is set to ‘True’ for naive map and ‘False’ for op?mal map.MADmap uses maximum likelihood and conjugate gradient solvers to find the op?mal solu?on. The parameters maxRelError and maxItera?ons control both the convergence tolerance and the number of itera?ons in finding the op?mal solu?on. See the above reference for details.

Select and execute this block of commands





Check # 8: Output map

print madmap


Expected output: The output from madmap or naivemap making is a simpleImage product class with several datasets.





Check # 9: Display the final map


Display(madmap)

Expected output: A mosiac of all images in your PACS data cube.



NHSC/PACS(Web(Tutorials Running(PACS(Photometer(pipelines...

Documents

Transcript of NHSC/PACS(Web(Tutorials Running(PACS(Photometer(pipelines...