The Web 2.0 and the NOAO NVO Portal Christopher J. Miller Data Products Program CTIO/NOAO.
Sep 4, 2008NVOSS 20081 VO From the Command-Line Mike Fitzpatrick NOAO.
-
Upload
lillian-willis -
Category
Documents
-
view
215 -
download
0
Transcript of Sep 4, 2008NVOSS 20081 VO From the Command-Line Mike Fitzpatrick NOAO.
Sep 4, 2008NVOSS 2008 1
VO From the Command-Line
Mike FitzpatrickNOAO
Sep 4, 2008NVOSS 2008 2
Introduction
• The Portals discussed yesterday show typical VO-access modes using the web.
• However, this isn’t easily scriptable (same is true for GUI applications).– Saving results is done manually– Data not easily saved for analysis in other
environments
• What we need are general VO client applications we can script
Sep 4, 2008NVOSS 2008 3
VOClient Overview
• Hides details of VO implementation from applications
• Provides: – Hi-level, easy-to-use interface between
applications and the distributed VO framework,– Client-side VO interfaces returning higher-level
structures (like tables) rather than a service API,– Equivalent functionality in all supported
languages
• Caches entire result until queried by app
Sep 4, 2008NVOSS 2008 4
Architecture
• Modeled after database server• VOClient Daemon
– Multi-threaded– Java access to VO svcs– Layered on DAL, Registry and services
• API– C-based, custom and SWIG bindings– Communicates w/ voclientd using proprietary RPC
protocol
• Connection Oriented– Allows voclientd to maintain state on behalf of client
PythonIRAF,
IDL, etc. Java
VOCLIENTD
dalclientregistry,
etc.
Other…
Sep 4, 2008NVOSS 2008 5
API Example
voc_initVOClient (config_opts)
voc_closeVOClient (shutdown_flag)
query = voc_regQuery (term, orValues) voc_regAddSearchTerm (query, term, orValue) voc_regRemoveSearchTerm (query, term)
count = voc_regGetSTCount (query) str = voc_regGetQueryString (query)
res = voc_regExecute (query) str = voc_regExecuteRaw (query)
Sep 4, 2008NVOSS 2008 6
From API to Apps
VOSESAME
Name resolver using Sesame service (Simbad/NED)Configurable output, multiple inputs
VOREGISTRYVO Registry search and resource resolution
VODATAUses capabilities of both of the aboveVO catalog and image access (spectra in progress)
Sep 4, 2008NVOSS 2008 7
VO-CLI in NVOSS
• Software in NVOSS_HOME$vo-cli directory•To build:
% cd $NVOSS_HOME/vo-cli% make install
This installs binaries in the package bin directory, to use these tools you must install to some directory in your path like the $NVOSS_HOME/bin
% cp bin/* $NVOSS_HOME/bin % cp voclientd $NVOSS_HOME/bin % cp -r voclient.jars $NVOSS_HOME/bin
Sep 4, 2008NVOSS 2008 8
VOSesame: Name Resolution
Usage:
vosesame [<flags>] [<objfile> | [<target> ...] ] -a print all information about the object -d print position in decimal degrees (default) -e print position errors -h print help summary -n print object name -t print object type -s print position as sexagesimal coordinates -v print verbose warning output -i invert search to print unresolved objects -f force Sesame svc invocation and ignore cached values -o specify output file (default: stdout) -A output an ASCII table of results (default) -C output a CSV table of results -T output a TSV table of results -H output a table heading
Sep 4, 2008NVOSS 2008 9
VOSesame: Examples
1) Print the coordinates of NGC4456 decimal degrees
% vosesame ngc4456 186.960000 -30.120000
2) Print the sexagesimal coordinates of multiple objects include the type:
% vosesame -st m31 00:42:44.31 +41:16:09.4 LINER
3) Print the decimal coordinates of those same objects listed in the file 'myobjs.txt', output as CSV, include a header, and print the id, coords, type:
% vosesame -CHndt myobjs.txt #Name,DRA,DDEC,Type, m31,10.684625,41.269278,LINER m51,202.468208,47.194667,Seyfert_2 m99,184.706333,14.416778,HII_G : : : :
Sep 4, 2008NVOSS 2008 10
VOSesame: Examples
• Calculator Mode:% vosesame -nd
m98 m98 183.45154 14.89944
[EOF]
• Print the sexagesimal and decimal values for multiple user coords:
% vosesame -s -c 12:30:0.0 -45:00:0.0 -c 187.5 2.05
12:30:00.0 -45:00:00.0 12:30:00.0 02:03:00.0
Sep 4, 2008NVOSS 2008 11
VORegistry: Registry Query/Resolve
• Command-line query of the VO Registry• Two Major Modes:
– Search (typical keyword search)– Resolve (ShortName/Identifier to something
else)
• List full resource record• List table metadata• Count matching records• Arbitrary ADQL searches• Constrain searches
– Bandpass, Service Type or ContentLevel
Sep 4, 2008NVOSS 2008 12
VORegistry: Examples
1) Find all services with radio data of Abell clusters.
% voregistry -b radio abell
2) Find all resources that mention Keck, how many are image services?
% voregistry -count keck% voregistry -t image -count
keck
Sep 4, 2008NVOSS 2008 13
VOData: Data Access
• Multi-threaded queries (hundreds/min)• Simple names for resources (e.g. ‘2mass-
xsc’ and not “ivo://irsa.ipac/2MASS-XSC”)• Various output formats
(XML/ASCII/CSV/KML…)• Not complete in terms of what VOClient API
provides• Callable from variety of scripting
languages• Interacts with NVO Portal• Targeted for power users
Sep 4, 2008NVOSS 2008 14
VOData: Usage
vodata [<flags>] [ <resource> [[ <objname> [ <sr> ]]] ]
vodata [<flags>] [ <resource> [[ <ra> <dec> [ <sr> ]]] ]
vodata [<flags>] [ <url> ]
Sep 4, 2008NVOSS 2008 15
VO-CLI
• For all tasks, a ‘-h’ flag will print help• The voclientd will be started
automatically• Web interface allows you to experiment
with options. Note the command being executed is printed.
• So, what can we actually do with this??
Sep 4, 2008NVOSS 2008 16
Scavenger Hunt
1) How many image services currently provide data for the GOODS survey?
Sep 4, 2008NVOSS 2008 17
Scavenger Hunt
1) How many image services currently provide data for the GOODS survey?
Solution:
% voregistry -t image goods% voregistry -count -t image goods
Sep 4, 2008NVOSS 2008 18
Scavenger Hunt
2) Where can you get an HST 4350A image of the center of the HDF?
Sep 4, 2008NVOSS 2008 19
Scavenger Hunt
2) Where can you get an HST 4350A image of the center of the HDF?
Solution:% vodata -O hdf -eu HST.GOODS.Cutout \
"Hubble Deep Field" 0.01
• the file 'hdf.urls' will contain the acrefs, last image is the F435
• possible to use the "-get" flag to simply download them all, or elsepass the hdf.urls in to download
Sep 4, 2008NVOSS 2008 20
Scavenger Hunt
3) How many Bautz-Morgan type II Abell clusters are within 20 degrees of the south pole?
Sep 4, 2008NVOSS 2008 21
Scavenger Hunt
3) How many Bautz-Morgan type II Abell clusters are within 20 degrees of the south pole?
Solution:
% vodata -V -O - ivo://nasa.heasarc/abell 0.0 -90.0 20 | \
stilts tpipe ifmt=votable in=- \ cmd='select "bmtype == \"II\""'
omode="count"
Sep 4, 2008NVOSS 2008 22
Scavenger Hunt
4) How many of them are in the ROSAT FSC (within 10')
Sep 4, 2008NVOSS 2008 23
Scavenger Hunt
4) How many of them are in the ROSAT FSC (within 10')Solution:Regenerate the data query, but save results: % vodata -V -O - ivo://nasa.heasarc/abell 0.0 -90.0 20 | \
stilts tpipe ifmt=votable in=- \ cmd='select "bmtype == \"II\""' \ ofmt=votable out=bmII.xml
Submit to Inventory, looking for ROSAT catalogs:% vodata -I -rm 10 any bmII.xml | grep ROSAT
total of only 4 clusters match. Query the RASS FSC for the data:% vodata -O - -A -rm 10 RASS/FSC bmII.xml
We see that one of the clusters (A3628) matches to 5 distinct X-ray sources, but in this tool we didn't crossmatch to the Abell number. (This could be done easily using STILTS, however)
Sep 4, 2008NVOSS 2008 24
Scavenger Hunt
5) How many chromospherically active binaries are EUV sources
… seen by the ROSAT WFC? …seen by EUVE?
Sep 4, 2008NVOSS 2008 25
Scavenger Hunt
5) How many chromospherically active binaries are EUV sources? … seen by the ROSAT WFC? …seen by EUVE?
Solution:Do a registry query to find the catalog and download it
as an all-sky table: % voregistry chromospherically active binary
% vodata -V -O sources ChrAcBin 0.0 0.0 180.0
Send it to Inventory and grep for results:% vodata -I -rs 10 any source.xml | egrep -i "rosat|euv"
Second EUVE catalog shows 30 matches, WFC 2RE shows 9 matches
Sep 4, 2008NVOSS 2008 26
Scavenger Hunt
6) How many stars with known exoplanets have been observed by ST?
Solution:
- Try to find a catalog in the Registry:% voregistry -R exoplanet
- Access the entire table and send it to Inventory: % vodata -O planets -V J/ApJ/646/505 0.0 0.0
180.0 % vodata -I -rm 1 any planets_table2.xml | grep
HST
Sep 4, 2008NVOSS 2008 27
Scavenger Hunt
7) Write a script that will show you the rough number of HST, Chandra and Spitzer observations at a given location.
Sep 4, 2008NVOSS 2008 28
Scavenger Hunt
7) Write a script that will show you the rough number of HST, Chandra and Spitzer observations at a given location.
Solution 1:
vodata -n hst $1 $2 .02 2>/dev/null | grep hst vodata -n spitzer $1 $2 .2 2>/dev/null | grep spitzer vodata -n chandra $1 $2 .2 2>/dev/null | grep chandra
When given 3c273 as the object this gives: hst 238 C Hubble Space Telescope spitzer 23 C Spitzer Space Telescope Observation Log chandra 35 I Chandra X-Ray Observatory Data Archive
Sep 4, 2008NVOSS 2008 29
Scavenger Hunt
7) Write a script that will show you the rough number of HST, Chandra and Spitzer observations at a given location.
Solution 2:
#!/bin/csh -fvodata -n hst,chandra,spitzer $1 $2 $3 |& grep -v -e
"#"
Sep 4, 2008NVOSS 2008 30