Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.
-
Upload
bertha-tucker -
Category
Documents
-
view
219 -
download
2
Transcript of Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.
![Page 1: Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.](https://reader035.fdocuments.net/reader035/viewer/2022062722/56649f315503460f94c4bff4/html5/thumbnails/1.jpg)
Web Services for the National Virtual Observatory
Tamás Budavári Johns Hopkins University
![Page 2: Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.](https://reader035.fdocuments.net/reader035/viewer/2022062722/56649f315503460f94c4bff4/html5/thumbnails/2.jpg)
3/31/2005 Tamás Budavári (JHU) 2
Outline
Motivations and trendsHow will the VO work?The NVO (hi)storySample VO servicesWhere we are going
![Page 3: Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.](https://reader035.fdocuments.net/reader035/viewer/2022062722/56649f315503460f94c4bff4/html5/thumbnails/3.jpg)
3/31/2005 Tamás Budavári (JHU) 3
The World is Exponential
Astrophysical data is growing exponentiallyDoubling every year (Moore’s Law+): both data sizes and number of data sets
Computational resources scale the same way
Constant $$$ will keep up with the data
Main problem is the software componentCurrently components are not reusedSoftware costs are increasingly larger fractionAggregate costs are growing exponentially
![Page 4: Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.](https://reader035.fdocuments.net/reader035/viewer/2022062722/56649f315503460f94c4bff4/html5/thumbnails/4.jpg)
3/31/2005 Tamás Budavári (JHU) 4
Making Discoveries
When and where are discoveries made?Always at the edges and boundariesGoing deeper, using more colors…
Metcalfe’s lawUtility of computer networks grows as the number of possible connections: O(N2)
VO: Federation of N archivesPossibilities for new discoveries grow as O(N2)
Current sky surveys have proven thisVery early discoveries from SDSS, 2MASS, DPOSS
![Page 5: Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.](https://reader035.fdocuments.net/reader035/viewer/2022062722/56649f315503460f94c4bff4/html5/thumbnails/5.jpg)
3/31/2005 Tamás Budavári (JHU) 5
Publishing Data
Roles
Authors
Publishers
Curators
Consumers
Traditional
Scientists
Journals
Libraries
Scientists
Emerging
Collaborations
Project website
Bigger archives
Scientists
![Page 6: Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.](https://reader035.fdocuments.net/reader035/viewer/2022062722/56649f315503460f94c4bff4/html5/thumbnails/6.jpg)
3/31/2005 Tamás Budavári (JHU) 6
Changing RolesExponential growth: data never centralized
A project last 3-5 years (linear rate but new projects)Data sent upwards only at the end of the project
More responsibility on projectsBecoming Publishers and CuratorsLarger fraction of budget spent on software
More standards are neededEasier data interchange, fewer tools
More templates are neededRecycling software components
![Page 7: Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.](https://reader035.fdocuments.net/reader035/viewer/2022062722/56649f315503460f94c4bff4/html5/thumbnails/7.jpg)
3/31/2005 Tamás Budavári (JHU) 7
Emerging New Concepts
Standardizing distributed dataWeb Services, supported on all platformsCustom configure remote data dynamicallyXML: Extensible Markup LanguageSOAP: Simple Object Access ProtocolWSDL: Web Services Description Language
Standardizing distributed computingGrid ServicesCustom configure remote computing dynamicallyBuild your own remote computer (and discard)Virtual Data: new data sets on demand
![Page 8: Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.](https://reader035.fdocuments.net/reader035/viewer/2022062722/56649f315503460f94c4bff4/html5/thumbnails/8.jpg)
3/31/2005 Tamás Budavári (JHU) 8
Shielding Users
Users do not want to deal with… XML – they just want their dataConfiguring grid computing – want results
Web servicesData appears in user memory, XML is invisibleSOAP: just a remote procedure call
![Page 9: Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.](https://reader035.fdocuments.net/reader035/viewer/2022062722/56649f315503460f94c4bff4/html5/thumbnails/9.jpg)
3/31/2005 Tamás Budavári (JHU) 9
NVO: How Will It Work?Define commonly used ‘core’ servicesBuild higher level toolboxes/portals on topWe do not build ‘everything for everybody’Use the 90-10 rule:
Define the standardsand interfacesBuild the frameworkBuild the 10% of services that are used by 90%Let the users build the rest from the components
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5
# of services
# o
f u
sers
![Page 10: Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.](https://reader035.fdocuments.net/reader035/viewer/2022062722/56649f315503460f94c4bff4/html5/thumbnails/10.jpg)
3/31/2005 Tamás Budavári (JHU) 10
Atomic Services
Metadata information about resourcesWavelength: optical, ultraviolet, etc.Sky coverage: where on the sky?Translation of names to universal dictionary (UCD)
Simple search patterns on the resourcesCone SearchImage mosaicUnit conversions
Simple filtering, statistics, histogramsOn-the-fly recalibrations
![Page 11: Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.](https://reader035.fdocuments.net/reader035/viewer/2022062722/56649f315503460f94c4bff4/html5/thumbnails/11.jpg)
3/31/2005 Tamás Budavári (JHU) 11
Higher Level Services
Built on basic Core ServicesPerform more complex tasks, e.g.
Automated resource discoveryCross-identificationsPhotometric redshiftsOutlier detectionsVisualization facilities
Expectation:Build custom services in days from existingbuilding blocks (like today in IRAF or IDL)
![Page 12: Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.](https://reader035.fdocuments.net/reader035/viewer/2022062722/56649f315503460f94c4bff4/html5/thumbnails/12.jpg)
3/31/2005 Tamás Budavári (JHU) 12
Early Standards
VOTableUniversal container for tables (in XML)First VO standard (from the DTD era)
ConeSearchSimple catalog access based on locationFirst VO standard interface (http get)
Many implemented them!
![Page 13: Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.](https://reader035.fdocuments.net/reader035/viewer/2022062722/56649f315503460f94c4bff4/html5/thumbnails/13.jpg)
3/31/2005 Tamás Budavári (JHU) 13
VO Registry
Searchable (fast)Various resourcesOAI harvesting
Web site andWeb services
5,459 resources
![Page 14: Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.](https://reader035.fdocuments.net/reader035/viewer/2022062722/56649f315503460f94c4bff4/html5/thumbnails/14.jpg)
3/31/2005 Tamás Budavári (JHU) 14
Sloan Digital Sky Survey
The cosmic genome projectMap the northern sky ~7000 sq.degSophisticated science archive (SQL)Many ways to query and filter dataVarious data types: Images, spectra, catalogs
http://skyserver.sdss.org/
![Page 15: Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.](https://reader035.fdocuments.net/reader035/viewer/2022062722/56649f315503460f94c4bff4/html5/thumbnails/15.jpg)
3/31/2005 Tamás Budavári (JHU) 15Courtesy of Robert Lupton
![Page 16: Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.](https://reader035.fdocuments.net/reader035/viewer/2022062722/56649f315503460f94c4bff4/html5/thumbnails/16.jpg)
3/31/2005 Tamás Budavári (JHU) 16
SDSS Image Cutout
Build JPEG mosaicsFor given (RA, Dec)Arbitrary zoom levelsOverplot catalog dataOverplot geometryMapQuest like navigationList of objects
Web servicesJPEG returned in DIMEor byte array in XML Kudos to Maria
Nieto-Santisteban
Kudos to Maria Nieto-
Santisteban
![Page 17: Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.](https://reader035.fdocuments.net/reader035/viewer/2022062722/56649f315503460f94c4bff4/html5/thumbnails/17.jpg)
3/31/2005 Tamás Budavári (JHU) 17
Spectrum Services
Public repository500,000 spectra (2B row)
Spatial search and more
Register & submit yours
Web siteOn-the-fly plotting
Building composites
Web servicesMore options and tools
Kudos to László Dobos
Kudos to László Dobos
![Page 18: Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.](https://reader035.fdocuments.net/reader035/viewer/2022062722/56649f315503460f94c4bff4/html5/thumbnails/18.jpg)
3/31/2005 Tamás Budavári (JHU) 18
Catalog Access
ConeSearchVery limited but in placeMany resources available
Need simple queriesSQL-like queries, e.g.
SELECTSELECT <columns> FROMFROM <table> WHEREWHERE <conditions>
What else we need?
To specify area on sky• Could be very complicated
Standard interface to DBs
• XML Web Services for interop
Tools to federate archives
• Dynamically for flexibility
![Page 19: Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.](https://reader035.fdocuments.net/reader035/viewer/2022062722/56649f315503460f94c4bff4/html5/thumbnails/19.jpg)
3/31/2005 Tamás Budavári (JHU) 19
Catalog Access: New Standard
Virtual Observatory Query Language
Level 3. Semantically rich XML language (TBD)
Level 2. Portals combining astro archivese.g. SkyQuery and Open SkyQuery
Level 1. Query a single databaseAstronomical Data Query Language VOTable
![Page 20: Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.](https://reader035.fdocuments.net/reader035/viewer/2022062722/56649f315503460f94c4bff4/html5/thumbnails/20.jpg)
3/31/2005 Tamás Budavári (JHU) 20
Open SkyQuery
SkyNode interface to archives
Implements ADQL returns VOTableBasic node understands “REGION”Full node understands “XMATCH”
SkyQuery portalKnows the SkyNodes from RegistryUnderstands federated query
http://openskyquery.net/
![Page 21: Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.](https://reader035.fdocuments.net/reader035/viewer/2022062722/56649f315503460f94c4bff4/html5/thumbnails/21.jpg)
3/31/2005 Tamás Budavári (JHU) 21
Architecture
Image cutout
SkyNodeSDSS
SkyNode2MASS
SkyNodeFIRST
SkyQuery
Web App
![Page 22: Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.](https://reader035.fdocuments.net/reader035/viewer/2022062722/56649f315503460f94c4bff4/html5/thumbnails/22.jpg)
3/31/2005 Tamás Budavári (JHU) 22
SkyNode
Metadata functionsTables, Columns, FunctionsUptime, Footprint, etc…
Query functionsPerformQuery (ADQL q)
• Simple query interface
QueryCost (ADQL q)• Cost estimate for query
ExecutePlan (XPlan plan)• Recursive probabilistic join
![Page 23: Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.](https://reader035.fdocuments.net/reader035/viewer/2022062722/56649f315503460f94c4bff4/html5/thumbnails/23.jpg)
3/31/2005 Tamás Budavári (JHU) 23
Data flow
SkyQuerySkyQuery
querySkyNode 1SkyNode 1
SkyNode 2SkyNode 2
SkyNode 3SkyNode 3
http://openskyquery.net
Registry
![Page 24: Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.](https://reader035.fdocuments.net/reader035/viewer/2022062722/56649f315503460f94c4bff4/html5/thumbnails/24.jpg)
3/31/2005 Tamás Budavári (JHU) 24
SkyQuery portal
Hides complexitySimple web services
MethodsGet to metadata of nodesFind location of nodesSend query to one nodeSend federated query
• Create execution plan• Submit execution plan
![Page 25: Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.](https://reader035.fdocuments.net/reader035/viewer/2022062722/56649f315503460f94c4bff4/html5/thumbnails/25.jpg)
3/31/2005 Tamás Budavári (JHU) 25
![Page 26: Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.](https://reader035.fdocuments.net/reader035/viewer/2022062722/56649f315503460f94c4bff4/html5/thumbnails/26.jpg)
3/31/2005 Tamás Budavári (JHU) 26
Other VO Services
Density map plottingCosmic distance calc.Footprint intersectionWrappers for legacy
TCP services, e.g. NEDAnalysis tools & libraries e.g. CFITSIO (in C#)
http://voservices.org/
![Page 27: Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.](https://reader035.fdocuments.net/reader035/viewer/2022062722/56649f315503460f94c4bff4/html5/thumbnails/27.jpg)
3/31/2005 Tamás Budavári (JHU) 27
NVO Summer School
Aspen, Colorado13-17 September 2004
Full house~40 “students”
![Page 28: Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.](https://reader035.fdocuments.net/reader035/viewer/2022062722/56649f315503460f94c4bff4/html5/thumbnails/28.jpg)
3/31/2005 Tamás Budavári (JHU) 28
QuestionnaireVery good scores !!!Needs to be longer (10d)Great software package
• ZIP file on the NVO twiki
ProceedingsOn the Web
HERE
NVO Summer SchoolAll possible OS
Real test of interoperabilityChallenging to teach
WS learning curveDepending on background
Winning projectsC. Miller & S. KrughoffExtinction map serviceR. Lucas and friendsVLA science project
![Page 29: Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.](https://reader035.fdocuments.net/reader035/viewer/2022062722/56649f315503460f94c4bff4/html5/thumbnails/29.jpg)
3/31/2005 Tamás Budavári (JHU) 29
NVO Comes of Age
Special NVO session at AAS ‘05Featuring ~10 science talks using NVO technology
First refereed astronomy publicationsUsing the NVO framework
Next Summer School is being organized
NASA+NSF: Long Term NVO FacilityStarting in 2006
![Page 30: Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.](https://reader035.fdocuments.net/reader035/viewer/2022062722/56649f315503460f94c4bff4/html5/thumbnails/30.jpg)
3/31/2005 Tamás Budavári (JHU) 30
NVO is here!
![Page 31: Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University.](https://reader035.fdocuments.net/reader035/viewer/2022062722/56649f315503460f94c4bff4/html5/thumbnails/31.jpg)
3/31/2005 Tamás Budavári (JHU) 31
Where to go…
Links in the talkhttp://www.us-vo.org/http://www.ivoa.net/http://voservices.org/http://openskyquery.net/http://skyserver.sdss.org/ http://nvo.stsci.edu/voregistry/