The SMA On-Line Data Arc › sma › memos › 138.pdf · SMA Astronomi cal Data Arc hiv e and...

SMA Technical Memo# 138

The SMA On-Line Data Archive and Storage System:

Software Development and Hardware Prospects

Jun-Hui Zhao, Patricia Mailhot & Takahiro Tsutsumi

January 26, 2000

ABSTRACT

As the MIT/SAO correlator comes on line, the maximum data

production rate from the Submillimeter Array (SMA) on Mauna Kea

in Hawaii could reach up to 2.75 Mbytes per sampling. For a typical

integration time of 10 sec, the daily data production rate of the SMA

would be 20 Gbytes/day. The raw SMA visibility data is required

to be accessible to the scientists at both CfA in Cambridge and

ASIAA in Taipei as well as eventually the international astronomical

community. How to archive the data and manage the database will

become a challenging issue to the SMA project. A good solution

by utilizing the existing resources can be achieved. In this memo,

we present the state of the art design for the SMA Astronomical

Data Archive and Storage System. The detailed description on the

development in the archiving software is also provided. The hardware

devices required for implementing the massive data storage system

are reviewed and evaluated based on the current technology and their

market performance.

1

Contents

1. Introduction : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :4

2. Software Design and Development : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 4

2.1. The Server Architecture : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 5

2.1.1 Data Transfer and Storage Server : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 5

2.1.2 Sybase SQL Server : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 7

2.1.3 Replication Server : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :8

2.2. SMADB { The SMA Astronomical Database : : : : : : : : : : : : : : : : : : : : : : : 10

2.3. Java Database Connectivity : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 17

2.3.1 Jconnect and HTTP Server : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 17

2.3.2 SmaJIsql Applet { The SMA Database GUI : : : : : : : : : : : : : : : : : : 17

2.4. System Setup and Backup Plan : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :20

3. Hardware Prospects : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :20

3.1. Data Rate : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 21

3.2. Requirement : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 21

3.3. Choice of Hardware : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 22

3.3.1. Removable Media : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 22

3.4. Recommendation: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :23

4. Remarks : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 25

4.1. Secondary Replication Sites : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 25

4.2. Other SMA On-line Databases : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :25

4.3. Dedicated Network Link : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :26

4.4. Temporary Solution for Data Storage : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 26

5. Summary : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 27

Acknowledgement : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 27

References : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 28

2

List of Figures

1. The SMA On-Line Archive Server Architecture : : : : : : : : : : : : : : : : : : : : : : : : : : :6

2. Replication Sites of SMA On-Line Data Archive : : : : : : : : : : : : : : : : : : : : : : : : : :9

3. SMADB { SMA Astronomical Data Model in Sybase : : : : : : : : : : : : : : : : : : : : 16

4. SmaJIsql Applet { The SMA Astronomical Database GUI : : : : : : : : : : : : : : :18

List of Tables

1. The Table Model of The SMA Astronomical Database : : : : : : : : : : : : : : : : : : 11

2. Hardware Properties for Mass Storage System : : : : : : : : : : : : : : : : : : : : : : : : : : 24

3

1. Introduction

Data transfer and archive are a major step in the processing of astronomical

data. Because of high cost in the mass storage hardware and the speed limitation

in network communication, an half-decade ago relatively little e�ort was expended

on the archiving of astronomical data with emphasis towards making the raw data

permanently on-line and immediately accessible. Due to the recent development in

technology the cost of designing and building an on-line archive system has dropped

signi�cantly. It becomes possible and can be a�orded by a project like the SMA to

develop and maintain an on-line archive system. A solid design provides easy and

quick access to users and also provides high e�ciency and good performance in data

management.

The storage and archive techniques used in the modern computer systems are

composed of several di�erent hierarchical levels. Four levels are needed to make an

archive successful, which can be divided into the two major categories, i. e. hardware

and software.

For the hardware,

� 1. the storage medium (such as juke box, disk array, and CD-ROM, tape reel

...);

� 2. the reading or backing up devices (disk drive, tape drive ...).

For the software,

� 3. the data structure on the medium (�le system or standard format);

� 4. control and management procedures.

A general discussion on these levels can be found in Pasian and Pirenne (1995).

In this memo, we present the state of the art design for the SMA Astronomical

Data Archive System by giving the details of each level. The rest of the memo is

organized as follows: In Section 2, we describe the software design and development

for the SMA on-line archive system. In Section 3, we will review the prospects of the

hardwares that are necessary to implement the entire system. Section 4 gives a few

remarks on this system. Section 5 is a summary.

2. Software Design and Development

The design of this on-line archive system is based on the existing resources

available to the SMA; this includes software, hardware and manpower. Therefore

the cost in the data processing can be minimized. We also consider that the �nal

4

system (at least the software part) must be portable and hardware independent. Our

goal is to create a system that can be easy to maintain and upgrade.

A key commercial database management system (DBMS) that has been used

at the SMA is Sybase. Recently, this system has been upgraded to the Sybase

SQL Server System 11 at both the Cambridge site and the Mauna Kea site.

This relational DBMS (or RDBMS) provides a good development environment for

enterprise client/server application. The database software discussed in this memo

utilizes the RDBMS.

2.1. The Server Architecture

Fig. 1 shows an overview of the SMA on-line archive server architecture. The

visibility data produced from the SMA correlator (Crates) along with the ancillary

data is transferred via RPC to the data computer Smadata (Sun Ultra 60/Solaris).

Smadata is a central host in which almost all the post-correlator data processing

in terms of real-time correction, agging, formating and archiving does occur. In

addition, this data computer also hosts the Sybase SQL server, Replication server

and the HTTP server.

2.1.1. Data Transfer and Storage server

The RPC server smadata svc provides several data services to process the data

received from real-time computers (Crates and Hal). As soon as the server receives a

wake up signal (that signi�es the observing run to start) from a real-time computer,

several FITS-IDI (Diamond et al.1997; and Flatters 1998) table �les are created with

a post�x indicating the type of data table (such as FILENAME.UV, FILENAME.AG,

FILENAME.AN, FILENAME.FQ, FILENAME.SU...). The UV data transferred from

the Crates will be assembled by the server process smadata svc along with the

random parameters for each integration and each baseline. These data are appended

continuously into the FILENAME.UV as the observing and the real-time correlation

process goes on. The ancillary data is appended into the associated tables if

any updating information occurs either as the source changes or as the correlator

con�guration changes. Immediately after each observing track, these FITS tables

are packed into a single FITS-IDI �le. This FITS-IDI will be moved to the On-

line Storage System. Meanwhile, in the process FITStoDB (Tsutsumi & Zhao 1999),

the FITS header information along with the other ancillary data such as the setup

parameters for the correlator con�guration and the information for the FITS-IDI �le

5

Hal

smadata_svc

FITS_IDI

files

FTIStoDBprocess

jConnect

Server HTTP

ServerRPC

JavaApplets

jConnect drivers

download

Clients

Web Browsers

Store the FITS_IDI files into the on-line

Storage System

Java-Database connection

with

jCon

nect

Com

pile

the

Java

App

let G

UI

FTP

or H

TT

PHTTP

The SMA On-Line Archive Server Architecture

Java applets

with Web Browser

Store the FITS header information into DBMS

JDBC (drivers)

SQL Server

SMA CorrelatorCrates

Real-time computer

Smadata (Sun Ultra 60 / Solaris)

(Power PC, RPC client/server)(Power-PC, RPC clients)

Any Client Computers

Data Storage Device (TBD)

smadb docdb dersdb ......

DBMS

ReplicationServer

to Network

Fig.1: The SMA Archive Server Architecture. The host computer is SMADATA, Sun

Ultra 60 /Solaris. The RDBMS is Sybase. The JDBC utilizes jConnect from Sybase.

This con�guration is for the primary site currently located on the top of Mauna Kea.

This system will be moved to the SMA headquarter in Hilo. There already exits a

dedicated network link (45Mb/s) between Mauna Kea and Hilo. This system can be

shared by SMADB and other SMA databases, such as DERSDB, DOCDB etc..

6

location is stored into the SMA Astronomical Database (SMADB see Section 2.2) in

Sybase. The details regarding the data transfer and data structure for the FITS-IDI

standard have been discussed in the SMA Technical Memo 126 (Zhao 1998).

Two points that we would like to emphasize: (1) the storage format for the data

taken from a single observing run must follow an international standard which is

platform and medium independent. The FITS-IDI is the best standard for packing

up the raw visibility data in real-time. A FITS binary table is a collection of row data

�elds, organized into columns. This structure is also similar to the table structure

used in a relational database. The information contained in the FITS-IDI �le can

be easily organized into our RDBMS. (2) the reasons why we do not store all the

interferometer data, including the visibility and other ancillary data into Sybase, is

because the SMA data sets will be very large in particular if the integration time is

shorter than 10 sec. The procedures and I/O within Sybase Servers would not be

e�cient to handle large data �les and will make the archiving process complicated

and slow. This will also cause di�culty and problems in database management. The

data in the FITS header and some of the associated tables are stored in SMADB. The

individual FITS-IDI data �le can be stored on an on-line �le system that requires

special hardware devices. The information about and location of each FITS-IDI �le

will also go to the RDBMS. These data are su�cient for users (astronomers) to review

the basic information for each SMA observing run and determine where to get the

FITS-IDI data �le that they want. Thus, with minimal I/O activity the archive data

model for our system becomes simple and easy to understand. Also in this system,

the individual FITS-IDI �les can be put on a portable medium for manually delivery

without interruption or slow down to the SQL server.

This storage design meets two basic requirements of (1) the exibility in data

retrieval process and (2) high e�ciency for data management.

2.1.2 Sybase SQL Server

The RDBM system is a commercial package, provided by Sybase, Inc. Its

relatively low cost (compared with other commercial package such as Oracle) is

suitable to the size of a project like the SMA.With the standard ANSI SQL (Structure

Query Language), Sybase also has an excellent reputation as a SQL Server. The

software functions supported by Sybase are also enough to meet our requirements for

archiving documentation and data management carried out at the SMA. Sybase SQL

Server is available on many hardware platforms under a variety of operating systems.

7

The SQL Server-client platforms can be any of the following system: MS-DOS or

Windows PCs, OS/2 PCs, Next PCs, Unix workstation, Unix terminal servers, or

Apple Macintoshes. The exibility in hardware and OS system makes our on-line

archive system portable and easy for upgrading in hardware.

One of the major components in the Sybase SQL Server architecture is the Server

Nucleus, that is the collection of software installed on a host platform that acts as a

server to a client platforms. It includes the SQL Server Kernel, Process Manager,

Query Optimizer, system databases, and application data.

2.1.3 Replication Server

Data replication is now one of the most commonways to distribute data to remote

sites. If a large number of remote users and remote applications need to perform ad

hoc queries and transaction processing or a large volume data transfer is involved,

the network can bog down. In particular, for the SMA, we have a primary database

server at the Mauna Kea site, most of the users will remotely access the data. Based

on their location, the remote users can be divided into two groups, namely Cambridge

(Massachusetts) based or Nankang (Taipei) based. For a large volume of the SMA

visibility data, users and applications may have to wait unacceptable lengths of time

for receiving a complete data set. This will also generate a large amount of network

tra�c. Instead, we can replicate the data to local servers, and users can access the

data locally.

Sybase Replication Server provides an excellent solution for replication of a database

at a remote secondary site. As illustrated in Fig. 2, Replication Server transmits data

from the primary database server in Hilo (where the data originates) to secondary

site (the Cambridge site and/or the Nankang site) on the network. Rather than

requiring each user to access the remote database server over a long-distance network,

this Replication Server handles data transmission and ensures that each local server

has an up-to-date copy of the data.

We note that in this on-line archive system, the RDBM (Sybase) handles only

the header information stored in SMADB (the SMA Astronomical Database, see 2.2)

while the visibility data in the massive storage system is handled separately. Since

the volume of the header information data is small, it can be easily replicated through

the network. During the testing period, the visibility data might be small enough to

allow us to duplicate all the data including the data in SMADB and the FITS-IDI

�les at the remote sites via the existing network link. As the full correlator operation

8

FITStoDB

FITS-IDI

smadata_svcRPCServer

files

smadb

Network

Primary Site

Cambridge

Nankang

Secondary Site

DataStorageDevice

DataStorageDevice

DataStorageDevice

Replication Sites of SMA On-Line Data Archive

SQL Server ReplicationServer

ReplicationServer

SQL Server

ReplicationServer

SQL Server

smadb

smadb

Express Air Delivery Service

Special Link

toClients

toClients

toClients

Hilo i

45Mb/s (30T1)

From Mauna Kea

Fig.2: A concept of the replication sites for SMA Astronomical DATA Archive. Hilo

is considered as the primary site; the secondary sites are in Cambridge and Nankang.

The data in SMADB can be replicated through normal network link. However, the

large FITS-IDI �les need either a dedicated link or portable medium with an express

air delivery service. For the ASIAA, it is not necessary to follow this design. They

can develop a database system based on their resources. The FITS-IDI �les can be

duplicated from the primary site.

9

comes on line, the duplication through network will run into some problems due to

the limited bandwidth in network transfer. Either a special network link or other

portable medium combined with some commercial express delivery service is required.

The approach proposed here is a remedy for the problems associated with the

distribution of and direct access to the SMA data. Using Replication Server is e�cient,

because Replication Server only replicates original data that is added, modi�ed, or

deleted. It also is fast, because Replication Server copies all of the data to the

remote server, so the remote users can access it over the Local Area Network (LAN).

Replication Server provides an additional important advantage. If the local data server

or local network is down and transactions need to be replicated, Replication Server

performs all of the necessary synchronization when the local network is available to

each other. This is an excellent function for disaster recovery.

Replication Server will also e�ectively reduce the manpower required for the

maintenance of the SMA Astronomical Database.

However, Replication Server has not been created in our system yet. We strongly

propose to implement the Sybase Replication Server to our database server system in

the next software upgrade. The cost would be minor as compared with the returns.

2.2. SMADB { The SMA Astronomical Database

The data model design in Fig. 3 is based on the data structure of the FITS-

IDI �le and the information obtained via the real-time data collection process. Ten

relational Sybase tables are needed to model the SMADB. Using this data model, the

physical schema is designed (see Tables 1{10). Table 1 (RUN LOG) contains the

general information for each observing run. The information about the correlator

that generates the visibility data is included in Table 2 (CORR). The mandatory

keywords for each FITS-IDI �le are stored in Table 3 (FITS KEY). The general

information on FITS tables in each FITS-IDI �le is stored in Table 4 (TAB NM).

The parameters for frequency setup, source coordinates and velocities are stored in

Tables 5 (FREQ), Table 6 (SOUR), and Table 7 (VELO). The information regarding

the array geometry is saved in Table 8 (ARR GEO). The information on the visibility

table and information about each FITS-IDI �le can be found in Table 9 (VIS) and

Table 10 (DFILE), respectively. The tables are laid out as: The 1st column is the

variable name; the 2nd column is the data type; and the comments on each variable

is given in the 3rd column.

10

Tables: The SMA Astronomical Data Model

1. RUN LOG

obscode char(20) /* observing program code */

job id smallint /* job ID for current script �le */

observer char(20) /* observer's name */

telescope char(16) /* telescope name */

start double precision /* job start time (MJD) */

stop double precision /* job stop time (MJD) */

date char(10) /* date of observation */

clustered index: obscode

index: job id, observer, start, date

byte size: 84

2. CORR


corr name char(12) /* correlator name */

corr vers char(12) /* correlator software version */

corr mode char(12) /* correlator mode */

no pol tinyint /* # of polarizations*/

ave time oat /* data averaging time */

transition char(48) /* transition name */

restfreq double precision /* line rest frequency for each band */

sideband smallint /* sideband ag */

smooth char(12) /* smooth function */

band idx smallint /* band index */

block ct freq oat /* block center frequency */

no chan chck1 int /* number of channel of 1st chunck */

no chan chck2 int /* number of channel of 2nd chunck */

no chan chck3 int /* number of channel of 3rd chunck */

no chan chck4 int /* number of channel of 4th chunck */


index: corr name, corr vers, corr mode, band idx

byte size: 153

3. FITS KEY

11


no stkd smallint /* number of Stokes parameters */

stk 1 smallint /* �rst Stokes parameter in data */

no band smallint /* number of 'bands' in data */

no chan int /* number of spectral channels in data */

ref freq double precision /* reference frequency */

chan bw oat /* frequency channel bandwidth */

ref pixel double precision /* coordinate for reference frequency*/

rdate int /* reference date (mjd) */


composite index: rdate

byte size: 54

4. TBL NM


�lename char(32) /* FITS IDI �le name */

tb names char(48) /* list of FITS tables in FITS IDI */

tb numb int /* number of FITS tables */


index: tb name

byte size: 104

5. FREQ


freqid tinyint /* frequency setup number */


bandfreq double precision /* frequency o�set + ref frequency */

ch width oat /* individual channel width */

total BW oat /* total bandwidth of a band */

sideband smallint /* sideband ag */

transition char(48) /* transition name */


index: freqid, bandfreq, band idx, transition

byte size: 89

12

6. SOUR


sou id tinyint /* source ID number */

sou name char(16) /* source name */

qual smallint /* source quali�er */

calcode char(2) /* calibrator code */

freqid tintyint /* frequency ID */

time onsource char(6) /* on-source time */

epoch double precision /* epoch for the source coordinates */

cra char(12) /* RA at the epoch in character string*/

cdec char(12) /* Dec at the epoch in character string*/

ra double precision /* RA at the epoch in deg.*/

dec double precision /* Dec at the epoch in deg.*/

pmra double precision /* proper motion in RA */

pmdec double precision /* proper motion in Dec */

parallax oat /* parallax of source */


index: sou id, sou name, freqid, qual, calc, ra, dec

byte size: 116

7. VELO


sou id tinyint /* source ID number */

freqid tinyint /* frequency ID */


sysvel double precision /* systematic velocity for each band */

veltyp char(8) /* velocity type */

veldef char(8) /* velocity de�nition */

restfreq double precision /* line rest frequency for each band */


index: sou id, band idx, freqid, sysvel, veltyp,veldef, restfreq

byte size: 56

8. ARR GEO


13

arrname char(12) /* array name */

frame char(12) /* coordinate frame */

arrayx oat /* x coordinate of array center */

arrayy oat /* y coordinate of array center */

arrayz oat /* z coordinate of array center */

freq oat /* reference frequency for the array*/

timesys char(12) /* time system */


gstia0 oat /* GST at 0h on reference date */

ut1utc oat /* UT1-UTC */

iatutc oat /* IAT-UTC */

polarx oat /* x coordinate of north pole */

polary oat /* y coordinate of north pole */


index: arrname, arrayx, arrayy, arrayz, rdate

byte size: 96

9. VIS


�lename char(32) /* �le name */


begin time double precision /* begin time */

end time double precision /* end time */

number baseline tinyint /* number of baselines */

array id tinyint /* sub-array ID */

total inttime oat /* total integration time */

#vis int /* total number of visibility */


index: rdate, begin time, array id

byte size: 82

10.DFILE


�lename char(32) /* FITS IDI �le name */

location char(32) /* location of FITS �le on on-line system*/

14

size Mb oat /* �le size in Mbyte */

#source tinyint /* number of sources */

arch date char(10) /* archiving date */

status int /* a ag to identify if the data is open

to public or restricted: 1 � > Yes;

{1 � > No */


composite index: full �lename (�lename, location)

byte size: 103

The parameter \obscode" is unique for each observing run and has been designed

as the cluster index in this particular relational database. The \cluster" index of a

table (there can only be one per table) is used to indicate the physical ordering of

table's row. Therefore, SMADB is organized based on the order of obscode assigned

for each observing run.

In each of the tables, we also listed the \nonclustered" index. This index provides

access to the table's data in an alternate order. Although access time via this type

of index is not as good as by using the cluster index, nonclustered indexes allow the

user to look at the table's data in more than one way. For example, we can look at

the data in an order as the source's coordinates (either ra, or dec).

One \composite" index, full �lename (�lename, location), is used in this database.

This index is composed of two column variables in DFILE. Composite indexes are

helpful when two or more columns are best searched as a unit.

The tables can be linked to each other through the common parameters (primary

key) as indicated in Fig. 3.

We also list the byte size in the end of each table. A total of 1 Kbytes is estimated

for each set of table data. For a typical observation track, the observing time is about

8 hrs. In other words, the SMA will daily produce three sets of table data that need to

be stored into SMADB. Assuming 300 observing days each year, only about 1 Mbyte

table data will be produced for the database per year if no text �les of scienti�c

proposals are considered to be stored in Sybase. Optionally, we could also store the

text �les of scienti�c proposal into Sybase. The size of the database would be roughly

increased by an order of magnitude.

15

SMA Astronomical Data Model in Sybase

RUN_LOG

CORR

FITS_KEY

FREQ

SOUR

VIS

ARR_GEO

VELO

DFILE

TBL_NM

freq

id

band

inx

sou_

id

rdat

e

obsc

ode

rest

freq

tran

sitio

n

file

nam

e

SMADB

Fig.3: The data table model for SMADB. Ten relational tables are used in this

database. The table can be linked to each other with the parameters labelled along

the connection lines.

16

2.3. Java Database Connectivity

JDBC (Java Database Connectivity) is a Java API (Application Programming

Interface) for executing SQL statements. It consists of a set of classes and interfaces

written in the Java programming language. JDBC provides a standard API for

tool/database developers and makes it possible to write database applications using

a pure Java API.

2.3.1. Jconnect and HTTP Server

A JDBC driver, jConnect purchased from the Sybase company, has been installed

in the Server host computer Smadata. The basic con�guration for the SMA On-Line

Archive System is illustrated in Fig. 1. JDBC provides standard Java API codes

that allows us to develop a speci�c Java Applet GUI (Graphical User Interface) to

communicate with SMADB via SQL Server. The data computer also hosts an HTTP

Server. This Server provides a port for outside clients to download Java Applets

and therefore to establish a connection with the database Server. As soon as the

Client/Server connection is established, the data transaction can proceed via the

network. In principle, a client can be anywhere in the world. As long as the client is

allowed to access the computer network (such as the internet) and the client computer

is equipped with a Java supported Web browser, the client should be able to access

the SMA On-Line Database.

2.3.2. SmaJIsql Applet { The SMA Database GUI

SmaJIsql is a Graphic User Interface (GUI) specially designed for SMADB using

Java Applet. This is a pure Java code utilizing JDBC components. The design of

this GUI has the exibility to access the data in SMADB to meet the needs for general

users in astronomy. The current version is SmaJIsql 1.0. The GUI of SmaJIsql 1.0,

as illustrated in Fig. 4, consists of �ve major parts:

1. The data fields for the input parameters: This part is composed of

four rows. The �rst rows are the parameters for establishing a connection to

SMADB, including the IP address of the host computer, the SQL Server port

number, the username and password for the database access and the database

name. In the current version (SmaJIsql1.0), the password for SMADB is encoded;

the input �eld of this parameter is locked. For security, we will take out

the �rst row in the next version upgrading. The next three rows involve the

searching parameters. The object name, its coordinates, observing time on

17

SmaJIsql Applet

SQL Server host na

131.142.12.246

SQL Server port nu

4100

Username:

jzhao

Password:

*******

Database:

smadb

Search

Object Name:

Any

Object RA:

hh:mm:ss.pp

Object Dec:

+dd:am:as.p

Radius:

All Sky

Equinox:

J2000

Observing Date:

’mm/dd/yyyy’

Time on source (H

Any

Observer Name:

Any

Array Configuratio

Any array

Correlator Mode:

Any mode

Observing Band:

Any band

Band Width (MHz)

Any

Program ID:

Any

Title of Proposal:

Any

Dataset Name:

Any

f t p

Query:

get_SourInfo

Results:

Fig.4: The SMA Astronomical Database GUI built in Java Applet using JDBC

components. It can be downloaded from a Web browser with Java support and run

on a client computer. The data transaction between users and SMADB can be done

via this interface.

18

source and observing bandwidth are listed in the second row. The third row

is the searching parameters such as the Array con�guration, Searching radius

on the sky, Equinox of the coordinates, Correlator mode and Observing band;

each of them is attached with a multiple choice list. These choice lists, which are

created from the Choice class in Java, are components that enable a single item

to be picked from a pull-down manu. The fourth row includes the searching

parameters as follows: Observing date, Observer name, Program ID, Title of

Proposal, Data Set Name. In SmaJIsql1.0, not all the searching parameters

have been applied to the query processes.

2. Query options: Three query options are provided in SmaJIsql1.0. Each

of them corresponds to a stored procedure embedded in SMADB. Stored

procedures are programs that are written using ANSI SQL commands combined

with standard programming constructs. Stored procedures are secure and

performance e�cient. The �rst query option get SourInfo is to search for the

source information by selecting Object Name and/or Object RA (input format

= hh:mm:ss.pp), Dec (input format = dd:am:as.p) and Searching radius. The

returned results are SMA program ID code (Obs code), Object name (Source),

Object right ascension (Ra), Object declination (Dec), Calibration code (calc,

T:target, C:calibrator), Total integration time (Otime), Observing date in

mm/dd/yyyy (Obs date), and Principal observer name (P. I. name).

The second query option get FileInfo to search for the data �le information by

selecting Program ID or Principal observer name (Observer Name), or Observing

date (in format mm/dd/yyyy). The returned results are SMA program ID code

(Obs code), Principal observer name (P. I. name), Observing date (Obs date),

Number of visibilities (#vis), Size of the data �le in Mbytes (Size), Number of

sources contained in each �le (#source), Location of the �les (the �rst character

code C: Cambridge, H: Hilo, M: Mauna Kea) , Data �le name.

The third query option get FreqInfo is to search for frequency setup

information by selecting Object Name or Observing Band or Program ID.

The returned variables are SMA program ID code (Obs code), Object name

(Source), Observing frequency (Obs freq in Hz), Channel width (Ch width in

Hz), Bandwidth (BW in Hz), Transition Name (Transition), Systematic Velocity

(Sys vel km/s).

More query options can be added in order to meet users' requirements.

19

3. Searching Key: The Search bar is an action key. By clicking it, The searching

action will take place based on the input parameters and the selected query.

4. FTP or HTTP: By clicking the ftp bar, the Applet will link the user to the FTP

and HTTP page from where the FITS-IDI �les can be found. The data can be

transferred via either ftp or http.

5. Results: This panel lists the results returned from each searching action based

on the query option.

2.4. System Setup and Backup Plan

SQL server for SMA data resides on SUN Ultra Sparc 60 running Solaris 2.7.

SMA is running Sybase 11.5.1 at both Cambridge and Hawaii sites.

Although Sybase SQL can use either a �le system or raw partitions, we opt for the

raw partitions for security of data as is recommended by Sybase. The Unix partitions

are owned by `sybase' (a user's account in the Unix system) and the server daemon

along with all DBCC (database consistency checker) and backup scripts are executed

by `sybase'. The master database (which contains all the system information) is

mirrored. In case the original device for master database is damaged, SQL Server

could start the mirrored device.

The backup and disaster recovery plan is two-level as follows:

Level One: User database recovery. Any database owner can retrieve a

copy of their database on-line by executing an sql command

within Sybase.

Level Two: System recovery. Master database can be retrieved on-line.

Key system tables and data are backed up.

The incorporation of the Sybase Replication Server is scheduled for

implementation next year. The dual purpose of this server is for disaster recovery

and data reliability. This server will also allow us to eliminate the need for mirroring

of databases with high levels of I/O.

3. Hardware Prospects

As the SMA becomes fully operational (with all 8 antennas and full set of the

MIT/SAO correlator), the storage of the data becomes a major issue for the on-line

data archive system described in the previous sections. We will inevitably need a

high capacity mass storage system.

20

Fortunately, the technology for storing large volume data is rapidly advancing

and numerous choices are available today. However hardware can be di�cult to

replace than software, thus we should choose the mass storage devices carefully. In

this section, we discuss criteria for choosing the hardware and compare the devices

that the current mass storage technology o�ers. At the end, recommendations are

given to aid future decision making on the purchase of the hardware.

3.1. Data Rate

In SMA Tech Memo #99 (Masson 1996), the data rate and the backup storage

for the SMA were discussed. Here we update the discussion based on the current

speci�cation of the SMA.

With the eight-element SMA in full operation, 2688 chips with 128 lags/chip

(64 spectral channels), or 172032 channels are available. The maximum data rate

will be 172032�8bytes (complex vis.) �2 (DSB) = 2.75 Mbytes per sampling. For

a typical full track synthesis observation of 8 hours, assuming typical sampling rate

of 10 seconds, 7.9 Gbytes of the data per each track or about 16 - 24 Gbytes per

day will be produced. For high resolution array modes including SMA + JCMT +

CSO combined array mode, higher sampling rates are expected if phase correction

techniques are to be applied. If we accumulate the data with the above mentioned

rate and assuming 256 observing days per year(assuming operational e�ciency of

70%), about 4 - 6 Tbytes of data will be produced in one year. The data can be

temporarily put into hard disks but it is apparent that we need massive storage

spaces for more long-term data storage. Note that above discussion assumes that all

the data are stored without any on-line calibration (e.g. on-line phase correction)

which can reduce the data volume drastically. Also, it is likely that many projects

(for example continuum observations) will produce much less data per unit time. In

average, we expect that the data production rate would be 1 Tbyte per year.

3.2. Requirements

The current proposed design of the replication sites as discussed in the previous

sections requires installation of data storage devices in Hilo, Cambridge, and

Nankang. At the primary site in Hilo, we should consider both temporary and long-

term data storage to hold at least few months worth of the maximum data rate. At

Cambridge and Nankang, each should have a mass storage device to hold about 1 to

2 years full operation data. In the future, if all the SMA scienti�c data need to be

21

available to the wider astronomical community, the Cambridge and Nankang sites

are likely to become science data center and may need to increase storage capacity.

3.3. Choice of Hardware

There are two types of mass storage devices: magnetic disks and removable

media. For storage system based on hard disks, one can use disk array such as RAID

(Redundant Array of Inexpensive Disks) to increase data redundancy for reliability.

One of the advantages of such a system is that all the data can be available on-line

all the time. However, additional cost of secondary backup on tapes may need to be

considered. And the initial setup to create data redundancy may be expensive.

The other alternative is the use of removable media. For most of the high density

removable media, devices that uses robotic mechanisms are available to build mass

storage systems. These are sometimes called auto-changers or juke boxes, or in the

case of tape media, (tape) libraries. With such devices one can build a semi-online

data archiving system with large volume capacity.

3.3.1 Removable media

There are many factors in choosing the right hardware for mass storage for our

needs. Some of the key issues are: volume, easy access, easy maintenance, reliability,

cost of media and the system as whole, I/O speed, durability, and lifetime of the

technology. In Table 2, high density media most commonly used for mass storage

system are evaluated in terms of these criteria.

Tapes: Since tapes utilize sequential access, the data transfer rate is generally

slow. The 8-mm exabyte and DAT tapes have been one standard storage medium

for astronomical data. The tape auto-exchanger, or library can store many 100 or

1000 Gbytes of the data. While these media are still very good for transporting

and backing up the data, these are not suitable for long term storage because of

the limited lifetime of the media. A recent development in tape storage technology is

DLT (Digital Linear Tape). DLT is a cartridge tape, data are recorded in longitudinal

multi-tracks versus the slanted stripes of helical scan technology which signi�cantly

increases data accessing rate. DLT is a durable (media lifetime �30 years) and has

very large capacity (�10 - 80G[compressed] per volume), with a low cost. The DLT

libraries are currently available from many vendors that can hold up to few tens of

Tbytes. Super DLT is the newest technology which may be available on the market

in the future, with about ten times capacity on a single cartridge than DLT. There

22

are other variants of high capacity tapes but the longevity of the such technologies

may be in question.

MO: Magneto-optical (MO) disks are also a popular choice among the high

density media with reasonable storage capacity (5.2GBytes on a single MO disk).

Juke boxes to store up to � 1 TBytes are available.

WORM: Write-Once-Read-Many(WORM) optical disks and the juke boxes have

been used in the astronomical data archive such as the HST. The capacity can go up

to 12 Gbytes per medium. But as CD/DVD becomes more popular, this technology

is now obsolete. The media and system are more expensive than others.

CD-R: The \write-once recordable" CD (CD-R) is inexpensive and CD juke

boxes have been available for quite sometime. But a single CD-R can hold only 0.65

Gbytes while our data at maximum rate can produce a single FITS �le of a single

observing run of a few G bytes.

DVD-R: The \write-once recordable" digital versatile disk (DVD-R), which

has been considered to replace CD, are now starting to appear on the market.

Current single-sided DVD-R holds 4.7 Gbytes. The storage capacity will doubled

when double-sided disks become available in future. DVD-R drives in juke box

con�guration (capacity �4 T bytes) are just appearing on the market. These juke

boxes(drives) are often backward compatible as the devices can handle mixture of

CDs and DVDs. The ESO is considering the DVD-R juke boxes for their mass storage

system for the VLT (Pirenne 1999).

3.4. Recommendation

Based on our investigation, the DLT library or DVD-R juke box systems would

be our preferred choice among the mass storage devices. While the DVD-R technology

is relatively new and still developing, the prospects and longevity of the DVD

technology appear promising. As the price of media is going down, the price/GByte

of the DVD juke box becomes comparable to the DLT library of the similar capacity.

As of December 1999, a juke box that can store over 700 disks is available. This

device can hold >3 TBytes of data with single-sided DVD-R disks.

For the minimum requirements at the summit, additional hard disks to hold up

to �60 Gbytes of the data temporarily and a mass storage system to hold 1Tbytes

are necessary with backup rate of �20 Gbytes/day. A single unit of the DVD-R juke

box would be su�cient for the storage system. The data are copied to media and can

be delivered to the Cambridge and Nankang sites. At these sites, each should have

23

Table2:HardwarePropertiesforMassStorageMedia

8mmexabyte

5.25"

12"

&4mm

DLT

MO

WORM

CD-R

DVD-R

keyword

DAT/DDS

opticaldisk

volume(Gbytes)

1-5a

10-40b

5.2

12

0.65

4.7

medialifetime(yr)

2-5

30

30

30

100

100c

datatransferrate(MB/sec)

<

1

1-5

2-5

2-6

5

1-2

durability

low

high

high

high

high

high

easyaccess

no?

yes

yes

yes

yes

yes

maintenance

?

easy

easy

easy

easy

easy

jukeboxesavailable?

yes

yes

yes

yes

yes

yes(1999)

cost/GB(media)

<

$1-2

$2

$10

$15

$2

$6

a;b

uncompressedcapacity

c

AsquotedinPioneer'stheDVDwhitepaper(URLhttp://www.pioneerusa.com/dvdwhite.html)

24

2-4 Tera byte juke boxes at initial installment. These are based on on-line archive

facility model discussed in the previous sections. If the data are to open to the wider

astronomical community, separate storage systems may need to be dedicated for data

search and retrieval for archival data researchers.

Investigation of the hardware on the market as well as the actual applications at

other astronomical institutions should be continued. In particular, optical and X-ray

astronomy �elds have more experience in on-line database and mass storage system,

and visits to such institutions would be very valuable.

4. Remarks

As we presented, we have designed and developed the SMA Astronomical Data

Archive System with the cutting-edge technology. The exibility designed in this

system allows easy expansion for other archiving purposes. It also can be upgraded

by adding more software and hardware components.

4.1. Secondary Replicate Sites

We recommend building additional replication sites for the SMA Astronomical

Data Archive besides the primary site in Hilo in order to reduce the data transfer

tra�c. The current development site in Cambridge can be used as the secondary

site for replicating the data archive. For the ASIAA, there will be signi�cant number

of users who will access the SMA data. The replication data archive suggested here

will enable the researchers to access the data locally; therefore the data transaction

time will be shortened adequately. The maintenance and continuing development are

required in order to maintain robust performance. It is necessary for the SAO to have

the hardware and software identical for both the primary site and replication sites.

The uniformity will e�ectively eliminate the redundant work in system management

and software upgrading.

For the ASIAA, they can develop a storage/archive system based on their

resources.

4.2. Other SMA On-Line Databases

In addition to the SMA astronomical database, this system can also be used

for other archiving purposes at the SMA. For example, the DERSDB { the SMA

engineering database for the Diagnostic and Error Reporting System (DERS), the

DOCDB { the SMA design documentation database, the STAR { the star catalogue

25

for optical pointing and etc. are also managed by Sybase. The system con�guration

and setup can be shared between all the databases for di�erent archiving purposes.

The template of the utility software, e.g. the Java GUI developed for SMADB, can

be used for other databases.

Besides the archive for the SMA raw visibility data, we also need to develop

and implement additional databases for calibrators at millimeter and submillimeter

wavelengths. In this system, we need to incorporate all the calibration information,

such as the results from observations for ux density calibration, phase calibrators,

bandpass pro�les, etc..

In addition, as discussed in the SMA project book (by Karl Menten and Patricia

Monger 1993), a comprehensive, reliable, and continually maintained spectral line

catalogue is needed to store within the on-line data system.

4.3. A Dedicated Network Link

The maximum data production rate of the SMA correlator is 2.75 Mbytes per

sampling. The SMA plans to support an integration time up to 1 sec. In order to

keep all the SMA data on line via real-time network transfer, a dedicated link with a

bandwidth of 22 Mbits/s at least is required between the primary site (Hilo) and each

replication site. Since the market price for network link goes down continually, we

could wait for lending a dedicated link until we feel comfortable. Before the dedicated

links are available, an alternative method can be used to make this data storage and

archive system work. We can keep the SMA visibility data on line for a period of

time (say 2 months) at the primary site on Mauna Kea through the real-time data

transfer in the local network. The data can be duplicated and be delivered expressly

to each of the replication sites once a week with a portable media. Virtually, we can

make the SMA visibility data on line at each sites with a time delay of a week.

4.4. Temporary Solution For Data Storage

While we continue to investigate hardware devices for data storage, we have a

temporary solution for keeping the visibility data on line. Currently we have 8 Gbyte

disk attached to SMADATA (Ultra 60) in Cambridge. Also an 18 Gbyte disk, which

will be installed at the Mauna Kea site, is being purchased. The testing data can

be continually stored in these disks until the SMA data production rate grows to a

critical number. However, this is only a temporary solution. In the summer of 2000,

26

we have to make a decision for choosing hardware devices to build our mass storage

system for the SMA project. Otherwise, we will face a crisis in data management

5. Summary

In the previous sections, we have presented the state of the art design for

the SMA Astronomical Data Archive and Storage System. The software

server architecture has been con�gured. Individual software components have been

developed and tested for the prototype model except for the Sybase replication

server. We strongly recommend incorporating the Sybase replication server into

this system. We plan to implement this system with the replication server in our

next software upgrade. Although the prototype system has been tested and shows

a good performance, this system needs to be continually supported at both system

and astronomy/programming levels for software upgrading and integration.

In order to keep the SMA visibility data on line, the hardware becomes a critical

issue. We have started an investigation for hardware devices to identify and construct

a mass data storage system for the SMA project. Based on our investigation, either

DLT library or DVD-R juke box is recommended as our preferred choices among the

mass storage devices on the current market. Both systems can handle several TBytes

of data on line based on their current capacity. We will continue our investigation

of the mass storage hardware on the market as well as the the actual application at

other astronomical institutions before a �nal decision is made.

Acknowledgement

We are grateful to Ken Harbour (Taco) Young for his detailed comments on this

memo. we also thank other SMA sta�s for their helpful comments and discussions.

27

Reference:

Diamond, P. J., Benson, J., Cotton, W. D., Wells, D. C., Romney, J. D. and Hunt,

G. \FITS Interferometry Data Interchange", VLBA correlator memo No. 108

(1997)

Flatters, Chris, \The FITS Interferometry Data Interchange Format" (1998)

(http://www.cv.nrao.edu/�ts/documents/drafts/idi-format.html )

Masson, C., \Data Reduction Computing for the SMA" SMA Tech Memo #99 (1996)

Pirenne, B. & Durand, D. \Data Storage Technology for Astronomy", in Information

& On-Line Data in Astronomy, eds. D. Egret and M. A. Albrecht p. 243 (1995)

Pirenne, B. & Albrecht, M. \The Prospects of DVD-R for Storing Astronomical

Archive Data", in Astronomical Data Analysis Software and Systems VIII, ASP

Conf. Ser. 172, eds. D. M. Mehringer, R.L. Plante, and D. A. Roberts (1999).

Tsutsumi, T. & Zhao, J.-H. \Librdsma�ts.a: a C library for reading SMA FITS

tables" SMA Memo #135 (1999)

Zhao, J.-H., \SMA Interferometry Data Transfer and Storage", SMA Memo #126

(1998)

28

The SMA On-Line Data Arc › sma › memos › 138.pdf · SMA Astronomi cal Data Arc hiv e and...

Documents

Transcript of The SMA On-Line Data Arc › sma › memos › 138.pdf · SMA Astronomi cal Data Arc hiv e and...