JSOC Overview - 1HMI/AIA Team Meeting – Feb 2006 JSOC Summary- Phil Scherrer Pipeline Processing -...

JSOC Overview - 1 HMI/AIA Team Meeting – Feb 2006

JSOC Summary- Phil Scherrer

Pipeline Processing - Rasmus Larsen

Data Access - Rick Bogart

Data Visualization - Neal Hurlburt

JSOC Overview


JSOC Scope

• The HMI/AIA Joint SOC consists of two parts:– Science Data Processing (SDP) – at Stanford and LMSAL

– Joint Operations Center (JOC) – at LMSAL

• JSOC JOC includes:– HMI and AIA Commanding and Health Monitoring

– HMI and AIA Engineering support as needed

• JSOC SDP includes:– HMI and AIA Telemetry Data capture (from DDS) and archive

– HMI and AIA Level-0 processing and archive

– HMI processing through to level-2 with archiving of end products

– AIA processing through level-1a with online archive at Stanford

– AIA level-2 processing at LMSAL

– Data export of the above and other HMI and AIA products as needed

• JSOC does not include tasks such as:– Science analysis beyond level-2 products

– HMI and AIA EPO

– HMI & AIA Co-I science support


SDO Ground System Architecture10/21/03

Telemetry & Command System

ASIST / FEDSTelemetry Monitoring

Command Management

HK Data ArchivalHK Level-0 Processing

Ground Station Control

DDS ControlAutomated Operations

Anomaly detection

Flight DynamicsSystem

Maneuver PlanningProduct Generation

R/T Attitude Determination

Sensor/Actuator Calibration

SDO Mission Operations Center

EVE SOC

Acquisition Data

Observatory Commands

Observatory HK Telemetry

Tracking Data

Integrated Trending& Plotting System

Mission Planning& Scheduling

Plan daily/periodic eventsCreate engineering planGenerate Daily Loads

HMI Science Data (55Mbps)

Ka-Band:150 Mbps

Science Data

Instrument Commands/Loads

Data DistributionSystem

(Incl. 30-Day Science Data Storage)

Ka Science Data

AIA R/T HK Telemetry/ Science Planning and FDS Products

EVE R/T HK Telemetry Science Planning and FDS Products

Universal Space Network

S-Band HK Tlm

CMD, Acquisition

Data

Station/DDS Control

Station/DDS Status

SDO Ground StationWSGT

Ka-Band:150 Mbps

Science Data

S-Band: TRK, Cmd & HK Tlm

S-Band: TRK,Cmd & HK Tlm

Alert NotificationSystem

Flight Software Maintenance Lab

Flight software loadsSimulated housekeeping telemetry

S/C Memory dumpsSimulated commands

Same Interfaces

as WSGT Ground Station

HMI AIA JSOC

Stanford Univ.

Science Data Capture

LMSAL

Instrument Monitoring& Control Instrument Commands/Loads

SDO Ground StationSTGT Flight Dynamics

FacilityOrbit DeterminationProduct Generation

Tracking Data

OD Products

Space Network(L&EO only)

S-Band: TRK,Cmd & HK Tlm

S-Band HK Tlm

Tracking Data

CMD

Acquisition DataS-Band: TRK & HK Tlm

DDS & Ground Station Control

AIA Science Data (67Mbps) EVE Science Data (7Mbps)

(Palo Alto, CA)

(Stanford, CA)

Tracking Data

Ka Science Data

Status and

Control

LASP(Boulder, CO)

Science Data CaptureInstrument Monitoring

& Control

Data Ack. & Retrans. RequestsData Ack. & Retrans. Requests

FLATSAT

Instrument Commands/LoadsHMI R/T HK Telemetry/ Science Planning and FDS Products

SDO

WSC

FSW SupportTool Suite

DDS & SDOGSIntegrated Manager

GSFC

S-Band RF &FEP system

Ka-Band RF system

(Includes 72-hr storage)

DDS FEP(Incl. 120-hr

storage)

S-Band RF &FEP system

Ka-Band RF system

(Includes 72-hr storage)

DDS FEP(Incl. 120-hr

storage)

Status and

Control

Mini MOC


HMI & AIA JSOC Architecture

Redundant

Data Capture System

Redundant Data Capture System

AIA HMI

30-DayArchive

Science TeamForecast Centers

EPOPublic

Catalog

Primary Archive

HMI & AIAOperations

House-keeping

Database

OffsiteArchiv

e

OfflineArchiv

e

JSOC Pipeline Processing System

HMI/AIA Level-0, 1, HMI-level2

DataExport& WebService

Stanford

LMSAL

High-LevelData Import

AIA Analysis System

Local Archive

QuicklookViewing

MOCDDShousekeeping

GSFCWhite Sands

World


JSOC Data Requirements

this version modified to show the links to the hardware plan

img size channels

cadence

compress

HMI: 55,000,000 bps ** SU 553 30 16 200% 395 90

AIA: 67,000,000 bps ** SU 674 30 20 200% 482 90

HMI: 4k*4k*2 bytes/2-seconds*(pi/4) 3.4E+07 2 4 0.39 SU 530 100 52 100% 189 180

AIA: 4k*4k*2 bytes * 8 imgs per 10 seconds 3.4E+07 8 10 0.50 SU 1,080 30 32 100% 386 1,900

HMI: V,M,Ic @ 45s & B, ld, ff @ 90s*(pi/4) 3.4E+07 5.5 45 0.39 SU 130 0 0 46 0

AIA: Level 1.0 same as level-0 3.4E+07 8 10 0.50 tbd 1,080 90 95 10% 39

HMI: See below 7.5E+10 1 86400 1.00 SU 70 0 0 25 100% 25 0

AIA (lev1a): movies & extracted regions. @ 20% 6.7E+06 8 10 0.50 LM 216 0 0 77 100% 77 0

HMI: Magnetograms (M, B) 3.4E+07 5 90 0.39 na 59 100 6 0

AIA: Full Level-0 data+lev1_extract 3.5E+07 8 10 0.50 na 1,134 100 111 0

HMI: 2 * Higher Level products + 5*10 min B SU 149 60 1 0

AIA: 3* higher Level products (TRACE < 1) SU 648 60 6 0

HMI: tlm SU 553 100% 198

AIA: tlm SU 674 100% 241

HMI: Lev0, Lev-1, All Higher SU 730 412

AIA: Lev0, Lev1a SU 1,296 743

HMI Totals 68 71 610

AIA Totals 146 77 984

Combined (TB) 214 148 1,594

Tape shelf size (TB) 7,968

Tape shelf number of tapes - mixed density 11,257

Export

Near-line retain days

Tape Archive Fraction

Totals

2,026Local tape

LMSAL Link 1,193

1,227Offsite tape

Higher level 286

Level-1

Data Path Assumptions Combined (GB/day)

Fixed Disk cache (TB)

Online disk cache days

Perm disk per year (TB)

Level-01,610

Volume (GB/day)

Processed at

In from DDS

Tape per year (TB)

1,227

1,210

797

TotalsCapture System Offsite Archive SUMS Archive Transient, 10 day


JSOC Systems

JSOC System

Need Date

Function

Assumptions

Processor System

Disk System

Tape system

UPS

Mounting

Misc

Basis of Estimate

count price each total cost count price each total cost count price each total cost count price each total cost count price each total cost count price each total cost count price each total cost count price each total cost

Processor 3 10.000 30 2 10.000 20 3 5.000 15 2 25.000 50 2 25.000 50 5 25.000 125 16 60.000 960 1 25.000 25

Disk 2 25.000 50 0 1 15.000 15 2 25.000 50 0 375 1.250 469 0 100 0.125 13

Tape system 2 25.000 50 2 20.000 40 0 1 10.000 10 0 2 180.000 360 0 0

Tape media 1281 0.100 128 0 0 0 0 1150 0.160 184 0 0

Switch 2 10.000 20 0 1 50.000 50 0 0 4 32.000 128 4 3.000 12 0

UPS 2 5.000 10 0 0 1 5.000 5 0 0 0 0

Initial Cost 288 60 80 115 50 1266 972 38

per year additions

Tape media 1125 0.100 113 0 0 0 0 1150 0.160 184 0 0

Disk 0 0 0 0 0 150 1.000 150 0 0

Networking 0 0 1 18.000 18 0 0 0 0 0

initial yearly cost 113 0 18 0 0 334 0 0

Reference for requirement

Rack

3 - Dual dual-core Opteron or other X86-64, 16 GB memory, 72 GB RAID 1 internal

10 TB for web and export services.

yes, 10 minutes

JSOC Support

Aug-07

Misc systems for operators (pui, SUMS, etc.), web, and export, also LMSAL network link.

LMSAL connection 10Gb/s link and fiber lease shown as switch. Yearl $2K SU, $16K PA, Initial 10K setup + 2x20K "switch"

Two identical systems with one cold spare. Assume 400GB tapes at $100 each. Assume tape deflate at 0.8/yr

Off-site and White Sands emergency system are similar systems.

Master for JSOC and slave system for outside users. Possibly buy slave system a year later.

Connect via 10Gb switches to cluster of file servers each connected viaredundant FC switches to RAID disk arrays and tape libraries.

Gets data from Capture system and SUMS

like MDI /scrxx disks, not backed up but connected via main archive FC switches

4 1-Gb->10Gb switches

HMI Science Cache

Mar-08

Working disk for analysis, outside SUMS

quad dual-core X86-64 NAS server, same as main archive

100TB user disk

yes, 10 minutes

Rack at SUMS archive

HMI Pipeline

Mar-08

Primary processing pipeline for HMI level-2,science analysis, and on demand processing.

16 - octet dual-core X86-64 systems, with 64 GB memory each.

no

2 racks, 8 units per rack.

10 Gb to users, FC to disks.

prototype file server, vendors

JSOC Pipeline

Aug-07

Processors for level-0, level-1 pipeline for HMI and AIA, and quicklook for JIOC

2 quad dual core X86-64 systems, 64 GB memory, 1TB local disk

no

Rack with JSOC support systems

Archive - SUMS

Jan-08

Primary data archive

5 - quad dual-core X86-64, 32 GB memory servers, FC support. Switches are 2 at 16K and 2 at 48K present prices.

350TB RAID 5 initial. Add 130TB per year.

2 1000 slot 6-drive tape libraries, for SAIT tapes., initial media for 1 year included, add 1150TB per year

yes, 10 minutesRack of servers, rack per 100TB, libraries.

2005 purchase of EGSE

JSOC Catalog

Aug-07

Database system for SUMS and DRMS

quad dual-core Xeon/Optern X86-64, 64GB memory

10TB RAID 10

tape auto loader for backups

yes, 10 minutes

rack with JSOC support

FC switch

2005 purchase of EGSE

Offsite

Jan-07

Receive archive copy from capture system, verify and copy as needed.

2-Dual dual-core Opteron or other X86-64, 16 GB memory, 72 GB RAID 1 internal, FC controller

n.a.

2-single drive, 24-slot library for LTO tapes.

no

small rack

2 (HMI and AIA) 20TB RAID 5, FC disks.

2 (HMI and AIA) dual-drive, 24-slot library for LTO tapes., assume 400GB tapes.

yes, 1 day

two racks, HMI and AIA

Capture

Data receipt from DDS, Archive onto main system and offsite. Assume 1.4TB tape per day.

Jan-07

3 (HMI, AIA, 1-spare) Dual dual-core Opteron or other X86-64, 16 GB memory, 72 GB RAID 1 internal, FC controller

CDRL 326a shows "c. 200" processors. Concern for vector field, coronal MHD for wpace weather compute load indicates an increase in capacity. This capability is largest area of uncertainty. We may still be low.

This need not identified in prior documents but is implied in the discussion in 326a

Processors shown in 326a, disk required for security to isolate external accessible data from archive. Export processors as in 326a. Control processors not detailed.

CDRL 326a shows 40TB each vs 20TB here. The need for two tape copies identified in developing the IT security plan. 326a showed only one permanent copy of the telemetry data. One copy now for offsite shown here and the other copy made in main archive.

IT Security planJSOC Peer review input said master/slave approach

CDRL 326a says 332TB initial plus 73TB per year disk and 1084TB tape per year. AIA wants larger cache nearline so library needs more slots

CDRL 326a shows 18 processors. Improved definition of AIA level-1 increases the SU needs here.


Operations HW Architecture

HMIQLQL image displayPlanningAnalysis

SP

AR

EO

PS

1

SP

AR

EO

PS

2

SPARE EGSE

Dedicated T1/FT1 Lines (redundant pair)

External network

Tlm

R

T H

K

PB

K H

K

Cm

d so

cket

1

Tlm

soc

ket 1

Tlm

soc

ket 2

DDS OC3 Lines

SDP

QL image production

StanfordStanfordUniversityUniversity

HMI & AIAWeb Server(PC or Sun)

HMI LMSAL EGSE

HMIOPS1(Sun)

HMIOPS2(Sun)

AIA LMSAL EGSE

AIAOPS1(Sun)

AIAOPS2(Sun)

Tlm

soc

ket 2

Cm

d so

cket

1

Tlm

soc

ket 1

HMIOPS3LMSAL

EGSE

LMSAL (Room in B. 252)

AIAQLQL image displayPlanningAnalysis

AIAOPS3LMSAL

EGSE

Cm

dsC

md

logs

Cm

d lo

gs

Tlm

R

T H

K

PB

K H

K

Cmd logs

Tlm RT HK PBK HK

MO

C(G

SF

C)

DD

SR

oute

rs(W

SC

)


JSOC Pipeline Processing System Components

Database Server

SUMSStorage Unit

Management System

DRMSData Record

Management SystemSUMS Tape Farm

SUMS Disks

Pipeline Program, “module”

Record Manage

ment

Keyword Access

Data Access

DRMS Library

Link Manage

ment

Utility LibrariesJSOC Science

Libraries

Record Cache

PUIPipeline User

Interface

Pipeline processing

plan

Processing script, “mapfile”

List of pipeline modules with needed datasets for input, output

Pipeline Operato

r

Processing History Log


HMI Processing Pipeline and Standard Products

Global Helioseismology

Processing

Local Helioseismology

Processing

Filtergrams

Line-of-sightMagnetograms

Vector Magnetograms

DopplerVelocity

ContinuumBrightness

Brightness Images

Line-of-SightMagnetic Field Maps

Coronal magneticField Extrapolations

Coronal andSolar wind models

Far-side activity index

Deep-focus v and cs

maps (0-200Mm)

High-resolution v and cs

maps (0-30Mm)

Carrington synoptic v and cs

maps (0-30Mm)

Full-disk velocity, v(r,Θ,Φ),And sound speed, cs(r,Θ,Φ),

Maps (0-30Mm)

Internal sound speed,cs(r,Θ) (0<r<R)

Internal rotation Ω(r,Θ)(0<r<R)

Vector MagneticField Maps

Tachocline

Differential Rotation

Meridional Circulation

Near-Surface Shear Layer

Activity Complexes

Active Regions

Sunspots

Irradiance Variations

Magnetic Shear

Flare Magnetic Config.

Flux Emergence

Magnetic Carpet

Coronal energetics

Large-scale Coronal Fields

Solar Wind

Far-side Activity Evolution

Predicting A-R Emergence

IMF Bs Events

Science ObjectiveData ProductProcessing

Observables

HMI Data

DopplerVelocity

HeliographicDoppler velocity

maps

Tracked TilesOf Dopplergrams

StokesI,V

Filtergrams

ContinuumBrightness

Tracked full-disk1-hour averagedContinuum maps

Brightness featuremaps

Solar limb parameters

StokesI,Q,U,V

Full-disk 10-minAveraged maps

Tracked Tiles

Line-of-sightMagnetograms

Vector MagnetogramsFast algorithm

Vector MagnetogramsInversion algorithm

Egression andIngression maps

Time-distanceCross-covariance

function

Ring diagrams

Wave phase shift maps

Wave travel times

Local wave frequency shifts

SphericalHarmonic

Time seriesTo l=1000

Mode frequenciesAnd splitting

Brightness Images

Line-of-SightMagnetic Field Maps

Coronal magneticField Extrapolations

Coronal andSolar wind models

Far-side activity index

Deep-focus v and cs

maps (0-200Mm)

High-resolution v and cs

maps (0-30Mm)

Carrington synoptic v and cs

maps (0-30Mm)

Full-disk velocity, v(r,Θ,Φ),And sound speed, cs(r,Θ,Φ),

Maps (0-30Mm)

Internal sound speed,cs(r,Θ) (0<r<R)

Internal rotation Ω(r,Θ)(0<r<R)

Vector MagneticField Maps

HMI Data Level 2 Data ProductProcessing

Level-1

Level-1

Level-0


JSOC Data Export System

DRMS

Pa

cka

ge

Format

Custom Keywords

Utilities

Selected

DataRecords

API

Dril

ldo

wn

OverviewNew/AvailStatistics

KeywordsRange

Se

arc

hB

row

se

Researcher A

General Public

Grid AdaptorGrid

VSO AdaptorVSO

CoSEC CoSEC Adaptor

Researcher B

Script Access

Space Weather

VSO – Virtual Solar ObservatoryDRMS – Data Record Mgmt Sys


JSOC SDP Development Milestones

• HMI and AIA Data EGSE installed

– Prototype for I/F testing with GS March 2005

– Version 2 to support flight inst. June 2005

• JSOC Capture System

– Purchase computers Fall 2006

– Final system installed Spring 2007

– Support DDS testing Summer 2007

• JSOC SDP Infrastructure, SUMS, DRMS, PUI

– Prototype testing of core system June 2005

– Fully functional Jan, 2006

• Purchase computers for JSOC Spring, 2007

• Infrastructure Operational Summer, 2007

• Data Product Modules Spring, 2008

• Test in I&T and with DDS,MOC as called for in SDO Ground System schedule


CY05

Launch

Ground System Readiness Testing

DDS to SOCs Science I/F Tests

T&CGSRT #1

4/06-04/07

CY06 CY07 CY08

MOC to SOCs T/C I/F Tests

Element Acceptance Testing

CTV E-T-EGSRT #48/07-4/08

Launch Site

GSRT #54-6/08

SCIGSRT #2

1/07-04/08

RFGSRT #33-12/07

CTV 72 DayE-T-E

Performance Test

2-4/08

REL #1 AT Complete

12/05

REL #2 ATComplete

6/06

REL #3 ATComplete

12/06

REL #4 ATComplete

10/07

EVE6-9/06

HMI2-3/07

AIA3-4/07

SDO Testing Timeline

Support needed from SOC(s)

All actual dates and test durations are included in the SDO Project Schedule

EVE2-9/07

HMI07-08/07

AIA8-9/07

Operations Test Opportunities with Observatory

1st S/C H/KDataflow with MOC

(6/06)

MOC Flight Proc Testing with Spacecraft

(3/07)

ScienceETE Test 1C

&Mission Sim Day in Life

(12/07 *)

LaunchRehearsals

(8/08)

1st Instrument HousekeepingDataflows with

MOC/SOC(6/07)

MOC Flight Procedure

Testing with Instruments

(11/07)

EVE

HMI

AIA

Mission Simsduring TV/TB(1/08 – 2/08)

Mission Simsduring TV/TB(1/08 – 2/08)

Mission SimsL&EO Ops

(3/08)

Mission SimsL&EO Ops

(3/08)

Ka ETU/SDOGS

1A (4/07)

Mission SimsContingency

Ops(2/08)

Mission SimsContingency

Ops(2/08)


JSOC SDP Development Plan

0

200

400

600

800

1000

1200

FY2006Q1

FY2006Q2

FY2006Q3

FY2006Q4

FY2007Q1

FY2007Q2

FY2007Q3

FY2007Q4

FY2008Q1

FY2008Q2

FY2008Q3

FY2008Q4

FY2009Q1

Per

cen

t F

TE

On-Demand SupportIrradianceForecast - farside…Coronal InferencesMag Field - Vector FieldMag. Field - Line-of-SightLevel 2 - Local HS HolographyLevel 2 - Local HS Ring DiagramLevel 2 - Local HS Time DistanceLevel 2 - Global HSLevel 2 - Quick LookHK & FDSLevel-0JSOC Verification & TestData Quality and Proc. MetadataGeneral Env.: cvs,os,oracle,compileArchive h/w arch.Processing Hardware Arch.User Tools: API, Data Export, & BrowsingpuiData CaptureDRMSSUMSHMI Develop and CalibrationEGSE and Inst testingSupport for AIA

JSOC Overview - 1HMI/AIA Team Meeting – Feb 2006 JSOC Summary- Phil Scherrer Pipeline Processing -...

Documents

Transcript of JSOC Overview - 1HMI/AIA Team Meeting – Feb 2006 JSOC Summary- Phil Scherrer Pipeline Processing -...