JSOC Overview - 1 HMI/AIA Team Meeting – Feb 2006
JSOC Summary- Phil Scherrer
Pipeline Processing - Rasmus Larsen
Data Access - Rick Bogart
Data Visualization - Neal Hurlburt
JSOC Overview
JSOC Overview - 2 HMI/AIA Team Meeting – Feb 2006
JSOC Scope
• The HMI/AIA Joint SOC consists of two parts:– Science Data Processing (SDP) – at Stanford and LMSAL
– Joint Operations Center (JOC) – at LMSAL
• JSOC JOC includes:– HMI and AIA Commanding and Health Monitoring
– HMI and AIA Engineering support as needed
• JSOC SDP includes:– HMI and AIA Telemetry Data capture (from DDS) and archive
– HMI and AIA Level-0 processing and archive
– HMI processing through to level-2 with archiving of end products
– AIA processing through level-1a with online archive at Stanford
– AIA level-2 processing at LMSAL
– Data export of the above and other HMI and AIA products as needed
• JSOC does not include tasks such as:– Science analysis beyond level-2 products
– HMI and AIA EPO
– HMI & AIA Co-I science support
JSOC Overview - 3 HMI/AIA Team Meeting – Feb 2006
SDO Ground System Architecture10/21/03
Telemetry & Command System
ASIST / FEDSTelemetry Monitoring
Command Management
HK Data ArchivalHK Level-0 Processing
Ground Station Control
DDS ControlAutomated Operations
Anomaly detection
Flight DynamicsSystem
Maneuver PlanningProduct Generation
R/T Attitude Determination
Sensor/Actuator Calibration
SDO Mission Operations Center
EVE SOC
Acquisition Data
Observatory Commands
Observatory HK Telemetry
Tracking Data
Integrated Trending& Plotting System
Mission Planning& Scheduling
Plan daily/periodic eventsCreate engineering planGenerate Daily Loads
HMI Science Data (55Mbps)
Ka-Band:150 Mbps
Science Data
Instrument Commands/Loads
Data DistributionSystem
(Incl. 30-Day Science Data Storage)
Ka Science Data
AIA R/T HK Telemetry/ Science Planning and FDS Products
EVE R/T HK Telemetry Science Planning and FDS Products
Universal Space Network
S-Band HK Tlm
CMD, Acquisition
Data
Station/DDS Control
Station/DDS Status
SDO Ground StationWSGT
Ka-Band:150 Mbps
Science Data
S-Band: TRK, Cmd & HK Tlm
S-Band: TRK,Cmd & HK Tlm
Alert NotificationSystem
Flight Software Maintenance Lab
Flight software loadsSimulated housekeeping telemetry
S/C Memory dumpsSimulated commands
Same Interfaces
as WSGT Ground Station
HMI AIA JSOC
Stanford Univ.
Science Data Capture
LMSAL
Instrument Monitoring& Control Instrument Commands/Loads
SDO Ground StationSTGT Flight Dynamics
FacilityOrbit DeterminationProduct Generation
Tracking Data
OD Products
Space Network(L&EO only)
S-Band: TRK,Cmd & HK Tlm
S-Band HK Tlm
Tracking Data
CMD
Acquisition DataS-Band: TRK & HK Tlm
DDS & Ground Station Control
AIA Science Data (67Mbps) EVE Science Data (7Mbps)
(Palo Alto, CA)
(Stanford, CA)
Tracking Data
Ka Science Data
Status and
Control
LASP(Boulder, CO)
Science Data CaptureInstrument Monitoring
& Control
Data Ack. & Retrans. RequestsData Ack. & Retrans. Requests
FLATSAT
Instrument Commands/LoadsHMI R/T HK Telemetry/ Science Planning and FDS Products
SDO
WSC
FSW SupportTool Suite
DDS & SDOGSIntegrated Manager
GSFC
S-Band RF &FEP system
Ka-Band RF system
(Includes 72-hr storage)
DDS FEP(Incl. 120-hr
storage)
S-Band RF &FEP system
Ka-Band RF system
(Includes 72-hr storage)
DDS FEP(Incl. 120-hr
storage)
Status and
Control
Mini MOC
JSOC Overview - 4 HMI/AIA Team Meeting – Feb 2006
HMI & AIA JSOC Architecture
Redundant
Data Capture System
Redundant Data Capture System
AIA HMI
30-DayArchive
Science TeamForecast Centers
EPOPublic
Catalog
Primary Archive
HMI & AIAOperations
House-keeping
Database
OffsiteArchiv
e
OfflineArchiv
e
JSOC Pipeline Processing System
HMI/AIA Level-0, 1, HMI-level2
DataExport& WebService
Stanford
LMSAL
High-LevelData Import
AIA Analysis System
Local Archive
QuicklookViewing
MOCDDShousekeeping
GSFCWhite Sands
World
JSOC Overview - 5 HMI/AIA Team Meeting – Feb 2006
JSOC Data Requirements
this version modified to show the links to the hardware plan
img size channels
cadence
compress
HMI: 55,000,000 bps ** SU 553 30 16 200% 395 90
AIA: 67,000,000 bps ** SU 674 30 20 200% 482 90
HMI: 4k*4k*2 bytes/2-seconds*(pi/4) 3.4E+07 2 4 0.39 SU 530 100 52 100% 189 180
AIA: 4k*4k*2 bytes * 8 imgs per 10 seconds 3.4E+07 8 10 0.50 SU 1,080 30 32 100% 386 1,900
HMI: V,M,Ic @ 45s & B, ld, ff @ 90s*(pi/4) 3.4E+07 5.5 45 0.39 SU 130 0 0 46 0
AIA: Level 1.0 same as level-0 3.4E+07 8 10 0.50 tbd 1,080 90 95 10% 39
HMI: See below 7.5E+10 1 86400 1.00 SU 70 0 0 25 100% 25 0
AIA (lev1a): movies & extracted regions. @ 20% 6.7E+06 8 10 0.50 LM 216 0 0 77 100% 77 0
HMI: Magnetograms (M, B) 3.4E+07 5 90 0.39 na 59 100 6 0
AIA: Full Level-0 data+lev1_extract 3.5E+07 8 10 0.50 na 1,134 100 111 0
HMI: 2 * Higher Level products + 5*10 min B SU 149 60 1 0
AIA: 3* higher Level products (TRACE < 1) SU 648 60 6 0
HMI: tlm SU 553 100% 198
AIA: tlm SU 674 100% 241
HMI: Lev0, Lev-1, All Higher SU 730 412
AIA: Lev0, Lev1a SU 1,296 743
HMI Totals 68 71 610
AIA Totals 146 77 984
Combined (TB) 214 148 1,594
Tape shelf size (TB) 7,968
Tape shelf number of tapes - mixed density 11,257
Export
Near-line retain days
Tape Archive Fraction
Totals
2,026Local tape
LMSAL Link 1,193
1,227Offsite tape
Higher level 286
Level-1
Data Path Assumptions Combined (GB/day)
Fixed Disk cache (TB)
Online disk cache days
Perm disk per year (TB)
Level-01,610
Volume (GB/day)
Processed at
In from DDS
Tape per year (TB)
1,227
1,210
797
TotalsCapture System Offsite Archive SUMS Archive Transient, 10 day
JSOC Overview - 6 HMI/AIA Team Meeting – Feb 2006
JSOC Systems
JSOC System
Need Date
Function
Assumptions
Processor System
Disk System
Tape system
UPS
Mounting
Misc
Basis of Estimate
count price each total cost count price each total cost count price each total cost count price each total cost count price each total cost count price each total cost count price each total cost count price each total cost
Processor 3 10.000 30 2 10.000 20 3 5.000 15 2 25.000 50 2 25.000 50 5 25.000 125 16 60.000 960 1 25.000 25
Disk 2 25.000 50 0 1 15.000 15 2 25.000 50 0 375 1.250 469 0 100 0.125 13
Tape system 2 25.000 50 2 20.000 40 0 1 10.000 10 0 2 180.000 360 0 0
Tape media 1281 0.100 128 0 0 0 0 1150 0.160 184 0 0
Switch 2 10.000 20 0 1 50.000 50 0 0 4 32.000 128 4 3.000 12 0
UPS 2 5.000 10 0 0 1 5.000 5 0 0 0 0
Initial Cost 288 60 80 115 50 1266 972 38
per year additions
Tape media 1125 0.100 113 0 0 0 0 1150 0.160 184 0 0
Disk 0 0 0 0 0 150 1.000 150 0 0
Networking 0 0 1 18.000 18 0 0 0 0 0
initial yearly cost 113 0 18 0 0 334 0 0
Reference for requirement
Rack
3 - Dual dual-core Opteron or other X86-64, 16 GB memory, 72 GB RAID 1 internal
10 TB for web and export services.
yes, 10 minutes
JSOC Support
Aug-07
Misc systems for operators (pui, SUMS, etc.), web, and export, also LMSAL network link.
LMSAL connection 10Gb/s link and fiber lease shown as switch. Yearl $2K SU, $16K PA, Initial 10K setup + 2x20K "switch"
Two identical systems with one cold spare. Assume 400GB tapes at $100 each. Assume tape deflate at 0.8/yr
Off-site and White Sands emergency system are similar systems.
Master for JSOC and slave system for outside users. Possibly buy slave system a year later.
Connect via 10Gb switches to cluster of file servers each connected viaredundant FC switches to RAID disk arrays and tape libraries.
Gets data from Capture system and SUMS
like MDI /scrxx disks, not backed up but connected via main archive FC switches
4 1-Gb->10Gb switches
HMI Science Cache
Mar-08
Working disk for analysis, outside SUMS
quad dual-core X86-64 NAS server, same as main archive
100TB user disk
yes, 10 minutes
Rack at SUMS archive
HMI Pipeline
Mar-08
Primary processing pipeline for HMI level-2,science analysis, and on demand processing.
16 - octet dual-core X86-64 systems, with 64 GB memory each.
no
2 racks, 8 units per rack.
10 Gb to users, FC to disks.
prototype file server, vendors
JSOC Pipeline
Aug-07
Processors for level-0, level-1 pipeline for HMI and AIA, and quicklook for JIOC
2 quad dual core X86-64 systems, 64 GB memory, 1TB local disk
no
Rack with JSOC support systems
Archive - SUMS
Jan-08
Primary data archive
5 - quad dual-core X86-64, 32 GB memory servers, FC support. Switches are 2 at 16K and 2 at 48K present prices.
350TB RAID 5 initial. Add 130TB per year.
2 1000 slot 6-drive tape libraries, for SAIT tapes., initial media for 1 year included, add 1150TB per year
yes, 10 minutesRack of servers, rack per 100TB, libraries.
2005 purchase of EGSE
JSOC Catalog
Aug-07
Database system for SUMS and DRMS
quad dual-core Xeon/Optern X86-64, 64GB memory
10TB RAID 10
tape auto loader for backups
yes, 10 minutes
rack with JSOC support
FC switch
2005 purchase of EGSE
Offsite
Jan-07
Receive archive copy from capture system, verify and copy as needed.
2-Dual dual-core Opteron or other X86-64, 16 GB memory, 72 GB RAID 1 internal, FC controller
n.a.
2-single drive, 24-slot library for LTO tapes.
no
small rack
2 (HMI and AIA) 20TB RAID 5, FC disks.
2 (HMI and AIA) dual-drive, 24-slot library for LTO tapes., assume 400GB tapes.
yes, 1 day
two racks, HMI and AIA
Capture
Data receipt from DDS, Archive onto main system and offsite. Assume 1.4TB tape per day.
Jan-07
3 (HMI, AIA, 1-spare) Dual dual-core Opteron or other X86-64, 16 GB memory, 72 GB RAID 1 internal, FC controller
CDRL 326a shows "c. 200" processors. Concern for vector field, coronal MHD for wpace weather compute load indicates an increase in capacity. This capability is largest area of uncertainty. We may still be low.
This need not identified in prior documents but is implied in the discussion in 326a
Processors shown in 326a, disk required for security to isolate external accessible data from archive. Export processors as in 326a. Control processors not detailed.
CDRL 326a shows 40TB each vs 20TB here. The need for two tape copies identified in developing the IT security plan. 326a showed only one permanent copy of the telemetry data. One copy now for offsite shown here and the other copy made in main archive.
IT Security planJSOC Peer review input said master/slave approach
CDRL 326a says 332TB initial plus 73TB per year disk and 1084TB tape per year. AIA wants larger cache nearline so library needs more slots
CDRL 326a shows 18 processors. Improved definition of AIA level-1 increases the SU needs here.
JSOC Overview - 7 HMI/AIA Team Meeting – Feb 2006
Operations HW Architecture
HMIQLQL image displayPlanningAnalysis
SP
AR
EO
PS
1
SP
AR
EO
PS
2
SPARE EGSE
Dedicated T1/FT1 Lines (redundant pair)
External network
Tlm
R
T H
K
PB
K H
K
Cm
d so
cket
1
Tlm
soc
ket 1
Tlm
soc
ket 2
DDS OC3 Lines
SDP
QL image production
StanfordStanfordUniversityUniversity
HMI & AIAWeb Server(PC or Sun)
HMI LMSAL EGSE
HMIOPS1(Sun)
HMIOPS2(Sun)
AIA LMSAL EGSE
AIAOPS1(Sun)
AIAOPS2(Sun)
Tlm
soc
ket 2
Cm
d so
cket
1
Tlm
soc
ket 1
HMIOPS3LMSAL
EGSE
LMSAL (Room in B. 252)
AIAQLQL image displayPlanningAnalysis
AIAOPS3LMSAL
EGSE
Cm
dsC
md
logs
Cm
d lo
gs
Tlm
R
T H
K
PB
K H
K
Cmd logs
Tlm RT HK PBK HK
MO
C(G
SF
C)
DD
SR
oute
rs(W
SC
)
JSOC Overview - 8 HMI/AIA Team Meeting – Feb 2006
JSOC Pipeline Processing System Components
Database Server
SUMSStorage Unit
Management System
DRMSData Record
Management SystemSUMS Tape Farm
SUMS Disks
Pipeline Program, “module”
Record Manage
ment
Keyword Access
Data Access
DRMS Library
Link Manage
ment
Utility LibrariesJSOC Science
Libraries
Record Cache
PUIPipeline User
Interface
Pipeline processing
plan
Processing script, “mapfile”
List of pipeline modules with needed datasets for input, output
Pipeline Operato
r
Processing History Log
JSOC Overview - 9 HMI/AIA Team Meeting – Feb 2006
HMI Processing Pipeline and Standard Products
Global Helioseismology
Processing
Local Helioseismology
Processing
Filtergrams
Line-of-sightMagnetograms
Vector Magnetograms
DopplerVelocity
ContinuumBrightness
Brightness Images
Line-of-SightMagnetic Field Maps
Coronal magneticField Extrapolations
Coronal andSolar wind models
Far-side activity index
Deep-focus v and cs
maps (0-200Mm)
High-resolution v and cs
maps (0-30Mm)
Carrington synoptic v and cs
maps (0-30Mm)
Full-disk velocity, v(r,Θ,Φ),And sound speed, cs(r,Θ,Φ),
Maps (0-30Mm)
Internal sound speed,cs(r,Θ) (0<r<R)
Internal rotation Ω(r,Θ)(0<r<R)
Vector MagneticField Maps
Tachocline
Differential Rotation
Meridional Circulation
Near-Surface Shear Layer
Activity Complexes
Active Regions
Sunspots
Irradiance Variations
Magnetic Shear
Flare Magnetic Config.
Flux Emergence
Magnetic Carpet
Coronal energetics
Large-scale Coronal Fields
Solar Wind
Far-side Activity Evolution
Predicting A-R Emergence
IMF Bs Events
Science ObjectiveData ProductProcessing
Observables
HMI Data
DopplerVelocity
HeliographicDoppler velocity
maps
Tracked TilesOf Dopplergrams
StokesI,V
Filtergrams
ContinuumBrightness
Tracked full-disk1-hour averagedContinuum maps
Brightness featuremaps
Solar limb parameters
StokesI,Q,U,V
Full-disk 10-minAveraged maps
Tracked Tiles
Line-of-sightMagnetograms
Vector MagnetogramsFast algorithm
Vector MagnetogramsInversion algorithm
Egression andIngression maps
Time-distanceCross-covariance
function
Ring diagrams
Wave phase shift maps
Wave travel times
Local wave frequency shifts
SphericalHarmonic
Time seriesTo l=1000
Mode frequenciesAnd splitting
Brightness Images
Line-of-SightMagnetic Field Maps
Coronal magneticField Extrapolations
Coronal andSolar wind models
Far-side activity index
Deep-focus v and cs
maps (0-200Mm)
High-resolution v and cs
maps (0-30Mm)
Carrington synoptic v and cs
maps (0-30Mm)
Full-disk velocity, v(r,Θ,Φ),And sound speed, cs(r,Θ,Φ),
Maps (0-30Mm)
Internal sound speed,cs(r,Θ) (0<r<R)
Internal rotation Ω(r,Θ)(0<r<R)
Vector MagneticField Maps
HMI Data Level 2 Data ProductProcessing
Level-1
Level-1
Level-0
JSOC Overview - 10 HMI/AIA Team Meeting – Feb 2006
JSOC Data Export System
DRMS
Pa
cka
ge
Format
Custom Keywords
Utilities
Selected
DataRecords
API
Dril
ldo
wn
OverviewNew/AvailStatistics
KeywordsRange
Se
arc
hB
row
se
Researcher A
General Public
Grid AdaptorGrid
VSO AdaptorVSO
CoSEC CoSEC Adaptor
Researcher B
Script Access
Space Weather
VSO – Virtual Solar ObservatoryDRMS – Data Record Mgmt Sys
JSOC Overview - 11 HMI/AIA Team Meeting – Feb 2006
JSOC SDP Development Milestones
• HMI and AIA Data EGSE installed
– Prototype for I/F testing with GS March 2005
– Version 2 to support flight inst. June 2005
• JSOC Capture System
– Purchase computers Fall 2006
– Final system installed Spring 2007
– Support DDS testing Summer 2007
• JSOC SDP Infrastructure, SUMS, DRMS, PUI
– Prototype testing of core system June 2005
– Fully functional Jan, 2006
• Purchase computers for JSOC Spring, 2007
• Infrastructure Operational Summer, 2007
• Data Product Modules Spring, 2008
• Test in I&T and with DDS,MOC as called for in SDO Ground System schedule
JSOC Overview - 12 HMI/AIA Team Meeting – Feb 2006
CY05
Launch
Ground System Readiness Testing
DDS to SOCs Science I/F Tests
T&CGSRT #1
4/06-04/07
CY06 CY07 CY08
MOC to SOCs T/C I/F Tests
Element Acceptance Testing
CTV E-T-EGSRT #48/07-4/08
Launch Site
GSRT #54-6/08
SCIGSRT #2
1/07-04/08
RFGSRT #33-12/07
CTV 72 DayE-T-E
Performance Test
2-4/08
REL #1 AT Complete
12/05
REL #2 ATComplete
6/06
REL #3 ATComplete
12/06
REL #4 ATComplete
10/07
EVE6-9/06
HMI2-3/07
AIA3-4/07
SDO Testing Timeline
Support needed from SOC(s)
All actual dates and test durations are included in the SDO Project Schedule
EVE2-9/07
HMI07-08/07
AIA8-9/07
Operations Test Opportunities with Observatory
1st S/C H/KDataflow with MOC
(6/06)
MOC Flight Proc Testing with Spacecraft
(3/07)
ScienceETE Test 1C
&Mission Sim Day in Life
(12/07 *)
LaunchRehearsals
(8/08)
1st Instrument HousekeepingDataflows with
MOC/SOC(6/07)
MOC Flight Procedure
Testing with Instruments
(11/07)
EVE
HMI
AIA
Mission Simsduring TV/TB(1/08 – 2/08)
Mission Simsduring TV/TB(1/08 – 2/08)
Mission SimsL&EO Ops
(3/08)
Mission SimsL&EO Ops
(3/08)
Ka ETU/SDOGS
1A (4/07)
Mission SimsContingency
Ops(2/08)
Mission SimsContingency
Ops(2/08)
JSOC Overview - 13 HMI/AIA Team Meeting – Feb 2006
JSOC SDP Development Plan
0
200
400
600
800
1000
1200
FY2006Q1
FY2006Q2
FY2006Q3
FY2006Q4
FY2007Q1
FY2007Q2
FY2007Q3
FY2007Q4
FY2008Q1
FY2008Q2
FY2008Q3
FY2008Q4
FY2009Q1
Per
cen
t F
TE
On-Demand SupportIrradianceForecast - farside…Coronal InferencesMag Field - Vector FieldMag. Field - Line-of-SightLevel 2 - Local HS HolographyLevel 2 - Local HS Ring DiagramLevel 2 - Local HS Time DistanceLevel 2 - Global HSLevel 2 - Quick LookHK & FDSLevel-0JSOC Verification & TestData Quality and Proc. MetadataGeneral Env.: cvs,os,oracle,compileArchive h/w arch.Processing Hardware Arch.User Tools: API, Data Export, & BrowsingpuiData CaptureDRMSSUMSHMI Develop and CalibrationEGSE and Inst testingSupport for AIA
Top Related