Serco Business

Copernicus Sentinels Data Hub Service

Collaborative Ground Segment WorkshopESRIN, 4-5 December 2018

Serco Business

Data Access Operational StatusOutline

• Data Hub Service Scope• ColHub Operational History• Service utilisation statistics• Service Availability• Service Architectural overview• Main issues and resolutions• Short Term Outlook

Serco Business

Ops Centre2

Ops Centre3

Ops Centre1

Data Access

ReportingS-5P PDGS

S-3 PDGS

S-2 PDGS

S-1 PDGS

DHuS Service

Data Hub Service Scope

Serco Business

Data Hub Service ScopeData Access Services

Copernicus Open Access Hub

Collaborative Data Hub

Copernicus Services Data Hub

International Access Hub

DIAS Access Hub

PrimaryServices

Commissioning Data Hub

Expert Users Data Hub

Pre-operational Data Hub

Temporary Data Access Hub

Demo Hub

Africa Cast Hub

Data Access Services

AncillaryServices

Serco Business

Data Hub Service Operation highlights

Serco Business

Collaborative Data Hub Service Timeline

2015 20192016 2017 2018

26-Jan-15Node 2

18-Apr-16Node 1

14-Mar-18Node 3

Jan-15S1A NTC

Sep-15S1A NRT

May-16S2A

Sep-16S1B

Nov-17S3A SRAL Oct-18

S3A OLCI

Oct-17S2B

DHRColl

2

Coll

9 1

DHRColl

13 5

DHRColl

18 7

DHRColl

15 6

Use

rsG

row

thD

ata

Pub

licat

ions

Acc

ess

Nod

es

Serco Business

0

200

400

600

800

1000

1200

2015

-03

2015

-04

2015

-05

2015

-06

2015

-07

2015

-08

2015

-09

2015

-10

2015

-11

2015

-12

2016

-01

2016

-02

2016

-03

2016

-04

2016

-05

2016

-06

2016

-07

2016

-08

2016

-09

2016

-10

2016

-11

2016

-12

2017

-01

2017

-02

2017

-03

2017

-04

2017

-05

2017

-06

2017

-07

2017

-08

2017

-09

2017

-10

2017

-11

2017

-12

2018

-01

2018

-02

2018

-03

2018

-04

2018

-05

2018

-06

2018

-07

2018

-08

2018

-09

2018

-10

2018

-11

Collaborative Data Hub Service Utilisation Timeline

2015 20192016 2017 2018

26-Jan-15Node 2

18-Apr-16Node 1

14-Mar-18Node 3

Jan-15S1A NTC

Sep-15S1A NRT

May-16S2A

Sep-16S1B

Nov-17S3A SRAL Oct-18

S3A OLCI

Oct-17S2B

[TiB]

DHR

Coll

Downloaded Volumes of Data per month

13 PiB

15 PiB

TotalsSince2015

Serco Business

Collaborative Data Hub Service Availability

Fire in Greece

Hosting site Maintenance

Serco Business

Collaborative Data Hub Service High level Setup

Ops Centre2

Ops Centre1

Sentinel-1PDGS

Sentinel-2PDGS

Sentinel-3PDGS

Ops Centre3

ColHubNode 2

ColHubNode 1

ColHub Node 3

S1A, S1B NRT+NTC

S1A, S1BNRT

S2A, S2BS3A

L1, L2NRT+ STC+ NTC

S1A, S1B NRT+NTC

S2A, S2B

S3A NRT+STC+NTC

S1A, S1B NTC

S2A, S2B

S3A NRT+STC+NTC

S-1(A+B) NRT+NTCS-2 (A+B)S-3A SRAL OLCI

2 weeks 3 weeks1 month

Data Offer

Rolling Policy

Collections S1 NRT Collection S1 NRT Collection S1 NRT Collection

Downl. Quota 10 10 10

S-1(A+B) NRT+NTCS-2 (A+B)S-3A SRAL OLCI

S-1(A+B) NRT+NTCS-2 (A+B)S-3A SRAL OLCI

Serco Business

Data fetching Data routing Backend Ingestion Frontend

publicationDissemination

Architectural layout at centre 1

S1

S2

A NTC B NTCA+B NRT

A L2A B L1CA L1C B L2A Repack.

OLCIA NTC B NRTA NRT B NTC Repro.

SLSTRA NTC B NRTA NRT B NTC Repro.

SRALA NTC B NRT+STCA NRT+STC B NTC Repro.

SYNA SYN B SYN B VGTA VGT

S3

(27)

Col Hub1

Sci Hub

API Hub

Cop Hub

DIAS Hub

(16+1)(12)

= Download manager instance

= Integrity check and compression

= DHuS software instance

= Data routing and circulation

= Data synchronization

Open Access Hub

Collaborative Data Hub

Copernicus ServicesData Hub

DIAS Access Hub

S1

S2

S3

Data Synch

(4)

Monitoring and Reporting ElementsPo

stG

res

Influ

xDB

Gra

fana

= Long Term Data warehouse (DWH)

= Real Time Database (RTDB)

= Analytics Platform Server (APS)

= Sysma Agent (SA)

Web application for the visualization of the metrics stored in the RTDB

MyS

QL,

Py

hton Metrics Collector serving both the DWH and the

RTDB. With caching capability

Stores the metrics used for reporting and data analysis with a long term data preservation (> 4 years).

Stores the metrics used for real-time monitoring and alarming with a short term data preservation (4 months).

Bash

, PL

SQ

L,TW

iki

= Automatic Weekly Reporting (AWR) Scripts for the automatic extraction of metrics from the DWH and web application for the visualization of the reports.

= Alarming System

Bosu

n

Application for triggering alarms notification based on thresholds set on metrics stored in the RTDB

= Sentinels Data Dashboard Web application for reporting metrics to the service stakeholders. Based on the data stored in the DWH

Serco Business

Main Issues and Resolution

• Sentinel-3 quicklook and thumbnail synchronization issue

• Database performances (optimization and maintenance)

• Data Gaps filling

Serco Business

Short Term Outlook

• Publication of Sentinel-3B products• Publication of Sentinel-2 Level-2 Global• Sentinel-2 Level-2 processing on demand dedicated presentation• Access to offline data from Long Term Archives dedicated presentation

Serco Business

Conclusions