Hosting CLARIN services in the EGI cloud · PDF fileUbercloud 19/09/17 7 The EGI Service...
Transcript of Hosting CLARIN services in the EGI cloud · PDF fileUbercloud 19/09/17 7 The EGI Service...
![Page 1: Hosting CLARIN services in the EGI cloud · PDF fileUbercloud 19/09/17 7 The EGI Service Catalogue . 19/09/17 8 Introduction to EGI cloud . 19/09/17 9 EGI Cloud • Grid of clouds](https://reader031.fdocuments.net/reader031/viewer/2022020120/5aa3ccb57f8b9a80378eb5dd/html5/thumbnails/1.jpg)
www.egi.eu EGI-Engage is co-funded by the Horizon 2020 Framework Programme
of the European Union under grant number 654142
[email protected] http://go.egi.eu/cloud
Hosting CLARIN services in the EGI cloud
Gergely Sipos EGI Foundation Boris Parak
CESNET Willem Elbers CLARING ERIC
![Page 2: Hosting CLARIN services in the EGI cloud · PDF fileUbercloud 19/09/17 7 The EGI Service Catalogue . 19/09/17 8 Introduction to EGI cloud . 19/09/17 9 EGI Cloud • Grid of clouds](https://reader031.fdocuments.net/reader031/viewer/2022020120/5aa3ccb57f8b9a80378eb5dd/html5/thumbnails/2.jpg)
2 19/09/17
EGI: Advanced Computing for Research
![Page 3: Hosting CLARIN services in the EGI cloud · PDF fileUbercloud 19/09/17 7 The EGI Service Catalogue . 19/09/17 8 Introduction to EGI cloud . 19/09/17 9 EGI Cloud • Grid of clouds](https://reader031.fdocuments.net/reader031/viewer/2022020120/5aa3ccb57f8b9a80378eb5dd/html5/thumbnails/3.jpg)
3 19/09/17
EGI Membership • Majornationale-Infrastructures:22NGIs• EIRO:CERN(EuropeanIntergovernmentalResearchOrganisation)• Membershipfeessustainthefederationandcommunity
– EGIFoundation– Collaboration&federation
![Page 4: Hosting CLARIN services in the EGI cloud · PDF fileUbercloud 19/09/17 7 The EGI Service Catalogue . 19/09/17 8 Introduction to EGI cloud . 19/09/17 9 EGI Cloud • Grid of clouds](https://reader031.fdocuments.net/reader031/viewer/2022020120/5aa3ccb57f8b9a80378eb5dd/html5/thumbnails/4.jpg)
4 19/09/17
EGI Federation Largest distributed compute e-Infrastructure of the world
300+HTCproviders23Cloudproviders
650kCPUcores
285PBonlinestorage
1.7Millionjobs/day
2.6BillionCPUhours/
year
240VirtualOrganisaKons
>48000users
EGI Foundation
(Amsterdam)
![Page 5: Hosting CLARIN services in the EGI cloud · PDF fileUbercloud 19/09/17 7 The EGI Service Catalogue . 19/09/17 8 Introduction to EGI cloud . 19/09/17 9 EGI Cloud • Grid of clouds](https://reader031.fdocuments.net/reader031/viewer/2022020120/5aa3ccb57f8b9a80378eb5dd/html5/thumbnails/5.jpg)
5 19/09/17
International Partnerships
AfricaandArabiaCouncilforScienKficandIndustrialResearch,SouthAfrica
IndiaCentreforDevelopmentofAdvancedComp.
ChinaInst.OfHEPChineseAcademyofSciences
La/nAmericaUniversidadeFederaldoRiodeJaneiro
UkraineUkrainianNaKonalGrid
USA
Canada
AsiaPacificRegionAcademiaSinicaatTaiwan
![Page 6: Hosting CLARIN services in the EGI cloud · PDF fileUbercloud 19/09/17 7 The EGI Service Catalogue . 19/09/17 8 Introduction to EGI cloud . 19/09/17 9 EGI Cloud • Grid of clouds](https://reader031.fdocuments.net/reader031/viewer/2022020120/5aa3ccb57f8b9a80378eb5dd/html5/thumbnails/6.jpg)
6 19/09/17
EGI serves researchers and innovators
ESFRIs,FETflagships
Sizeofindividualgroups
Mul;na;onalcommuni;es,(e.g.H2020projects)
‘Longtailofscience’
WLCGELICTAELIXIREPOSBBMRIINSTRUCTCLARINEISCAT_3DDARIAHLOFARLifeWatchICOSEMSOCORBELENVRIplus…
VREprojectsOpenDreamKitWeNMRDRIHMVERCEMuGAgINFRACMMSTLSGCSuperSitesExploitaKonEnvironmentalsci.neuGRID…
PeachNoteCEBAGalaxyeLabSemiconductordesignMain-beltcometsQuantumpysicsstudiesVirtualimaging(LS)BovinetuberculosisspreadConvergentevol.ingenomesGeographyevoluKonSeafloorseismicwaves3DlivermapswithMRIMetabolicratemodellingGenomealignmentTapewormsinfecKononfish…
Industry,SMEs
AgroknowCloudEOCloudSMEEcohydrosgnubilaSinergiseSixSqTEISSTerradueUbercloud…
![Page 7: Hosting CLARIN services in the EGI cloud · PDF fileUbercloud 19/09/17 7 The EGI Service Catalogue . 19/09/17 8 Introduction to EGI cloud . 19/09/17 9 EGI Cloud • Grid of clouds](https://reader031.fdocuments.net/reader031/viewer/2022020120/5aa3ccb57f8b9a80378eb5dd/html5/thumbnails/7.jpg)
7 19/09/17
The EGI Service Catalogue www.egi.eu/services
![Page 8: Hosting CLARIN services in the EGI cloud · PDF fileUbercloud 19/09/17 7 The EGI Service Catalogue . 19/09/17 8 Introduction to EGI cloud . 19/09/17 9 EGI Cloud • Grid of clouds](https://reader031.fdocuments.net/reader031/viewer/2022020120/5aa3ccb57f8b9a80378eb5dd/html5/thumbnails/8.jpg)
8 19/09/17
Introduction to EGI cloud
![Page 9: Hosting CLARIN services in the EGI cloud · PDF fileUbercloud 19/09/17 7 The EGI Service Catalogue . 19/09/17 8 Introduction to EGI cloud . 19/09/17 9 EGI Cloud • Grid of clouds](https://reader031.fdocuments.net/reader031/viewer/2022020120/5aa3ccb57f8b9a80378eb5dd/html5/thumbnails/9.jpg)
9 19/09/17
EGI Cloud • Gridofclouds• Uniformeduserinterfaces• Harmonisedoperational
behaviour• Basedonopenstandardsand
opentechnologies• PrimarilyanIaaScloud,but
withcommunity-hostedPaaSandSaaSontop
• InfrastructureàAccessANDTechnologystackàDeploy
![Page 10: Hosting CLARIN services in the EGI cloud · PDF fileUbercloud 19/09/17 7 The EGI Service Catalogue . 19/09/17 8 Introduction to EGI cloud . 19/09/17 9 EGI Cloud • Grid of clouds](https://reader031.fdocuments.net/reader031/viewer/2022020120/5aa3ccb57f8b9a80378eb5dd/html5/thumbnails/10.jpg)
10 19/09/17
The current infrastructure Today:• 23providersfrom14NGIs
• 15OpenStack• 7OpenNebula• 1Synnefo
![Page 11: Hosting CLARIN services in the EGI cloud · PDF fileUbercloud 19/09/17 7 The EGI Service Catalogue . 19/09/17 8 Introduction to EGI cloud . 19/09/17 9 EGI Cloud • Grid of clouds](https://reader031.fdocuments.net/reader031/viewer/2022020120/5aa3ccb57f8b9a80378eb5dd/html5/thumbnails/11.jpg)
11 19/09/17
Typical usage models
• Computeanddataintensiveworkloads– Batchandinteractive(e.g.Jupyternotebooks)withscalableand
customizedenvironments
• ServiceHosting– Long-runningservices(e.g.webserver,database,applicationserver)
• Datasetsrepository– Storeandmanagelargedatasets(inastoragevolume)
• Disposableandtestingenvironments– Hosttrainingenvironments,testapplications
![Page 12: Hosting CLARIN services in the EGI cloud · PDF fileUbercloud 19/09/17 7 The EGI Service Catalogue . 19/09/17 8 Introduction to EGI cloud . 19/09/17 9 EGI Cloud • Grid of clouds](https://reader031.fdocuments.net/reader031/viewer/2022020120/5aa3ccb57f8b9a80378eb5dd/html5/thumbnails/12.jpg)
12 19/09/17
Computing models
HTCGridservice(e.g.clusters;HTC)
Commercialcloudservice(e.g.Azure)
EGIcloudservice(federatedIaaScloud)
• ForbatchcompuKng• Forembarrassingly
parallelapplicaKons• UsingmulKplesitesat
aKme
• ForinteracKveandbatchcompuKng
• ForservicehosKng• FlexibleOSandapplicaKon
use• Typicallyonesiteisused• Pay-as-you-go• Nousersupport(orcentral
paidservice)
• ForinteracKveandbatchcompuKng
• ForservicehosKng• FlexibleOSandapplicaKon
use• UsesingleormulKplesites
ataKme• VirtualOrganisaKonbased
access• Freeatthepointofuse(for
research)• Withlocalusersupport
![Page 13: Hosting CLARIN services in the EGI cloud · PDF fileUbercloud 19/09/17 7 The EGI Service Catalogue . 19/09/17 8 Introduction to EGI cloud . 19/09/17 9 EGI Cloud • Grid of clouds](https://reader031.fdocuments.net/reader031/viewer/2022020120/5aa3ccb57f8b9a80378eb5dd/html5/thumbnails/13.jpg)
13 19/09/17
Main resource about EGI Cloud
EGI Federated Cloud User Guides: • https://wiki.egi.eu/wiki/Federated_Cloud_user_support
![Page 14: Hosting CLARIN services in the EGI cloud · PDF fileUbercloud 19/09/17 7 The EGI Service Catalogue . 19/09/17 8 Introduction to EGI cloud . 19/09/17 9 EGI Cloud • Grid of clouds](https://reader031.fdocuments.net/reader031/viewer/2022020120/5aa3ccb57f8b9a80378eb5dd/html5/thumbnails/14.jpg)
14 19/09/17
CLARIN services in the EGI Cloud
![Page 15: Hosting CLARIN services in the EGI cloud · PDF fileUbercloud 19/09/17 7 The EGI Service Catalogue . 19/09/17 8 Introduction to EGI cloud . 19/09/17 9 EGI Cloud • Grid of clouds](https://reader031.fdocuments.net/reader031/viewer/2022020120/5aa3ccb57f8b9a80378eb5dd/html5/thumbnails/15.jpg)
15 19/09/17
Brokering resources for a community
Project/Communityrepresen;ngtheVO
Nego;ator Gridprovider
Cloudprovider
OperaKonLevel
Agreement
ServiceLevelAgreement
SaKsfacKonreview(every3/6/12months)
Storageprovider
Servicerequirements Condi;ons
Applic.provider
Performancereports
SupportTraining
Type,number,size,cost,availability,etc.
TriggertheprocesswithaservicerequestontheEGIwebsite
![Page 16: Hosting CLARIN services in the EGI cloud · PDF fileUbercloud 19/09/17 7 The EGI Service Catalogue . 19/09/17 8 Introduction to EGI cloud . 19/09/17 9 EGI Cloud • Grid of clouds](https://reader031.fdocuments.net/reader031/viewer/2022020120/5aa3ccb57f8b9a80378eb5dd/html5/thumbnails/16.jpg)
16 19/09/17
CLARIN cloud provider:
• National grid & cloud infrastructure including storage
• Multiple production IaaS clouds operated by CESNET
– MetaCloud (OpenNebula-based) • 2,900 CPU, ≈ 30 TB RAM1
– EGI Federated Cloud (also OpenNebula-based) • ≈300 CPU + 1,600GB RAM
– Full-fledged & certified FedCloud site – Numerous testbed deployments (also OpenNebula-based) – ownCloud personal cloud-based storage
![Page 17: Hosting CLARIN services in the EGI cloud · PDF fileUbercloud 19/09/17 7 The EGI Service Catalogue . 19/09/17 8 Introduction to EGI cloud . 19/09/17 9 EGI Cloud • Grid of clouds](https://reader031.fdocuments.net/reader031/viewer/2022020120/5aa3ccb57f8b9a80378eb5dd/html5/thumbnails/17.jpg)
17 19/09/17
CLARIN services in the CESNET cloud
1. CLARIN component registry (http://clarin.eu/cmdi), the backbone of the Component-based MetaData Infrastructure.
2. The CLARIN docker registry (http://docker.clarin.eu), which plays an important role in the CLARIN deployment workflow.
3. The CLARIN maven repository (http://nexus.clarin.eu), intensively used during CLARIN software development projects.
4. A grafana instance (http://metrics.clarin.eu), to visualize server status and metrics.
5. Document archive for the CLARIN ERIC central office.
![Page 18: Hosting CLARIN services in the EGI cloud · PDF fileUbercloud 19/09/17 7 The EGI Service Catalogue . 19/09/17 8 Introduction to EGI cloud . 19/09/17 9 EGI Cloud • Grid of clouds](https://reader031.fdocuments.net/reader031/viewer/2022020120/5aa3ccb57f8b9a80378eb5dd/html5/thumbnails/18.jpg)
18 19/09/17
Future outlook – The European perspective European Open Science Cloud
“The European Open Science Cloud (EOSC) is a vision for a federated, globally accessible,
multidisciplinary environment where researchers, innovators, companies and citizens can publish,
find, use and reuse each other's data, tools, publications and other outputs for research,
innovation and educational purposes.”
Credits: Open Science Policy Platform EOSC wg
à EOSCpilotH2020project:2017.Jan1–2018Dec31.à EOSC-hubH2020project:2018.Jan1–2020Dec31.
![Page 19: Hosting CLARIN services in the EGI cloud · PDF fileUbercloud 19/09/17 7 The EGI Service Catalogue . 19/09/17 8 Introduction to EGI cloud . 19/09/17 9 EGI Cloud • Grid of clouds](https://reader031.fdocuments.net/reader031/viewer/2022020120/5aa3ccb57f8b9a80378eb5dd/html5/thumbnails/19.jpg)
19 19/09/17
Next Community Event Digital Infrastructures for Research (DI4R)
hlps://www.digitalinfrastructures.euCo-locatedwithEOSCpilot1stStakeholderEngagementEvent(28-29Nov)
CALLFORCONTRIBUTIONISNOWOPEN!
![Page 20: Hosting CLARIN services in the EGI cloud · PDF fileUbercloud 19/09/17 7 The EGI Service Catalogue . 19/09/17 8 Introduction to EGI cloud . 19/09/17 9 EGI Cloud • Grid of clouds](https://reader031.fdocuments.net/reader031/viewer/2022020120/5aa3ccb57f8b9a80378eb5dd/html5/thumbnails/20.jpg)
20 19/09/17
Summary
• EGI Foundation: Cloud service broker • CESNET: Cloud service provider • CLARIN ERIC: Cloud customer • CLARIN community: Cloud user