Second EOSCpilot Stakeholders Forum · 2019. 1. 7. · The European Open Science Cloud for Research...

Enablinginteroperability--experiencefromtheEOSCpilotwork

DoinaCristinaDuma(INFN-CNAF)OnbehalfofWP6

Second EOSCpilot Stakeholders Forum

Vienna 21 – 22 November 2018

TheEuropeanOpenScienceCloudforResearchpilotprojectisfundedbytheEuropeanCommission,DGResearch&Innovationundercontractno.739563

EOSCpilot: High Level Aims & Impact

www.eoscpilot.eu 2

TheEOSCpilotprojectsupportsthefirstphaseofdevelopmentofEOSC:

Engagewithabroadrangeofstakeholders,crossingbordersandcommunities,tobuildthetrustandskillsrequiredforadoptionofanopenapproachtoscientificresearch

Developanumberofdemonstratorsfunctioningashigh-profilepilotsthatintegrateservicesandinfrastructurestoshowinteroperabilityanditsbenefitsinanumberofscientificdomains

EstablishthegovernanceframeworkfortheEOSCandcontributetothedevelopmentofEuropeanopensciencepolicyandbestpractice

ImpactReducefragmentationbetweendatainfrastructuresbyworkingacrossscientificandeconomicdomains,countriesandgovernancemodels;andImproveinteroperabilitybetweendatainfrastructuresbydemonstratinghowdataandresourcescanbesharedevenwhentheyarelargeandcomplexandinvariedformats

Threetypesofchallenges:ScientificChallenges:deployingtheEOSCtodeliverOpenScience

NeedsandprovisionWhatdoresearchcommunitiesneedfroman“OpenDataScienceEnvironment”?

TechnicalChallenges:developingtechnicalsolutionsthatmeetthescientificneeds

ServicesandintegrationHowcanEOSCdeliverintegratedservicesthatarerelevanttocommunityneeds?

CulturalChallenges:adoptingnew,moreopenwaysofworking

SkillsandengagementWhatchangesareneededincapabilityandpractices?

GovernanceandpolicyHowshouldprovisionbeoverseentomaximizebenefit?

WP5-6


Interoperability:a crucial aspect of the EOSC

www.eoscpilot.eu 3


EOSC Interoperability – WP6

InfrastructureInteroperability:TheproviderviewcentricThecomplementaryusageofCloud,Grid,HTCandHPCinfrastructures,includinglargedata-stores,throughhighspeednetworksandperformantdatatransferprotocolsandtools.ThehighlevelobjectiveistofacilitatethemostadequateinfrastructuresforthetreatmentofextensiveamountsofdataThefederatedinfrastructurepilotshavetobesetup…willenabletheanalysesoftheexistinginteroperationmechanismsforsoftwarecomponents,services,workflows,usersandresourceaccesswithinexistingRIsystems.

ResearchandDataInteroperability:Providestheresearchinfrastructureanddomainexpertviewintheworkprogrammewithfocusondatainteroperability.ThedefinitionofaDataInteroperabilityframeworkinEOSCisbasedontheFAIRprinciples-dataandservicesneedtobeFindable,Accessible,InteroperableandReusable.

www.eoscpilot.eu 4

Through-instantiationofmulti-infrastructure,multi-communitypilots.Services&SDs–deployedandvalidated=>maturity,scalability,andusabilityforafutureEOSC.


e-infrastructure gap analysis & interoperability architecture – T6.1

www.eoscpilot.eu 5

D6.1:e-infrastructuregapanalysis&D6.2:EOSCarchitecturedesignandvalidationprocedure• describestheframeworktobesettoallowtheinteroperabilitybetweenthee-infrastructuresandResearch

infrastructuresinvolvedinEOSC

PerformagapanalysisEstablishsynergiesandcollaborationsDefineproceduresandtestbedsDefinethearchitectureforinteroperatingvarioustypesofinfrastructuresintheEOSC

https://www.eoscpilot.eu/themes/wp6-interoperability/e-infrastructure-gap-analysis

https://www.eoscpilot.eu/themes/wp6-interoperability/e-infrastructure-gap-analysis

https://www.eoscpilot.eu/content/d62-eosc-architecture-design-and-validation-procedure

https://www.eoscpilot.eu/content/d62-eosc-architecture-design-and-validation-procedure


Risk1:LackofAAI

interoperability

Risk2:Lackorpoor

networkservices

Risk3:Lackoftrustbetweeninfras,brokentraceability,poor

security,lackofaccounting,poorprivacyRisk4:

E-infrastructures-lessresearchers

Risk5:Lackofcommonvocabulary,lackofglobalservicescatalogue

Risk6:Infrastructuresfall

behindoninteroperability

T6.1 : D 6.8 – Final EOSC architecture

D6.8,dueonM24

www.eoscpilot.eu 6

Risksandrecommendationsforeachgapandbridgeidentified

Recc.1:FollowAARCBlueprint

architecture&guidelines

Recc.2:Fosterthecollab.BetweenEOSCand

NRENs

Recc.3:Traceability,security,accounting,privacy

Recc.4:Buildamultidisciplinary

mutualisedspace

Recc.5:BuildEOSCcommonvocabulary&EOSCglobalservicescatalogue

Recc.6:Training&expertise

sharingforinfrastructuresstaff


EOSC Research & Data Interoperability – T6.2

Plan

Recommendations

Architecture

define

validate

Data int. demonstratorsAdoption

& testing

Data resources discovery

Research schemas markup

RDA metadata elements

PrinciplesScope|Direction

2017

2018

FAIR

FAIREDMImetadatacataloguesecosystem

AimstoestablishprinciplesanddevelopmechanismsthatenabletheEOSCtoprovideresearchanddatainteroperabilityacrossthediversityofexisting(andpotentialfuture)researchcommunities,researchinfrastructuresandotherresearchorganizations.

analysestheexistinginteroperationmechanismsfordata,softwarecomponents,workflows,usersandresourceaccesswithparticularattentiontotheuseofstandardsandtheirsyntacticandsemanticrepresentations.

providestheknowledgemanagementframework-thecontentdescriptions-consumedbytheservicesestablishedinWP5andtechnicalinteroperabilitydefinedintask6.1and6.3.

gathersrequirementsfromtheparticipatingRIsandsciencepartners

OrganizedfollowingtheFAIRprinciples,andaddresstheFindability,Accessibility,InteroperabilityandReusabilityofresearchassets.

Goal:Todemonstratehowtoensureavailabilityofscientificdataanddata-analysisservicestousersandservicesthroughacloudinfrastructureanddesignastakeholderdrivengovernanceframework

7www.eoscpilot.eu


“Develop a number of pilots that integrate services and infrastructures to demonstrate interoperability in a number of scientific domains”

setupdemonstratorstoshowinteroperabilityamonginfrastructuresandtofostertheadoptionofthesolutionsaccordingtotheFAIRprinciples,assessthematuritylevelofsolutionsforwhatregardsTRL,openness,scalability,usercommunityadoptionandsustainability.Validationregarding:

AAIrequirements-distributedauthorizationpolicies,supportformultipleauthenticationmethodsusingstandardsResourcebrokeringsolutions-high-levelresourcediscoverabilityandaddressabilityAccessibility-EOSClocal,Grid,HPCandCloudresources<-bymultiplecommunitiesviadifferentinterfacesDataaccessibility–throughpersonalresources,scientificportals,CLIInteroperability-underlyingdistributedstoragesystemswiththeEOSCplatformservicesTransparentnetworkingsolutions-connectingmultipleinfrastructuresandcommunities,spanningEuropeancountriesDatainteroperability-dataplacement,findabilityandaccessibilityinpilotsolutionsreusingcommoncomponents

www.eoscpilot.eu 8

Interoperabilitypilots–T6.3


Interoperability pilots: Activities

SupportthesetupoftheScienceDemonstratorpilots,followingtheirinteroperabilityrequirementsandmatchingthemagainwithavailableservicesandsolutionsSetupofdifferentpilotaddressingdifferentinteroperabilityaspects:

TransparentNetworking–PiCo2(PilotforCOnnectionbetweenCOmputingcenters)Grid&Cloudinteroperability–pilotdemonstratorforoneoftheHEPexperimentsAAI–throughthesetupofascopedinteroperabilitypilotaspartoftheWLCGAuthorizationWG&AARC&EOSCpilotcollaborationResourceBrokering&orchestration–leveragingINDIGO-DataCloudsolutionsDataaccessibility&interoperabilityofunderlyingstoragesystems–distributedOnedatadeployment

www.eoscpilot.eu 9


Interoperability pilots: Transparent Networking

PiCO2(PilotforCOnnectingCOmputingcenters)oneofthefirstinteroperabilitypilotsbetweengeneric,communityagnostic,infrastructures,especiallyTier-1(NationalHPC/HTCcentres),andTier-2(HPC/HTCregionalcentres).

Itsmainobjectiveistheautomationoffrequent,communityagnostic,dataflow(manylargefiles)andcodeexchangebetweenHPC(National,Regional)andHTC(national,grid)infrastructurestechnicalgroupshavebeensetup:

a)oneforbuildinganetworkofpeertopeerfederationsbetweeniRODSzones(datastorageservice),betweenTier1&Tier2,betweenTier2,andbetweenTier2andthegridb)oneforconnectingtheinfrastructureswithinaL3VPNnetworkandmonitoringtheperformanceofthenetworkbetweensitesc)oneforfacilitatingthemobilityanduseofcodesbetweendifferentmachines,usingcontainers,packagesforconfigurationmanagement,andnotebooks

www.eoscpilot.eu 10


Interoperability pilots: Transparent Networking

www.eoscpilot.eu 11

Currentparticipants:

GRICAD,Tier2HPC/HTC/Grid/cloud,Grenoble

MCIA,Tier2HPC/Grille/Cloud,Bordeaux

Plafrim,clusterINRIABordeaux,Tier2HPC

IDRIS,CNRSNationalComputingCenter,Tier1HPC

CC-IN2P3,IN2P3/CNRSComputingCenter,Tier1HTC/Grid/Cloud

RENATER,REN,France

France-Grille,EGINGI,France;Tier1Grid/Cloud

GRIF/CEA,Tier2,Saclay

DESY,Tier-2,Hamburg


Interoperability pilots: Grids & Clouds …

DynamicOnDemandAnalysisServiceaPaaStoolbuiltcombiningseveralsolutionsandproductsdevelopedbytheINDIGO-DataCloudextensivelytestedonadedicatedinteroperabilitytestbedundertheumbrellaoftheEOSCpilotprojectoriginallydesignedfortheCMSExperimentatLHC=>ThematicServicethatwillprovidemulti-disciplinarysolutionsintheEOSC-hubproject

Weaknessesidentifiedandrecommendations:FederatedAAI->crucialtohaveitasaEOSCprovidedservice.TransparentDataAccess:-scalablesolution(XrootD)-Amoregenericsolutionwouldbeabigplus.Resourcemonitoring–commonstrategyforcloudresourcesmonitoringPaaSOrchestration-improvementbothintheinterfaceandinthemanagementofIaaSranking.

www.eoscpilot.eu 12


Collaborationonthefieldofauthorizationandauthentication,policiesandrecommendationsregardingsolutionsdesign->https://twiki.cern.ch/twiki/bin/view/LCG/WLCGAuthorizationWGMainobjectives:

DesignandtestingofaWLCG Membership Management and Token Translation service,facilitatedbypilotprojectswiththesupportofAARCDefinitionofatoken-based authentication and authorization profilefor WLCG

www.eoscpilot.eu 13

Interoperability pilots: AAI

[email protected]

https://twiki.cern.ch/twiki/bin/view/LCG/WLCGAuthorizationWG

mailto:[email protected]?subject=


IAMinstancedeployed@INFN-CNAFsinceJanuary2018toshowcasemainfeaturesandintegrationcapabilitieshttps://wlcg-authz-wg.cloud.cnaf.infn.it/login

ThisdeploymentisbeingmigratedtoCERNinfrastructureforfurthervalidation&feedbackonRCAuth.euandCERNHRdatabaseintegrationRegistration&administrationmanagementfunctionality

www.eoscpilot.eu 14

Interoperability pilots: AAI

INDIGOIAM

https://wlcg-authz-wg.cloud.cnaf.infn.it/login

http://RCAuth.eu


PhotonandNeutronScienceDemonstratorInteroperabilitySolutions

InteroperabilitybetweencloudstorageandvirtualizedcomputeclustersintheEOSCdCachestorageevents(KafkaandServerSentEvents(SSE))RESTfulAPIbasedinteractionSingleSignOn(SSO)e.g.OpenIDConnect(OIDC)Securedelegationofread/writeaccessrightstostatelesscloudfunctions(usingMacaroons)

Cloudfunctionsasaservice(FaaS)usingdockerruntimesPreservingenvironments,versions,configurationsforfunctioncataloguesandusercodesEnhancedportabilityonfederatedcloudenvironmentsintheEOSCSamearchitectureforautomatedpipelinesalsousedforinteractiveworkflows(JupyterNotebooks)CI/CDfordockercontainers(scalingloadwithGitLabRunners)Seamlessscalingforheadlessapplications;graphicalinteractionneedsco-developmenttofacilitateintegrationofscientificapplicationsoncloudbasedinfrastructuresandplatforms

www.eoscpilot.eu 15

Flexiblepetascalestorage • QoS,ACLs • Storageevents

Workflows&scientificdataanalysis • UsingFaaS

CodeManagement • CI • Container

Cloudfunctions ● Interactive ● Automated

r/w data

containerized codes


ObjectivesInfrastructureinteroperability:facilitatethemostadequateinfrastructuresforthetreatmentofextensiveamountsofdata.Demonstratewithmulti-infrastructure,multi-communitypilots(sciencedemonstratorsfromWP3&WP4)ResearchandDataInteroperability:Data&servicestobefindable,accessible,interoperableandreusable(FAIR)Testbedsforinteroperability:PuttoworkScienceDemonstratorandlearnaboutinteroperabilityissuesandsolutions

KeyOutput:ThedesignofafutureEOSCbasedonfederatedinteroperableservicesmeetingtheneedsofthethematicresearchdomainsandwideruserbase.

WPOutputssofar:

• Gapanalysis• EOSCarchitecture• Reportsondata

interoperabilityandtestbeds• InitialandUpdated

Requirementsfortestbeds

Moretocome:

▪ FinalEOSCarchitecture▪ Finalreportondata

interoperability▪ FinalInteroperabilityTestbeds

report

www.eoscpilot.eu 16

Interoperability in EOSCpilot – Summary & Conclusions

TheEuropeanOpenScienceCloudforResearchpilotprojectisfundedbytheEuropeanCommission,DGResearch&Innovationundercontractno.739563www.eoscpilot.eu 17

Interoperability:Key element of the EOSCWP lead: Volker Beckmann (CNRS / IN2P3), T6.1 lead: Geneviève Romier, Eric Fede (CNRS / IN2P3) T6.2 lead: Rafael C. Jimenez Domenech (ELIXIR) T6.3 lead: Doina Cristina Duma (INFN)

Second EOSCpilot Stakeholders Forum · 2019. 1. 7. · The European Open Science Cloud for Research...

Documents

Transcript of Second EOSCpilot Stakeholders Forum · 2019. 1. 7. · The European Open Science Cloud for Research...