Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR...

PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY OF CALIFORNIA, BERKELEY

Connected Corridors: I-210 Pilot Integrated Corridor Management System

Core System High-Level Design

November 29, 2017 DRAFT

Partners for Advanced Transportation Technology works with researchers, practitioners, and industry to implement transportation research and innovation, including products and services that improve the efficiency, safety, and security of the transportation system.

I-210Pilot:CoreSystemHigh-LevelDesign

Thispageleftblankintentionally

TableofContents

1. Introduction ...................................................................................................................................1

1.1. PurposeofDocument ............................................................................................................. 1

1.2. RelationtoSystemsEngineeringProcess................................................................................ 1

1.3. IntendedAudience.................................................................................................................. 2

1.4. DocumentOrganization .......................................................................................................... 2

2. SystemPrimaryObjectivesandPurpose.........................................................................................4

2.1. ProjectGoalsandObjectives .................................................................................................. 5

2.2. TechnicalCapabilitiesSought.................................................................................................. 7

3. HighLevelDesignObjectives,Constraints,andPrinciples ...............................................................8

4. CoreSystemHighLevelDesign .....................................................................................................10

4.1. MajorComponents ............................................................................................................... 10

4.2. FieldElements ....................................................................................................................... 11

4.3. DataHub ............................................................................................................................... 12

4.4. DecisionSupport ................................................................................................................... 13

4.5. CorridorManagement .......................................................................................................... 14

4.6. Primaryprocessflow............................................................................................................. 17

5. DataHubDesign...........................................................................................................................18

5.1. DataSources ......................................................................................................................... 21

5.2. DataPipelines........................................................................................................................ 23

5.2.1. SensingDataPipeline .......................................................................................... 235.2.2. CaltransFreewayTrafficDataPipeline................................................................ 245.2.3. HeterogeneousDataPipeline.............................................................................. 255.2.4. HomogenousDataPipeline ................................................................................. 275.2.5. PipelineControl ................................................................................................... 275.2.6. PipelineStatusandLogging................................................................................. 295.2.7. CorridorManagementSystem-DecisionSupportSystem(CMS-DSS)Pipeline ... 305.2.8. CMSCommandStatusPipeline ........................................................................... 34

5.3. ExternalInterface/DataGateway.......................................................................................... 35

5.3.1. Subscriptionendpoints........................................................................................ 375.3.2. Request/Responseendpoints ............................................................................. 37

5.4. DataHubCommandGateway............................................................................................... 37

5.4.1. Conductor............................................................................................................ 38

5.4.2. Camel................................................................................................................... 395.4.3. ActiveMQWorkflowStatusTopic ....................................................................... 395.4.4. ActiveMQWorkflowTaskTopic .......................................................................... 395.4.5. Monitor ............................................................................................................... 39

6. DecisionSupportSystemDesign...................................................................................................39

6.1. DSShighleveldesign............................................................................................................. 40

6.2. ResponsePlanManagement................................................................................................. 41

6.3. RulesEngine .......................................................................................................................... 43

6.4. Modeling ............................................................................................................................... 43

6.4.1. Modelingtechniques........................................................................................... 43

1. INTRODUCTION

ThisConnectedCorridorsHighLevelDesigndocumentprovidesthehighlevelsystemarchitectureforthesystemtobedeployedontheI-210corridor.ThesystemarchitecturedescribedhereisadirectresultoftheConnectedCorridorsSystemRequirementsdocumentandtheworkdoneatUCBerkeleyintrafficmodelingandcontrol.Thisdocumentprovidesthesystemarchitecture,highleveldesignoftheprimarysubsystems,thedecisions,assumptions,constraints,andreasoningbehindthatarchitecture,andcriticalfunctionseachsubsystemprovidesforthesystem.

Thesystem,tobepilotedalongasectionoftheI-210corridorintheSanGabrielValleyareaofLosAngelesCounty,aimstoimproveoverallcorridorperformanceduringincidents,unscheduledevents,andplannedevents.Thisistobeachievedbymoreefficientlymanagingexistingsystemsandinfrastructures,promotingcross-jurisdictionaloperations,andusingmulti-modaltrafficanddemandmanagementstrategiesthatconsiderallrelevantmodesoftransportation.

1.1. PURPOSEOFDOCUMENT

Thisdocumentprovidesthehighleveldesign,servingastheidentificationoftheprimarysubsystemsandmajorcomponentsaswellasthebasisfortheselection,development,andintegrationoftheseintoasystemthatsatisfiesthesystemrequirementsasdefinedintheSystemsRequirementsDocument.ThishighleveldesignwillgovernthetechnologyplatformanddirectionoftheI-210PilotICMSystemandserveasthebasisforotherCaltrans-ledICMeffortsstatewide.

1.2. RELATIONTOSYSTEMSENGINEERINGPROCESS

ThedevelopmentofhighleveldesignispartofthesystemsengineeringprocessthattheFederalHighwayAdministration(FHWA)requiresbefollowedfordevelopingIntelligentTransportationSystem(ITS)projectswhenfederalfundsareinvolved.Whilenotrequiredforprojectsonlyusingstateorlocalfunds,useofthesystemsengineeringprocessisstillencouragedinsuchcases.

TheoverallsystemsengineeringprocessisillustratedinFigure1-1.DevelopinghighleveldesignrepresentsthenextstepoftheSystemDefinitionandDesignphaseofaproject(Phase2inthefigure)followingthecompletionoftheSystemRequirements.HighLevelDesignistypicallyderivedfromtherequirements.Theresultingdesignelementsareinturnusedtoinformandguidethemoredetaileddesignofthevarioussystemandsubsystemcomponents.

Figure1-1–SystemRequirementsSpecificationwithinSystemsEngineeringProcess

1.3. INTENDEDAUDIENCE

TheprimaryaudiencefortheSystemHighLevelDesigndocumentincludespersonnelresponsiblefordesigningandimplementingtheICMsystem.TheaudiencealsoincludesindividualsfromCaltransDistrict7,CaltransHeadquarters,andtheUniversityofCalifornia,Berkeley,taskedwithprojectmanagementduties.

1.4. DOCUMENTORGANIZATION

Theremainderofthisdocumentisorganizedasfollows:

• Section2summarizestheprimarysystemobjectivesidentifiedwithintheSystemRequirementsthatshapethesystemdesign.

• Section3presentstheprimaryguidingdesignprinciplesandbaseassumptionsthatshapethesystemdesign.

• Section4presentsthekeysystemdesigncomponentsandprimarydataflows.

• Section 5 presents the key system components of the Data Hub including data sources,interfaces/gatewaysandpipelines.

• Section6presentsthekeysystemcomponentsoftheDecisionSupportSystem(DSS)includingtherulesengine,modelinginterfaces,andresponseplangeneration.

• Section7presentskeysecuritydesignissuesandimplementationplans.

• Section8providesdesigninformationforthemessagingsystemsanddescribeshowinformationispassedbetweensubsystems.

• Section9describesthesysteminterfaces.

Inaddition,othersupportingdocumentsareavailableintheDocumentLibraryoftheI-210Pilotwebsiteathttp://ccdocs.berkeley.edu/content/document-library:

2. SYSTEMPRIMARYOBJECTIVESANDPURPOSE

TheoverridingpurposeoftheI-210PilotistoreducecongestionandimprovemobilityalongasectionoftheI-210corridorinLosAngelesCountythroughthecoordinatedmanagementofitsmajornetworks:theI-210freeway,keysurroundingarterials,andlocalandregionaltransitservices.Thegoalistoenableallcorridor“actors”—transportationsystemmanagersandoperators,controlsystems,vehicles,andtravelers—toworktogetherinanefficientandcoordinatedway.

TheseimprovementswillbeachievedbydevelopinganddeployingtheICMsystemdescribedinthishighleveldesigndocument.AttheheartoftheproposedsystemwillbeaDecisionSupportSystem(DSS)designedtohelpcorridorsystemoperatorsmanageincidents,unscheduledevents,andplannedeventsmoreeffectively.Thissystemwilluseinformationgatheredfrommonitoringsystemsandprovidedbypredictiveanalyticaltoolstoestimatecurrentandnear-futureoperationalperformance.Theinformationwillbeusedtodeveloprecommendedcoursesofactiontoaddressproblemscausedbyidentifiedincidentsandevents.Morespecifically,thissystemisexpectedto:

• Improvereal-timemonitoringoftravelconditionswithinthecorridor• Enableoperatorstobettercharacterizetravelpatternswithinthecorridorandacrosssystems• Providepredictivetrafficandsystemperformancecapabilities• Beabletoevaluatealternativesystemmanagementstrategiesandrecommenddesiredcourses

ofactioninresponsetoplannedevents,unscheduledevents,andincidents• Improvedecision-makingbytransportationsystemmanagers• Improvecollaborationamongagenciesoperatingtransportationsystemsinthecorridor• Improvetheutilizationofexistinginfrastructureandsystems• Moreefficientlyusesparecapacitytoaddressnon-recurringcongestion• Reducedelaysandtraveltimesalongfreewaysandarterials• Improvetraveltimereliability• Helpreducethenumberofaccidentsoccurringalongthecorridor• Reducetheperiodduringwhichthecongestionresultingfromanincidentoreventaffects

corridoroperations• Reducegreenhousegasemissions• Generatehighertravelersatisfactionrates• IncreasetheoveralllivabilityofcommunitiesinandaroundtheI-210corridor

WhiledevelopmentoftheproposedsystemisunderthefinancialsponsorshipofCaltransHeadquarters,thesystemwillbedevelopedprimarilybythelocaltransportationagenciesthathaveagreedtoparticipateinitsoperation,incoordinationwithPATH.Projectactivitieswillincludethedesign,development,installation,testing,andoperationofvariouscomponentsoftheICMsystem,aswellasthedevelopmentofinterfaceswithexistingmonitoringandcontrolsystems.Forexample,theICMCoreSystemwillbeinterfacedwithtrafficmanagementsystemsownedbyCaltrans,suchastheAdvancedTrafficManagementSystem(ATMS).

2.1. PROJECTGOALSANDOBJECTIVES

TheprimarygoaloftheI-210PilotICMprojectistoimproveoverallcorridorperformancealongasectionoftheI-210corridor.Thistranslatesintothefollowingspecificgoals:

1. Improveoperationalsituationalawareness2. Promotecollaborationamongcorridorstakeholders3. Improveresponsetoincidentsandevents4. Improvetravelreliability5. Improveoverallcorridormobility6. Empowertravelerstomakeinformedtraveldecisions7. Facilitatemulti-modalmovementsacrosstheregion8. Promotetransportationsustainabilitybyreducingimpactsontheenvironment9. Improvecorridorsafety

Foreachofthesegoals,Table2-1furtheridentifiesthemainoperationalobjectives.Manyoftheobjectivesaresimilartothoseoftraditionaltransportationimprovementprojects.Many,however,alsofocusonimplementingmorecomprehensivetravelandsystemstatusmonitoringsystems,improvedoperationalforecasting,improvedinformationdisseminationtotravelers,enhanceddata-sharingcapabilities,demandmanagementapproaches,andimprovedcollaborationamongtransportationsystemoperators.

Table2-1–ICMSystemGoalsandObjectives

Goals Objectives

1.Improvesituationalawareness

• Establishminimumrequirementsfordatacollectiontosupportsystemmanagement• Increasedatacollectionopportunitiesfromarterialsandlocalroads• Improvethecollectionofreal-timeoperationaldatafromnon-traditionalsources,suchasprobevehicles

• Developacomprehensivecorridorinformationaldatabasecoveringallrelevanttravelmodeswithinthecorridor

• Improvethequality,accuracy,andvalidationprocessofcollecteddata• Increasetheabilitytoestimatetraveldemandpatternsinamulti-modalenvironment• Improvetheabilitytoforecastnear-futuretravelconditionsbasedonknownincidents,roadconditions,weather,andlocalevents

• Developperformancemetricsconsideringallavailabletravelmodes

2.Promotecollaborationamongcorridorstakeholders

• Strengthenexistingcommunicationchannelsamongthecorridor’sinstitutionalstakeholders• Exploreopportunitiesfornewcommunicationlinksbetweencorridorstakeholders• Improvecooperationandcollaborationamongcorridorstakeholders• Developregional/jointoperationsconcepts• Identifynewmethodsofcollaboration• Extendcorridorperformancemetricstothenetworklevel• Investigatenewtypesofagreementsbetweenparticipatingagencies

Goals Objectives

3.Improveresponsetoincidentsandunexpectedevents

• Reducethetimeneededtoidentifytheexistenceofanincidentorunexpectedevent• Reducethetimeneededtorespondtoincidentsorunscheduledevents• Enhancethecoordinationofactivitiesamongfirstresponders,trafficmanagementagencies,andtransitagenciestominimizeimpactsonsystemoperations

• Reducethetimeneededtoimplementcontrolactionstoaddresscongestionresultingfromanincidentorevent

• Reducethetimeneededtodisseminaterecommendeddetoursaroundanincidentorevent

4.Improvetravelreliability

• Improvetraveltimepredictabilityalongthecorridor• Reducetheimpactsofincidentsandeventsonnetworkoperations• Improveincident/eventnotificationforfirstrespondersandnetworkoperators• Improveincident/eventnotificationtotravelersandfleetoperators• Providetravelersandcommercialvehicleoperatorsaffectedbyanincidentoreventanenhancedabilitytoseekalternateroutesormodeoftransportation

5.Improveoverallcorridormobility

• Reducedelaysincurredbytravelers• Reducetheimpactsofincidentsandeventsonnetworkoperations• Efficientlyusesparecapacityalongcorridorroadwaystoplannecessarydetoursaroundincidentsorevents

• Promotestrategiestoinducedesirabletraveldemandpatterns• Coordinatethemanagementoffreewayandarterialbottlenecks• Promoteincreasesinvehicleoccupancy• Promoteincreasesintransitridership

6.Empowersystemuserstomakeinformedtraveldecisions

• Improvethedisseminationofreal-time,multi-modaltravelinformation• Enhancetheuseofinfrastructure-basedinformationaldevices(freewayCMS,arterialtrailblazersigns,kiosks,etc.)toprovideen-routeinformationtotravelers

• Enableindividualstoreceivetravelinformationonconnectedmobiledevices• Makearchivedhistoricaldataavailableto511servicesandinformationserviceproviders• Supportthedisseminationoftravelinformationby511servicesandthird-partyproviders

7.Facilitateregionalmulti-modalmovements

• Promotetheintegrationofcommuterrailandbusserviceswithcorridoroperations• Facilitatetransfersacrossmodesduringincidentsandevents• Providerelevantregionaltravelinformationtotravelers• Directtravelerstopark-and-ridefacilitieswithavailablespaces

8.Promotetransportationsustainability

• Reducefuelconsumption• Reducevehicleemissions• Identifyfinanciallysustainablesolutionsforlong-termsystemoperationsandmaintenance• Encouragetheuseoftransit,walking,andbicyclingwhereappropriate• Supportlocallypreferredalternativescompatiblewithcorridorobjectives• Developandimplementperformancemetricsreflectingenvironmentalgoals

9.Improvecorridorsafety

• Reducecollisionrates• Reducetheseverityofcollisions• Reducethenumberoffatalities• Reducetheimpactsofprimaryandsecondaryincidentsonnetworkoperationsthroughimprovedincidentmanagement

• Improvesafetyforbicycles,pedestrians,andtransit

2.2. TECHNICALCAPABILITIESSOUGHT

Tohelpmanagetravelactivitieswithinthecorridorduringincidents,unscheduledevents,andplannedevents,theprojectisseekingthefollowingtechnicalcapabilitiestosupportthegoalsandobjectivesidentifiedinSection2.1:

• Gatherandarchiveinformationcharacterizingtrafficoperations,transitoperations,andtheoperationalstatusofrelevantcontroldeviceswithintheI-210corridor.

• IdentifyunusualtravelconditionsontheI-210freewayornearbyarterialsbasedonmonitoringdataprovidedbyvarioustraffic,transit,andtravelmonitoringsystems.

• Identifysituationsinwhichanincidentonroadwaysortransitfacilitiessignificantlyaffectstravelconditionswithinthecorridor.

• Providecorridor-wideoperationalevaluationstotrafficmanagers,transitdispatchers,andotherrelevantsystemmanagers,includingprojectedassessmentsofnear-futuresystemoperationsundercurrentandalternatecontrolscenarios.

• Identifyrecommendeddetoursaroundincidentsorroutesleadingtothesiteofanevent,consideringobservedtravelconditionswithinthecorridor.Dependingontheneed,andfinalsystemcapabilities,specificdetoursmayberecommendedformotoristsandtransitvehicles.

• Identifyrecommendedsignaltimingplanstouseatsignalizedintersectionstoimproveand/oraccommodatetrafficflowinfluxduringincidentsandeventsandimproveoverallcorridormobility.

• IdentifyrecommendedrampmeteringratestouseonindividualI-210freewayon-rampsandconnectorstomaintainoverallcorridormobility.

• IdentifymessagestopostonavailablefreewayandarterialCMSstoinformmotoristsofincidentsandevents.

• ProvideguidancetomotoristsontheI-210freewayandsurroundingarterialsusingavailablefreewayCMSs,arterialCMSs,andarterialdynamictrailblazersignsregardingwhichdetourtotaketogoaroundanincidentorwhichroutetofollowtoreachthesiteofanevent.

• Provideinformationtomotoristsabouttheavailabilityofparkingandtransitservicestohelptravelersmakealternatemode-choicedecisions.

• Provideuniformtrafficmanagementstrategiesacrossjurisdictionalboundariesduringincidentsandevents.

• Provideinformationtomotoriststhroughthird-partyoutlets,suchas511services,navigationapplicationproviders,etc.

3. HIGHLEVELDESIGNOBJECTIVES,CONSTRAINTS,ANDPRINCIPLES

Thecoresystemdesignisgovernedbyafewkeyprimarydesignobjectivesdictatedbytheprojectandtherequirements:

• Realtimeoperation–Thesystemmustoperateinanearrealtimeenvironment.Thetimebetweenanincidentoccurringandaresponseplanimplementationisacriticaltimeperiod.Responsetimesshouldbedictatedprimarilybythetimetonotificationandconfirmationoftheincident,andthetimetofullyimplementadecision.Thetimerequiredbythesystemtoprocessinformationshouldbeminimizedsoastohavethemaximumpositiveimpactoncorridoroperations.

• Speedtodecision–Thereisasignificantamountofdataprocessingrequiredtomaintainbothoperatorawarenessandtoensurethemodelingwithinthedecisionsupportsystemisupdatedwiththelatestavailableinformation.Dataprocessinganddecisionprocessingtimeshouldbeminimized.

• Qualityofdecision–Thequalityoftheresponseplansisadirectresultofthetrafficestimationquality,trafficpredictionquality,dataprocessingtime,dataquality,andrulesusedtogenerateresponseplans.

• Abilitytomeasureoutcomes–Theremustbeahighlevelofconfidenceinthesystemoutcomes,andtheseoutcomesmustbecontinuouslymeasuredandmonitoredwithprocessesinplacetoimprovethesystemseffectivenessovertime.

• Incrementaldeployment–Thesystemmustbeabletobedeployedincrementally,addingnewcapabilitiesovertime.

• Systemflexibility–Thesystemmustbebuilttobeflexible,asitisintendedasapilotthatisadjustedandmodifiedasexperienceisgainedwithoperationsandgrowasnewcapabilitiesareadded.Inaddition,thisflexibilitywillbekeytofutureimplementationsinothercorridorswithdifferentintegrationneedsanddifferentrequirements.Itisalsoexpectedthattransportationitselfwillbechangingradicallyoverthelifetimeofthesystem,soflexibilityiskeytoitslongtermviability.

• Secureoperations–Asthesystemhasthecapabilitytorequestchangestotrafficcontrolsacrossalarge,complexurbancorridor,securityofthesystemiscriticaltoitssuccess.

Coresystemdesignislimitedbythefollowingconstraints:

• AbilitytobeoperatedandmaintainedbyCaltranswithoutsignificantlicensingcosts–Thesystemmustbedesignedwithopensourcecomponentsasmuchaspossible.Itisnotdesiredtohavesignificantlicensingcostsrequiredforsubsequentdeploymentsforfuturecorridorsoroverlongperiodsoftime.ItisintendedthatCaltranswillbecapableofdeploying,maintaining,andoperatingthisandfuturedeployments.

• Limitedtimetodeploy–Thedeploymentscheduleisaggressive,creatingtheneedtoensuresignificantreusabilitywithinthedesignsothatmultipledatasourcescanbeincorporatedintothesystemwithoutdesigningnewdatapipelinesforeachone.

• Constraineddevelopmentresources–Caltransprovidedafixedamountoffundingsothesystemdesignmustnotexceedourresourceswithcomplexityandadditionalfeatures.

Thesecoredesignobjectivesandconstraintsareimplementedwiththefollowingdesignprinciples:

• Clouddevelopment,deployment,andoperations–Inordertoachievemaximumflexibilityinbothsystemdevelopmentandfutureconfigurations,minimizeresourceutilizationindevelopmentanddeployment,andminimizedeploymenttime,thecoresystemisdesignedfor100%cloudoperationswithinanAmazonAWScloudenvironment.

• Thesystemisapilotsystemanditexpectedthataswelearnfromthedeploymentofthesystemchangeswillbeneededandrecommended.

• ThesystemshallbecapableofbeingsupportedbyCaltransinternalresources.• Decisionsproducedintheformofresponseplansshallbebasedoncurrentcorridorinformation

withlimiteddelaysininformationprocessing.• Datamaintainedwithinthecoresystemwillbelimitedtothedatarequiredforsystem

operation.LongtermdatastorageshallbeafunctionofPeMS.• Thesystemwillbedesignedforfuturegrowth,incrementalimprovements,future

transportationsystemchanges,changesintransportationitself,geographicchanges,andnewandupdatedinformationsourcesandneeds.

• Easeofdeployment• Easeofmaintenance• Portabilityofthesystemdesignandcomponentstoothercorridors• Highlydecoupleddesignforflexibility,scalability,redundancy,andreliability• Useavailabledatastandardswheneverpossible• Standardizedexternalinterfacesforeaseofportabilityandinclusionofnewsubsystems• Maintenanceofaseparationofconcernsbetweendatamanagement,decisionsupport,and

controlfunctions.• Designwithstate,regional,andlocallayersforfuturescalabilityandstandardizeddeployment

acrossthestate.

4. CORESYSTEMHIGHLEVELDESIGN

ThecoresystemhighleveldesignisillustratedinFigure4-1.Thishighleveldesignprovidesclearseparationbetweenexternalsystemsprovidingdataandreceivingcontrolrequests(green),thedatahubreceivingandprocessinginformationfromthoseexternalsystems(red),decisionsupportprovidingresponseplansforincidents(blue),andthecorridormanagementsystemprovidinguserinterface,responseplanselectionandapproval,andsendingofcontrolrequeststoexternalsystems(purple).

Figure4-1CoreSystemHighLevelDesign

4.1. MAJORCOMPONENTS

AsshowninFigure4-1,theICMCoreSystemiscomposedofthreemajorsubsystems:theDataHub,theDecisionSupportSystem,andtheCorridorManagementsubsystem.Thesesubsystems,plustheexternalfieldelementstheCoreSysteminteractswith,aresummarizedinthefollowingtable(color-codedtomatchFigure4-1)anddescribedindetailinthefollowingpages.

Component Description

DataHub Providesforreceiptandprocessingofallcorridordataandamethodofcommunicationamongthethreesubsystems,usingacommonsetofdatadefinitions.Providesoperationaldatapersistenceandretrieval.

DecisionSupport Providescurrenttrafficstate,responseplandevelopment,andresponseplanperformancepredictiontoprovideoneormorerecommendedresponseplansforincidentsandevents.

CorridorManagement Providestheprimaryuserinterfaceforsystemoperators,amethodforusersto

monitorcurrentcorridorandcorridorassetstates,interactwiththeDecisionSupportSystem,review,evaluate,select,andapproveresponseplans,andinteractwithexternalsystems,particularlyforexecutionandmanagementofresponseplans.

FieldElements Representexternalprovidersandconsumersofinformationandrequestsforactionsuchasintersectionsignals,rampmeters,corridorsensing,andotherelements,usuallythroughrespectivetransportationmanagementsystemsandTMCs

ThedesignfortheI-210Pilotwillnothavealayeredgeographicapproach,butthedesignisintendedtoallowsuchanapproachforfutureuse.Therearethreegeographiclayerstothedesign:state,regional,andlocal.TheintentisthattheDataHuboperatesonaregionallevel(Caltransdistrict),withthepotentialforservingmultiplecorridorswithinaregion.DatawouldbeconsolidatedandaggregatedatastatelevelformultipleregionsviaPeMS,tosupportlargerscalearchiving,businessintelligenceanalysis,andcontinuedimprovementofcorridorandICMsystemcapabilitiesatthestatelevel.TheDecisionSupportandCorridorManagementcomponentsoperateonalocallevel,withinasinglecorridor.Withstateconsolidationofdata,regionaldatacommunication,andlocaldecisionsupportandresponseplanexecution,amethodofsharinginformationbetweencorridorsisenabled,whileensuringlocaljurisdictioncontroloftrafficeventandincidentresponse.

4.2. FIELDELEMENTS

GreenelementsinFigure4-1representthevariouscorridorfielddatasourcesandfieldelementcontrols,includingvariousstate,regional,andlocaltransportationsystems,regionaltransportationdatanetworks,andprivateinformationproviders.

• Ontheleftsideofthediagram,theseelementsrepresentthevariousdatasourcesforinformationconsumedandprocessedbytheICMsystem.

• Ontherightside,theserepresentthevariouscontrolinterfacestoexecuteresponseplanelementsselectedbyoperatorsoftheICMsystem.

ThesystemsthatwillbeprovidingthedatawithintheI-210corridorinclude:

Table4-1–FIELDSYSTEMS

Source InformationTypePasadenaTMC* PasadenaintersectionsignalsArcadiaTMC* Arcadiaintersectionsignals

LACountyintersectionsignalsDuarteintersectionsignals

CountyTMC*

MonroviaintersectionsignalsFreewayloopsensing(traffic)Rampmeters

CaltransATMS**

Dynamicmessagesystem

IncidentinformationCaltransLCS* LaneclosuresCaltransTSMSS* IntersectionsignalTrailblazerSystem* Dynamicmessagesystemforrerouting

LocalvideoRIITSCaltransvideoEnvironmentalsensingCalPolyODSTransit

HSRLaneClosure* CitylaneclosuresNextBus Goldlinetransit

*Indicatesfieldsystemthatacceptscontrolmessages

**Caltrans’ATMSsystemacceptscontrolmessagesandprovideslimitedICMcoresystemcontrol

Futurefieldsystemsmayincludeprobedataproviders(GPSlocation/speed),additionaltransitinformation,parkinginformation,andothers.

4.3. DATAHUB

DatafromeachofthefieldsourcesarereceivedbytheICMDataHub,representedinredinFigure4-1.TheprimaryfunctionsoftheDataHubare:

• Receivedatafromfieldelementsviaexistingcorridortrafficmanagementcentersandregionaldatanetworks:VariousdatareceiversreceivedataandprepareitforprocessingbytheICMsystem.Thesereceiversaregenerallyexpectedtobebuiltforthespecificinterfacesdefinedbyeachfielddatasource,bothintransmissionmethod(RESTorSOAPwebservices,socket-basedstreaming,file-based,orothers)andtheintendedinitialpathrequiredforsystemdataprocessing.ThepreferredmethodofinformationtransferforthesystemistheTrafficManagementDataDictionary(TMDD)standarddevelopedbytheInstituteforTrafficEngineers(ITE),currentlyatversion3.03d(withcertainmodifications).

• Processdatareceivedfromfieldelements:Datafromfieldelementsmustbevalidatedforcompletenessanddataqualitypriortousebydownstreamsystemcomponents.Withsuchavarietyofdatasources,oftenforthesametypeoffieldelements,datamustbetransformedintoacommonsetofdatasemantics.TheDataHubprocessesallincomingdataintoastandardizedformat,TMDDfortransportationassets,GTFSfortransitinformation,orothersdependinguponthedatabeingreceived,withacommonsetofdatadefinitionsaswell(suchasasinglenamingstandardforallstreetswithinthecorridor).However,itiscriticaltonotethatthistransformationintoastandardizedformatforprocessingwithinthedatahub,whileitmaintainsthedatastructureswithinTMDD,doesnotgenerallymaintainanXMLdataformatandinternalcommunicationswithinthedatahub(andtheDSS)donotuseSOAPprotocols.ThedatahubandDSSuseJMSprotocols,eitherwithinanActiveMQorKafkamessagingsystem.

Additionalprocessingisalsocompletedwithinthedatahub,eitherforspecificmetrics,calculatedparameters,orpredictiveanalytics.

• Datamessagingandcommunications:Dataprovidedtodownstreamsystems,aswellasinternalsystemcommand,control,andstatusdata—indeed,alldatawithintheICMsystem—ismadeavailableviaaninternaldatabus.Themethodofdatatransportisviadatamessagingtechnologies,namelyActiveMQJMSmessagingorKafkadatamessagingsystems.Thespecificmessagetechnologyisbasedonthetype,size,frequency,andmessagepersistencerequirementsofthedata,dataproducer,anddataconsumers.Exceptionstothesemethodswithinthedatahubarelimited,andaregenerallyusedforlargeblockbulkdataarchiving/datatransportbetweenpersistencestores.InordertoaccommodatemultipleCMSvendors,andtoprovidecommunicationbetweenthedatahub’sKafkamessagingsystemandtheDSSActiveMQmessagingsystem,anApacheCamelbasedinterfacebetweenthedatahubandtheDSSandCMSsystemsisincludedwithinthedatahub.ThisinterfaceprovidestheabilitytoprovideSOAPandRESTbasedinterfacesasnecessary,andtranslationbetweenmessagingsystems.

• Datapersistence:TheDataHubalsoprovidesdatapersistencecapabilities,allowingforpersistenceofrawandprocesseddata,systemcommand/control/status,andothersysteminformationinacentralrepository.Thisdatapersistenceisbrokenintodifferenttimelayers,includinglivereal-timedata,youngdata(0-90days),aggregateddata,andarchiveddata.ItincludesleveragingstateEDWandBIresourcesasacomponentofthedatapersistencecapabilities,specificallyasthesingledatastorefornon-operationaldata(dataolderthan90days).Sincedatapersistencewithinthedatahubislimitedtooperationaluses,andthearchitectureisbasedonapatternofcoreservicesconnectedbymessaging,multipledatapersistencetechnologiesareutilizedspecifictotheneedsofthesystemandthedatabeingstored.Largetimeseriescollectionsofdata,suchassensordata,ismaintainedinaCassandradatabase;largerelationalstructureswithsmallerupdatefrequenciesaremaintainedwithinaPostgresdatabase,andMongoDBisusedwithindatatranslationpipelineswhenmultiplesourcesprovidethesametypeofinformationindifferingformatsandimplementations.NOTE:MongoDBmaybeusedindevelopmentandearlyversionsofthesysteminsteadofPostgresforsomeoftherelationaldatastores.

4.4. DECISIONSUPPORT

TheDecisionSupportSystem,portrayedinblueinFigure4-1,providesthefollowingcapabilities:

• Corridortrafficstatedetermination:TheDecisionSupportSystemprovidescorridortrafficstateestimation,providingbothgeospatialtrafficstateinformationandtrafficstatemetrics.Separatearterialandfreewaytrafficmodelsareusedandmergedtoprovidefullcorridortrafficstateatalltimes.

• Corridortrafficstateprediction:TheDecisionSupportSystemprovidescorridortrafficstatepredictionsforusebycorridoroperatorsandforpredictionofincidentandeventresponseplanperformance.Acommercialsimulationengine(TSSAimsun)isusedtoprovidetraffic

predictions.Trafficpredictionsaregeneratedusingthetrafficestimatedstateasaninitialtrafficstate,andcurrentcorridorassetstatefromthedatahub.

• Responseplandevelopmentandevaluation:TheDecisionSupportSystem,uponnotificationofaconfirmedincident,willusearules-basedapproachalongwithtrafficstateestimationandpredictioncapabilitiestodevelop,evaluate,andrankresponseplansforusebycorridoroperators.

4.5. CORRIDORMANAGEMENT

TheCorridorManagementsubsystem,portrayedinpurpleinFigure4-1,shallprovidethefollowingprimarycapabilities:

• Presentationofcorridorstate:Displayofcorridorstate,includingcorridorassets(signals,ramps,cameras,dynamicmessaging,transit,etc.),trafficestimatedstate,trafficvisualization(cameraoutput),humanassets,physicalassets(vehicles,responsecrews).Assetstateincludescurrentoperationalcapabilities,currentoperatingstate,currentfunctionalstate(operating,failed,degradedstates),timeavailability,controllability,etc.

• Controlofcorridorassets:TheCorridorManagementsubsystemprovidesthecapabilitytocontrolcorridorassets,takingresponseplansapprovedforexecutionandexecutingeachoftheindividualresponseplanelementsbysendingcommandstotheappropriatestate,regional,andlocalsystemsandentities.TheCorridorManagementsubsystemdoesnotdirectlycontrolcorridorassets,butonlysendscommandstothestate,regional,orlocalsystemsthatdirectlycommandcorridorassets.

• Presentationofresponseplans:TheCorridorManagementsubsystempresentseachresponseplandevelopedbytheDecisionSupportSystem,displayinginmeaningfulwaysthevariousresponseplanelements,howtheywillchangefromthecurrentcorridorstate,howtheychangeduringresponseplanexecution,whatthepredictedoutcomesareforeachresponseplan,whattheactualoutcomesareforaresponseplan,aswellasthevariousmetricsonhowtheresponseplansareevaluatedandtheireffectivenessmeasured.Thispresentationisexpectedtobeinmultipledomains,includinggeospatial,tabular,comparative,graphical,andothersolution-specificmethods.

• Responseplanlifecyclemanagement:Eachresponseplandevelopedbythesystemhasaspecificlifecyclethatincludes:

1. Initiation2. Development3. Evaluation4. Selection5. Approval

6. Execution7. Monitoring8. Close9. Post-incident/eventanalysis

TheCorridorManagementsubsystemprovidesamanagementinterfaceforthislifecycle:

Initiation TheCorridorManagementsubsystempresentsincidentandeventinformationreceivedfromtheDataHubforreview,edit,andconfirmationbythecorridoroperators.

Development TheCorridorManagementsystemprovidesdisplayofstatusinformationreceivedfromtheDecisionSupportSystem(DSS)regardingitsdecisiontodevelopasetofresponseplanstoanincidentorevent,aswellasthedevelopmentofthoseresponseplans.Inaddition,itcansendcommandstotheDecisionSupportSystemtoinitiatespecificDSSfunctions,suchasmockincidentevaluation,manualresponseplandevelopmentorevaluation,orotherfunctionsrequiredfortheICMsystem.

Evaluation TheCorridorManagementsubsystemreceivestheresultsofresponseplanevaluationandtrafficstatepredictionanddisplaysthoseresults,alongwiththeresponseplans,tothecorridoruser.Displayshallincludeindividualplananalysis,aswellasrankingandcomparativeanalysisofresponseplans.

Selection TheCorridorManagementsubsystemprovidestheuserthecapabilitytoselectaspecificresponseplanforsubsequentapprovalandimplementation.

Modification TheCorridorManagementsubsystemshallprovideuserstheabilitytomakeminormodificationstoresponseplans.NOTE:Initialversionsofthesoftwarewillnothavethiscapability.

Approval TheCorridorManagementsubsystemprovidesaselectedresponseplantothecorridorstakeholdersatthestate,regional,andlocallevelsforreviewandapproval.Theseapprovalsshallincludeminormodificationstotheresponseplanandmayincludereevaluationandresubmittalforapproval.

Execution TheCorridorManagementsubsystemisthesolecomponentinthesystemcapableofsendingcommandstothevarioussystemsincontrolofcorridorassetsforindividualresponseplancomponentexecution.Itshallhavedisplaysthatallowcontrolandmonitoringofcorridorassetsviathestate,regional,andlocaltrafficmanagementsystemsandassets.

Monitoring Giventhecapabilitytodisplayassetstate,managecorridorassets,andexecuteresponseplanelements,theCorridorManagementsubsystemshallprovideanintegrateddisplayofresponseplanmonitoringoncetheexecutioncommandsaresenttothefieldsystems.Monitoringshallincludedisplayingwhencommandshavebeenexecutedandarecompleteatthefieldassetlevel,alertinguserswhencommandsorassetshavefailed,identifyinganddisplayingvariancesbetweenexpectedandactualtrafficstate,andtheabilitytotakeactionintheeventofresponseplanelementfailureforanyspecificcorridorasset.

Close TheCorridorManagementsubsystemshallbeabletoreturncorridorassetstotheirnormalstateoncetraffichasreturnedtoanormalstate.

Post-incident/eventanalysis

TheCorridorManagementsubsystemshalldisplayacorridorpost-eventanalysisforeachincidentorevent.TheanalysisshallincludeinformationfromtheCorridorManagementsubsystemitself,especiallywithregardtothecorridorassetresponseplanexecution,responseplanexecutionissuesand

failures,trafficanalysis(expectedvs.actual),andothersystemperformancemeasures.Thepost-eventanalysisisprimarilylimitedtoinformationusedbytheICMsystemanditscomponents,aswellasmetricscalculatedpostincident/eventregardingresponseplanandsystemperformance.Itisnotintendedtobeanexhaustiveanalysiswithextensiveadditionalmodelinganalysisand“what-if”scenarios.

• ICMSystemmanagement:TheCorridorManagementsubsystemprovidesmanagementcapabilitiesfortheICMsystem,including:

Rulesmanagement TheCorridorManagementsubsystemprovidesauserinterfaceforthemanagementoftherulesusedwithintherulesengine.Thiswillincludetheabilitytocreatebasicrulesandrulesets,editrulesandrulesets,archiverulesandrulesets,provideconfigurationmanagementforrulesandrulesets,makerulesactive,executerulesandrulesets,andimportrulesandrulesets.Generally,rulesthatcanbeeditedwillbesimplerrulesandrulessets,asmorecomplexrulesarelikelytorequireprofessionaldevelopmenteffortandintegration.NOTE:Theinitialimplementationofthesystemwillnotincludethiscapability.

Responseplanmanagement

TheCorridorManagementsubsystemprovidesuserstheabilitytoupdatetheavailableresponseplansandresponseplanelements.Thisincludesadding,archiving,updating,andactivationordeactivationofresponseplansorresponseplanelementsforusewithinthecorridor.

Systemmonitoring TheCorridorManagementsubsystemprovidesuserstheabilitytomonitortheICMsystemandcomponents.Systemslogs,alerts,statusinformation,controlandexecutionofsystemfunctionsshallbeaccomplishedbytheCorridorManagementsubsystem.

Security TheCorridorManagementsubsystemprovidesuserauthenticationandauthorizationfortheCorridorManagementsubsystemandallfunctionsavailablewithintheCorridorManagementsubsystemthataffectotherICMsystemfunctions,includingDecisionSupportandtheDataHub.ThisdoesnotincludeuserauthenticationandauthorizationofindividualcomponentssuchasCassandra,Postgres,ormessaging;rather,itprovidestheseservicesforthefunctionsthatusethosecomponents.

Configuration TheCorridorManagementsubsystemprovidesmethodsandinterfaceforallowinguserstochangeanyandallconfigurationelementswithintheICMsystem.Aswithsecurity,thisdoesnotincludeconfigurationofindividualsystemcomponentssuchasCassandraandPostgres,butratherforthefunctionsthatusethesecomponents.NOTE:Theinitialimplementationofthesystemwillhaveminimalcapabilitiesforthisfunction.

4.6. PRIMARYPROCESSFLOW

Figure4-2providestheprimarysystemworkflowillustratingthebasicintegrationbetweeneachofthesubsystems.Whilethisillustrationisnotintendedtocaptureallofthedetailsoftheinteractionsbetweensystems,andtheprocesslifetimesarenotintendedtobeaccuratewithinthissequencediagram,itdoesprovidethebasicsequenceofeventsandprimarycommunicationchannelsfortheprimaryworkflowwithinthesystem,aresponsetoanincidentonthecorridor.Secondaryactionssuchaspersistence,logging,communicationdialogs,andothersarenotdepictedinthisdiagram.Alsonotethat,asdescribedinFigure4-1,allcommunicationbetweentheDSSandCMSisviatheDataHub’sdatabus,althoughthespecificsofthecommunicationchannelarenotillustratedinFigure4-2.

Figure4-2PrimarySystemIncidentFlow(Subsystem)

Thebasicworkflowdescribedinthisdiagramisasfollows:

• Anincidentoccurswithinthecorridor.Thediagramshowstheincidentinformationbeingprocessedthroughthedatahub.

• TheincidentisconfirmedbyanoperatorwithintheCMSorCaltran’sATMS.Theinitialpilotsystemwillalwaysuseanoperatortoconfirmanincident.

• TheincidentinformationispassedtotheDSS’sresponseplandevelopmentcomponent.• Theresponseplandevelopmentcomponentrequestsapredictionoftheimpactoftheincident

onthecorridor.• Thepredictionengineusesthecurrentestimatedtrafficstateandcorridortrafficstateto

initializeandrunapredictionwithnoresponseplanexecution(“donothing”prediction).• Theresponseplandevelopmentcomponentusesthe“donothing”predictionandtherules

enginetodetermineifaresponseplanshouldbedeveloped.Iftheresultisnoresponseplanshouldbedeveloped,executionoftheworkflowwouldendwithnotificationtotheCMSofthatresult.

• Iftheresultisthataresponseplanshouldbedeveloped,theresponseplandevelopmentcomponentagainusestherulesengine,resultsofthe“do-nothing”prediction,currentcorridorstate,andasetoffixedresponseplancomponentstodevelopalimitednumberofresponseplansforevaluation.Itsubmitsthoseresponseplanstothepredictionengine.

• Thepredictionenginerunsapredictionforeachresponseplan,alongwithanother“donothing”prediction”basedoncurrentcorridorconditions.Itprovidestheresultsofthosepredictionstotheresponseplandevelopmentcomponent.

• Theresponseplandevelopmentcomponentusestheresultsofeachprediction,currentcorridorstate,andtherulesenginetoevaluate,rank,andselectarecommendedresponseplan.

• TheresultsofresponseplanpredictionsandevaluationisprovidedtotheCMSforoperatorselection.

• TheCMSprovidestheresultstoanoperator,whoselectstheresponseplantobeimplementedandobtainsapprovalfortheresponseplanimplementation.

• Iftheresponseplanisapproved,theCMSexecutestheresponseplanbysubmittingC2CcommandstotheTMCsandothersystemsrequiredtoexecutecommandstoindividualcorridorassets.

• TheindividualTMCsandothersystemsexecutingtheresponseplancommandsreportthesuccessorfailuretoexecutethedesiredresponseplanelements,andcontinuetoreportcorridorassetstate.

Uponcompletionofthisprimaryworkflow,furtherprocessesinvolvedinresponseplanlifecyclemanagementwilloccur,suchasresponseplanevaluation,responseplangenerationtoadjusttocurrentconditions,responseplancancellation,andresponseplancloseout.

5. DATAHUBDESIGN

Thedatahubhasfiveprimaryroleswithinthesystem:

1. Receiveinformationfromexternalprovidersofcorridorstate2. Processinformationreceivedfromprovidersofcorridorstate,evaluatingthedataforquality,

providingcommonmetricsandanalysisofthedatareceived,conductingpredictiveanalyticstosupportestimationandpredictionwithintheDSS,andstandardizingtheformatandcontentfordownstreamconsumptionbytheDSSandCMSsystems.

3. Persistinformationreceived,resultsofprocessing,andoverallsystemstateandcommandinformation.Persistoperationallyrequiredinformationlocallyandsendtoarchiveolder

informationforlongertermstorage.Retrieveinformationstoredupondemand(operationaldataforimmediateretrieval,archiveddataforlater,scheduledretrieval).

4. Secureinformationreceived,processedandstored.5. ProvidedatacommunicationsbetweentheprimarysystemsoftheICMcoresystem.

Inordertofulfilltheseroles,thedatahubprovidesaseriesofdatapipelinestoreceiveinformation,processtheinformation,persisttheinformation,andsecuretheinformation.ThepipelinesuseKafkaandActiveMQmessagingsystemstotransmitinformationthatcanbetappedbypersistenceworkerstopersistinformationtothevariousdatastoreswithinthedatahub,andretrievetheinformationandplacetheinformationbackonthemessagingsystemsfordownstreamconsumption.Externalinterfaces,usingApacheCamel,provideinformationandcommunicationbetweenthedatahub,theDSS,andCMSsystems.

Figure5-1DataHubHighLevelDesign

Thisdesignconsistsofindividualservicesconnectedviamessaging.Itisahighlydecoupleddesignbasedonagroupofindependentservicesconnectedbytwomessagingplatforms.Figure5-1providesageneralizeddiagramthatillustratesthreebasictypesofdatapipelines.Sensordataisgenerallyahighfrequency,largerdatavolumepipelinethathandlesalargernumberofsmallermessages.Theheterogeneousdatapipelinetypehandleslargersizedmessagesatlowermessageprocessingfrequencies.Heterogeneousdataischaracterizedbymultiplesourcesprovidingthesametypeofdatainvaryingdegreesofformatandcontentdifferences.Thehomogeneousdatapipelinesaresimilartotheheterogeneousdatapipelines,butgenerallyarebuiltforasingledatasourcethatprovidesallofa

specifictypeofdata.Notethattheheterogeneousandhomogenousdatapipelinesprovidesomedataintothesensordatapipelinewhensensordataispartofthedatareceivedwithinthesepipelines.

Adescriptionofthedifferenttypesofcomponentsisprovidedbelow:

Reader–Readersarethebeginningofthedatapipeline.Theirroleissimple–toconnectandgatherthedatafromadatasourceusingtheprotocolandformatnativetothatdatasource,andtoplacetheinformationwithinadatamessagingchannelfordownstreamprocessing.Ingeneral,itdoeslittleornoprocessingofthedata,withanyprocessinglimitedtosimpleparsingoflargebatchesintoindividualmessages.

Spark–SparkisanApacheSparkinstancethatprovideshighspeed,highvolume,realtimedataprocessingforsensordata.Itsroleistoprovidedatatransformation,dataquality,dataprocessing,andpredictiveanalyticsforsensordata.Itreceivesdatafromreaders,processesthatdata,andplacesthedataintoKafkamessagetopicsforusebydownstreamprocesses.

Cassandra–CassandraisanApacheCassandrainstancethatprovidesstoragefortimeseriesdata,primarilysensingdata.Cassandraprovideshighspeed,highvolume,clusteredstorageforsensingdata.Thisdataislimitedtooperationaldatastorageandisnotintendedasalong-termdatastore.

Postgres–Postgresisanopensourcerelationaldatabaseinstancethatprovidesstorageforrelationalandgeospatialdata,suchasintersectionsignalplans,rampmeterplans,roadnetworks,assetinventories,organizationalinformation.Itisanoperationaldatastoreandnotintendedasalongtermdatastore.

Persistenceworkers–PersistenceworkersprovidebothstorageandretrievalservicesforbothPostgresandCassandradatastores.Persistenceworkerslistentoassignedmessagetopicsandqueues,bothKafkaandActiveMQ,andstoretheinformationontheappropriatedatastore.Requestscanbesenttoapersistenceworkerviaacommandmessagechanneltoretrievedataandplacethatdataonaspecifiedmessagetopicorqueue.

Processors–Processorsareusedinthehomogeneousandheterogeneousdatapipelinestoprocessdatareceivedfromthevariousdatasources.Processorsprovidedatatransformation,quality,anddataprocessingservicesfordatareceivedfromthesesources.

Mongo–MongoisaninstanceofaMongoDBdatabaseandisusedwithintheprocesstotransformdatareceivedonaheterogeneousdatapipeline.ReadersstorethedatareceivedfromasourceintoMongoDB.ProcessorsthanqueryMongofortheinformationrequiredforprocessinganddownstreamprocessesincludingdecisionsupportandcorridormanagementsystems.MongoDBprovidesanidealplatformforstoringinformationretrievedinanativeformatandfastqueryingofthisinformationwithminimalmaintenancerequiredwhensourceformattingischanged.

Datahublogger(DHlogger)–ThedatahubloggerisaninstanceofGreylogthatcollectsandindexesallloggingforeachindividualcomponentwithinthedatahub,andlistenstomessagingtocapturecontrolandprocessloggingmessages.Itsroleistocapturesystemstateanderrormessagesforsystemstatusandtroubleshootingneeds.

Datahubcontrolgateway(DHcontrolgateway)–ThedatahubcontrolgatewayisanAPIgatewaythatprovidesroutingandmanagementofcontrolandstatusmessagesforthedatahub.ItsroleistoreceivemessagesroutedfromtheinterfacesfortheDSSandCMS,aswellasinternaldatahubcontrolchannels,androutethosetotheappropriatechannelsforthereceivingprocess.Thisprovidesencapsulationofindividualservicesandensuresthatnewservices,servicechanges,andmessagesystemmodificationscanbeaccomplishedwithoutimpactingothersystems,andiskeytothedecouplingofthesystemservicesfortheentireICMsystem.ThisgatewayusesanimplementationofApacheCamel.

Externalinterface/Datagateway–Theexternalinterface/datagatewayencapsulatesthefunctionsofthedatahub,providingasortofswitchboardforusebytheCMSandDSSsystems.CMSandDSSsystemsarenotrequiredtoknowtheinternalsofthedatahubmessaging,butinsteadrelyontheexternalinterface/datagatewaytoprovideprotocolspecificcommunicationportsthatareappropriatefortheCMSandDSSandthentoroutemessagestoandfromthedatahubforthem.Thisexternalinterface/datagatewayusesApachecameltoprovidebothroutingandtranslationservices.Internaldatahubmessages(KafkaorActiveMQ)aretranslatedintotheprotocolsusedbytheCMSandDSS(ActiveMQ,RESTservices,orSOAPservices).CommonTMDDandGTFSmessageformatsareusedinmostcommunications,sonotranslationofdataformatsisrequiredbythegateway.

PeMS–PeMSisastatesystemthatcurrentlyprovidesstatetransportationdataservices.PeMSwillprovidelongtermstorageforallsystemdata,archivingservices,andreportingservicesfornon-realtimedataandreportingneeds.

BusinessIntelligence–BIserviceswillbeprovidedbytheCaltrans’OBIEEimplementation.

5.1. DATASOURCES

ThefollowingdatasourceshavebeendefinedfortheI-210ICMproject:

Source InformationType

System Vendor Product C2CTMDD?

Pasadena Intersectionsignal

PasadenaTMC

McCain Transparity Planned

Duarte Intersectionsignal

CountyTMC KimleyHorn

KITS Planned

Monrovia Intersectionsignal

KITS Planned

Arcadia Intersectionsignal

ArcadiaTMC Transcore Transuite Planned

CaltransFWTraffic Loopsensing CaltransATMS Parsons Custom N

CaltransFWRamps Rampmeters CaltransATMS Parsons Custom N

CaltransFWCMS DMS CaltransATMS Parsons Custom N

CorridorTrailblazer DMS Notyetidentified

Custom Planned

CaltransIntersections Intersectionsignal

TSMSS Transcore Transuite Planned

Bluetoothtraffic1 Traveltime Vendor Iteris/? Unknown

Bluetoothtraffic2 Traveltime CountyTMC ? Unknown

Environmentalsensing Environmental ? CalPoly ODS ?

RIITSTransit Transit ? CalPoly ODS ?

RIITSVideo* Video RIITS

CaltransVideo* Video viaRIITS

CaltransFWLaneclosure Lanestatus LCS

CityLaneclosure Lanestatus StateHSRsystem

StateofCA Custom N

LACounty Intersectionsignal

KITS Planned

CHPCAD Incident CAD manualinputbyoperator

Caltransincident Incident CaltransATMS Parsons Custom Planned

Goldlinetransit Transit NextBus NextBus

*Videoisnotpassedthroughorstoredwithinthedatahub.

5.2. DATAPIPELINES

5.2.1. SENSINGDATAPIPELINE

ThesensingdatapipelinedesignisprovidedinFigure5-2.Inthisillustration,thedifferentsensorfeedscanbeseenontheleft.Thisillustrationshowsthereaderreceivingdatafromadatasource,placingthatdataonaKafkatopic,SparkconsumingthatdataandprocessingthedatautilizingtheMLLibandStreamingSparklibraries,andthenplacingtheprocesseddataonaKafkatopic.DecisionSupportandCorridorManagementsystemscanaccessthatprocesseddatadirectlyviathedatahubdatainterfaces.SparkalsostoresresultsontheCassandracluster.Persistenceworkerscanbeusedtoretrieveprocesseddatawhenrequested,anddatathatisstoredinPostgres,suchassensorinventories,isavailabletoSparkifneededforprocessingthatmayrequiresuchdata.

Figure5-2SensingDataPipelineDesign

ThedatasourceslistedinSection5.1thatwillbeprocessedusingthistypeofpipelineinclude:

Source InformationType

System DataType C2CTMDD?

Pasadena Intersectionsignal

PasadenaTMC

SignalSensing Planned

Duarte Intersectionsignal

CountyTMC SignalSensing Planned

Monrovia Intersectionsignal

Arcadia Intersectionsignal

ArcadiaTMC SignalSensing Planned

CaltransFWTraffic Loopsensing CaltransATMS

FWsensing N

CaltransFWRamps Rampmeters CaltransATMS

Rampsensing N

CaltransIntersections Intersectionsignal

TSMSS SignalSensing Planned

Bluetoothtraffic1 Traveltime Vendor Traveltime Unknown

Bluetoothtraffic2 Traveltime CountyTMC Traveltime Unknown

RIITSEnvironmentalsensing

Environmental ? EnvironmentalSensing

RIITSTransit Transit ? Transitlocation(future)

LACounty Intersectionsignal

Goldlinetransit Transit NextBus Transitlocation(future)

5.2.2. CALTRANSFREEWAYTRAFFICDATAPIPELINE

TheCaltransFreewaytrafficdatapipelineisasensingdatapipeline.Itconsistsofallofthestandardsensingpipelinecomponents:

• Kafkatopicsfordatatransmission• Readerforretrievingdata• Sparkfordatatransformation,qualitychecks,machinelearning• Cassandraforpersistence

Figure5-3illustratesthisdatapipeline.

Figure5-3FreewaySensingDataPipeline(PeMS)

PeMS30secondfilesareaflatformatdatafile.DataisretrievedbythereaderviaFTP.Thereaderutilizesaconfigurableschedulerthatwillpollfordatafiles.Whenoneormoredatafilesisavailable,thereaderprocessesthefilesintimeorder,withtheoldestfilefirst.Eachrowwithinthefilecontainstheinformationforasinglesensoratasingletimestep.Allsensorsforasingletimesteparecontainedwithinthefile.Thetimeofeachreadingislocatedinthefilename.ThereaderparseseachrowofthefileandcreatesasingleKafkamessageforeachsensorandtimestep,appendingthetimesteptotheinformationpassedwithinthemessage.Uponcompletionofthefileprocessing,astatusmessageforthefile,includingthetimestepandtheKafkapositionofeachmessageissenttoastatustopic.

Uponreceivingthestatusmessage,Sparkprocessesthemessages,completingthefollowingsteps:

1. ThemessagesaretransformedintoaTMDDdatastructure(Note:thisisnotaTMDDformat,andKafkamessagesareJSON,notXML,anditisnotpassedviaaSOAPprotocol.However,thedatastructureisidenticaltoaTMDDstructure.)

2. Aninventorycheckiscompletedforeachtimestep.3. Aflowbalancecheckiscompletedforallsensorson4and6hourrollingtimewindowsanda24

hourstatictimewindow.4. Ademandpredictioniscompletedforthenexthourforeachfreewayramp.

SparkstorestheresultofitsprocessinginCassandraandpassesthesensinginformationtotheDH_pems_processedtopic.

5.2.3. HETEROGENEOUSDATAPIPELINE

TheheterogeneousdatapipelinedesignisprovidedinFigure5-4.Inthisillustration,thedifferentintersectionsignalfeedscanbeseenontheleft.

ThisillustrationshowsthereaderreceivingdatafromadatasourceandinsertingthatdatainarawmessageformatintoMongoDB.Thereaderretrievesdatafromthedatasourceusingthedatasource’s

nativeprotocolsandformats.ItisdesiredthattheformatbeTMDDformat,andtheprotocolisastandardSOAPprotocol,eitherarequestreceiveorsubscriptionmethodasspecifiedintheTMDDspecification.ThereadertransformsthemessagefromaSOAP/XMLformattoaJSONformatpriortoinsertingintoMongo.

Amessageissentfromthereadertotheprocessortoinformtheprocessoroftheavailabilityofnewdata.

TheprocessorthenqueriesMongotoobtainadesireddocumentwiththespecificattributesrequiredtomakeaTMDDmessagecontainingastandardizedsetofinformationrequiredfordownstreamprocessing.Theprocessoralsoutilizesastandarddictionaryofdatavaluessothatdatafromdifferentsourcesthatmaybedifferentinspecificdefinition,butrepresentthesamevaluearestandardizedfordownstreamconsumption.Forexample,ifonesourceabbreviatesboulevardasBlvd,andaseconddoesnotabbreviatetheword,theprocessorwillstandardizetoasingledefinitionthatcanbeunderstoodbythedownstreamprocesses.Theprocessoralsoconductsdataqualitychecksandanyotherprocessingrequired.

OncetheTMDDtransformation,qualitychecks,andanyprocessingiscompleted,theprocessorconstructsthefinalTMDDstructuredJSONmessageandplacesitontheoutgoingtopic.Persistenceworkerslisteningtothetopicpersisttheinformationforlateranalysisorreplay,andtheCMSandDSSsystemsmayconsumethemessagedirectlyfromtheprocesseddatatopic.

Forintersectionsthatprovideintersectionsensing,theprocessorwillretrievethesensinginformationandsenditviaaseparatetopicforfurtherprocessingwithinthesensing/Sparkstream.

Figure5-4HeterogeneousDataPipeline

5.2.4. HOMOGENOUSDATAPIPELINE

ThehomogenousdatapipelinedesignisprovidedinFigure5-5.Inthisillustration,arampmeterinventoryandstatesourceisusedastheexamplefeedontheleft.Thehomogenousdatapipelineisidenticaltotheheterogeneousdatapipelinewithonlyonedifference.Homogenousdatapipelinesareusedwhenthereisonlyasingledatasourceforthetypeofinformationbeingprocessed.Whenthisoccurs,thereisnoneedtostorethedocumentsreceivedina“schema-less”database,sincetherearenotmultipledatastandardsandimplementationstoaccommodate.Becauseofthis,thehomogenousdatasourceremovestheMongoDBinstallationfromthepipeline,andthereaderandprocessordirectlycommunicateforanydatatransformation,qualitychecks,orprocessingrequired.

Figure5-5HomogenousDataPipeline

5.2.5. PIPELINECONTROL

Allpipelines,regardlessofpipelinetype,areprovidedwiththesamebasecontrolset.Thecontrolsavailableforeachpipelineinclude:

1. Startpipelineprocessing2. Stoppipelineprocessing3. Pipelinestatus4. Replaydata

Allpipelinecontrolrequestsarehandledbythedatahubcontrolgateway.Requestsreceivedbythegatewayareroutedtoanappropriateendpoint.

Startpipelineprocessing–thestartpipelineprocessingwillberoutedviathedatahubcontrolgatewaytotheappropriatereaderwithinapipeline.Therequestwillresultinthesepossibleoutcomes:

1. Ifthepipelineiscurrentlynotstarted,thereaderwillbeginprocessingdataandinsertingitintothepipeline.Thedataprocessedwillbebasedontheconfigurationofthespecificreader.

2. Ifthepipelineisalreadystarted,anerrorwillbereturnedfromthereaderindicatingtheprocessisalreadystarted.

3. Ifthepipelinecannotbestartedforanyreason,anerrorwillbereturned,fromthereaderifpossible,orfromthedatahubcontrolgatewayifnomessageisreceivedfromthereaderorifnoreaderinstanceexists.

4. Anothererror,suchaspipelinedoesnotexistorthattherequestisnotproperlyformatted.

Stoppipelineprocessing–thestoppipelineprocessingwillberoutedviathedatahubcontrolgatewaytotheappropriatereaderwithinapipeline.Therequestwillresultinthesepossibleoutcomes:

1. Ifthepipelineiscurrentlyrunning,thereaderwillstopprocessingdataandwillcompleteinsertionofanycurrentlyprocessingdataintothepipeline.

2. Ifthepipelineisnotcurrentlyrunning,anerrorwillbereturnedfromthereaderindicatingtheprocessisalreadystopped.

3. Ifthepipelinecannotbestoppedforanyreason,anerrorwillbereturned,fromthereaderifpossible,orfromthedatahubcontrolgatewayifnomessageisreceivedfromthereader.

4. Anothererror,suchaspipelinedoesnotexistorthattherequestisnotproperlyformatted.

Pipelinestatus–thepipelinestatusrequestisarequestforasubscriptiontopipelinestatusmessagesthatareproducedatalltimesbyeachprocessingpointwithinapipeline,includingreaders,spark,processors,messagingsystems,andworkers.Thisincludestimesforwhichthepipelineisnotcurrentlyrunning,butthecomponentsarerunning(forinstancewhenthepipelineiscurrentlystopped,butavailableforprocessing).Allpipelinecomponentsprovideataminimumaheartbeatindicatingtheyarecurrentlyupandrunning,evenifnodataisbeingprocessed.

Foreachpipelinestatusrequest,thedatahubcontrolgatewaywillprovideachannelforasubscriptiontoakafkatopiccontainingallstatusmessagesforaspecificpipeline.Pipelinestatusrequestsshallalwaysincludewithintheresponsethetopicidentifier,andmostrecentkafkamessageoffset.Requestersmaythenconnecttothekafkachannelprovidedbythedatahubcontrolgatewayandsetupaconsumerforstatusmessages.

Thefollowingresponsesarepossiblewhenrequestingapipelinestatus:

1. Aresponsewithakafkatopicidentifierandaddress,alongwiththecurrentkafkamessageoffsetandcurrenttime.

2. Ifthestatusmessagesarenotcurrentlyavailable,anerrorwillbereturned.3. Anothererror,suchaspipelinedoesnotexistorthattherequestisnotproperlyformatted.

Figure5-6StreamingPrimaryControlLayer

5.2.6. PIPELINESTATUSANDLOGGING

Allpipelinecomponentsprovideloggingandstatusinformation,regardlessofpipelinetypeorcomponent.LoggingincludestheloggingattheEC2orapplicationlevelthatisprovidedtotheEC2logsattheOSlevelandispublishedtotheGrayloginstancewithinthedatahub.StatusincludesstatusmessagespublishedtostatusKafkatopics,againcollectedbythedatahubGrayloginstance.

Loggingshallbeconfigurableandsetoncomponentstartandshallincludeataminimumthefollowinglogginglevels:

1. All2. Debug3. Fatal4. Error5. Warn6. Info

Logginglevelwillbesetuponcomponentinitializationwhenitisstarted.Ingeneral,duringnormaloperation,thefollowinglevelsshallreport–Error,Fatal,Warn,andInfo.TheinformationgeneratedbyloggingshallonlybeavailablethrougheitherinspectionoftheEC2instancewheretheyweregeneratedandviathedatahubGrayloginstance.

StatusinformationisproducedbythecomponentsandpublishedtoapipelinestatusKafkatopic.Thisinformationisavailabletoexternalconsumersviathedatahubcontrolgateway(usingadatahubstatusrequest)orthedatahubGrayloginstance.Datapipelinestatusmessagesshallbeprovidedforthefollowingataminimum:

1. Pipelinecomponentheartbeat–astatusmessageindicatingthecomponentisrunningsentataregularinterval.

2. Pipelineerrormessage–astatusmessagesentindicatingthatanerrorhasoccurred.Generally,thiswillincludeFatalorErrorlogmessages.

3. Pipelinewarningmessage–astatusmessagesentprovidingawarningindicatingpotentialissuesinoperation.Generally,thiswillincludeallWarnloggingmessages.

4. Pipelineprocessingmessage–thisincludesmessagesindicatingprocessingofasetofdata,oftenatatimesteporfilelevel.Forinstance,inaTMDDsubscription,aprocessingmessagewouldbesetfromeachpipelinecomponentindicatingwhentheinformationfromamessageresultingfromthesubscriptionhasbeenprocessed.Itisnotguaranteedthatdifferentpipelineprocesseswillprocessinformationatthesamegranularity,however,eachstatusmessagewillprovidethecorridorentityidentifiersandtheoriginalmessageidentifierfromtheoriginatingsystemthatstartedthedataprocessing.

Pipelinecomponentsmayincludeotherstatusmessagesasneeded.

Figure5-7Status/LoggingLayer

5.2.7. CORRIDORMANAGEMENTSYSTEM-DECISIONSUPPORTSYSTEM(CMS-DSS)PIPELINE

TheCorridorManagementSystem–DecisionSupportSystem(CMS-DSS)PipelineprovidesthecommunicationbetweentheDecisionSupportSystem,DataHubandtotheCorridorManagementSystem:

• Trafficestimationresults,includinggeospatialinformationandtrafficmetrics• Trafficprediction,includinggeospatialinformationandtrafficmetrics• Responseplansandresponseplanevaluations• DSSStatus• CMSStatus

• DSSControl• Trafficestimationandpredictionscenarioinformation

Inaddition,thepipelineprovidespersistencewithinthedatahubofallDSS–CMScommunications.ThepipelinedoesnotincludeanycommunicationsfromtheCorridorManagementSystemthatdonotinvolvetheDecisionSupportSystem.

AllCMS-DSSpipelinessupportTMDDcompliantSOAPdialogsinaccordancewiththeConnectedCorridorsDataCommunicationSpecification.SOAPendpointsarepresentattheCMSDataHubGatewaytosupporttheseconversationsandareexposedtotheCMS.ThedatafortheseSOAPinterfacesarecommunicatedtotheDSSviatheDataHubanditsKafkaandActiveMQmessagingsystem.AllcommandrequestsandresponsesarealsoprovidedtotheDataHubControlgatewayforanydatahubcontrolactionsrequired.WhileonlytobeprovidedasnecessarytosupportthedifferentCMSvendors,REST,ActiveMQmessaging,andKafkamessagingmayalsobeprovidedasinterfacestotheCMSinsteadofSOAPinterfaces.

AllCMS-DSSpipelinesarealwaysonpipelines.Forfutureuse,datahubswillsupportmultipleCMSandDSSinstances,soallindividualpipelinesarecreatedforeachCMS/DSSpair.AllmessagesbetweenCMSandDSSinstancesshallbecommunicatedthroughonlypipelinesdesignatedforthatspecificCMS-DSSpairandshallincludewithinthemessagesthesourceandtargetsystemidentifiers.

Figure5-8providesanillustrationoftheCMS–DSSpipeline,anditsassociatedprimarydatatopics,including:

• Incidents• ResponsePlans• DecisionResults• StatusRequest• StatusResponse• EstimationRequest• EstimationMetrics• EstimationResults• EstimationNetwork• EstimationScenario• PredictionRequest• PredictionResult• PredictionMetrics• PredictionNetwork• PredictionScenario

Figure5-8CorridorManagement-DecisionSupportSystemPipeline

5.2.7.1. Incident

TheincidentpipelineprovidesconfirmedincidentinformationfromtheCMStotheDSS.ThispipelinemaystartasaRESTrequest,Kafkamessage,ActiveMQmessage,orSOAPincident.AlsonotethatthispipelinerepresentsaTMDDeventclassdialogsetwithbothrequests,responses,andsubscriptions.Subscriptionsarethepreferredmethodofcommunication.TheincidentpipelineincludesanincidentsubscriptionrequesttotheCMS,aswellasthecorrespondingsubscriptionresponses.IncidentrequestsareoriginatedfromtheCMSDataGateway.TheresponsesareprovidedtotheDSSviathedatahubasActiveMQmessages.

5.2.7.2. ResponsePlan

TheresponseplanpipelineprovidesresponseplanstotheCMS.AsthereisnoTMDDdialogforresponseplans,thisisaConnectedCorridorscustomcommunication.Responseplansareprovidedasan

ActiveMQmessageviatheCMSDataGateway.ResponseplansareprovidedinaccordancewiththeConnectedCorridorsDataSpecification,andincludetheresponseplanelements,event/incidentindexidentifier,andassociatedperformanceindicatorsandmetrics.

5.2.7.3. DecisionResult

ThedecisionresultpipelineprovidestheresultsfromtheDSS,providingtheindexoftheselecteddecision,andthekeyvaluepairsforeachresponseplanindicatorandscoreelements.ThedecisionresultisnotaTMDDdialog,andisavailableviaActiveMQmessageviatheCMSDataGateway.

5.2.7.4. StatusRequest

ThestatusrequestisacommunicationchannelforrequestingDSSanddatahubstatus.ItisanActiveMQqueue/message.Requestsmustincludeanidentifierfortherequestingsystem.Aresponsewillincludeanaddressforreceivingstatusmessages.Statusrequestsareessentiallyasubscriptionrequest,withtheresponsesimplyalocationforsubscribingtoacontinuousstreamofstatusmessages.

5.2.7.5. Status

ThedecisionresultpipelineprovidestheresultsfromtheDSS,providingtheindexoftheselecteddecision,andthekeyvaluepairsforeachresponseplanindicatorandscoreelements.ThedecisionresultisnotaTMDDdialog,andisavailableviaActiveMQmessageviatheCMSDataGateway.DatahubandCMSstatusmessagesareprovidedonthesametopicspecifictotheCMS-DSSpair.

5.2.7.6. EstimationRequest

TheestimationrequestpipelineprovidesamethodfortheCMStorequestanestimation.Ifanestimationiscurrentlyrunning,theresponsetotherequestshallindicatetheaddresswhereestimationresultsarepublished.Iftheestimationisnotcurrentlyrunning,theDSSshallstartanestimationandtheresponseshallincludeamessageindicatingtheDSSisbeginninganestimation,andtheaddresswhereestimationresultsarepublished.

5.2.7.7. EstimationResult

Theestimationresultpipelineprovidestherawresultsofanestimationrequest,includingforfreewaystheestimateddensity,flow,andspeedforeachnetworklinkwithinthefreewayestimationnetwork,andforarterials,theestimateddensityforeachnetworklinkwithinthearterialestimationnetwork.

5.2.7.8. EstimationMetrics

Theestimationmetricspipelineprovidesestimationmetricresults,includingtraveltimeandcurrentdelayestimatesforthefreewayandarterialnetworks.

5.2.7.9. EstimationNetwork

TheestimationnetworkpipelineprovidesthecurrentestimationnetworkinaTMDDformat.ThispipelinesupportsTMDDmessaging.

5.2.7.10. EstimationScenario

Theestimationscenariopipelineprovidesthecurrentestimationscenariodatastructure(arterialandfreeway).Itisnotexpectedthatthispipelinewillbeavailableintheinitialsystemdeployment.

5.2.7.11. PredictionRequest

ThepredictionrequestpipelineprovidesamethodfortheCMStorequestaprediction.Theresponsetotherequestshallindicatetheaddresswherepredictionresultsarepublished.Intheinitialversionofthesoftware,predictionsshallonlybeprovidedeitherasaresponsetoanincidentaspartoftheDSSprocess,orshallbeprovidedasareplaytoapreviousprediction.

5.2.7.12. PredictionResult

Thepredictionresultpipelineprovidestherawresultsofapredictionrequest.TODO:determinewhatfromAimsunwillbeprovided.

5.2.7.13. PredictionMetrics

Thepredictionmetricspipelineprovidespredictionmetricresults,includedtraveltimeandcurrentdelaypredictions.

5.2.7.14. PredictionNetwork

ThepredictionnetworkpipelineprovidesthecurrentpredictionnetworkinaTMDDformat.ThispipelinesupportsTMDDmessaging.

5.2.7.15. PredictionScenario

ThepredictionscenariopipelineprovidesthecurrentAimsunpredictionfile.Itisnotexpectedthatthispipelinewillbeavailableintheinitialsystemdeployment.

5.2.8. CMSCOMMANDSTATUSPIPELINE

CommunicationsbetweentheCMSandexternalsystemsorcentersarecapturedbytheDataHub.TheyarealsoprovidedtotheCMScontrolgatewaysothatthedatahubmayrespondtocommandsissuedbytheCMSsystemtoexternalsystems.Figure5-9providesanillustrationoftheCMSCommandStatuspipeline.

Figure5-9CMSCommandStatusPipeline

ThisCMSCommandStatuspipelineisasimpleimplementationofanActiveMQtopicthatisprovidedacopyofallCMScommunicationswithanyexternalcenterorsystem.AllsuchcommunicationsareprovidedviaActiveMQmessagecontainingatextrepresentationofanymessagesentorreceived.Messagesarekeyedwiththeexternalsystemorcentercommunicatedwith,aswellaseithersendorreceivetoindicatewhetherthemessagewassenttotheexternalcenterorsystem(send)orreceivedfromtheexternalcenterorsystem(receive)aswellasthetimeofmessagesentorreceipt,asappropriate.AllmessagesaresavedviapersistenceworkertoaPostgresdatabase.TheDSSdatahubcontrolgatewayalsosubscribestothedatahubCMScommandstatustopicandmayrespondwithspecificcontrolactionswithinthedatahub.

Initially,thestoreforcommandswillbeGraylog,notPostgres.Graylogprovidesauserinterfaceandqueryenginethatallowsuserstointerpretandreviewsystemprocessesthroughthecapturedcommandstatusmessages.

5.3. EXTERNALINTERFACE/DATAGATEWAY

Thedatagatewayprovidesanexternalinterfacetothedatahubforthecorridormanagementsystemandthedecisionsupportsystem.Thepurposeofthisdatagatewayis:

• Providefortwo-waycommunicationbetweentheDataHubandtheDecisionSupportSystem• Providefortwo-waycommunicationbetweentheDataHubandtheCorridorManagement

System• EncapsulatethefunctionsofthedatahubfromotherICMcomponentsystems• Provideasecureinterfacetothedatahub• AllowforthechangefromKafkaorActiveMQprotocolsusedwithinthedatahubtoRESTor

SOAPbasedservicesaswellasKafkaorActiveMQthatmayberequiredbyotherICMcomponentsystems

• AllowforaswitchboardlikefunctionalitythatcanbeconfiguredfordifferentCorridorManagementSystemprovidersorDecisionSupportSystems

• Provideformultipleoutputchannelstopublishinformationtomultipledownstreamconsumers

TheDataHubDataGatewayusesApacheCameltoprovidetheseservices.TwoprimarydesignpatternsareusedasshowninFigure5-10,firstforcommunicationtoexternalActiveMQbrokers,andsecondtoexternalSOAPorRESTwebservicesendpoints.

Figure5-10DataHubDataGateway–ActiveMQandWebServicesDesignPatterns

EachofthetwodesignsusesApachecamelandimplementsaKafkaconsumertoretrievemessagesfromadefinedKafkatopic.Eachalsoprovidesforaprocessorfortransformationandroutingofmessages.ForpassingdatausinganActiveMQtopicorqueuehostedbyanexternalbroker,Camel’sSedacomponentisusedforasynchronousmessagingandmessageflowregulation.ForpassingdataviaSOAPorRESTwebservices,anappropriateSOAPorRESTendpointisprovidedforcommunicationwithanexternalSOAPorRESTendpoint.

NOTE:AllinternaldatacommunicationswithinthedatahubareconductedviaKafkaorActiveMQdatamessagingandJSONformatting.NativedatacommunicationsareimplementedusingTMDDorotherappropriatestandardformatswhenavailable.WhilethesestandardsmaynotimplementJSONformatting,themessageswithinthedatahubthatcontainthisinformationareJSONformatted,retainingthedatastructuresandrelationshipswithintheapplicablestandard.Whenexternallyexposed,thesedatamessagesareserializedbacktotheappropriatedataformatdictatedbythestandardused(TMDDorother).

5.3.1. SUBSCRIPTIONENDPOINTS

Subscribe/Notifypattern(http://www.enterpriseintegrationpatterns.com/patterns/conversation/SubscribeNotify.html)-usealeasetoendifnorenew/continuationmessagereceived?

5.3.2. REQUEST/RESPONSEENDPOINTS

BothActiveMQandKafka–CMSgetsconnectionviacamel–camelmaintainssubscriptiontotopic/queue,ActiveMQusesaLastImageSubscriptionRecoveryPolicysubscriptionrecoverypolicy.Getmostrecentmessage.Kafka–getmostrecentmessage

5.4. DATAHUBCOMMANDGATEWAY

ThedatahubcommandgatewayisaninternaldatahubcomponentthatmanagesallcommandcommunicationandcoordinationactivitiesofthedatahubandorchestratescommunicationandcontrolofallcoreICMsystemcomponents.SincetheDSSandCMSdonotcommunicatedirectly,thedatahuborchestratesworkflowsandcommunicationsbetweenthetwocomponentsandtheinternalcomponentsofthedatahub.Similartothedatahubdatagateway,itusesApacheCameltorouteandprocessCoreICMSystemcommands.Inaddition,itusesaworkflowcomponentdesignedforamicroservicearchitectureusedbyNetflixinitsownproductionenvironmentcalledConductor.Itprovidesthefollowingcapabilities:

• Receiveallcoresystemrequestsforservices• Provideexecutionmanagementofallrequestsfordatahubservices• Provideexecutionmanagementofworkflowprocessingforrequestsinvolvingsimplesinglestep

workflowsorcomplexmulti-stepworkflows• Persistservicerequestsandtheiroutcomes(successorfailure)• Manageinternaloperationsofthedatahub(starting/stoppingservices,restartonfailure,etc)• OrchestrateactionsbetweentheDSS,CMS,anddatahub.

Notethatthedatahubcommandgatewayisaninternaldatahubcomponent.Atnotimearethemessagequeuesandtopicsthatitcommunicateswithdirectlyconnectedtoexternalsystems.Allqueuesandtopicsthatcommunicatewiththiscomponentareinternalqueuesandtopicsconnectedtodatahubcomponentsorthedatahubdatagateway(forcommunicationwithexternalsystems).

Figure5-11DataHubCommandGateway

Thedatahubcommandgatewayconsistsofjustafewprimarycomponenttypes.Theseinclude:

• Conductorandrelatedworkflowcomponents• Camelandrelatedroutingandtransformationcomponents• ActiveMQworkflowstatustopic• ActiveMQworkflowtasktopic• Monitor

5.4.1. CONDUCTOR

Conductorisusedfororchestrationoftheindividualpipelinesandbothinternalandexternalrequestsforservices.ConductorisanopensourceprojectdevelopedbyNetflixtomanagesomeofitsownproductionworkflows,particularlywithamicro-servicearchitecture.ThereareanumberofsimilaritiesbetweentheusecasesforwhichConductorisdesignedandtheorchestrationoftheindividualpipelinesandtherequestsforservicesfromtheDSSandCMS,aswellasorchestrationofthecommunicationsbetweenDSSandCMSthatmakeitwellsuitedforuse.

Conductorisusedinan“as-is”configuration“out-of-the-box”.ThedatahubimplementationusestheRESTinterfacesandin-memorydatabasethatcomeswithConductor.NomodificationsaremadetoConductor.Itprovidesworkflowmanagementandcontrol,withworkflowsdescribedinJSONformat.

5.4.2. CAMEL

ToadaptConductortothedatahubinterfaces,CamelisusedasafacadetotheActiveMQandKafkadatahubinterfaces.CamelprovidestheinterfacebetweenConductor’snativeRESTinterfaceandthedatahub’sActiveMQandKafkadatabuselements.

5.4.3. ACTIVEMQWORKFLOWSTATUSTOPIC

TheActiveMQworkflowstatustopicisusedtoallowindividualdatahubcomponentstoprovideworkflowstatusmessagestoConductor,viatheCamelinterface.Italsoisusedtoprovidegeneralsystemandcomponentstatusinformationforuseinworkflowmanagement.Thedatahubdatagatewayalsoprovidesworkflowandexternalsystemstatus(CMSandDSS)messagesviatheworkflowstatustopic.TheCMSandDSSsystemsareneverrequired(orallowed)todirectlymessageviatheWorkflowstatustopic,butinstead,theircommunicationsaremanagedbythedatahubdatagateway.

5.4.4. ACTIVEMQWORKFLOWTASKTOPIC

TheActiveMQworkflowtasktopicishowworkflowtasksaredistributedfromConductortotheindividualdatahubcomponents,ortheDSSandCMSsystems(viathedatahubdatagateway).Aswiththeworkflowstatustopic,DSSandCMSsystemsareneverdirectlyallowedtocommunicatewiththeworkflowtasktopic.Instead,thedatahubdatagatewaymanagesthedistributionoftasksandtheresultingresponsesviatheDSSandCMSAPIs.

5.4.5. MONITOR

Themonitorlistenstotheworkflowstatustopicforsystemandcomponentstatusmessages,alwaysprovidingdatahub,DSS,andCMSstateandstatusinformationtotheConductorworkflowmanager.ThisallowsConductortorespondtodegradedsystemstate,eitherbydisablingspecificworkflowsorstartingotherworkflowstorespondtosystemcomponentfailures.

6. DECISIONSUPPORTSYSTEMDESIGN

Thedecisionsupportsystem’s(DSS)primaryrolesinclude:

• Provideresponseplansfollowingreceiptofaconfirmedincident• Evaluateresponseplansandrankthembasedonuserdefinedcriteria• ProvideoneormorerecommendationstocorridoroperatorsandstakeholdersviatheCorridor

ManagementSystem,whethertoimplementaresponseplanandwhichresponseplantoimplementforeachreceivedconfirmedincident

• Evaluateimplementedresponseplaneffectivenessandrecommendnewresponseplanswhenappropriate

• ProvideresponseplanevaluationresultsforcorridoroperatorsreviewviatheCorridorManagementSystem

TheDSSaccomplishesthiswiththreeprimarycomponents:

• ResponsePlanManagement• RulesEngine• Modeling

Theresponseplanmanagementcomponentreceivesincidentinformationandcoordinatesthedevelopmentandevaluationofresponseplans.Therulesengineprovidesconfigurablelogic/rulesforthedecisiontocreateresponseplans,responseplancreation,andresponseplanranking.Themodelingcomponentprovidestrafficestimationandpredictioncapabilitiestosupportresponseplancreation,evaluation,andranking.

6.1. DSSHIGHLEVELDESIGN

TheDSS’sthreeprimarycomponentsaredescribedinFigure6-1.TheDSSisoneofthethreeprimaryICMsystemsubsystems,andcommunicateswiththeICMsystemviathedatahub’sdatagateway.

Figure6-1DSSArchitecture

Allcorridorinformationisreceivedfromthedatahubdatagateway.Themodelingandresponseplanmanagementcomponentsreceiveallcorridordataandsystemstatusviaaseriesofdatachannelsavailableonthedatahubdatagateway.

AllcommandstotheDSSandrequestsforDSSstatusarereceivedviaasetofcommandchannelsexposedonthedatahubdatagateway.AllcommandrequestsfromentitiesexternaltotheDSSarereceivedbytheresponseplanmanagementcomponent.Requeststothemodelingcomponentareoriginatedbytheresponseplanmanagementcomponent.DirectrequeststothemodelingcomponentfromsystemsexternaltotheDSSarenotallowed,andmustbemanagedviathecommandsexposedbytheresponseplanmanagementcomponent.

AllcommunicationswiththeDSSviatheDataHubDataGatewayaremanagedviaActiveMQmessaging.CommunicationsbetweentheResponsePlanManagementandModelingareenabledviaRESTorActiveMQmessaging.TheresponseplanmanagementandrulesenginearetightlycoupledviatheirjavabasedAPIs.

OutputfromtheDSSconsistsprimarilyofresponseplansandtheirevaluationsfromtheresponseplanmanagementsystem,andestimationandpredictionresultsfromthemodelingsystem.

Figure6-2DecisionSupportSystemHighLevelDesign

Figure6-2providesaslightlymoredetailedviewoftheDSS.

6.2. RESPONSEPLANMANAGEMENT

Responseplanmanagementprovidesfunctionsforthedevelopmentandselectionofresponseplans,includingcoordinationoftheactionsoftheresponseplanmanagement,rulesengine,andthemodelingcomponentsoftheDSS.

Theresponseplanmanagementcomponentisprimarilyaneventbased,asynchronousservicethatlistensforaconfirmedincidentfromthedatahub,anduponreceipt,orchestratesactionsbetweenitself,therulesengine,andmodelingcomponentstodetermineifaresponseplanmightpositivelyimpacttrafficconditionsandifso,developseveralresponseplans,evaluatethem,andrecommenda

responseplanforimplementation.Theresponseplanmanagementcomponentalsohasalimitedselectionofmethodsimplementedsothatotherspecificrequestscanbemadeviathedatahub.

Theresponseplanmanagementalsomonitorsthedecisionsupportsystemanditscomponents.IthaslimitedcapabilitiestoperformspecificstartupactivitiesrequiredforDSSoperation,monitoreachDSScomponent,andperformlimitedactionsshouldastartupactivityfail(forinstance,ifthemodelingestimationprocessshouldstop,theresponseplanmanagementcomponentshallattempttorestarttheestimationprocess).

Figure6-3ResponsePlanManagementHighLevelDesign

ThecircledareaofFigure6-3illustratesthefundamentalcomponentsoftheresponseplanmanagementcomponent.

Ontheleftoftheresponseplanmanagementcomponentinthefigureisthedatahubinterface.ThisisanActiveMQbasedinterfacethatsubscribestothedatahubdatainterfaceandprovidesassetinformation,routeinformation,andincidentinformationtotheresponseplanmanagementcomponent.Thisinformationismaintainedinalocaldatastore.ItalsolistensforspecificrequestssenttotheDSSviathedatahub.

Theresponsemanager,uponreceiptofaspecificrequestorincidentmanagesthespecificworkflowsrequiredtorespondtotherequestortoprovidearesponseplanrecommendation.Thisincludesrequestingdecisionsandresultsfromtherulesengine,informationregardingcurrenttrafficstatefromthemodelingsystem,orspecificpredictionsforresponseplansreceivedfromtherulesengine.Theresponseplanmanageralsoassemblesthecompleteresponseplan,andprovidesitasaresponsetothedatahubforstorageanddistributiontotheCMS.

6.3. RULESENGINE

TherulesengineisacombinationofjavacomponentsandtheopensourceDroolsrulesenginedesignedtorespondtoalimitednumberofquestionsthatcanbeaskedbytheresponseplanmanager.Theselimitedquestionsinclude:

• Doesthecurrentconfirmedincident,withtheresultsofa“do-nothing”prediction,requiredevelopmentandevaluationofresponseplans?

• Whatresponseplansshouldbeevaluated?Createthoseresponseplansfromalimitedsetofresponseplancomponents.

• Whichresponseplan,giventheresponseplansdevelopedandtheirrespectivepredictions,shouldberecommended,listedinarankedorder?

TherulesengineisaJavalibrarythatisincludedintheresponseplanmanagercomponentwithadefinedapispecifictoansweringthesedefinedquestions,giventhecurrentstateofthecorridorassets,definedlimitswithinasetofuserspecifiedrules(suchas“donotusethisroutebetweenthehoursof3and5pm”),currenttrafficstate,currenttime,andincidentinformation.

Itisintendedthatthefinaluserinterfacefordeveloping,editing,testing,andevaluatingrulesandtheirperformancewillbeintheCMS.However,initially,thiswillbelimitedtorequiringITsupporttoassistwithrulesdevelopment,anduploadingofspreadsheetstoaknownlocationwhererulescanbeaddedorupdated.

6.4. MODELING

Themodelingsystemprovidestwoprimaryservices;trafficstateestimationandtrafficprediction.

6.4.1. MODELINGTECHNIQUES

Threetrafficmodelsareusedwithinthemodelingsystem.Theseinclude:

• AcustomfreewaytrafficestimationmodeldevelopedbyUCBerkeleyusingacelltransmissionmodel(CTM)method.

• AcustomarterialtrafficestimationmodeldevelopedbyUCBerkeleyusinganintersectionqueuemodelmethod.

• AcommercialpredictionmodelusingTSS’sAimsunmodelingsystemandamicro-modelingbehavioralmodel.

Figure6-4-ModelingComponentDesign

Thesystemiscomposedofseveralprimarysubsystems–• Datamanagement(inblue)–receivesandprocessestrafficandtrafficinfrastructuredataso

thatitcanbeconsumedbythemodelingsoftware.• Estimationmodelingsystem(inpink)–buildsandrunsestimations,producesoutputof

estimatedtrafficstate.• Predictionmodelingsystem(inorange)–buildsandrunspredictedtrafficstate,inparticular

predictedresponseplanperformance,includingperformancemetrics.• Calculation/metrics(inlightorange)–consumesestimatedtrafficstateandproducescurrent

trafficandnetworkperformancemeasures.• ProjectManager(inbrown)–providesoverallsystemorchestration,cloudscalingandEC2

management,workflow,andexternalRESTinterfaces.Auserinterfaceisavailableforestimationmodeldevelopment.ThisiscurrentlyusedforinternaltestingandQA.

• Persistence–(green/gray)–persiststraffic/trafficinfrastructuredata,modelandmodelbuildingdefinitions,rawandprocessedresults.

CommunicationwithexternalsystemsandthemodelingsystemisviaaRESTAPIexposedbytheProjectManager.DatafeedsfromthedatahubareprovideddataviaActiveMQdatatopics.InternalcomponentcommunicationbetweenmodelingsystemcomponentsisviaActiveMQtopics(data)orqueues(command).

Technologystack:• PrimarycomponentsarebuiltusingJavawiththeSpringframework.Hibernateisusedwith

componentsrequiringPostgresaccess.• PersistenceisbuiltwithJavabasedpersistenceworkers(bothforstoreandretrieveoperations)

andPostgresv9.6.2orCassandrav3.10forpersistence.• MessagingviaActiveMQv5.14.5• ProjectManagerRESTserviceshostedonTomcatv8.5.15• LogManagementviaGraylog• AmazonWebServices,includingthefollowingAWSservices

o EC2o RDS(Postgres)o S3o VPCo IAMo Otherservicesareusedforcoderepository,build,anddeploymentservices

Platform/ApplicationServers/OS• MostanyversionofLinuxOSisacceptable,Ubuntu14.04isthecurrentstandard• AmazonWebServices• Tomcatv8.5.15

GeneralArchitecturalApproach:

Theapproachusedhasbeentodevelopspecificapplicationcomponentstargetedtospecifictasks,connectthemtoeachotherviamessagingorRESTservicesdependingupontheirapplicationrole(backendprocessingorservicedelivery),andprovidescalingcomponentsthatalloweachindividualcomponenttodetectload(CPU/threadutilizationanditsfeedingqueuestate)andscalethenumberofinstancesoftheresourcetypebasedondemand.

Primarycomponentdescriptions:

ModelEngine–Therearetwoversionsofthemodelengine.Thefirstisusedforfreewayestimation.ThefreewaymodelengineisthecorecomponentofthesystemthatimplementstheCTMalgorithm.Thisimplementationisasimulationusingauserdefinedroadnetwork,accompanyinginfrastructureelementsincludingrampmetersandsensors,trafficdemand,trafficbehaviorinformation(splitratiosatintersectionsandfreewayramps),androad/trafficcharacteristicsrequiredbytheCTMalgorithm(fundamentaldiagrams).Thesimulationcanbeusedtoprovideestimationsofcurrentstateorpredictionsoffuturestate,butisusedonlyforfreewaycurrenttrafficstatewithintheICMsystem.

Thesecondversionofthemodelengineisusedforarterialestimation.Thisarterialmodelengineusesanalgorithmusingintersectionsensingandsignaltimingandcycleinformationtoproduceanestimatedtrafficstateofthecorridorarterials.

Bothtypesofmodelenginereceivearequestforamodelruncontainingthescenariotoberunandthedesiredconfigurationparameters.Themodelenginethenrunsthemodelandoutputsthelinkstate(trafficdensityandspeed)foreachroadnetworklink.

ThemodelengineisamultithreadedJavaapplication,buttheprimaryworkcurrentlyremainssingle-threaded.

Predictionengine–ThepredictionengineisanimplementationofTSS’sAimsunsoftware.ItincludestheAimsunsimulationsoftwareaswellasacomponentthatbuildssimulationmodelsforuseinthepredictionworkflow.Thesimulationbuildcomponentcombinesabasesimulationmodelwithdatafromthedatahubindicatingcurrentcorridorassetstate(signals,ramps,signs,etc),currenttrafficstatefromthefreewayandarterialmodelengines,andtheresponseplansfromtheresponseplanmanagementsystemtocreateasetofoneormoremodelsthatcanbeusedtoevaluatearesponseplan.Italsoprovidesthepredictionmetricsrequiredforpresentationtotheuserandtherulesengineforresponseplanrecommendationandselection.

ProjectManager/ProjectManagerApp-Theprojectmanagerappallowsexternalsystemstointeractwiththemodelingsystem.ProjectmanagerappprovidesRESTservicesandmessagingtoinitiateactionsbytherestofthesystem.TheprojectmanagerappisservedviaaTomcatapplicationserver.

ScenarioBuilder-Thescenariobuilderisusedtocreateascenariotoberun.Itisrarelyused,butcanbeusedwhencorridorinfrastructureelementsaremodified,suchasarampmeter.Itisaback-endcomponentthatreceivesmodelcomponents(roadnetworks,intersectionsignals,rampmeters,trafficdemandandsplitsets,etc)fromtheprojectmanagerappandbuildsamodelsuitableforrunninginthefreewaymodelengine.

Calculators-Thecalculatorsareusedtotakethemodelenginelinkstatedataandprocessitforconsumptioneitherasspecificmetricsforanalysis,orasvisualizationinformationfortheCMS.Eachcalculatorjobissinglethreadedandisdesignedforaspecificcalculation.

PersistenceWorker-Persistenceworkersarecomponentsthataredesignedtoeithertakedatafromaqueueortopic(modelenginedata,model/scenarioinformation,orcalculatoroutput)andpersistitinoneofthedatastores(PostgresorCassandra)or,uponrequestretrievethedatafromoneofthedatastoresandplaceinontheappropriatequeueortopicforconsumptionbydownstreamprocesses.

ReadersandProcessors-Thesecomponentsareusedtoretrieveoracceptpusheddatafromthedatahubandplaceitintoaqueueortopicforconsumption(reader)andtoretrievedatafromaqueueortopicandprocessitforconsumptionbythemodel(processor)orpredictionengine.

Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR...

Documents

Transcript of Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR...

Figure 37. Corridor Location Map - Waxahachie, Texas 5 - Design Strategies for Transportation... · Chapter 955 | Design Strategies for Transportation Corridors U.S. Highway 287 is

U.S. States Affected by Mexican Drug Trafficking Organizations · U.S. map showing regional drug transportation corridors: • Corridor A follows I-8 from southern California and

CONNECTING TO OPPORTUNITIES NWT Transportation Corridors · SGP Corridor – Access to Resources • GNWT is early in process for developing project but there are significant opportunities

Transportation Corridor Case Study Espanol

GTA West Transportation Corridor Planning and ...€¦ · design for a new transportation corridor within the Route Planning Study Area. The new transportation corridor will include:

Addressing Mine Hazards under Transportation Corridors – The

European corridors: Strategies for corridor development ...content.alterra.wur.nl/webdocs/internet/corporate/prodpubl/boekjes... · corridor development for target species ... Das

PROPOSED ACTIVE TRANSPORTATION CORRIDORS · BRIGHAM CITY EAST-WEST CORRIDORS Trail Corridor Map Key Recommended Corridor Type Recommended Corridor Use Difficulty Length (miles) US

CENTRAL AND EAST CORRIDORS - TheLensNews › ... › 201410_STCentralEastHCT_CorridorReport_… · ES‐2 Corridor Report: I‐405 Corridor September 2014 As part of the I‐405 Corridor

Multimodal Maturity of Virginia’s Transportation Corridors

Alternatives Analysis for Phase 2B Metro Corridor · Alternatives analysis is about finding best alternative to address the transportation related problems for specific corridors

Emanuel’s Industrial Corridor Modernization Kinzie ... · North Branch Industrial Corridor Receiving Corridors Ineligible Corridors New fee could generate tens of millions of dollars;

Corridors of Statewide Significance...Corridors of Statewide Significance: Southside Corridor. March 2010. Page 1-1. 3 Commonwealth Transportation Board. VTrans 2035: Virginia’s

Washington State Freight and Goods...Feb 13, 2006 · Rail freight corridors ... means a transportation corridor of great economic importance within an integrated freight system that

Regional Perspectivesolympiawa.gov/.../Planning/urban-corridors/OPC-Corridors-Presentati… · Strategy Corridors: Major transportation corridors where widening is not an option for

Innovation Corridor Transportation Visioning Forum ...

SW Corridor Plan Active Transportation

European corridors: Strategies for corridor development for - ECNC

Transportation Corridor Study

GIS for Transportation Corridor Planning West Corridor.