PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY OF CALIFORNIA, BERKELEY
Connected Corridors: I-210 Pilot Integrated Corridor Management System
Core System High-Level Design
November 29, 2017 DRAFT
Partners for Advanced Transportation Technology works with researchers, practitioners, and industry to implement transportation research and innovation, including products and services that improve the efficiency, safety, and security of the transportation system.
I-210Pilot:CoreSystemHigh-LevelDesign
ii
Thispageleftblankintentionally
I-210Pilot:CoreSystemHigh-LevelDesign
iii
I-210Pilot:CoreSystemHigh-LevelDesign
iv
TableofContents
1. Introduction ...................................................................................................................................1
1.1. PurposeofDocument ............................................................................................................. 1
1.2. RelationtoSystemsEngineeringProcess................................................................................ 1
1.3. IntendedAudience.................................................................................................................. 2
1.4. DocumentOrganization .......................................................................................................... 2
2. SystemPrimaryObjectivesandPurpose.........................................................................................4
2.1. ProjectGoalsandObjectives .................................................................................................. 5
2.2. TechnicalCapabilitiesSought.................................................................................................. 7
3. HighLevelDesignObjectives,Constraints,andPrinciples ...............................................................8
4. CoreSystemHighLevelDesign .....................................................................................................10
4.1. MajorComponents ............................................................................................................... 10
4.2. FieldElements ....................................................................................................................... 11
4.3. DataHub ............................................................................................................................... 12
4.4. DecisionSupport ................................................................................................................... 13
4.5. CorridorManagement .......................................................................................................... 14
4.6. Primaryprocessflow............................................................................................................. 17
5. DataHubDesign...........................................................................................................................18
5.1. DataSources ......................................................................................................................... 21
5.2. DataPipelines........................................................................................................................ 23
5.2.1. SensingDataPipeline .......................................................................................... 235.2.2. CaltransFreewayTrafficDataPipeline................................................................ 245.2.3. HeterogeneousDataPipeline.............................................................................. 255.2.4. HomogenousDataPipeline ................................................................................. 275.2.5. PipelineControl ................................................................................................... 275.2.6. PipelineStatusandLogging................................................................................. 295.2.7. CorridorManagementSystem-DecisionSupportSystem(CMS-DSS)Pipeline ... 305.2.8. CMSCommandStatusPipeline ........................................................................... 34
5.3. ExternalInterface/DataGateway.......................................................................................... 35
5.3.1. Subscriptionendpoints........................................................................................ 375.3.2. Request/Responseendpoints ............................................................................. 37
5.4. DataHubCommandGateway............................................................................................... 37
5.4.1. Conductor............................................................................................................ 38
I-210Pilot:CoreSystemHigh-LevelDesign
v
5.4.2. Camel................................................................................................................... 395.4.3. ActiveMQWorkflowStatusTopic ....................................................................... 395.4.4. ActiveMQWorkflowTaskTopic .......................................................................... 395.4.5. Monitor ............................................................................................................... 39
6. DecisionSupportSystemDesign...................................................................................................39
6.1. DSShighleveldesign............................................................................................................. 40
6.2. ResponsePlanManagement................................................................................................. 41
6.3. RulesEngine .......................................................................................................................... 43
6.4. Modeling ............................................................................................................................... 43
6.4.1. Modelingtechniques........................................................................................... 43
I-210Pilot:CoreSystemHigh-LevelDesign
1
1. INTRODUCTION
ThisConnectedCorridorsHighLevelDesigndocumentprovidesthehighlevelsystemarchitectureforthesystemtobedeployedontheI-210corridor.ThesystemarchitecturedescribedhereisadirectresultoftheConnectedCorridorsSystemRequirementsdocumentandtheworkdoneatUCBerkeleyintrafficmodelingandcontrol.Thisdocumentprovidesthesystemarchitecture,highleveldesignoftheprimarysubsystems,thedecisions,assumptions,constraints,andreasoningbehindthatarchitecture,andcriticalfunctionseachsubsystemprovidesforthesystem.
Thesystem,tobepilotedalongasectionoftheI-210corridorintheSanGabrielValleyareaofLosAngelesCounty,aimstoimproveoverallcorridorperformanceduringincidents,unscheduledevents,andplannedevents.Thisistobeachievedbymoreefficientlymanagingexistingsystemsandinfrastructures,promotingcross-jurisdictionaloperations,andusingmulti-modaltrafficanddemandmanagementstrategiesthatconsiderallrelevantmodesoftransportation.
1.1. PURPOSEOFDOCUMENT
Thisdocumentprovidesthehighleveldesign,servingastheidentificationoftheprimarysubsystemsandmajorcomponentsaswellasthebasisfortheselection,development,andintegrationoftheseintoasystemthatsatisfiesthesystemrequirementsasdefinedintheSystemsRequirementsDocument.ThishighleveldesignwillgovernthetechnologyplatformanddirectionoftheI-210PilotICMSystemandserveasthebasisforotherCaltrans-ledICMeffortsstatewide.
1.2. RELATIONTOSYSTEMSENGINEERINGPROCESS
ThedevelopmentofhighleveldesignispartofthesystemsengineeringprocessthattheFederalHighwayAdministration(FHWA)requiresbefollowedfordevelopingIntelligentTransportationSystem(ITS)projectswhenfederalfundsareinvolved.Whilenotrequiredforprojectsonlyusingstateorlocalfunds,useofthesystemsengineeringprocessisstillencouragedinsuchcases.
TheoverallsystemsengineeringprocessisillustratedinFigure1-1.DevelopinghighleveldesignrepresentsthenextstepoftheSystemDefinitionandDesignphaseofaproject(Phase2inthefigure)followingthecompletionoftheSystemRequirements.HighLevelDesignistypicallyderivedfromtherequirements.Theresultingdesignelementsareinturnusedtoinformandguidethemoredetaileddesignofthevarioussystemandsubsystemcomponents.
I-210Pilot:CoreSystemHigh-LevelDesign
2
Figure1-1–SystemRequirementsSpecificationwithinSystemsEngineeringProcess
1.3. INTENDEDAUDIENCE
TheprimaryaudiencefortheSystemHighLevelDesigndocumentincludespersonnelresponsiblefordesigningandimplementingtheICMsystem.TheaudiencealsoincludesindividualsfromCaltransDistrict7,CaltransHeadquarters,andtheUniversityofCalifornia,Berkeley,taskedwithprojectmanagementduties.
1.4. DOCUMENTORGANIZATION
Theremainderofthisdocumentisorganizedasfollows:
• Section2summarizestheprimarysystemobjectivesidentifiedwithintheSystemRequirementsthatshapethesystemdesign.
• Section3presentstheprimaryguidingdesignprinciplesandbaseassumptionsthatshapethesystemdesign.
• Section4presentsthekeysystemdesigncomponentsandprimarydataflows.
I-210Pilot:CoreSystemHigh-LevelDesign
3
• Section 5 presents the key system components of the Data Hub including data sources,interfaces/gatewaysandpipelines.
• Section6presentsthekeysystemcomponentsoftheDecisionSupportSystem(DSS)includingtherulesengine,modelinginterfaces,andresponseplangeneration.
• Section7presentskeysecuritydesignissuesandimplementationplans.
• Section8providesdesigninformationforthemessagingsystemsanddescribeshowinformationispassedbetweensubsystems.
• Section9describesthesysteminterfaces.
Inaddition,othersupportingdocumentsareavailableintheDocumentLibraryoftheI-210Pilotwebsiteathttp://ccdocs.berkeley.edu/content/document-library:
I-210Pilot:CoreSystemHigh-LevelDesign
4
2. SYSTEMPRIMARYOBJECTIVESANDPURPOSE
TheoverridingpurposeoftheI-210PilotistoreducecongestionandimprovemobilityalongasectionoftheI-210corridorinLosAngelesCountythroughthecoordinatedmanagementofitsmajornetworks:theI-210freeway,keysurroundingarterials,andlocalandregionaltransitservices.Thegoalistoenableallcorridor“actors”—transportationsystemmanagersandoperators,controlsystems,vehicles,andtravelers—toworktogetherinanefficientandcoordinatedway.
TheseimprovementswillbeachievedbydevelopinganddeployingtheICMsystemdescribedinthishighleveldesigndocument.AttheheartoftheproposedsystemwillbeaDecisionSupportSystem(DSS)designedtohelpcorridorsystemoperatorsmanageincidents,unscheduledevents,andplannedeventsmoreeffectively.Thissystemwilluseinformationgatheredfrommonitoringsystemsandprovidedbypredictiveanalyticaltoolstoestimatecurrentandnear-futureoperationalperformance.Theinformationwillbeusedtodeveloprecommendedcoursesofactiontoaddressproblemscausedbyidentifiedincidentsandevents.Morespecifically,thissystemisexpectedto:
• Improvereal-timemonitoringoftravelconditionswithinthecorridor• Enableoperatorstobettercharacterizetravelpatternswithinthecorridorandacrosssystems• Providepredictivetrafficandsystemperformancecapabilities• Beabletoevaluatealternativesystemmanagementstrategiesandrecommenddesiredcourses
ofactioninresponsetoplannedevents,unscheduledevents,andincidents• Improvedecision-makingbytransportationsystemmanagers• Improvecollaborationamongagenciesoperatingtransportationsystemsinthecorridor• Improvetheutilizationofexistinginfrastructureandsystems• Moreefficientlyusesparecapacitytoaddressnon-recurringcongestion• Reducedelaysandtraveltimesalongfreewaysandarterials• Improvetraveltimereliability• Helpreducethenumberofaccidentsoccurringalongthecorridor• Reducetheperiodduringwhichthecongestionresultingfromanincidentoreventaffects
corridoroperations• Reducegreenhousegasemissions• Generatehighertravelersatisfactionrates• IncreasetheoveralllivabilityofcommunitiesinandaroundtheI-210corridor
WhiledevelopmentoftheproposedsystemisunderthefinancialsponsorshipofCaltransHeadquarters,thesystemwillbedevelopedprimarilybythelocaltransportationagenciesthathaveagreedtoparticipateinitsoperation,incoordinationwithPATH.Projectactivitieswillincludethedesign,development,installation,testing,andoperationofvariouscomponentsoftheICMsystem,aswellasthedevelopmentofinterfaceswithexistingmonitoringandcontrolsystems.Forexample,theICMCoreSystemwillbeinterfacedwithtrafficmanagementsystemsownedbyCaltrans,suchastheAdvancedTrafficManagementSystem(ATMS).
I-210Pilot:CoreSystemHigh-LevelDesign
5
2.1. PROJECTGOALSANDOBJECTIVES
TheprimarygoaloftheI-210PilotICMprojectistoimproveoverallcorridorperformancealongasectionoftheI-210corridor.Thistranslatesintothefollowingspecificgoals:
1. Improveoperationalsituationalawareness2. Promotecollaborationamongcorridorstakeholders3. Improveresponsetoincidentsandevents4. Improvetravelreliability5. Improveoverallcorridormobility6. Empowertravelerstomakeinformedtraveldecisions7. Facilitatemulti-modalmovementsacrosstheregion8. Promotetransportationsustainabilitybyreducingimpactsontheenvironment9. Improvecorridorsafety
Foreachofthesegoals,Table2-1furtheridentifiesthemainoperationalobjectives.Manyoftheobjectivesaresimilartothoseoftraditionaltransportationimprovementprojects.Many,however,alsofocusonimplementingmorecomprehensivetravelandsystemstatusmonitoringsystems,improvedoperationalforecasting,improvedinformationdisseminationtotravelers,enhanceddata-sharingcapabilities,demandmanagementapproaches,andimprovedcollaborationamongtransportationsystemoperators.
Table2-1–ICMSystemGoalsandObjectives
Goals Objectives
1.Improvesituationalawareness
• Establishminimumrequirementsfordatacollectiontosupportsystemmanagement• Increasedatacollectionopportunitiesfromarterialsandlocalroads• Improvethecollectionofreal-timeoperationaldatafromnon-traditionalsources,suchasprobevehicles
• Developacomprehensivecorridorinformationaldatabasecoveringallrelevanttravelmodeswithinthecorridor
• Improvethequality,accuracy,andvalidationprocessofcollecteddata• Increasetheabilitytoestimatetraveldemandpatternsinamulti-modalenvironment• Improvetheabilitytoforecastnear-futuretravelconditionsbasedonknownincidents,roadconditions,weather,andlocalevents
• Developperformancemetricsconsideringallavailabletravelmodes
2.Promotecollaborationamongcorridorstakeholders
• Strengthenexistingcommunicationchannelsamongthecorridor’sinstitutionalstakeholders• Exploreopportunitiesfornewcommunicationlinksbetweencorridorstakeholders• Improvecooperationandcollaborationamongcorridorstakeholders• Developregional/jointoperationsconcepts• Identifynewmethodsofcollaboration• Extendcorridorperformancemetricstothenetworklevel• Investigatenewtypesofagreementsbetweenparticipatingagencies
I-210Pilot:CoreSystemHigh-LevelDesign
6
Goals Objectives
3.Improveresponsetoincidentsandunexpectedevents
• Reducethetimeneededtoidentifytheexistenceofanincidentorunexpectedevent• Reducethetimeneededtorespondtoincidentsorunscheduledevents• Enhancethecoordinationofactivitiesamongfirstresponders,trafficmanagementagencies,andtransitagenciestominimizeimpactsonsystemoperations
• Reducethetimeneededtoimplementcontrolactionstoaddresscongestionresultingfromanincidentorevent
• Reducethetimeneededtodisseminaterecommendeddetoursaroundanincidentorevent
4.Improvetravelreliability
• Improvetraveltimepredictabilityalongthecorridor• Reducetheimpactsofincidentsandeventsonnetworkoperations• Improveincident/eventnotificationforfirstrespondersandnetworkoperators• Improveincident/eventnotificationtotravelersandfleetoperators• Providetravelersandcommercialvehicleoperatorsaffectedbyanincidentoreventanenhancedabilitytoseekalternateroutesormodeoftransportation
5.Improveoverallcorridormobility
• Reducedelaysincurredbytravelers• Reducetheimpactsofincidentsandeventsonnetworkoperations• Efficientlyusesparecapacityalongcorridorroadwaystoplannecessarydetoursaroundincidentsorevents
• Promotestrategiestoinducedesirabletraveldemandpatterns• Coordinatethemanagementoffreewayandarterialbottlenecks• Promoteincreasesinvehicleoccupancy• Promoteincreasesintransitridership
6.Empowersystemuserstomakeinformedtraveldecisions
• Improvethedisseminationofreal-time,multi-modaltravelinformation• Enhancetheuseofinfrastructure-basedinformationaldevices(freewayCMS,arterialtrailblazersigns,kiosks,etc.)toprovideen-routeinformationtotravelers
• Enableindividualstoreceivetravelinformationonconnectedmobiledevices• Makearchivedhistoricaldataavailableto511servicesandinformationserviceproviders• Supportthedisseminationoftravelinformationby511servicesandthird-partyproviders
7.Facilitateregionalmulti-modalmovements
• Promotetheintegrationofcommuterrailandbusserviceswithcorridoroperations• Facilitatetransfersacrossmodesduringincidentsandevents• Providerelevantregionaltravelinformationtotravelers• Directtravelerstopark-and-ridefacilitieswithavailablespaces
8.Promotetransportationsustainability
• Reducefuelconsumption• Reducevehicleemissions• Identifyfinanciallysustainablesolutionsforlong-termsystemoperationsandmaintenance• Encouragetheuseoftransit,walking,andbicyclingwhereappropriate• Supportlocallypreferredalternativescompatiblewithcorridorobjectives• Developandimplementperformancemetricsreflectingenvironmentalgoals
9.Improvecorridorsafety
• Reducecollisionrates• Reducetheseverityofcollisions• Reducethenumberoffatalities• Reducetheimpactsofprimaryandsecondaryincidentsonnetworkoperationsthroughimprovedincidentmanagement
• Improvesafetyforbicycles,pedestrians,andtransit
I-210Pilot:CoreSystemHigh-LevelDesign
7
2.2. TECHNICALCAPABILITIESSOUGHT
Tohelpmanagetravelactivitieswithinthecorridorduringincidents,unscheduledevents,andplannedevents,theprojectisseekingthefollowingtechnicalcapabilitiestosupportthegoalsandobjectivesidentifiedinSection2.1:
• Gatherandarchiveinformationcharacterizingtrafficoperations,transitoperations,andtheoperationalstatusofrelevantcontroldeviceswithintheI-210corridor.
• IdentifyunusualtravelconditionsontheI-210freewayornearbyarterialsbasedonmonitoringdataprovidedbyvarioustraffic,transit,andtravelmonitoringsystems.
• Identifysituationsinwhichanincidentonroadwaysortransitfacilitiessignificantlyaffectstravelconditionswithinthecorridor.
• Providecorridor-wideoperationalevaluationstotrafficmanagers,transitdispatchers,andotherrelevantsystemmanagers,includingprojectedassessmentsofnear-futuresystemoperationsundercurrentandalternatecontrolscenarios.
• Identifyrecommendeddetoursaroundincidentsorroutesleadingtothesiteofanevent,consideringobservedtravelconditionswithinthecorridor.Dependingontheneed,andfinalsystemcapabilities,specificdetoursmayberecommendedformotoristsandtransitvehicles.
• Identifyrecommendedsignaltimingplanstouseatsignalizedintersectionstoimproveand/oraccommodatetrafficflowinfluxduringincidentsandeventsandimproveoverallcorridormobility.
• IdentifyrecommendedrampmeteringratestouseonindividualI-210freewayon-rampsandconnectorstomaintainoverallcorridormobility.
• IdentifymessagestopostonavailablefreewayandarterialCMSstoinformmotoristsofincidentsandevents.
• ProvideguidancetomotoristsontheI-210freewayandsurroundingarterialsusingavailablefreewayCMSs,arterialCMSs,andarterialdynamictrailblazersignsregardingwhichdetourtotaketogoaroundanincidentorwhichroutetofollowtoreachthesiteofanevent.
• Provideinformationtomotoristsabouttheavailabilityofparkingandtransitservicestohelptravelersmakealternatemode-choicedecisions.
• Provideuniformtrafficmanagementstrategiesacrossjurisdictionalboundariesduringincidentsandevents.
• Provideinformationtomotoriststhroughthird-partyoutlets,suchas511services,navigationapplicationproviders,etc.
I-210Pilot:CoreSystemHigh-LevelDesign
8
3. HIGHLEVELDESIGNOBJECTIVES,CONSTRAINTS,ANDPRINCIPLES
Thecoresystemdesignisgovernedbyafewkeyprimarydesignobjectivesdictatedbytheprojectandtherequirements:
• Realtimeoperation–Thesystemmustoperateinanearrealtimeenvironment.Thetimebetweenanincidentoccurringandaresponseplanimplementationisacriticaltimeperiod.Responsetimesshouldbedictatedprimarilybythetimetonotificationandconfirmationoftheincident,andthetimetofullyimplementadecision.Thetimerequiredbythesystemtoprocessinformationshouldbeminimizedsoastohavethemaximumpositiveimpactoncorridoroperations.
• Speedtodecision–Thereisasignificantamountofdataprocessingrequiredtomaintainbothoperatorawarenessandtoensurethemodelingwithinthedecisionsupportsystemisupdatedwiththelatestavailableinformation.Dataprocessinganddecisionprocessingtimeshouldbeminimized.
• Qualityofdecision–Thequalityoftheresponseplansisadirectresultofthetrafficestimationquality,trafficpredictionquality,dataprocessingtime,dataquality,andrulesusedtogenerateresponseplans.
• Abilitytomeasureoutcomes–Theremustbeahighlevelofconfidenceinthesystemoutcomes,andtheseoutcomesmustbecontinuouslymeasuredandmonitoredwithprocessesinplacetoimprovethesystemseffectivenessovertime.
• Incrementaldeployment–Thesystemmustbeabletobedeployedincrementally,addingnewcapabilitiesovertime.
• Systemflexibility–Thesystemmustbebuilttobeflexible,asitisintendedasapilotthatisadjustedandmodifiedasexperienceisgainedwithoperationsandgrowasnewcapabilitiesareadded.Inaddition,thisflexibilitywillbekeytofutureimplementationsinothercorridorswithdifferentintegrationneedsanddifferentrequirements.Itisalsoexpectedthattransportationitselfwillbechangingradicallyoverthelifetimeofthesystem,soflexibilityiskeytoitslongtermviability.
• Secureoperations–Asthesystemhasthecapabilitytorequestchangestotrafficcontrolsacrossalarge,complexurbancorridor,securityofthesystemiscriticaltoitssuccess.
Coresystemdesignislimitedbythefollowingconstraints:
• AbilitytobeoperatedandmaintainedbyCaltranswithoutsignificantlicensingcosts–Thesystemmustbedesignedwithopensourcecomponentsasmuchaspossible.Itisnotdesiredtohavesignificantlicensingcostsrequiredforsubsequentdeploymentsforfuturecorridorsoroverlongperiodsoftime.ItisintendedthatCaltranswillbecapableofdeploying,maintaining,andoperatingthisandfuturedeployments.
• Limitedtimetodeploy–Thedeploymentscheduleisaggressive,creatingtheneedtoensuresignificantreusabilitywithinthedesignsothatmultipledatasourcescanbeincorporatedintothesystemwithoutdesigningnewdatapipelinesforeachone.
I-210Pilot:CoreSystemHigh-LevelDesign
9
• Constraineddevelopmentresources–Caltransprovidedafixedamountoffundingsothesystemdesignmustnotexceedourresourceswithcomplexityandadditionalfeatures.
Thesecoredesignobjectivesandconstraintsareimplementedwiththefollowingdesignprinciples:
• Clouddevelopment,deployment,andoperations–Inordertoachievemaximumflexibilityinbothsystemdevelopmentandfutureconfigurations,minimizeresourceutilizationindevelopmentanddeployment,andminimizedeploymenttime,thecoresystemisdesignedfor100%cloudoperationswithinanAmazonAWScloudenvironment.
• Thesystemisapilotsystemanditexpectedthataswelearnfromthedeploymentofthesystemchangeswillbeneededandrecommended.
• ThesystemshallbecapableofbeingsupportedbyCaltransinternalresources.• Decisionsproducedintheformofresponseplansshallbebasedoncurrentcorridorinformation
withlimiteddelaysininformationprocessing.• Datamaintainedwithinthecoresystemwillbelimitedtothedatarequiredforsystem
operation.LongtermdatastorageshallbeafunctionofPeMS.• Thesystemwillbedesignedforfuturegrowth,incrementalimprovements,future
transportationsystemchanges,changesintransportationitself,geographicchanges,andnewandupdatedinformationsourcesandneeds.
• Easeofdeployment• Easeofmaintenance• Portabilityofthesystemdesignandcomponentstoothercorridors• Highlydecoupleddesignforflexibility,scalability,redundancy,andreliability• Useavailabledatastandardswheneverpossible• Standardizedexternalinterfacesforeaseofportabilityandinclusionofnewsubsystems• Maintenanceofaseparationofconcernsbetweendatamanagement,decisionsupport,and
controlfunctions.• Designwithstate,regional,andlocallayersforfuturescalabilityandstandardizeddeployment
acrossthestate.
I-210Pilot:CoreSystemHigh-LevelDesign
10
4. CORESYSTEMHIGHLEVELDESIGN
ThecoresystemhighleveldesignisillustratedinFigure4-1.Thishighleveldesignprovidesclearseparationbetweenexternalsystemsprovidingdataandreceivingcontrolrequests(green),thedatahubreceivingandprocessinginformationfromthoseexternalsystems(red),decisionsupportprovidingresponseplansforincidents(blue),andthecorridormanagementsystemprovidinguserinterface,responseplanselectionandapproval,andsendingofcontrolrequeststoexternalsystems(purple).
Figure4-1CoreSystemHighLevelDesign
4.1. MAJORCOMPONENTS
AsshowninFigure4-1,theICMCoreSystemiscomposedofthreemajorsubsystems:theDataHub,theDecisionSupportSystem,andtheCorridorManagementsubsystem.Thesesubsystems,plustheexternalfieldelementstheCoreSysteminteractswith,aresummarizedinthefollowingtable(color-codedtomatchFigure4-1)anddescribedindetailinthefollowingpages.
Component Description
DataHub Providesforreceiptandprocessingofallcorridordataandamethodofcommunicationamongthethreesubsystems,usingacommonsetofdatadefinitions.Providesoperationaldatapersistenceandretrieval.
DecisionSupport Providescurrenttrafficstate,responseplandevelopment,andresponseplanperformancepredictiontoprovideoneormorerecommendedresponseplansforincidentsandevents.
CorridorManagement Providestheprimaryuserinterfaceforsystemoperators,amethodforusersto
I-210Pilot:CoreSystemHigh-LevelDesign
11
monitorcurrentcorridorandcorridorassetstates,interactwiththeDecisionSupportSystem,review,evaluate,select,andapproveresponseplans,andinteractwithexternalsystems,particularlyforexecutionandmanagementofresponseplans.
FieldElements Representexternalprovidersandconsumersofinformationandrequestsforactionsuchasintersectionsignals,rampmeters,corridorsensing,andotherelements,usuallythroughrespectivetransportationmanagementsystemsandTMCs
ThedesignfortheI-210Pilotwillnothavealayeredgeographicapproach,butthedesignisintendedtoallowsuchanapproachforfutureuse.Therearethreegeographiclayerstothedesign:state,regional,andlocal.TheintentisthattheDataHuboperatesonaregionallevel(Caltransdistrict),withthepotentialforservingmultiplecorridorswithinaregion.DatawouldbeconsolidatedandaggregatedatastatelevelformultipleregionsviaPeMS,tosupportlargerscalearchiving,businessintelligenceanalysis,andcontinuedimprovementofcorridorandICMsystemcapabilitiesatthestatelevel.TheDecisionSupportandCorridorManagementcomponentsoperateonalocallevel,withinasinglecorridor.Withstateconsolidationofdata,regionaldatacommunication,andlocaldecisionsupportandresponseplanexecution,amethodofsharinginformationbetweencorridorsisenabled,whileensuringlocaljurisdictioncontroloftrafficeventandincidentresponse.
4.2. FIELDELEMENTS
GreenelementsinFigure4-1representthevariouscorridorfielddatasourcesandfieldelementcontrols,includingvariousstate,regional,andlocaltransportationsystems,regionaltransportationdatanetworks,andprivateinformationproviders.
• Ontheleftsideofthediagram,theseelementsrepresentthevariousdatasourcesforinformationconsumedandprocessedbytheICMsystem.
• Ontherightside,theserepresentthevariouscontrolinterfacestoexecuteresponseplanelementsselectedbyoperatorsoftheICMsystem.
ThesystemsthatwillbeprovidingthedatawithintheI-210corridorinclude:
Table4-1–FIELDSYSTEMS
Source InformationTypePasadenaTMC* PasadenaintersectionsignalsArcadiaTMC* Arcadiaintersectionsignals
LACountyintersectionsignalsDuarteintersectionsignals
CountyTMC*
MonroviaintersectionsignalsFreewayloopsensing(traffic)Rampmeters
CaltransATMS**
Dynamicmessagesystem
I-210Pilot:CoreSystemHigh-LevelDesign
12
IncidentinformationCaltransLCS* LaneclosuresCaltransTSMSS* IntersectionsignalTrailblazerSystem* Dynamicmessagesystemforrerouting
LocalvideoRIITSCaltransvideoEnvironmentalsensingCalPolyODSTransit
HSRLaneClosure* CitylaneclosuresNextBus Goldlinetransit
*Indicatesfieldsystemthatacceptscontrolmessages
**Caltrans’ATMSsystemacceptscontrolmessagesandprovideslimitedICMcoresystemcontrol
Futurefieldsystemsmayincludeprobedataproviders(GPSlocation/speed),additionaltransitinformation,parkinginformation,andothers.
`
4.3. DATAHUB
DatafromeachofthefieldsourcesarereceivedbytheICMDataHub,representedinredinFigure4-1.TheprimaryfunctionsoftheDataHubare:
• Receivedatafromfieldelementsviaexistingcorridortrafficmanagementcentersandregionaldatanetworks:VariousdatareceiversreceivedataandprepareitforprocessingbytheICMsystem.Thesereceiversaregenerallyexpectedtobebuiltforthespecificinterfacesdefinedbyeachfielddatasource,bothintransmissionmethod(RESTorSOAPwebservices,socket-basedstreaming,file-based,orothers)andtheintendedinitialpathrequiredforsystemdataprocessing.ThepreferredmethodofinformationtransferforthesystemistheTrafficManagementDataDictionary(TMDD)standarddevelopedbytheInstituteforTrafficEngineers(ITE),currentlyatversion3.03d(withcertainmodifications).
• Processdatareceivedfromfieldelements:Datafromfieldelementsmustbevalidatedforcompletenessanddataqualitypriortousebydownstreamsystemcomponents.Withsuchavarietyofdatasources,oftenforthesametypeoffieldelements,datamustbetransformedintoacommonsetofdatasemantics.TheDataHubprocessesallincomingdataintoastandardizedformat,TMDDfortransportationassets,GTFSfortransitinformation,orothersdependinguponthedatabeingreceived,withacommonsetofdatadefinitionsaswell(suchasasinglenamingstandardforallstreetswithinthecorridor).However,itiscriticaltonotethatthistransformationintoastandardizedformatforprocessingwithinthedatahub,whileitmaintainsthedatastructureswithinTMDD,doesnotgenerallymaintainanXMLdataformatandinternalcommunicationswithinthedatahub(andtheDSS)donotuseSOAPprotocols.ThedatahubandDSSuseJMSprotocols,eitherwithinanActiveMQorKafkamessagingsystem.
I-210Pilot:CoreSystemHigh-LevelDesign
13
Additionalprocessingisalsocompletedwithinthedatahub,eitherforspecificmetrics,calculatedparameters,orpredictiveanalytics.
• Datamessagingandcommunications:Dataprovidedtodownstreamsystems,aswellasinternalsystemcommand,control,andstatusdata—indeed,alldatawithintheICMsystem—ismadeavailableviaaninternaldatabus.Themethodofdatatransportisviadatamessagingtechnologies,namelyActiveMQJMSmessagingorKafkadatamessagingsystems.Thespecificmessagetechnologyisbasedonthetype,size,frequency,andmessagepersistencerequirementsofthedata,dataproducer,anddataconsumers.Exceptionstothesemethodswithinthedatahubarelimited,andaregenerallyusedforlargeblockbulkdataarchiving/datatransportbetweenpersistencestores.InordertoaccommodatemultipleCMSvendors,andtoprovidecommunicationbetweenthedatahub’sKafkamessagingsystemandtheDSSActiveMQmessagingsystem,anApacheCamelbasedinterfacebetweenthedatahubandtheDSSandCMSsystemsisincludedwithinthedatahub.ThisinterfaceprovidestheabilitytoprovideSOAPandRESTbasedinterfacesasnecessary,andtranslationbetweenmessagingsystems.
• Datapersistence:TheDataHubalsoprovidesdatapersistencecapabilities,allowingforpersistenceofrawandprocesseddata,systemcommand/control/status,andothersysteminformationinacentralrepository.Thisdatapersistenceisbrokenintodifferenttimelayers,includinglivereal-timedata,youngdata(0-90days),aggregateddata,andarchiveddata.ItincludesleveragingstateEDWandBIresourcesasacomponentofthedatapersistencecapabilities,specificallyasthesingledatastorefornon-operationaldata(dataolderthan90days).Sincedatapersistencewithinthedatahubislimitedtooperationaluses,andthearchitectureisbasedonapatternofcoreservicesconnectedbymessaging,multipledatapersistencetechnologiesareutilizedspecifictotheneedsofthesystemandthedatabeingstored.Largetimeseriescollectionsofdata,suchassensordata,ismaintainedinaCassandradatabase;largerelationalstructureswithsmallerupdatefrequenciesaremaintainedwithinaPostgresdatabase,andMongoDBisusedwithindatatranslationpipelineswhenmultiplesourcesprovidethesametypeofinformationindifferingformatsandimplementations.NOTE:MongoDBmaybeusedindevelopmentandearlyversionsofthesysteminsteadofPostgresforsomeoftherelationaldatastores.
4.4. DECISIONSUPPORT
TheDecisionSupportSystem,portrayedinblueinFigure4-1,providesthefollowingcapabilities:
• Corridortrafficstatedetermination:TheDecisionSupportSystemprovidescorridortrafficstateestimation,providingbothgeospatialtrafficstateinformationandtrafficstatemetrics.Separatearterialandfreewaytrafficmodelsareusedandmergedtoprovidefullcorridortrafficstateatalltimes.
• Corridortrafficstateprediction:TheDecisionSupportSystemprovidescorridortrafficstatepredictionsforusebycorridoroperatorsandforpredictionofincidentandeventresponseplanperformance.Acommercialsimulationengine(TSSAimsun)isusedtoprovidetraffic
I-210Pilot:CoreSystemHigh-LevelDesign
14
predictions.Trafficpredictionsaregeneratedusingthetrafficestimatedstateasaninitialtrafficstate,andcurrentcorridorassetstatefromthedatahub.
• Responseplandevelopmentandevaluation:TheDecisionSupportSystem,uponnotificationofaconfirmedincident,willusearules-basedapproachalongwithtrafficstateestimationandpredictioncapabilitiestodevelop,evaluate,andrankresponseplansforusebycorridoroperators.
4.5. CORRIDORMANAGEMENT
TheCorridorManagementsubsystem,portrayedinpurpleinFigure4-1,shallprovidethefollowingprimarycapabilities:
• Presentationofcorridorstate:Displayofcorridorstate,includingcorridorassets(signals,ramps,cameras,dynamicmessaging,transit,etc.),trafficestimatedstate,trafficvisualization(cameraoutput),humanassets,physicalassets(vehicles,responsecrews).Assetstateincludescurrentoperationalcapabilities,currentoperatingstate,currentfunctionalstate(operating,failed,degradedstates),timeavailability,controllability,etc.
• Controlofcorridorassets:TheCorridorManagementsubsystemprovidesthecapabilitytocontrolcorridorassets,takingresponseplansapprovedforexecutionandexecutingeachoftheindividualresponseplanelementsbysendingcommandstotheappropriatestate,regional,andlocalsystemsandentities.TheCorridorManagementsubsystemdoesnotdirectlycontrolcorridorassets,butonlysendscommandstothestate,regional,orlocalsystemsthatdirectlycommandcorridorassets.
• Presentationofresponseplans:TheCorridorManagementsubsystempresentseachresponseplandevelopedbytheDecisionSupportSystem,displayinginmeaningfulwaysthevariousresponseplanelements,howtheywillchangefromthecurrentcorridorstate,howtheychangeduringresponseplanexecution,whatthepredictedoutcomesareforeachresponseplan,whattheactualoutcomesareforaresponseplan,aswellasthevariousmetricsonhowtheresponseplansareevaluatedandtheireffectivenessmeasured.Thispresentationisexpectedtobeinmultipledomains,includinggeospatial,tabular,comparative,graphical,andothersolution-specificmethods.
• Responseplanlifecyclemanagement:Eachresponseplandevelopedbythesystemhasaspecificlifecyclethatincludes:
1. Initiation2. Development3. Evaluation4. Selection5. Approval
6. Execution7. Monitoring8. Close9. Post-incident/eventanalysis
I-210Pilot:CoreSystemHigh-LevelDesign
15
TheCorridorManagementsubsystemprovidesamanagementinterfaceforthislifecycle:
Initiation TheCorridorManagementsubsystempresentsincidentandeventinformationreceivedfromtheDataHubforreview,edit,andconfirmationbythecorridoroperators.
Development TheCorridorManagementsystemprovidesdisplayofstatusinformationreceivedfromtheDecisionSupportSystem(DSS)regardingitsdecisiontodevelopasetofresponseplanstoanincidentorevent,aswellasthedevelopmentofthoseresponseplans.Inaddition,itcansendcommandstotheDecisionSupportSystemtoinitiatespecificDSSfunctions,suchasmockincidentevaluation,manualresponseplandevelopmentorevaluation,orotherfunctionsrequiredfortheICMsystem.
Evaluation TheCorridorManagementsubsystemreceivestheresultsofresponseplanevaluationandtrafficstatepredictionanddisplaysthoseresults,alongwiththeresponseplans,tothecorridoruser.Displayshallincludeindividualplananalysis,aswellasrankingandcomparativeanalysisofresponseplans.
Selection TheCorridorManagementsubsystemprovidestheuserthecapabilitytoselectaspecificresponseplanforsubsequentapprovalandimplementation.
Modification TheCorridorManagementsubsystemshallprovideuserstheabilitytomakeminormodificationstoresponseplans.NOTE:Initialversionsofthesoftwarewillnothavethiscapability.
Approval TheCorridorManagementsubsystemprovidesaselectedresponseplantothecorridorstakeholdersatthestate,regional,andlocallevelsforreviewandapproval.Theseapprovalsshallincludeminormodificationstotheresponseplanandmayincludereevaluationandresubmittalforapproval.
Execution TheCorridorManagementsubsystemisthesolecomponentinthesystemcapableofsendingcommandstothevarioussystemsincontrolofcorridorassetsforindividualresponseplancomponentexecution.Itshallhavedisplaysthatallowcontrolandmonitoringofcorridorassetsviathestate,regional,andlocaltrafficmanagementsystemsandassets.
Monitoring Giventhecapabilitytodisplayassetstate,managecorridorassets,andexecuteresponseplanelements,theCorridorManagementsubsystemshallprovideanintegrateddisplayofresponseplanmonitoringoncetheexecutioncommandsaresenttothefieldsystems.Monitoringshallincludedisplayingwhencommandshavebeenexecutedandarecompleteatthefieldassetlevel,alertinguserswhencommandsorassetshavefailed,identifyinganddisplayingvariancesbetweenexpectedandactualtrafficstate,andtheabilitytotakeactionintheeventofresponseplanelementfailureforanyspecificcorridorasset.
Close TheCorridorManagementsubsystemshallbeabletoreturncorridorassetstotheirnormalstateoncetraffichasreturnedtoanormalstate.
Post-incident/eventanalysis
TheCorridorManagementsubsystemshalldisplayacorridorpost-eventanalysisforeachincidentorevent.TheanalysisshallincludeinformationfromtheCorridorManagementsubsystemitself,especiallywithregardtothecorridorassetresponseplanexecution,responseplanexecutionissuesand
I-210Pilot:CoreSystemHigh-LevelDesign
16
failures,trafficanalysis(expectedvs.actual),andothersystemperformancemeasures.Thepost-eventanalysisisprimarilylimitedtoinformationusedbytheICMsystemanditscomponents,aswellasmetricscalculatedpostincident/eventregardingresponseplanandsystemperformance.Itisnotintendedtobeanexhaustiveanalysiswithextensiveadditionalmodelinganalysisand“what-if”scenarios.
• ICMSystemmanagement:TheCorridorManagementsubsystemprovidesmanagementcapabilitiesfortheICMsystem,including:
Rulesmanagement TheCorridorManagementsubsystemprovidesauserinterfaceforthemanagementoftherulesusedwithintherulesengine.Thiswillincludetheabilitytocreatebasicrulesandrulesets,editrulesandrulesets,archiverulesandrulesets,provideconfigurationmanagementforrulesandrulesets,makerulesactive,executerulesandrulesets,andimportrulesandrulesets.Generally,rulesthatcanbeeditedwillbesimplerrulesandrulessets,asmorecomplexrulesarelikelytorequireprofessionaldevelopmenteffortandintegration.NOTE:Theinitialimplementationofthesystemwillnotincludethiscapability.
Responseplanmanagement
TheCorridorManagementsubsystemprovidesuserstheabilitytoupdatetheavailableresponseplansandresponseplanelements.Thisincludesadding,archiving,updating,andactivationordeactivationofresponseplansorresponseplanelementsforusewithinthecorridor.
Systemmonitoring TheCorridorManagementsubsystemprovidesuserstheabilitytomonitortheICMsystemandcomponents.Systemslogs,alerts,statusinformation,controlandexecutionofsystemfunctionsshallbeaccomplishedbytheCorridorManagementsubsystem.
Security TheCorridorManagementsubsystemprovidesuserauthenticationandauthorizationfortheCorridorManagementsubsystemandallfunctionsavailablewithintheCorridorManagementsubsystemthataffectotherICMsystemfunctions,includingDecisionSupportandtheDataHub.ThisdoesnotincludeuserauthenticationandauthorizationofindividualcomponentssuchasCassandra,Postgres,ormessaging;rather,itprovidestheseservicesforthefunctionsthatusethosecomponents.
Configuration TheCorridorManagementsubsystemprovidesmethodsandinterfaceforallowinguserstochangeanyandallconfigurationelementswithintheICMsystem.Aswithsecurity,thisdoesnotincludeconfigurationofindividualsystemcomponentssuchasCassandraandPostgres,butratherforthefunctionsthatusethesecomponents.NOTE:Theinitialimplementationofthesystemwillhaveminimalcapabilitiesforthisfunction.
I-210Pilot:CoreSystemHigh-LevelDesign
17
4.6. PRIMARYPROCESSFLOW
Figure4-2providestheprimarysystemworkflowillustratingthebasicintegrationbetweeneachofthesubsystems.Whilethisillustrationisnotintendedtocaptureallofthedetailsoftheinteractionsbetweensystems,andtheprocesslifetimesarenotintendedtobeaccuratewithinthissequencediagram,itdoesprovidethebasicsequenceofeventsandprimarycommunicationchannelsfortheprimaryworkflowwithinthesystem,aresponsetoanincidentonthecorridor.Secondaryactionssuchaspersistence,logging,communicationdialogs,andothersarenotdepictedinthisdiagram.Alsonotethat,asdescribedinFigure4-1,allcommunicationbetweentheDSSandCMSisviatheDataHub’sdatabus,althoughthespecificsofthecommunicationchannelarenotillustratedinFigure4-2.
Figure4-2PrimarySystemIncidentFlow(Subsystem)
Thebasicworkflowdescribedinthisdiagramisasfollows:
• Anincidentoccurswithinthecorridor.Thediagramshowstheincidentinformationbeingprocessedthroughthedatahub.
• TheincidentisconfirmedbyanoperatorwithintheCMSorCaltran’sATMS.Theinitialpilotsystemwillalwaysuseanoperatortoconfirmanincident.
I-210Pilot:CoreSystemHigh-LevelDesign
18
• TheincidentinformationispassedtotheDSS’sresponseplandevelopmentcomponent.• Theresponseplandevelopmentcomponentrequestsapredictionoftheimpactoftheincident
onthecorridor.• Thepredictionengineusesthecurrentestimatedtrafficstateandcorridortrafficstateto
initializeandrunapredictionwithnoresponseplanexecution(“donothing”prediction).• Theresponseplandevelopmentcomponentusesthe“donothing”predictionandtherules
enginetodetermineifaresponseplanshouldbedeveloped.Iftheresultisnoresponseplanshouldbedeveloped,executionoftheworkflowwouldendwithnotificationtotheCMSofthatresult.
• Iftheresultisthataresponseplanshouldbedeveloped,theresponseplandevelopmentcomponentagainusestherulesengine,resultsofthe“do-nothing”prediction,currentcorridorstate,andasetoffixedresponseplancomponentstodevelopalimitednumberofresponseplansforevaluation.Itsubmitsthoseresponseplanstothepredictionengine.
• Thepredictionenginerunsapredictionforeachresponseplan,alongwithanother“donothing”prediction”basedoncurrentcorridorconditions.Itprovidestheresultsofthosepredictionstotheresponseplandevelopmentcomponent.
• Theresponseplandevelopmentcomponentusestheresultsofeachprediction,currentcorridorstate,andtherulesenginetoevaluate,rank,andselectarecommendedresponseplan.
• TheresultsofresponseplanpredictionsandevaluationisprovidedtotheCMSforoperatorselection.
• TheCMSprovidestheresultstoanoperator,whoselectstheresponseplantobeimplementedandobtainsapprovalfortheresponseplanimplementation.
• Iftheresponseplanisapproved,theCMSexecutestheresponseplanbysubmittingC2CcommandstotheTMCsandothersystemsrequiredtoexecutecommandstoindividualcorridorassets.
• TheindividualTMCsandothersystemsexecutingtheresponseplancommandsreportthesuccessorfailuretoexecutethedesiredresponseplanelements,andcontinuetoreportcorridorassetstate.
Uponcompletionofthisprimaryworkflow,furtherprocessesinvolvedinresponseplanlifecyclemanagementwilloccur,suchasresponseplanevaluation,responseplangenerationtoadjusttocurrentconditions,responseplancancellation,andresponseplancloseout.
5. DATAHUBDESIGN
Thedatahubhasfiveprimaryroleswithinthesystem:
1. Receiveinformationfromexternalprovidersofcorridorstate2. Processinformationreceivedfromprovidersofcorridorstate,evaluatingthedataforquality,
providingcommonmetricsandanalysisofthedatareceived,conductingpredictiveanalyticstosupportestimationandpredictionwithintheDSS,andstandardizingtheformatandcontentfordownstreamconsumptionbytheDSSandCMSsystems.
3. Persistinformationreceived,resultsofprocessing,andoverallsystemstateandcommandinformation.Persistoperationallyrequiredinformationlocallyandsendtoarchiveolder
I-210Pilot:CoreSystemHigh-LevelDesign
19
informationforlongertermstorage.Retrieveinformationstoredupondemand(operationaldataforimmediateretrieval,archiveddataforlater,scheduledretrieval).
4. Secureinformationreceived,processedandstored.5. ProvidedatacommunicationsbetweentheprimarysystemsoftheICMcoresystem.
Inordertofulfilltheseroles,thedatahubprovidesaseriesofdatapipelinestoreceiveinformation,processtheinformation,persisttheinformation,andsecuretheinformation.ThepipelinesuseKafkaandActiveMQmessagingsystemstotransmitinformationthatcanbetappedbypersistenceworkerstopersistinformationtothevariousdatastoreswithinthedatahub,andretrievetheinformationandplacetheinformationbackonthemessagingsystemsfordownstreamconsumption.Externalinterfaces,usingApacheCamel,provideinformationandcommunicationbetweenthedatahub,theDSS,andCMSsystems.
Figure5-1DataHubHighLevelDesign
Thisdesignconsistsofindividualservicesconnectedviamessaging.Itisahighlydecoupleddesignbasedonagroupofindependentservicesconnectedbytwomessagingplatforms.Figure5-1providesageneralizeddiagramthatillustratesthreebasictypesofdatapipelines.Sensordataisgenerallyahighfrequency,largerdatavolumepipelinethathandlesalargernumberofsmallermessages.Theheterogeneousdatapipelinetypehandleslargersizedmessagesatlowermessageprocessingfrequencies.Heterogeneousdataischaracterizedbymultiplesourcesprovidingthesametypeofdatainvaryingdegreesofformatandcontentdifferences.Thehomogeneousdatapipelinesaresimilartotheheterogeneousdatapipelines,butgenerallyarebuiltforasingledatasourcethatprovidesallofa
I-210Pilot:CoreSystemHigh-LevelDesign
20
specifictypeofdata.Notethattheheterogeneousandhomogenousdatapipelinesprovidesomedataintothesensordatapipelinewhensensordataispartofthedatareceivedwithinthesepipelines.
Adescriptionofthedifferenttypesofcomponentsisprovidedbelow:
Reader–Readersarethebeginningofthedatapipeline.Theirroleissimple–toconnectandgatherthedatafromadatasourceusingtheprotocolandformatnativetothatdatasource,andtoplacetheinformationwithinadatamessagingchannelfordownstreamprocessing.Ingeneral,itdoeslittleornoprocessingofthedata,withanyprocessinglimitedtosimpleparsingoflargebatchesintoindividualmessages.
Spark–SparkisanApacheSparkinstancethatprovideshighspeed,highvolume,realtimedataprocessingforsensordata.Itsroleistoprovidedatatransformation,dataquality,dataprocessing,andpredictiveanalyticsforsensordata.Itreceivesdatafromreaders,processesthatdata,andplacesthedataintoKafkamessagetopicsforusebydownstreamprocesses.
Cassandra–CassandraisanApacheCassandrainstancethatprovidesstoragefortimeseriesdata,primarilysensingdata.Cassandraprovideshighspeed,highvolume,clusteredstorageforsensingdata.Thisdataislimitedtooperationaldatastorageandisnotintendedasalong-termdatastore.
Postgres–Postgresisanopensourcerelationaldatabaseinstancethatprovidesstorageforrelationalandgeospatialdata,suchasintersectionsignalplans,rampmeterplans,roadnetworks,assetinventories,organizationalinformation.Itisanoperationaldatastoreandnotintendedasalongtermdatastore.
Persistenceworkers–PersistenceworkersprovidebothstorageandretrievalservicesforbothPostgresandCassandradatastores.Persistenceworkerslistentoassignedmessagetopicsandqueues,bothKafkaandActiveMQ,andstoretheinformationontheappropriatedatastore.Requestscanbesenttoapersistenceworkerviaacommandmessagechanneltoretrievedataandplacethatdataonaspecifiedmessagetopicorqueue.
Processors–Processorsareusedinthehomogeneousandheterogeneousdatapipelinestoprocessdatareceivedfromthevariousdatasources.Processorsprovidedatatransformation,quality,anddataprocessingservicesfordatareceivedfromthesesources.
Mongo–MongoisaninstanceofaMongoDBdatabaseandisusedwithintheprocesstotransformdatareceivedonaheterogeneousdatapipeline.ReadersstorethedatareceivedfromasourceintoMongoDB.ProcessorsthanqueryMongofortheinformationrequiredforprocessinganddownstreamprocessesincludingdecisionsupportandcorridormanagementsystems.MongoDBprovidesanidealplatformforstoringinformationretrievedinanativeformatandfastqueryingofthisinformationwithminimalmaintenancerequiredwhensourceformattingischanged.
Datahublogger(DHlogger)–ThedatahubloggerisaninstanceofGreylogthatcollectsandindexesallloggingforeachindividualcomponentwithinthedatahub,andlistenstomessagingtocapturecontrolandprocessloggingmessages.Itsroleistocapturesystemstateanderrormessagesforsystemstatusandtroubleshootingneeds.
I-210Pilot:CoreSystemHigh-LevelDesign
21
Datahubcontrolgateway(DHcontrolgateway)–ThedatahubcontrolgatewayisanAPIgatewaythatprovidesroutingandmanagementofcontrolandstatusmessagesforthedatahub.ItsroleistoreceivemessagesroutedfromtheinterfacesfortheDSSandCMS,aswellasinternaldatahubcontrolchannels,androutethosetotheappropriatechannelsforthereceivingprocess.Thisprovidesencapsulationofindividualservicesandensuresthatnewservices,servicechanges,andmessagesystemmodificationscanbeaccomplishedwithoutimpactingothersystems,andiskeytothedecouplingofthesystemservicesfortheentireICMsystem.ThisgatewayusesanimplementationofApacheCamel.
Externalinterface/Datagateway–Theexternalinterface/datagatewayencapsulatesthefunctionsofthedatahub,providingasortofswitchboardforusebytheCMSandDSSsystems.CMSandDSSsystemsarenotrequiredtoknowtheinternalsofthedatahubmessaging,butinsteadrelyontheexternalinterface/datagatewaytoprovideprotocolspecificcommunicationportsthatareappropriatefortheCMSandDSSandthentoroutemessagestoandfromthedatahubforthem.Thisexternalinterface/datagatewayusesApachecameltoprovidebothroutingandtranslationservices.Internaldatahubmessages(KafkaorActiveMQ)aretranslatedintotheprotocolsusedbytheCMSandDSS(ActiveMQ,RESTservices,orSOAPservices).CommonTMDDandGTFSmessageformatsareusedinmostcommunications,sonotranslationofdataformatsisrequiredbythegateway.
PeMS–PeMSisastatesystemthatcurrentlyprovidesstatetransportationdataservices.PeMSwillprovidelongtermstorageforallsystemdata,archivingservices,andreportingservicesfornon-realtimedataandreportingneeds.
BusinessIntelligence–BIserviceswillbeprovidedbytheCaltrans’OBIEEimplementation.
5.1. DATASOURCES
ThefollowingdatasourceshavebeendefinedfortheI-210ICMproject:
Source InformationType
System Vendor Product C2CTMDD?
Pasadena Intersectionsignal
PasadenaTMC
McCain Transparity Planned
Duarte Intersectionsignal
CountyTMC KimleyHorn
KITS Planned
Monrovia Intersectionsignal
CountyTMC KimleyHorn
KITS Planned
Arcadia Intersectionsignal
ArcadiaTMC Transcore Transuite Planned
CaltransFWTraffic Loopsensing CaltransATMS Parsons Custom N
I-210Pilot:CoreSystemHigh-LevelDesign
22
CaltransFWRamps Rampmeters CaltransATMS Parsons Custom N
CaltransFWCMS DMS CaltransATMS Parsons Custom N
CorridorTrailblazer DMS Notyetidentified
Custom Planned
CaltransIntersections Intersectionsignal
TSMSS Transcore Transuite Planned
Bluetoothtraffic1 Traveltime Vendor Iteris/? Unknown
Bluetoothtraffic2 Traveltime CountyTMC ? Unknown
Environmentalsensing Environmental ? CalPoly ODS ?
RIITSTransit Transit ? CalPoly ODS ?
RIITSVideo* Video RIITS
CaltransVideo* Video viaRIITS
CaltransFWLaneclosure Lanestatus LCS
CityLaneclosure Lanestatus StateHSRsystem
StateofCA Custom N
LACounty Intersectionsignal
CountyTMC KimleyHorn
KITS Planned
CHPCAD Incident CAD manualinputbyoperator
Caltransincident Incident CaltransATMS Parsons Custom Planned
Goldlinetransit Transit NextBus NextBus
I-210Pilot:CoreSystemHigh-LevelDesign
23
*Videoisnotpassedthroughorstoredwithinthedatahub.
5.2. DATAPIPELINES
5.2.1. SENSINGDATAPIPELINE
ThesensingdatapipelinedesignisprovidedinFigure5-2.Inthisillustration,thedifferentsensorfeedscanbeseenontheleft.Thisillustrationshowsthereaderreceivingdatafromadatasource,placingthatdataonaKafkatopic,SparkconsumingthatdataandprocessingthedatautilizingtheMLLibandStreamingSparklibraries,andthenplacingtheprocesseddataonaKafkatopic.DecisionSupportandCorridorManagementsystemscanaccessthatprocesseddatadirectlyviathedatahubdatainterfaces.SparkalsostoresresultsontheCassandracluster.Persistenceworkerscanbeusedtoretrieveprocesseddatawhenrequested,anddatathatisstoredinPostgres,suchassensorinventories,isavailabletoSparkifneededforprocessingthatmayrequiresuchdata.
Figure5-2SensingDataPipelineDesign
ThedatasourceslistedinSection5.1thatwillbeprocessedusingthistypeofpipelineinclude:
Source InformationType
System DataType C2CTMDD?
Pasadena Intersectionsignal
PasadenaTMC
SignalSensing Planned
I-210Pilot:CoreSystemHigh-LevelDesign
24
Duarte Intersectionsignal
CountyTMC SignalSensing Planned
Monrovia Intersectionsignal
CountyTMC SignalSensing Planned
Arcadia Intersectionsignal
ArcadiaTMC SignalSensing Planned
CaltransFWTraffic Loopsensing CaltransATMS
FWsensing N
CaltransFWRamps Rampmeters CaltransATMS
Rampsensing N
CaltransIntersections Intersectionsignal
TSMSS SignalSensing Planned
Bluetoothtraffic1 Traveltime Vendor Traveltime Unknown
Bluetoothtraffic2 Traveltime CountyTMC Traveltime Unknown
RIITSEnvironmentalsensing
Environmental ? EnvironmentalSensing
RIITSTransit Transit ? Transitlocation(future)
LACounty Intersectionsignal
CountyTMC SignalSensing Planned
Goldlinetransit Transit NextBus Transitlocation(future)
5.2.2. CALTRANSFREEWAYTRAFFICDATAPIPELINE
TheCaltransFreewaytrafficdatapipelineisasensingdatapipeline.Itconsistsofallofthestandardsensingpipelinecomponents:
• Kafkatopicsfordatatransmission• Readerforretrievingdata• Sparkfordatatransformation,qualitychecks,machinelearning• Cassandraforpersistence
I-210Pilot:CoreSystemHigh-LevelDesign
25
Figure5-3illustratesthisdatapipeline.
Figure5-3FreewaySensingDataPipeline(PeMS)
PeMS30secondfilesareaflatformatdatafile.DataisretrievedbythereaderviaFTP.Thereaderutilizesaconfigurableschedulerthatwillpollfordatafiles.Whenoneormoredatafilesisavailable,thereaderprocessesthefilesintimeorder,withtheoldestfilefirst.Eachrowwithinthefilecontainstheinformationforasinglesensoratasingletimestep.Allsensorsforasingletimesteparecontainedwithinthefile.Thetimeofeachreadingislocatedinthefilename.ThereaderparseseachrowofthefileandcreatesasingleKafkamessageforeachsensorandtimestep,appendingthetimesteptotheinformationpassedwithinthemessage.Uponcompletionofthefileprocessing,astatusmessageforthefile,includingthetimestepandtheKafkapositionofeachmessageissenttoastatustopic.
Uponreceivingthestatusmessage,Sparkprocessesthemessages,completingthefollowingsteps:
1. ThemessagesaretransformedintoaTMDDdatastructure(Note:thisisnotaTMDDformat,andKafkamessagesareJSON,notXML,anditisnotpassedviaaSOAPprotocol.However,thedatastructureisidenticaltoaTMDDstructure.)
2. Aninventorycheckiscompletedforeachtimestep.3. Aflowbalancecheckiscompletedforallsensorson4and6hourrollingtimewindowsanda24
hourstatictimewindow.4. Ademandpredictioniscompletedforthenexthourforeachfreewayramp.
SparkstorestheresultofitsprocessinginCassandraandpassesthesensinginformationtotheDH_pems_processedtopic.
5.2.3. HETEROGENEOUSDATAPIPELINE
TheheterogeneousdatapipelinedesignisprovidedinFigure5-4.Inthisillustration,thedifferentintersectionsignalfeedscanbeseenontheleft.
ThisillustrationshowsthereaderreceivingdatafromadatasourceandinsertingthatdatainarawmessageformatintoMongoDB.Thereaderretrievesdatafromthedatasourceusingthedatasource’s
I-210Pilot:CoreSystemHigh-LevelDesign
26
nativeprotocolsandformats.ItisdesiredthattheformatbeTMDDformat,andtheprotocolisastandardSOAPprotocol,eitherarequestreceiveorsubscriptionmethodasspecifiedintheTMDDspecification.ThereadertransformsthemessagefromaSOAP/XMLformattoaJSONformatpriortoinsertingintoMongo.
Amessageissentfromthereadertotheprocessortoinformtheprocessoroftheavailabilityofnewdata.
TheprocessorthenqueriesMongotoobtainadesireddocumentwiththespecificattributesrequiredtomakeaTMDDmessagecontainingastandardizedsetofinformationrequiredfordownstreamprocessing.Theprocessoralsoutilizesastandarddictionaryofdatavaluessothatdatafromdifferentsourcesthatmaybedifferentinspecificdefinition,butrepresentthesamevaluearestandardizedfordownstreamconsumption.Forexample,ifonesourceabbreviatesboulevardasBlvd,andaseconddoesnotabbreviatetheword,theprocessorwillstandardizetoasingledefinitionthatcanbeunderstoodbythedownstreamprocesses.Theprocessoralsoconductsdataqualitychecksandanyotherprocessingrequired.
OncetheTMDDtransformation,qualitychecks,andanyprocessingiscompleted,theprocessorconstructsthefinalTMDDstructuredJSONmessageandplacesitontheoutgoingtopic.Persistenceworkerslisteningtothetopicpersisttheinformationforlateranalysisorreplay,andtheCMSandDSSsystemsmayconsumethemessagedirectlyfromtheprocesseddatatopic.
Forintersectionsthatprovideintersectionsensing,theprocessorwillretrievethesensinginformationandsenditviaaseparatetopicforfurtherprocessingwithinthesensing/Sparkstream.
Figure5-4HeterogeneousDataPipeline
I-210Pilot:CoreSystemHigh-LevelDesign
27
5.2.4. HOMOGENOUSDATAPIPELINE
ThehomogenousdatapipelinedesignisprovidedinFigure5-5.Inthisillustration,arampmeterinventoryandstatesourceisusedastheexamplefeedontheleft.Thehomogenousdatapipelineisidenticaltotheheterogeneousdatapipelinewithonlyonedifference.Homogenousdatapipelinesareusedwhenthereisonlyasingledatasourceforthetypeofinformationbeingprocessed.Whenthisoccurs,thereisnoneedtostorethedocumentsreceivedina“schema-less”database,sincetherearenotmultipledatastandardsandimplementationstoaccommodate.Becauseofthis,thehomogenousdatasourceremovestheMongoDBinstallationfromthepipeline,andthereaderandprocessordirectlycommunicateforanydatatransformation,qualitychecks,orprocessingrequired.
Figure5-5HomogenousDataPipeline
5.2.5. PIPELINECONTROL
Allpipelines,regardlessofpipelinetype,areprovidedwiththesamebasecontrolset.Thecontrolsavailableforeachpipelineinclude:
1. Startpipelineprocessing2. Stoppipelineprocessing3. Pipelinestatus4. Replaydata
Allpipelinecontrolrequestsarehandledbythedatahubcontrolgateway.Requestsreceivedbythegatewayareroutedtoanappropriateendpoint.
Startpipelineprocessing–thestartpipelineprocessingwillberoutedviathedatahubcontrolgatewaytotheappropriatereaderwithinapipeline.Therequestwillresultinthesepossibleoutcomes:
I-210Pilot:CoreSystemHigh-LevelDesign
28
1. Ifthepipelineiscurrentlynotstarted,thereaderwillbeginprocessingdataandinsertingitintothepipeline.Thedataprocessedwillbebasedontheconfigurationofthespecificreader.
2. Ifthepipelineisalreadystarted,anerrorwillbereturnedfromthereaderindicatingtheprocessisalreadystarted.
3. Ifthepipelinecannotbestartedforanyreason,anerrorwillbereturned,fromthereaderifpossible,orfromthedatahubcontrolgatewayifnomessageisreceivedfromthereaderorifnoreaderinstanceexists.
4. Anothererror,suchaspipelinedoesnotexistorthattherequestisnotproperlyformatted.
Stoppipelineprocessing–thestoppipelineprocessingwillberoutedviathedatahubcontrolgatewaytotheappropriatereaderwithinapipeline.Therequestwillresultinthesepossibleoutcomes:
1. Ifthepipelineiscurrentlyrunning,thereaderwillstopprocessingdataandwillcompleteinsertionofanycurrentlyprocessingdataintothepipeline.
2. Ifthepipelineisnotcurrentlyrunning,anerrorwillbereturnedfromthereaderindicatingtheprocessisalreadystopped.
3. Ifthepipelinecannotbestoppedforanyreason,anerrorwillbereturned,fromthereaderifpossible,orfromthedatahubcontrolgatewayifnomessageisreceivedfromthereader.
4. Anothererror,suchaspipelinedoesnotexistorthattherequestisnotproperlyformatted.
Pipelinestatus–thepipelinestatusrequestisarequestforasubscriptiontopipelinestatusmessagesthatareproducedatalltimesbyeachprocessingpointwithinapipeline,includingreaders,spark,processors,messagingsystems,andworkers.Thisincludestimesforwhichthepipelineisnotcurrentlyrunning,butthecomponentsarerunning(forinstancewhenthepipelineiscurrentlystopped,butavailableforprocessing).Allpipelinecomponentsprovideataminimumaheartbeatindicatingtheyarecurrentlyupandrunning,evenifnodataisbeingprocessed.
Foreachpipelinestatusrequest,thedatahubcontrolgatewaywillprovideachannelforasubscriptiontoakafkatopiccontainingallstatusmessagesforaspecificpipeline.Pipelinestatusrequestsshallalwaysincludewithintheresponsethetopicidentifier,andmostrecentkafkamessageoffset.Requestersmaythenconnecttothekafkachannelprovidedbythedatahubcontrolgatewayandsetupaconsumerforstatusmessages.
Thefollowingresponsesarepossiblewhenrequestingapipelinestatus:
1. Aresponsewithakafkatopicidentifierandaddress,alongwiththecurrentkafkamessageoffsetandcurrenttime.
2. Ifthestatusmessagesarenotcurrentlyavailable,anerrorwillbereturned.3. Anothererror,suchaspipelinedoesnotexistorthattherequestisnotproperlyformatted.
I-210Pilot:CoreSystemHigh-LevelDesign
29
Figure5-6StreamingPrimaryControlLayer
5.2.6. PIPELINESTATUSANDLOGGING
Allpipelinecomponentsprovideloggingandstatusinformation,regardlessofpipelinetypeorcomponent.LoggingincludestheloggingattheEC2orapplicationlevelthatisprovidedtotheEC2logsattheOSlevelandispublishedtotheGrayloginstancewithinthedatahub.StatusincludesstatusmessagespublishedtostatusKafkatopics,againcollectedbythedatahubGrayloginstance.
Loggingshallbeconfigurableandsetoncomponentstartandshallincludeataminimumthefollowinglogginglevels:
1. All2. Debug3. Fatal4. Error5. Warn6. Info
Logginglevelwillbesetuponcomponentinitializationwhenitisstarted.Ingeneral,duringnormaloperation,thefollowinglevelsshallreport–Error,Fatal,Warn,andInfo.TheinformationgeneratedbyloggingshallonlybeavailablethrougheitherinspectionoftheEC2instancewheretheyweregeneratedandviathedatahubGrayloginstance.
StatusinformationisproducedbythecomponentsandpublishedtoapipelinestatusKafkatopic.Thisinformationisavailabletoexternalconsumersviathedatahubcontrolgateway(usingadatahubstatusrequest)orthedatahubGrayloginstance.Datapipelinestatusmessagesshallbeprovidedforthefollowingataminimum:
I-210Pilot:CoreSystemHigh-LevelDesign
30
1. Pipelinecomponentheartbeat–astatusmessageindicatingthecomponentisrunningsentataregularinterval.
2. Pipelineerrormessage–astatusmessagesentindicatingthatanerrorhasoccurred.Generally,thiswillincludeFatalorErrorlogmessages.
3. Pipelinewarningmessage–astatusmessagesentprovidingawarningindicatingpotentialissuesinoperation.Generally,thiswillincludeallWarnloggingmessages.
4. Pipelineprocessingmessage–thisincludesmessagesindicatingprocessingofasetofdata,oftenatatimesteporfilelevel.Forinstance,inaTMDDsubscription,aprocessingmessagewouldbesetfromeachpipelinecomponentindicatingwhentheinformationfromamessageresultingfromthesubscriptionhasbeenprocessed.Itisnotguaranteedthatdifferentpipelineprocesseswillprocessinformationatthesamegranularity,however,eachstatusmessagewillprovidethecorridorentityidentifiersandtheoriginalmessageidentifierfromtheoriginatingsystemthatstartedthedataprocessing.
Pipelinecomponentsmayincludeotherstatusmessagesasneeded.
Figure5-7Status/LoggingLayer
5.2.7. CORRIDORMANAGEMENTSYSTEM-DECISIONSUPPORTSYSTEM(CMS-DSS)PIPELINE
TheCorridorManagementSystem–DecisionSupportSystem(CMS-DSS)PipelineprovidesthecommunicationbetweentheDecisionSupportSystem,DataHubandtotheCorridorManagementSystem:
• Trafficestimationresults,includinggeospatialinformationandtrafficmetrics• Trafficprediction,includinggeospatialinformationandtrafficmetrics• Responseplansandresponseplanevaluations• DSSStatus• CMSStatus
I-210Pilot:CoreSystemHigh-LevelDesign
31
• DSSControl• Trafficestimationandpredictionscenarioinformation
Inaddition,thepipelineprovidespersistencewithinthedatahubofallDSS–CMScommunications.ThepipelinedoesnotincludeanycommunicationsfromtheCorridorManagementSystemthatdonotinvolvetheDecisionSupportSystem.
AllCMS-DSSpipelinessupportTMDDcompliantSOAPdialogsinaccordancewiththeConnectedCorridorsDataCommunicationSpecification.SOAPendpointsarepresentattheCMSDataHubGatewaytosupporttheseconversationsandareexposedtotheCMS.ThedatafortheseSOAPinterfacesarecommunicatedtotheDSSviatheDataHubanditsKafkaandActiveMQmessagingsystem.AllcommandrequestsandresponsesarealsoprovidedtotheDataHubControlgatewayforanydatahubcontrolactionsrequired.WhileonlytobeprovidedasnecessarytosupportthedifferentCMSvendors,REST,ActiveMQmessaging,andKafkamessagingmayalsobeprovidedasinterfacestotheCMSinsteadofSOAPinterfaces.
AllCMS-DSSpipelinesarealwaysonpipelines.Forfutureuse,datahubswillsupportmultipleCMSandDSSinstances,soallindividualpipelinesarecreatedforeachCMS/DSSpair.AllmessagesbetweenCMSandDSSinstancesshallbecommunicatedthroughonlypipelinesdesignatedforthatspecificCMS-DSSpairandshallincludewithinthemessagesthesourceandtargetsystemidentifiers.
Figure5-8providesanillustrationoftheCMS–DSSpipeline,anditsassociatedprimarydatatopics,including:
• Incidents• ResponsePlans• DecisionResults• StatusRequest• StatusResponse• EstimationRequest• EstimationMetrics• EstimationResults• EstimationNetwork• EstimationScenario• PredictionRequest• PredictionResult• PredictionMetrics• PredictionNetwork• PredictionScenario
I-210Pilot:CoreSystemHigh-LevelDesign
32
Figure5-8CorridorManagement-DecisionSupportSystemPipeline
5.2.7.1. Incident
TheincidentpipelineprovidesconfirmedincidentinformationfromtheCMStotheDSS.ThispipelinemaystartasaRESTrequest,Kafkamessage,ActiveMQmessage,orSOAPincident.AlsonotethatthispipelinerepresentsaTMDDeventclassdialogsetwithbothrequests,responses,andsubscriptions.Subscriptionsarethepreferredmethodofcommunication.TheincidentpipelineincludesanincidentsubscriptionrequesttotheCMS,aswellasthecorrespondingsubscriptionresponses.IncidentrequestsareoriginatedfromtheCMSDataGateway.TheresponsesareprovidedtotheDSSviathedatahubasActiveMQmessages.
5.2.7.2. ResponsePlan
TheresponseplanpipelineprovidesresponseplanstotheCMS.AsthereisnoTMDDdialogforresponseplans,thisisaConnectedCorridorscustomcommunication.Responseplansareprovidedasan
I-210Pilot:CoreSystemHigh-LevelDesign
33
ActiveMQmessageviatheCMSDataGateway.ResponseplansareprovidedinaccordancewiththeConnectedCorridorsDataSpecification,andincludetheresponseplanelements,event/incidentindexidentifier,andassociatedperformanceindicatorsandmetrics.
5.2.7.3. DecisionResult
ThedecisionresultpipelineprovidestheresultsfromtheDSS,providingtheindexoftheselecteddecision,andthekeyvaluepairsforeachresponseplanindicatorandscoreelements.ThedecisionresultisnotaTMDDdialog,andisavailableviaActiveMQmessageviatheCMSDataGateway.
5.2.7.4. StatusRequest
ThestatusrequestisacommunicationchannelforrequestingDSSanddatahubstatus.ItisanActiveMQqueue/message.Requestsmustincludeanidentifierfortherequestingsystem.Aresponsewillincludeanaddressforreceivingstatusmessages.Statusrequestsareessentiallyasubscriptionrequest,withtheresponsesimplyalocationforsubscribingtoacontinuousstreamofstatusmessages.
5.2.7.5. Status
ThedecisionresultpipelineprovidestheresultsfromtheDSS,providingtheindexoftheselecteddecision,andthekeyvaluepairsforeachresponseplanindicatorandscoreelements.ThedecisionresultisnotaTMDDdialog,andisavailableviaActiveMQmessageviatheCMSDataGateway.DatahubandCMSstatusmessagesareprovidedonthesametopicspecifictotheCMS-DSSpair.
5.2.7.6. EstimationRequest
TheestimationrequestpipelineprovidesamethodfortheCMStorequestanestimation.Ifanestimationiscurrentlyrunning,theresponsetotherequestshallindicatetheaddresswhereestimationresultsarepublished.Iftheestimationisnotcurrentlyrunning,theDSSshallstartanestimationandtheresponseshallincludeamessageindicatingtheDSSisbeginninganestimation,andtheaddresswhereestimationresultsarepublished.
5.2.7.7. EstimationResult
Theestimationresultpipelineprovidestherawresultsofanestimationrequest,includingforfreewaystheestimateddensity,flow,andspeedforeachnetworklinkwithinthefreewayestimationnetwork,andforarterials,theestimateddensityforeachnetworklinkwithinthearterialestimationnetwork.
5.2.7.8. EstimationMetrics
Theestimationmetricspipelineprovidesestimationmetricresults,includingtraveltimeandcurrentdelayestimatesforthefreewayandarterialnetworks.
I-210Pilot:CoreSystemHigh-LevelDesign
34
5.2.7.9. EstimationNetwork
TheestimationnetworkpipelineprovidesthecurrentestimationnetworkinaTMDDformat.ThispipelinesupportsTMDDmessaging.
5.2.7.10. EstimationScenario
Theestimationscenariopipelineprovidesthecurrentestimationscenariodatastructure(arterialandfreeway).Itisnotexpectedthatthispipelinewillbeavailableintheinitialsystemdeployment.
5.2.7.11. PredictionRequest
ThepredictionrequestpipelineprovidesamethodfortheCMStorequestaprediction.Theresponsetotherequestshallindicatetheaddresswherepredictionresultsarepublished.Intheinitialversionofthesoftware,predictionsshallonlybeprovidedeitherasaresponsetoanincidentaspartoftheDSSprocess,orshallbeprovidedasareplaytoapreviousprediction.
5.2.7.12. PredictionResult
Thepredictionresultpipelineprovidestherawresultsofapredictionrequest.TODO:determinewhatfromAimsunwillbeprovided.
5.2.7.13. PredictionMetrics
Thepredictionmetricspipelineprovidespredictionmetricresults,includedtraveltimeandcurrentdelaypredictions.
5.2.7.14. PredictionNetwork
ThepredictionnetworkpipelineprovidesthecurrentpredictionnetworkinaTMDDformat.ThispipelinesupportsTMDDmessaging.
5.2.7.15. PredictionScenario
ThepredictionscenariopipelineprovidesthecurrentAimsunpredictionfile.Itisnotexpectedthatthispipelinewillbeavailableintheinitialsystemdeployment.
5.2.8. CMSCOMMANDSTATUSPIPELINE
CommunicationsbetweentheCMSandexternalsystemsorcentersarecapturedbytheDataHub.TheyarealsoprovidedtotheCMScontrolgatewaysothatthedatahubmayrespondtocommandsissuedbytheCMSsystemtoexternalsystems.Figure5-9providesanillustrationoftheCMSCommandStatuspipeline.
I-210Pilot:CoreSystemHigh-LevelDesign
35
Figure5-9CMSCommandStatusPipeline
ThisCMSCommandStatuspipelineisasimpleimplementationofanActiveMQtopicthatisprovidedacopyofallCMScommunicationswithanyexternalcenterorsystem.AllsuchcommunicationsareprovidedviaActiveMQmessagecontainingatextrepresentationofanymessagesentorreceived.Messagesarekeyedwiththeexternalsystemorcentercommunicatedwith,aswellaseithersendorreceivetoindicatewhetherthemessagewassenttotheexternalcenterorsystem(send)orreceivedfromtheexternalcenterorsystem(receive)aswellasthetimeofmessagesentorreceipt,asappropriate.AllmessagesaresavedviapersistenceworkertoaPostgresdatabase.TheDSSdatahubcontrolgatewayalsosubscribestothedatahubCMScommandstatustopicandmayrespondwithspecificcontrolactionswithinthedatahub.
Initially,thestoreforcommandswillbeGraylog,notPostgres.Graylogprovidesauserinterfaceandqueryenginethatallowsuserstointerpretandreviewsystemprocessesthroughthecapturedcommandstatusmessages.
5.3. EXTERNALINTERFACE/DATAGATEWAY
Thedatagatewayprovidesanexternalinterfacetothedatahubforthecorridormanagementsystemandthedecisionsupportsystem.Thepurposeofthisdatagatewayis:
• Providefortwo-waycommunicationbetweentheDataHubandtheDecisionSupportSystem• Providefortwo-waycommunicationbetweentheDataHubandtheCorridorManagement
System• EncapsulatethefunctionsofthedatahubfromotherICMcomponentsystems• Provideasecureinterfacetothedatahub• AllowforthechangefromKafkaorActiveMQprotocolsusedwithinthedatahubtoRESTor
SOAPbasedservicesaswellasKafkaorActiveMQthatmayberequiredbyotherICMcomponentsystems
• AllowforaswitchboardlikefunctionalitythatcanbeconfiguredfordifferentCorridorManagementSystemprovidersorDecisionSupportSystems
• Provideformultipleoutputchannelstopublishinformationtomultipledownstreamconsumers
I-210Pilot:CoreSystemHigh-LevelDesign
36
TheDataHubDataGatewayusesApacheCameltoprovidetheseservices.TwoprimarydesignpatternsareusedasshowninFigure5-10,firstforcommunicationtoexternalActiveMQbrokers,andsecondtoexternalSOAPorRESTwebservicesendpoints.
Figure5-10DataHubDataGateway–ActiveMQandWebServicesDesignPatterns
EachofthetwodesignsusesApachecamelandimplementsaKafkaconsumertoretrievemessagesfromadefinedKafkatopic.Eachalsoprovidesforaprocessorfortransformationandroutingofmessages.ForpassingdatausinganActiveMQtopicorqueuehostedbyanexternalbroker,Camel’sSedacomponentisusedforasynchronousmessagingandmessageflowregulation.ForpassingdataviaSOAPorRESTwebservices,anappropriateSOAPorRESTendpointisprovidedforcommunicationwithanexternalSOAPorRESTendpoint.
NOTE:AllinternaldatacommunicationswithinthedatahubareconductedviaKafkaorActiveMQdatamessagingandJSONformatting.NativedatacommunicationsareimplementedusingTMDDorotherappropriatestandardformatswhenavailable.WhilethesestandardsmaynotimplementJSONformatting,themessageswithinthedatahubthatcontainthisinformationareJSONformatted,retainingthedatastructuresandrelationshipswithintheapplicablestandard.Whenexternallyexposed,thesedatamessagesareserializedbacktotheappropriatedataformatdictatedbythestandardused(TMDDorother).
I-210Pilot:CoreSystemHigh-LevelDesign
37
5.3.1. SUBSCRIPTIONENDPOINTS
Subscribe/Notifypattern(http://www.enterpriseintegrationpatterns.com/patterns/conversation/SubscribeNotify.html)-usealeasetoendifnorenew/continuationmessagereceived?
5.3.2. REQUEST/RESPONSEENDPOINTS
BothActiveMQandKafka–CMSgetsconnectionviacamel–camelmaintainssubscriptiontotopic/queue,ActiveMQusesaLastImageSubscriptionRecoveryPolicysubscriptionrecoverypolicy.Getmostrecentmessage.Kafka–getmostrecentmessage
5.4. DATAHUBCOMMANDGATEWAY
ThedatahubcommandgatewayisaninternaldatahubcomponentthatmanagesallcommandcommunicationandcoordinationactivitiesofthedatahubandorchestratescommunicationandcontrolofallcoreICMsystemcomponents.SincetheDSSandCMSdonotcommunicatedirectly,thedatahuborchestratesworkflowsandcommunicationsbetweenthetwocomponentsandtheinternalcomponentsofthedatahub.Similartothedatahubdatagateway,itusesApacheCameltorouteandprocessCoreICMSystemcommands.Inaddition,itusesaworkflowcomponentdesignedforamicroservicearchitectureusedbyNetflixinitsownproductionenvironmentcalledConductor.Itprovidesthefollowingcapabilities:
• Receiveallcoresystemrequestsforservices• Provideexecutionmanagementofallrequestsfordatahubservices• Provideexecutionmanagementofworkflowprocessingforrequestsinvolvingsimplesinglestep
workflowsorcomplexmulti-stepworkflows• Persistservicerequestsandtheiroutcomes(successorfailure)• Manageinternaloperationsofthedatahub(starting/stoppingservices,restartonfailure,etc)• OrchestrateactionsbetweentheDSS,CMS,anddatahub.
Notethatthedatahubcommandgatewayisaninternaldatahubcomponent.Atnotimearethemessagequeuesandtopicsthatitcommunicateswithdirectlyconnectedtoexternalsystems.Allqueuesandtopicsthatcommunicatewiththiscomponentareinternalqueuesandtopicsconnectedtodatahubcomponentsorthedatahubdatagateway(forcommunicationwithexternalsystems).
I-210Pilot:CoreSystemHigh-LevelDesign
38
Figure5-11DataHubCommandGateway
Thedatahubcommandgatewayconsistsofjustafewprimarycomponenttypes.Theseinclude:
• Conductorandrelatedworkflowcomponents• Camelandrelatedroutingandtransformationcomponents• ActiveMQworkflowstatustopic• ActiveMQworkflowtasktopic• Monitor
5.4.1. CONDUCTOR
Conductorisusedfororchestrationoftheindividualpipelinesandbothinternalandexternalrequestsforservices.ConductorisanopensourceprojectdevelopedbyNetflixtomanagesomeofitsownproductionworkflows,particularlywithamicro-servicearchitecture.ThereareanumberofsimilaritiesbetweentheusecasesforwhichConductorisdesignedandtheorchestrationoftheindividualpipelinesandtherequestsforservicesfromtheDSSandCMS,aswellasorchestrationofthecommunicationsbetweenDSSandCMSthatmakeitwellsuitedforuse.
Conductorisusedinan“as-is”configuration“out-of-the-box”.ThedatahubimplementationusestheRESTinterfacesandin-memorydatabasethatcomeswithConductor.NomodificationsaremadetoConductor.Itprovidesworkflowmanagementandcontrol,withworkflowsdescribedinJSONformat.
I-210Pilot:CoreSystemHigh-LevelDesign
39
5.4.2. CAMEL
ToadaptConductortothedatahubinterfaces,CamelisusedasafacadetotheActiveMQandKafkadatahubinterfaces.CamelprovidestheinterfacebetweenConductor’snativeRESTinterfaceandthedatahub’sActiveMQandKafkadatabuselements.
5.4.3. ACTIVEMQWORKFLOWSTATUSTOPIC
TheActiveMQworkflowstatustopicisusedtoallowindividualdatahubcomponentstoprovideworkflowstatusmessagestoConductor,viatheCamelinterface.Italsoisusedtoprovidegeneralsystemandcomponentstatusinformationforuseinworkflowmanagement.Thedatahubdatagatewayalsoprovidesworkflowandexternalsystemstatus(CMSandDSS)messagesviatheworkflowstatustopic.TheCMSandDSSsystemsareneverrequired(orallowed)todirectlymessageviatheWorkflowstatustopic,butinstead,theircommunicationsaremanagedbythedatahubdatagateway.
5.4.4. ACTIVEMQWORKFLOWTASKTOPIC
TheActiveMQworkflowtasktopicishowworkflowtasksaredistributedfromConductortotheindividualdatahubcomponents,ortheDSSandCMSsystems(viathedatahubdatagateway).Aswiththeworkflowstatustopic,DSSandCMSsystemsareneverdirectlyallowedtocommunicatewiththeworkflowtasktopic.Instead,thedatahubdatagatewaymanagesthedistributionoftasksandtheresultingresponsesviatheDSSandCMSAPIs.
5.4.5. MONITOR
Themonitorlistenstotheworkflowstatustopicforsystemandcomponentstatusmessages,alwaysprovidingdatahub,DSS,andCMSstateandstatusinformationtotheConductorworkflowmanager.ThisallowsConductortorespondtodegradedsystemstate,eitherbydisablingspecificworkflowsorstartingotherworkflowstorespondtosystemcomponentfailures.
6. DECISIONSUPPORTSYSTEMDESIGN
Thedecisionsupportsystem’s(DSS)primaryrolesinclude:
• Provideresponseplansfollowingreceiptofaconfirmedincident• Evaluateresponseplansandrankthembasedonuserdefinedcriteria• ProvideoneormorerecommendationstocorridoroperatorsandstakeholdersviatheCorridor
ManagementSystem,whethertoimplementaresponseplanandwhichresponseplantoimplementforeachreceivedconfirmedincident
• Evaluateimplementedresponseplaneffectivenessandrecommendnewresponseplanswhenappropriate
• ProvideresponseplanevaluationresultsforcorridoroperatorsreviewviatheCorridorManagementSystem
I-210Pilot:CoreSystemHigh-LevelDesign
40
TheDSSaccomplishesthiswiththreeprimarycomponents:
• ResponsePlanManagement• RulesEngine• Modeling
Theresponseplanmanagementcomponentreceivesincidentinformationandcoordinatesthedevelopmentandevaluationofresponseplans.Therulesengineprovidesconfigurablelogic/rulesforthedecisiontocreateresponseplans,responseplancreation,andresponseplanranking.Themodelingcomponentprovidestrafficestimationandpredictioncapabilitiestosupportresponseplancreation,evaluation,andranking.
6.1. DSSHIGHLEVELDESIGN
TheDSS’sthreeprimarycomponentsaredescribedinFigure6-1.TheDSSisoneofthethreeprimaryICMsystemsubsystems,andcommunicateswiththeICMsystemviathedatahub’sdatagateway.
Figure6-1DSSArchitecture
Allcorridorinformationisreceivedfromthedatahubdatagateway.Themodelingandresponseplanmanagementcomponentsreceiveallcorridordataandsystemstatusviaaseriesofdatachannelsavailableonthedatahubdatagateway.
I-210Pilot:CoreSystemHigh-LevelDesign
41
AllcommandstotheDSSandrequestsforDSSstatusarereceivedviaasetofcommandchannelsexposedonthedatahubdatagateway.AllcommandrequestsfromentitiesexternaltotheDSSarereceivedbytheresponseplanmanagementcomponent.Requeststothemodelingcomponentareoriginatedbytheresponseplanmanagementcomponent.DirectrequeststothemodelingcomponentfromsystemsexternaltotheDSSarenotallowed,andmustbemanagedviathecommandsexposedbytheresponseplanmanagementcomponent.
AllcommunicationswiththeDSSviatheDataHubDataGatewayaremanagedviaActiveMQmessaging.CommunicationsbetweentheResponsePlanManagementandModelingareenabledviaRESTorActiveMQmessaging.TheresponseplanmanagementandrulesenginearetightlycoupledviatheirjavabasedAPIs.
OutputfromtheDSSconsistsprimarilyofresponseplansandtheirevaluationsfromtheresponseplanmanagementsystem,andestimationandpredictionresultsfromthemodelingsystem.
Figure6-2DecisionSupportSystemHighLevelDesign
Figure6-2providesaslightlymoredetailedviewoftheDSS.
6.2. RESPONSEPLANMANAGEMENT
Responseplanmanagementprovidesfunctionsforthedevelopmentandselectionofresponseplans,includingcoordinationoftheactionsoftheresponseplanmanagement,rulesengine,andthemodelingcomponentsoftheDSS.
Theresponseplanmanagementcomponentisprimarilyaneventbased,asynchronousservicethatlistensforaconfirmedincidentfromthedatahub,anduponreceipt,orchestratesactionsbetweenitself,therulesengine,andmodelingcomponentstodetermineifaresponseplanmightpositivelyimpacttrafficconditionsandifso,developseveralresponseplans,evaluatethem,andrecommenda
I-210Pilot:CoreSystemHigh-LevelDesign
42
responseplanforimplementation.Theresponseplanmanagementcomponentalsohasalimitedselectionofmethodsimplementedsothatotherspecificrequestscanbemadeviathedatahub.
Theresponseplanmanagementalsomonitorsthedecisionsupportsystemanditscomponents.IthaslimitedcapabilitiestoperformspecificstartupactivitiesrequiredforDSSoperation,monitoreachDSScomponent,andperformlimitedactionsshouldastartupactivityfail(forinstance,ifthemodelingestimationprocessshouldstop,theresponseplanmanagementcomponentshallattempttorestarttheestimationprocess).
Figure6-3ResponsePlanManagementHighLevelDesign
ThecircledareaofFigure6-3illustratesthefundamentalcomponentsoftheresponseplanmanagementcomponent.
Ontheleftoftheresponseplanmanagementcomponentinthefigureisthedatahubinterface.ThisisanActiveMQbasedinterfacethatsubscribestothedatahubdatainterfaceandprovidesassetinformation,routeinformation,andincidentinformationtotheresponseplanmanagementcomponent.Thisinformationismaintainedinalocaldatastore.ItalsolistensforspecificrequestssenttotheDSSviathedatahub.
Theresponsemanager,uponreceiptofaspecificrequestorincidentmanagesthespecificworkflowsrequiredtorespondtotherequestortoprovidearesponseplanrecommendation.Thisincludesrequestingdecisionsandresultsfromtherulesengine,informationregardingcurrenttrafficstatefromthemodelingsystem,orspecificpredictionsforresponseplansreceivedfromtherulesengine.Theresponseplanmanageralsoassemblesthecompleteresponseplan,andprovidesitasaresponsetothedatahubforstorageanddistributiontotheCMS.
I-210Pilot:CoreSystemHigh-LevelDesign
43
6.3. RULESENGINE
TherulesengineisacombinationofjavacomponentsandtheopensourceDroolsrulesenginedesignedtorespondtoalimitednumberofquestionsthatcanbeaskedbytheresponseplanmanager.Theselimitedquestionsinclude:
• Doesthecurrentconfirmedincident,withtheresultsofa“do-nothing”prediction,requiredevelopmentandevaluationofresponseplans?
• Whatresponseplansshouldbeevaluated?Createthoseresponseplansfromalimitedsetofresponseplancomponents.
• Whichresponseplan,giventheresponseplansdevelopedandtheirrespectivepredictions,shouldberecommended,listedinarankedorder?
TherulesengineisaJavalibrarythatisincludedintheresponseplanmanagercomponentwithadefinedapispecifictoansweringthesedefinedquestions,giventhecurrentstateofthecorridorassets,definedlimitswithinasetofuserspecifiedrules(suchas“donotusethisroutebetweenthehoursof3and5pm”),currenttrafficstate,currenttime,andincidentinformation.
Itisintendedthatthefinaluserinterfacefordeveloping,editing,testing,andevaluatingrulesandtheirperformancewillbeintheCMS.However,initially,thiswillbelimitedtorequiringITsupporttoassistwithrulesdevelopment,anduploadingofspreadsheetstoaknownlocationwhererulescanbeaddedorupdated.
6.4. MODELING
Themodelingsystemprovidestwoprimaryservices;trafficstateestimationandtrafficprediction.
6.4.1. MODELINGTECHNIQUES
Threetrafficmodelsareusedwithinthemodelingsystem.Theseinclude:
• AcustomfreewaytrafficestimationmodeldevelopedbyUCBerkeleyusingacelltransmissionmodel(CTM)method.
• AcustomarterialtrafficestimationmodeldevelopedbyUCBerkeleyusinganintersectionqueuemodelmethod.
• AcommercialpredictionmodelusingTSS’sAimsunmodelingsystemandamicro-modelingbehavioralmodel.
I-210Pilot:CoreSystemHigh-LevelDesign
44
Figure6-4-ModelingComponentDesign
Thesystemiscomposedofseveralprimarysubsystems–• Datamanagement(inblue)–receivesandprocessestrafficandtrafficinfrastructuredataso
thatitcanbeconsumedbythemodelingsoftware.• Estimationmodelingsystem(inpink)–buildsandrunsestimations,producesoutputof
estimatedtrafficstate.• Predictionmodelingsystem(inorange)–buildsandrunspredictedtrafficstate,inparticular
predictedresponseplanperformance,includingperformancemetrics.• Calculation/metrics(inlightorange)–consumesestimatedtrafficstateandproducescurrent
trafficandnetworkperformancemeasures.• ProjectManager(inbrown)–providesoverallsystemorchestration,cloudscalingandEC2
management,workflow,andexternalRESTinterfaces.Auserinterfaceisavailableforestimationmodeldevelopment.ThisiscurrentlyusedforinternaltestingandQA.
• Persistence–(green/gray)–persiststraffic/trafficinfrastructuredata,modelandmodelbuildingdefinitions,rawandprocessedresults.
CommunicationwithexternalsystemsandthemodelingsystemisviaaRESTAPIexposedbytheProjectManager.DatafeedsfromthedatahubareprovideddataviaActiveMQdatatopics.InternalcomponentcommunicationbetweenmodelingsystemcomponentsisviaActiveMQtopics(data)orqueues(command).
I-210Pilot:CoreSystemHigh-LevelDesign
45
Technologystack:• PrimarycomponentsarebuiltusingJavawiththeSpringframework.Hibernateisusedwith
componentsrequiringPostgresaccess.• PersistenceisbuiltwithJavabasedpersistenceworkers(bothforstoreandretrieveoperations)
andPostgresv9.6.2orCassandrav3.10forpersistence.• MessagingviaActiveMQv5.14.5• ProjectManagerRESTserviceshostedonTomcatv8.5.15• LogManagementviaGraylog• AmazonWebServices,includingthefollowingAWSservices
o EC2o RDS(Postgres)o S3o VPCo IAMo Otherservicesareusedforcoderepository,build,anddeploymentservices
Platform/ApplicationServers/OS• MostanyversionofLinuxOSisacceptable,Ubuntu14.04isthecurrentstandard• AmazonWebServices• Tomcatv8.5.15
GeneralArchitecturalApproach:
Theapproachusedhasbeentodevelopspecificapplicationcomponentstargetedtospecifictasks,connectthemtoeachotherviamessagingorRESTservicesdependingupontheirapplicationrole(backendprocessingorservicedelivery),andprovidescalingcomponentsthatalloweachindividualcomponenttodetectload(CPU/threadutilizationanditsfeedingqueuestate)andscalethenumberofinstancesoftheresourcetypebasedondemand.
Primarycomponentdescriptions:
ModelEngine–Therearetwoversionsofthemodelengine.Thefirstisusedforfreewayestimation.ThefreewaymodelengineisthecorecomponentofthesystemthatimplementstheCTMalgorithm.Thisimplementationisasimulationusingauserdefinedroadnetwork,accompanyinginfrastructureelementsincludingrampmetersandsensors,trafficdemand,trafficbehaviorinformation(splitratiosatintersectionsandfreewayramps),androad/trafficcharacteristicsrequiredbytheCTMalgorithm(fundamentaldiagrams).Thesimulationcanbeusedtoprovideestimationsofcurrentstateorpredictionsoffuturestate,butisusedonlyforfreewaycurrenttrafficstatewithintheICMsystem.
Thesecondversionofthemodelengineisusedforarterialestimation.Thisarterialmodelengineusesanalgorithmusingintersectionsensingandsignaltimingandcycleinformationtoproduceanestimatedtrafficstateofthecorridorarterials.
Bothtypesofmodelenginereceivearequestforamodelruncontainingthescenariotoberunandthedesiredconfigurationparameters.Themodelenginethenrunsthemodelandoutputsthelinkstate(trafficdensityandspeed)foreachroadnetworklink.
I-210Pilot:CoreSystemHigh-LevelDesign
46
ThemodelengineisamultithreadedJavaapplication,buttheprimaryworkcurrentlyremainssingle-threaded.
Predictionengine–ThepredictionengineisanimplementationofTSS’sAimsunsoftware.ItincludestheAimsunsimulationsoftwareaswellasacomponentthatbuildssimulationmodelsforuseinthepredictionworkflow.Thesimulationbuildcomponentcombinesabasesimulationmodelwithdatafromthedatahubindicatingcurrentcorridorassetstate(signals,ramps,signs,etc),currenttrafficstatefromthefreewayandarterialmodelengines,andtheresponseplansfromtheresponseplanmanagementsystemtocreateasetofoneormoremodelsthatcanbeusedtoevaluatearesponseplan.Italsoprovidesthepredictionmetricsrequiredforpresentationtotheuserandtherulesengineforresponseplanrecommendationandselection.
ProjectManager/ProjectManagerApp-Theprojectmanagerappallowsexternalsystemstointeractwiththemodelingsystem.ProjectmanagerappprovidesRESTservicesandmessagingtoinitiateactionsbytherestofthesystem.TheprojectmanagerappisservedviaaTomcatapplicationserver.
ScenarioBuilder-Thescenariobuilderisusedtocreateascenariotoberun.Itisrarelyused,butcanbeusedwhencorridorinfrastructureelementsaremodified,suchasarampmeter.Itisaback-endcomponentthatreceivesmodelcomponents(roadnetworks,intersectionsignals,rampmeters,trafficdemandandsplitsets,etc)fromtheprojectmanagerappandbuildsamodelsuitableforrunninginthefreewaymodelengine.
Calculators-Thecalculatorsareusedtotakethemodelenginelinkstatedataandprocessitforconsumptioneitherasspecificmetricsforanalysis,orasvisualizationinformationfortheCMS.Eachcalculatorjobissinglethreadedandisdesignedforaspecificcalculation.
PersistenceWorker-Persistenceworkersarecomponentsthataredesignedtoeithertakedatafromaqueueortopic(modelenginedata,model/scenarioinformation,orcalculatoroutput)andpersistitinoneofthedatastores(PostgresorCassandra)or,uponrequestretrievethedatafromoneofthedatastoresandplaceinontheappropriatequeueortopicforconsumptionbydownstreamprocesses.
ReadersandProcessors-Thesecomponentsareusedtoretrieveoracceptpusheddatafromthedatahubandplaceitintoaqueueortopicforconsumption(reader)andtoretrievedatafromaqueueortopicandprocessitforconsumptionbythemodel(processor)orpredictionengine.
Top Related