TFL Hackathon Solution 4-3-Comp - Mandrel - As presented · TFL Hackathon Solution 4-3-Comp -...
Transcript of TFL Hackathon Solution 4-3-Comp - Mandrel - As presented · TFL Hackathon Solution 4-3-Comp -...
![Page 2: TFL Hackathon Solution 4-3-Comp - Mandrel - As presented · TFL Hackathon Solution 4-3-Comp - Mandrel - As presented.pptx Author: Mary-Ann Claridge Created Date: 11/15/2015 8:24:55](https://reader036.fdocuments.net/reader036/viewer/2022062523/60570cd53f36a8255a774c62/html5/thumbnails/2.jpg)
Summary Slides …
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 2
![Page 3: TFL Hackathon Solution 4-3-Comp - Mandrel - As presented · TFL Hackathon Solution 4-3-Comp - Mandrel - As presented.pptx Author: Mary-Ann Claridge Created Date: 11/15/2015 8:24:55](https://reader036.fdocuments.net/reader036/viewer/2022062523/60570cd53f36a8255a774c62/html5/thumbnails/3.jpg)
Challenge 1 / Answer 3 ….... • Data mapping to a common geography • Within TfL our most granular view of our network is defined by the Ordinance Survey Integrated TOID (Topological IdenPfier) Network (OS ITN). • TOID = Unique ID of feature on the Ordinance Survey map.
• However, it not common that our data sets can map directly onto this. • Really?
• It is our vision that data fusion can enable a single view of performance which takes input from all available datasets and enable us to produce a consistent view of our network. • You bet! Our vision too. You asked …
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 3
![Page 4: TFL Hackathon Solution 4-3-Comp - Mandrel - As presented · TFL Hackathon Solution 4-3-Comp - Mandrel - As presented.pptx Author: Mary-Ann Claridge Created Date: 11/15/2015 8:24:55](https://reader036.fdocuments.net/reader036/viewer/2022062523/60570cd53f36a8255a774c62/html5/thumbnails/4.jpg)
Tiered Topology Model Mandrel “The InformaOon Architects”
Challenge 1 In 1933 Harry Beck revoluPonized maps. In 2015 we have to leave the map again… • Edges & Vertex
• Unique Id • Hierarchy
• Topology per Mode • Physical &
Logical
• Linked Topologies • Across Modes
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 4
1933
2015
![Page 5: TFL Hackathon Solution 4-3-Comp - Mandrel - As presented · TFL Hackathon Solution 4-3-Comp - Mandrel - As presented.pptx Author: Mary-Ann Claridge Created Date: 11/15/2015 8:24:55](https://reader036.fdocuments.net/reader036/viewer/2022062523/60570cd53f36a8255a774c62/html5/thumbnails/5.jpg)
Unconnected Mapping Data So Which Sensor relates To Which Bus Route & Are We On A Bus Lane?
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 5
![Page 6: TFL Hackathon Solution 4-3-Comp - Mandrel - As presented · TFL Hackathon Solution 4-3-Comp - Mandrel - As presented.pptx Author: Mary-Ann Claridge Created Date: 11/15/2015 8:24:55](https://reader036.fdocuments.net/reader036/viewer/2022062523/60570cd53f36a8255a774c62/html5/thumbnails/6.jpg)
Unconnected Mapping Data – Over OSM So Which Sensor relates To Which Bus Route & Are We On A Bus Lane?
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 6
![Page 7: TFL Hackathon Solution 4-3-Comp - Mandrel - As presented · TFL Hackathon Solution 4-3-Comp - Mandrel - As presented.pptx Author: Mary-Ann Claridge Created Date: 11/15/2015 8:24:55](https://reader036.fdocuments.net/reader036/viewer/2022062523/60570cd53f36a8255a774c62/html5/thumbnails/7.jpg)
Logical Model With Hierarchy Of Routes Per Transport Mode Links Down To Physical Model(s)
Bus 1 Stop at X,Y
(Vertex)
Bus 2 Stop at X,Y
(Vertex)
Bus 3 Stop at X,Y
(Vertex) Route Leg 2 (Edge)
Route Leg 1 (Edge)
Route (Vertex) Route Has Leg
(Edge) Route Has Leg(Edge)
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 7
Bus 3 Stop at X,Y
(Vertex)
Road Signals & Sensors (Vertex)
Road SecPons (Edge), Waypoints (Vertex), Bus Lanes
& JuncPons
We can now tell which Sensor relates To Which Bus Route & If We Are We On A Bus Lane
![Page 8: TFL Hackathon Solution 4-3-Comp - Mandrel - As presented · TFL Hackathon Solution 4-3-Comp - Mandrel - As presented.pptx Author: Mary-Ann Claridge Created Date: 11/15/2015 8:24:55](https://reader036.fdocuments.net/reader036/viewer/2022062523/60570cd53f36a8255a774c62/html5/thumbnails/8.jpg)
More Hierarchy Links Physical & Logical Views Two Physical Stops Either Side Of Road
Bus 1a Stop at X,Y
(Vertex)
Bus 2a Stop at X,Y
(Vertex)
Route
End
Bus 1a Stop at X,Y
(Vertex)
Route Leg 1a (Edge)
Bus 2a Stop at X,Y
(Vertex)
Route Leg 2a (Edge)
Marble Arch Bus “Stop” (Vertex)
Has Bus Stop (Edge)
Has Bus Stop (Edge)
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 8
![Page 9: TFL Hackathon Solution 4-3-Comp - Mandrel - As presented · TFL Hackathon Solution 4-3-Comp - Mandrel - As presented.pptx Author: Mary-Ann Claridge Created Date: 11/15/2015 8:24:55](https://reader036.fdocuments.net/reader036/viewer/2022062523/60570cd53f36a8255a774c62/html5/thumbnails/9.jpg)
Now Link Modes With Summary InformaOon
Different parts of the hierarchy are needed to map between modes
Logical model of Tube
journey ends at staPon
Physical model of staPon
has many exits
StaPon exit has walking route to
nearest bus stop of 5 min
Bus route runs from bus stop to bus stop near
home
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 9
![Page 10: TFL Hackathon Solution 4-3-Comp - Mandrel - As presented · TFL Hackathon Solution 4-3-Comp - Mandrel - As presented.pptx Author: Mary-Ann Claridge Created Date: 11/15/2015 8:24:55](https://reader036.fdocuments.net/reader036/viewer/2022062523/60570cd53f36a8255a774c62/html5/thumbnails/10.jpg)
Full Slide Deck …
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 10
![Page 11: TFL Hackathon Solution 4-3-Comp - Mandrel - As presented · TFL Hackathon Solution 4-3-Comp - Mandrel - As presented.pptx Author: Mary-Ann Claridge Created Date: 11/15/2015 8:24:55](https://reader036.fdocuments.net/reader036/viewer/2022062523/60570cd53f36a8255a774c62/html5/thumbnails/11.jpg)
Challenge 1 – An Unexpected Answer? • Data mapping to a common geography
• Within TfL our most granular view of our network is defined by the Ordinance Survey Integrated TOID (Topological IdenPfier) Network (OS ITN).
• However, it not common that our data sets can map directly onto this.
• It is our vision that data fusion can enable a single view of performance which takes input from all available datasets and enable us to produce a consistent view of our network.
• Our answer is a vision • In 1933 Harry Beck produced the beauPful
design for the London Underground. • It was simple, a topology, not a map. • For AnalyPcs 3.0 we have to take the idea
further breaking the data away from being directly dependent physical map locaPons.
• Adopt a vision to refine and link all of the reference data in shape files to a set of linked Pered topologies covering the TfL network and beyond.
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 11
![Page 12: TFL Hackathon Solution 4-3-Comp - Mandrel - As presented · TFL Hackathon Solution 4-3-Comp - Mandrel - As presented.pptx Author: Mary-Ann Claridge Created Date: 11/15/2015 8:24:55](https://reader036.fdocuments.net/reader036/viewer/2022062523/60570cd53f36a8255a774c62/html5/thumbnails/12.jpg)
Unconnected Mapping Data So Which Sensor relates To Which Bus Route & Are We On A Bus Lane?
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 12
![Page 13: TFL Hackathon Solution 4-3-Comp - Mandrel - As presented · TFL Hackathon Solution 4-3-Comp - Mandrel - As presented.pptx Author: Mary-Ann Claridge Created Date: 11/15/2015 8:24:55](https://reader036.fdocuments.net/reader036/viewer/2022062523/60570cd53f36a8255a774c62/html5/thumbnails/13.jpg)
Unconnected Mapping Data – Over OSM So Which Sensor relates To Which Bus Route & Are We On A Bus Lane?
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 13
![Page 14: TFL Hackathon Solution 4-3-Comp - Mandrel - As presented · TFL Hackathon Solution 4-3-Comp - Mandrel - As presented.pptx Author: Mary-Ann Claridge Created Date: 11/15/2015 8:24:55](https://reader036.fdocuments.net/reader036/viewer/2022062523/60570cd53f36a8255a774c62/html5/thumbnails/14.jpg)
AnalyOcs 2.0 Early Big Data IntegraOon Pair-‐wise Linkage Of Data Sources
Find a way to smash data
together
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 14
![Page 15: TFL Hackathon Solution 4-3-Comp - Mandrel - As presented · TFL Hackathon Solution 4-3-Comp - Mandrel - As presented.pptx Author: Mary-Ann Claridge Created Date: 11/15/2015 8:24:55](https://reader036.fdocuments.net/reader036/viewer/2022062523/60570cd53f36a8255a774c62/html5/thumbnails/15.jpg)
AnalyOcs 3.0 Coherent reference data model Joins mulOple data sources
Single Reference Data Model. Tiered Linked Topologies.
Slow changing reference data:
staPons, routes, ….
High volume data. E.g. sensor and trip data.
PosiPvely Link related enPtles. Not just rely on similar map locaPons.
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 15
![Page 16: TFL Hackathon Solution 4-3-Comp - Mandrel - As presented · TFL Hackathon Solution 4-3-Comp - Mandrel - As presented.pptx Author: Mary-Ann Claridge Created Date: 11/15/2015 8:24:55](https://reader036.fdocuments.net/reader036/viewer/2022062523/60570cd53f36a8255a774c62/html5/thumbnails/16.jpg)
How Do We Know You Need This? • We wanted to do clever things at the Hackathon but …. • We spent too much Pme trying to merge
GIS and reference informaPon and tear apart Shape files • Not recommended in R or Python
• Other network based industries have gone through exactly the same pain. • For example the mobile phone networks
are (too late) desperately trying to build unified topology models of their mulP-‐layered mulP-‐Pered mulP-‐technology networks.
• New big data tooling is built to handle data network model. • E.g. Spark’s GraphX
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 16
![Page 17: TFL Hackathon Solution 4-3-Comp - Mandrel - As presented · TFL Hackathon Solution 4-3-Comp - Mandrel - As presented.pptx Author: Mary-Ann Claridge Created Date: 11/15/2015 8:24:55](https://reader036.fdocuments.net/reader036/viewer/2022062523/60570cd53f36a8255a774c62/html5/thumbnails/17.jpg)
Some Detailed DirecOon
• Here is what we know interpreted from other industries ….
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 17
![Page 18: TFL Hackathon Solution 4-3-Comp - Mandrel - As presented · TFL Hackathon Solution 4-3-Comp - Mandrel - As presented.pptx Author: Mary-Ann Claridge Created Date: 11/15/2015 8:24:55](https://reader036.fdocuments.net/reader036/viewer/2022062523/60570cd53f36a8255a774c62/html5/thumbnails/18.jpg)
Everything Becomes Directed Line Graphs Stops (Vertex) & Route Leg (Edges)
Bus Route (One DirecPon)
Bus 1 Stop at X,Y
(Vertex)
Bus 2 Stop at X,Y
(Vertex)
Bus 3 Stop at X,Y
(Vertex) Route Leg 2 (Edge)
Route Leg 1 (Edge)
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 18
![Page 19: TFL Hackathon Solution 4-3-Comp - Mandrel - As presented · TFL Hackathon Solution 4-3-Comp - Mandrel - As presented.pptx Author: Mary-Ann Claridge Created Date: 11/15/2015 8:24:55](https://reader036.fdocuments.net/reader036/viewer/2022062523/60570cd53f36a8255a774c62/html5/thumbnails/19.jpg)
Introduce Hierarchy In Logical Model More VerOces and Edges Build Routes
Bus 1 Stop at X,Y
(Vertex)
Bus 2 Stop at X,Y
(Vertex)
Bus 3 Stop at X,Y
(Vertex) Route Leg 2 (Edge)
Route Leg 1 (Edge)
Route (Vertex) Route
Has Leg (Edge)
Route Has Leg (Edge) 15/11/15 © 2015 Mandrel Systems Limited, All Rights
Reserved 19
![Page 20: TFL Hackathon Solution 4-3-Comp - Mandrel - As presented · TFL Hackathon Solution 4-3-Comp - Mandrel - As presented.pptx Author: Mary-Ann Claridge Created Date: 11/15/2015 8:24:55](https://reader036.fdocuments.net/reader036/viewer/2022062523/60570cd53f36a8255a774c62/html5/thumbnails/20.jpg)
More Hierarchy Links Physical & Logical Views Two Physical Stops Either Side Of Road
Bus 1a Stop at X,Y
(Vertex)
Bus 2a Stop at X,Y
(Vertex)
Route
End
Bus 1a Stop at X,Y
(Vertex)
Route Leg 1a (Edge)
Bus 2a Stop at X,Y
(Vertex)
Route Leg 2a (Edge)
Marble Arch Bus “Stop” (Vertex)
Has Bus Stop (Edge)
Has Bus Stop (Edge)
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 20
![Page 21: TFL Hackathon Solution 4-3-Comp - Mandrel - As presented · TFL Hackathon Solution 4-3-Comp - Mandrel - As presented.pptx Author: Mary-Ann Claridge Created Date: 11/15/2015 8:24:55](https://reader036.fdocuments.net/reader036/viewer/2022062523/60570cd53f36a8255a774c62/html5/thumbnails/21.jpg)
Route Legs Link To Map Physical Topology E.g. road Segments, Waypoints, Sensors Etc. (Very Simplified Drawing)
Route Leg 1 (Edge) Bus 1
Stop at X,Y
(Vertex)
Bus 2 Stop at X,Y
(Vertex)
Road Sensor 23423 (Vertex)
Road SecPons (Edge),
Waypoints (Vertex)
& JuncPons
For each transport mode you will have
one or more linked
physical and logical models.
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 21
![Page 22: TFL Hackathon Solution 4-3-Comp - Mandrel - As presented · TFL Hackathon Solution 4-3-Comp - Mandrel - As presented.pptx Author: Mary-Ann Claridge Created Date: 11/15/2015 8:24:55](https://reader036.fdocuments.net/reader036/viewer/2022062523/60570cd53f36a8255a774c62/html5/thumbnails/22.jpg)
Now Link Modes With Summary InformaOon
Different parts of the hierarchy are needed to map between modes
Logical model of Tube
journey ends at staPon
Physical model of staPon
has many exits
StaPon exit has walking route to
nearest bus stop of 5 min
Bus route runs from bus stop to bus stop near
home
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 22
![Page 23: TFL Hackathon Solution 4-3-Comp - Mandrel - As presented · TFL Hackathon Solution 4-3-Comp - Mandrel - As presented.pptx Author: Mary-Ann Claridge Created Date: 11/15/2015 8:24:55](https://reader036.fdocuments.net/reader036/viewer/2022062523/60570cd53f36a8255a774c62/html5/thumbnails/23.jpg)
Robust and PracOcable Model Only when all the model is built can you add in mulPple data sets…
…to map mulP-‐modal journeys
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 23
![Page 24: TFL Hackathon Solution 4-3-Comp - Mandrel - As presented · TFL Hackathon Solution 4-3-Comp - Mandrel - As presented.pptx Author: Mary-Ann Claridge Created Date: 11/15/2015 8:24:55](https://reader036.fdocuments.net/reader036/viewer/2022062523/60570cd53f36a8255a774c62/html5/thumbnails/24.jpg)
Step One … Add Your Own Unique IDs Everywhere
• Everything Has a Unique Numeric ID • So Anything can Link to Anything In Either
DirecPon • Bus Stop To Tube Exit • MulPple Tube Exit to Tube StaPon • Tube StaPon To Railway StaPon • Railway StaPon To Plaporms • Railway StaPon To Taxi rank
• Enables mulP-‐mode in single topology • Some things may have a locaPon or link to a map
TOID • But can’t just link everything via a OS map enPty
ID (TOID)
• Extra Nodes & Edges Build Hierarchy • A sensor is on a road segment • MulPple road segments make up a journey leg • MulPple journey legs make up a route run
Bus 1 Stop at X,Y
(Vertex)
Tube StaPon Exit X,Y (Vertex)
Walk: 3 Min (Edge)
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 24
![Page 25: TFL Hackathon Solution 4-3-Comp - Mandrel - As presented · TFL Hackathon Solution 4-3-Comp - Mandrel - As presented.pptx Author: Mary-Ann Claridge Created Date: 11/15/2015 8:24:55](https://reader036.fdocuments.net/reader036/viewer/2022062523/60570cd53f36a8255a774c62/html5/thumbnails/25.jpg)
Step Two … Consistent Time and Agribute Structure
• You have at least 7 different representaPons of Time • To simplify linkage seqle on a single reference representaPon of Pme and Pme intervals
• All your different data sources have an underlying structure • Use it to make searching and linking feasible and Pmely
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 25
![Page 26: TFL Hackathon Solution 4-3-Comp - Mandrel - As presented · TFL Hackathon Solution 4-3-Comp - Mandrel - As presented.pptx Author: Mary-Ann Claridge Created Date: 11/15/2015 8:24:55](https://reader036.fdocuments.net/reader036/viewer/2022062523/60570cd53f36a8255a774c62/html5/thumbnails/26.jpg)
To ConOnue …
• TacPcally • Please do release as much reference data as you can.
• Be kind to Web developers and also release shape files as GeoJson with Lat/Long.
• Strategically
• Build the coherent model. • Please involve us.
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 26
![Page 27: TFL Hackathon Solution 4-3-Comp - Mandrel - As presented · TFL Hackathon Solution 4-3-Comp - Mandrel - As presented.pptx Author: Mary-Ann Claridge Created Date: 11/15/2015 8:24:55](https://reader036.fdocuments.net/reader036/viewer/2022062523/60570cd53f36a8255a774c62/html5/thumbnails/27.jpg)
For More InformaOon
• Let’s Have A Chat • mary-‐[email protected] • [email protected]
• See Also: • Processing Graphs: Spark GraphX
• hqp://spark.apache.org/docs/latest/graphx-‐programming-‐guide.html
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 27