Admiral Group

Speakers

Simon Elliston Ball – Solutions Architect, Hortonworks

Adam Morton – Enterprise Data Architect, Admiral Group plc

• Over 10 years experience in Data Warehousing, Business Intelligence and Analytics

• Working at Admiral for the past 2 years delivering a greenfield Enterprise Data Warehouse as part of an overall Data Architecture modernisation programme

The Admiral Group

Admiral Group has grown from a small start up to one of the largest car insurance providers in the UK with a presence in seven countries.

Our strategy is simple: To continue to progress in the UK Car Insurance market whilst taking what we do well to new markets and products: keep doing what we’re doing and do it better year after year.

Admiral – International Operations

Admiral employs more than 7,000 people at its offices in the UK, Spain, Italy, France, USA, Canada and India.

"People who like what they do, do it better"

R&D at Admiral

• Strong history of using data to drive innovation which needs to be continued

• New function aimed at testing and learning through technology

• Time-boxed iterative efforts of no more than 4-6 weeks

• Fail fast, fail quickly approach; success or failure can end the PoC early

• Understand ‘Big Data’ and trial Hadoop ecosystem projects

Why Telematics?

• Scalability – A product with large potential and potentially huge volumes

• Timeliness - Data & Scoring was processed in batch – how quickly can this be done?

• Granularity - Suppliers provide aggregated data – could map matching be improved?

• Event Notification – Can we respond quickly to NRT events in the data?

• Data Enrichment - Opportunity to uncover further insights by integrating with interesting data sources

Objectives of the Telematics PoC

• Scalability - Prove that data storage and high performance analytics can be accomplished on large data sets cost effectively

• Timeliness - Reduce scoring time

• Data Enrichment

• NRT data processing – acting on events such as proximity to an airport

• Improve stability and flexibility

• Test the viability of a cloud solution

• Data Visualisation

Technical Challenges – Networking and Security

• Privacy Sensitive

• Third Party Sources

• Real-time data

8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

There’s a VPN, it will be fine!

Admiral vNET

Third Party vNET

Telematics Provider

DC

External Users

Internal Users


Kafka SSL

Admiral vNET

Telematics Provider

DC

External Users

Internal Users

K

SSL


Ingest with NiFi

Admiral vNET

Telematics Provider

DC

External Users

Internal Users

K

HDF

Other Providers

Other Providers

Other Providers


Real-time Scoring

Clean up done in NiFi– Basic data correctness– Format changes

Fed To Kafka

Spark Streaming– NEAR Real time requirement– Mixing Scala RDD and Data Frames code– Integrating with map matching library

Output fed into Kafka– Kafka to WebSockets bridge for real-time visualization


Batch Scoring

More Spark!

Zeppelin for ease of use, interaction

Productionized into batch Spark Jobs


SAS on Hive

Spark as ETL engine Hive for Large Scale processing SAS connector using Hive ORC as a file format

– Significantly smaller than JSON– So much faster to process

14

Technical Challenges – Map Matching

• GPS data is messy

• Open Data sources based on roads

• Nearest road is fast, but not very good

• Hidden Markov Models. Know where you’re going, and where you’ve been.

• Open source to the rescue…

15

Barefoot – Map Matching

• https://github.com/bmwcarit/barefoot

• Docker based service

• PostGIS map server loaded from OSM data

• Serializable map, distributed in Spark

https://github.com/bmwcarit/barefoot

Next Steps

Completing knowledge transfer workshops with Hortonworks

How to move from a POC to Production – ready?

Establishing a in-house R&D function

Deciding on the tools and frameworks to use within a POC environment in the future

Admiral Group

Technology

Transcript of Admiral Group