@commuterjoy

Democratising data at the Financial Times

Matt Chadburn, Principal Engineer, ft.com May 2016

@commuterjoy

@commuterjoy

"It's about getting every part of the organisation to be more effective

with what they do and democratising data across the

organisation," he explained.

!"The challenge is to move beyond a centralised data function to a

decentralised one, where data becomes part of everyone's

responsibility.

!"It's much more than a technology challenge - creating reports and

saying 'there you go'. It's a training challenge, how we help

everyone understand the impact of what they do.

!"Certainly there's a core of experts who sit in the middle offering

expertise and rigour, but this is about having everybody learn

about our customers and ultimately this is a decentralised

capability."

@commuterjoy

source: opendefinition.org

@commuterjoy

• Users of data at the FT

• Systems we built to support them

@commuterjoy

@commuterjoy

@commuterjoy

@commuterjoy

@commuterjoy

@commuterjoy

Matthias Müller

Volkswagen

Germany

Emissions scandal

Frankfurt Stock Exchange

Automotive

located

headquartered inlisted on

industry

CEOabout

@commuterjoy

User

follows

USA

General Motors

lives in

works for

@commuterjoy

@commuterjoy

@commuterjoy

@commuterjoy

@commuterjoy

@commuterjoy

@commuterjoy

@commuterjoy

Lots of users !

Diverse use cases !

Single version of the truth !

No analytics system can do all this :(

@commuterjoy

Events Data WarehouseAPI

@commuterjoy

category:action

email:open

signup:subscribe

page:view

comment:post

page:scroll

payment:failure

…

account:update

@commuterjoy

Event

Context

User

Device

System

Category

Action

Time

@commuterjoy

@commuterjoy

Data WarehouseAPI

Client-side

Offline

Server (go, java, node)

Webhooks

AMP etc

@commuterjoy

Events API

SQS

Kinesis

Neo4J

Elastic Search

RedShift

Ontotext

Keen.io

Hadoop

Redis

Pusher.io

Google Sheets

@commuterjoy

Events Event streamAPI Enrichment

@commuterjoy

@commuterjoy

@commuterjoy

@commuterjoy

@commuterjoy

@commuterjoy

Events Event streamAPI Enrichment

Maxmind

Time

Device Atlas

Content API

AB Test

Propensity

Propensity

License API

RFV

myFT

@commuterjoy

caniuse.com

Events Event streamAPI Enrichment

Weather

Market prices

Offers

Demographics

QV

Freebase

timezone

Sentiment

sharedcount.com

@commuterjoy

Events Event streamAPI EnrichmentValidation

(in progress!)

@commuterjoy

• Focus on the users need

• Learnable (standards)

• Ease of use (APIs to get stuff in and out)

• Iterative (700+ production releases)

• Open for contribution

Democracy?

@commuterjoy

https://bit.ly/ft-csv-job