Time Series Data: OpenTSDB and TSP (Betfair)
20
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. TIME SERIES DATA Richard Haigh Global Head of Reliability and Operations @rakh1
-
Upload
rakh1 -
Category
Technology
-
view
334 -
download
2
Transcript of Time Series Data: OpenTSDB and TSP (Betfair)
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
TIME SERIES DATA
Richard Haigh
Global Head of Reliability and Operations
@rakh1
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
Born in 1999UK, Romania, Portugal, Ireland, Malta, Gibraltar, USAEngineering Blog: www.betsandbits.com700+ Engineers and Growing
ExchangeSportsbookGames
~750k active users~70M daily transactions~2.5Bn daily API calls~100k/s Monitoring PointsE2E P95 transaction times <4ms
One of UK’s most successful .coms with ~£2Bn Market Cap
2
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 3
In the Beginning….
Bernard
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 4
In the Beginning…. there was DATA!
time
value
CPU 5, betfair-exchange-007
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 5
time
value
Time Series Data
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 6
time
value
Time Series Data
• Capture all of these metrics, thousands of them, hundreds of thousands of them
• From every machine in your estate
• At least every 10 seconds, maybe faster
• You can store this data for years
• With no down sampling
• Graph any combination you like
That would make you happy, wouldn’t it?
CONFIDENTIAL and not for reproduction without prior written consent. © of Te Sporting Exchange Limited. 7
What if….
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 8
Betfair’s first TSDB implementation
Source :http://mo.nitor.me/hadoop-hbase-opentsdb/
TSDB is great for root cause analysis
But it could be better!
OpenTSDB – in their own words
Source : http://opentsdb.net/overview.html
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 14
OpenTSDB – limitations
Aggregator
Site Feed
Source : http://opentsdb.net/overview.html
The Site Feed
Kale
“We’d like to introduce you to the Kale stack, which is our attempt to fix both of these problems. It consists of two parts: Skyline and Oculus. We first use Skyline to detect anomalous metrics. Then, we search for that metric in Oculus, to see if any other metrics look similar.” - Etsy
What next – the future?
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 17
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 18
http://opentsdb.net/
https://github.com/betfair/opentsp
http://riemann.io/
https://github.com/Ticketmaster/metrilyx-2.0
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 19
[email protected]@RAKH1
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 20
HOW TO GET IN TOUCH?
