HBaseConEast2016: OpenTSDB+BigTable
-
Upload
michael-stack -
Category
Engineering
-
view
40 -
download
1
Transcript of HBaseConEast2016: OpenTSDB+BigTable
OpenTSDB + BigtableIntegrating time series database withGoogle Cloud BigtableDanil Zburivsky, Big Data Practice Lead, Pythian — [email protected] Max Luebbe, SRE, Google — [email protected]
Pythian specializes in design, implementation, and management of systems that directly contribute to revenue and business success.
History19 years in business
Growing at 30+% per year
400+ employees
300+ customers worldwide
HQ Ottawa, Canada - global reach
Technology agnostic = trusted advisor
Deep expertise: Oracle, Oracle Apps, MySQL, AWS, SQL Server, Cassandra/DataStax, Azure, PostgreSQL, Cloudera, MapR, Hortonworks etc.
Google Premier Partner Status (as of end Aug)
5 Certified Developers (soon to be 12)
Dedicated Google Technical Champion
Launch partner for: Kubernetes, Dataflow, Cloud SQL, Dataproc
Integrated OpenTSDB with Bigtable
DW Explorers Program Partner
Upcoming BigQuery & Cloud ML Launch Partner
• Volume can be explosive• Data arrival and access
patterns are different
Storing time series data is a challenge
• Volume can be explosive
• Data arrival and access patterns are different
Storing time series data is a challenge
OpenTSDB ArchitectureServer Server Server Server
TSD TSD
Hbase or Bigtable
TSD RPC
Hbase RPC or Hbase API
Web UI
Scripts/Alerting
http
TSD RPC
• Open source• Uses HBase as a data
store• Data model optimized for
TS • REST API
OpenTSDB
<metric_uid><timestamp><tagk1><tagv1>[...<tagkN><tagvN>]
<col_t+1>[...<col_t+N>]
Origins of Bigtable• Updating search index in bulk: too slow!• Rather than a file system (GFS), needed random
access• Generic & scalable, now powers many diverse
products
Bigtable decouples compute & storage...
Node
Distributed filesystem
Node
Client Client Client Client Client Client
Processing
Storage
Clients
A B C D E
...making scalability fast & seamless
Node
Distributed filesystem
Node
Client Client Client Client Client Client
Processing
Storage
Clients
A B C D E
Node
async-bigtable● Uses standard HBase 1.0 API● BufferedMutator● Thread pool
https://github.com/OpenTSDB/asyncbigtable