HBase on Docker with clusterdock

Dima Spivak

OverviewIntroduction to Docker (5 min or less!)

clusterdock

Topology for Apache HBase

Demo!

hbase-it upstream

In progress

Introduction to Docker

Virtual machines Docker containers

https://www.docker.com/what-docker

Introduction to DockerIsolated namespaces, process trees

Daemon with REST API and a popular client

docker run …

Tooling

Networking

Images

Registries

Docker Swarm, Docker Compose, ...

clusterdock

Docker tooling designed to support one-process-per-container workflow

No mature tooling for “treat container like a host”

Python-based container-orchestration framework

Started as a hackathon at Cloudera, now open-source

https://github.com/cloudera/clusterdock

clusterdockBuild clusters once, start clusters

repeatedly

Abstractions for cluster-centric entities (e.g. Cluster, Node, NodeGroup)

cluster.ssh(‘wget …’)

workers_nodegroup.ssh(‘cp …’)

Topology defines how to build/start clusters

clusterdock

clusterdockprofile.cfg

actions.py

clusterdock | Topology for Apache HBaseStarted in HBASE-12721, now lives in dev-support

Supports building and starting clusters from binary tarballs or from source

Once image is built, starting a 5-node cluster takes less than 1 minute

clusterdock | Topology for Apache HBasehttps://github.com/apache/hbase/blob/master/dev-support/apache_hbase_topology/actions.py

clusterdock | Topology for Apache HBase

configurations.cfg <configuration> <property> <name>hbase.cluster.distributed</name> <value>true</value> </property> <property> <name>hbase.rootdir</name> <value>hdfs://node-1.cluster/hbase</value> </property> <property> <name>hbase.zookeeper.quorum</name> <value>node-1.cluster</value> </property> <property> <name>hbase.zookeeper.property.datadir</name> <value>/usr/local/zookeeper</value> </property></configuration>

clusterdock | Topology for Apache HBasesource /dev/stdin <<< "$(curl -sL http://tiny.cloudera.com/clusterdock.sh)"

CLUSTERDOCK_TOPOLOGY_IMAGE=apache-docker-hbase.bintray.io/dev/clusterdock:apache_hbase_topology \ clusterdock_run ./bin/start_cluster --registry-url apache-docker-hbase.bintray.io --namespace dev \ apache_hbase --hbase-version=1.2.3 --hadoop-version=2.5.1 --java-version=7u79 --secondary-nodes='node-{2..5}'

clusterdock_ssh node-1.cluster

DEMO

hbase-it upstreamUpstream system testing framework

It works

Finds a lot of serious (e.g. data loss) bugs

Fault-injection via chaos monkey

… but no one runs it upstream

Setting up clusters is hard

Tests take a long time to run

Manual

clusterdock can help

hbase-it upstreamBuild and push clusterdock clusters of

HBase dev branches

master, branch-1.3, branch-1.2, …

Use these pre-built clusters for testing

Can also improve release process

HBASE-16559

Ultimate goal: make passing ITs a quality gate for new features

DEMOclusterdock_ssh node-1.cluster \ hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList -m serverKilling loop 1 4 250000000 dir 4

In progressDocumentation

Moving to infrastructure that can support longer runs

Expanded matrix

More chaos monkeys, configuration options, Java versions, ...

Finding a permanent home for clusterdock

@dimaspivakdimaspivak@apache.org