ZooKeeper vs Consul

Ivan Glushkovivan.glushkov@gmail.com@gliush

v1.2, Nov 2014

ZooKeeper: Overview

❖ Written in Java

❖ Strongly consistent (CP)

❖ Zab protocol (Paxos-like)

❖ Ensemble of servers

❖ Quorum needed (majority)

❖ Dataset must fit in memory

ZooKeeper is a high-performance coordination service for distributed applications

ZooKeeper: Data

❖ In-memory data

❖ Shared hierarchal namespace (FS-like)

❖ Ephemeral nodes exists while session is alive

❖ Access Control List (ACL) to each node

❖ Watches on nodes.

ZooKeeper: API

❖ create

❖ delete

❖ exists

❖ get data

❖ set data

❖ get children

❖ sync

ZooKeeper: Problems

❖ Latency-dependent

❖ No multi datacenters support (very slow writes)

❖ Client number limit (out of sockets)

❖ Designed to store data on the order of kB in size

❖ Possibly lost changes while reenabling watch

❖ Need to read after every watch callback (to sync)

❖ TCP connection != service healthy (proxy)

❖ Change zk server is a problem

Consul

❖ Written in Go❖ Service discovery❖ Health checking❖ Key/Value store❖ Multi Datacenter support❖ Agent/Server concept❖ Gossip protocol for all the nodes (discovery)❖ Consensus protocol (Raft-based) for servers❖ 3 consistency modes❖ Access Control List (ACL)

Consul is a tool for discovering and configuring services in your infrastructure

Consul: HTTP API❖ kv - Key/Value store

❖ agent - Agent control

❖ catalog - Manages nodes and services

❖ health - Manages health checks

❖ session - Session manipulation

❖ acl - ACL creations and management

❖ event - User Events

❖ status - Consul system status

With strong consistency Key/Value 3 consistency modes

HTTP API Service registering Config file or HTTP API

TCP Connection Health CheckConnection to node + Any script on node +

HTTP API

HTTP API Service Discovery HTTP API + DNS + Configuration templates

ZooKeeper Consulvs

Key/Value

❖ ZooKeeper: REST API. Needs quorum, has only strong consistency

❖ Consul: REST API. 3 modes:

• ‘consistency’

• ‘default’ (potentially stale values = default for zk)

• ‘stale’ (read from any server, often stale values, fastest mode)

Service registering

❖ ZooKeeper: K/V via HTTP request to servers (ephemeral nodes)

❖ Consul: local agent registering

• through HTTP request

• config file

Health Check

❖ ZooKeeper: HTTP Connection. Might not be relevant:

• proxy holds connect

• node is alive, while the app hangs

❖ Consul: gossip for node aliveness, preconfigured health checks (any script), HTTP API

Service Discovery

❖ ZooKeeper: HTTP request to the K/V

❖ Consul: DNS or HTTP API

More Consul pros❖ Consul is built on top of Serf - node discovery and

orchestration tool (gossip model). It’s used for group membership, failure detection and event broadcast:

• LAN-pool for membership and failure detection (distributed checks, good for large clusters)

• WAN-pool for inter-datacenter consul servers communication

❖ Consul templates to update files and possibly restart applications (haproxy with backend servers)

Links❖ Zk overview: http://zookeeper.apache.org/doc/trunk/zookeeperOver.html

❖ Zk experience: http://whilefalse.blogspot.ru/2012/12/building-global-highly-available.html

❖ Test zk: http://aphyr.com/posts/291-call-me-maybe-zookeeper

❖ Restart zk server problem: http://zookeeper-user.578899.n2.nabble.com/How-to-replace-a-zookeeper-server-td7472559.html

❖ Sample consul use for HAproxy configuration: https://hashicorp.com/blog/haproxy-with-consul.html

❖ Consul templates feature: https://github.com/hashicorp/consul-template/releases

❖ Stale reads from ZooKeeper: http://stackoverflow.com/questions/5420087/apache-zookeeper-how-do-writes-work