Consul and Consul Pusher

12
Consul by Hashicorp

Transcript of Consul and Consul Pusher

Page 1: Consul and Consul Pusher

Consul by Hashicorp

Page 2: Consul and Consul Pusher

Agenda● Consul - overview● Components● Architecture● REST endpoints● REST for kv storage● Libraries● Demo of Consul and Consul Pusher

Page 3: Consul and Consul Pusher

Consul is like combining the features of a DNS server plus Consistent Key/Value Store like etcd plus features of ZooKeeper for service discovery, and health monitoring like Nagios but all rolled up into a consistent system.

Essentially, Consul is all the bits you need to have a coherent domain service model available to provide service discovery, health and replicated config, service topology and health status. Consul also provides a nice REST interface and Web UI to see your service topology and distributed service config.

Consul

Page 4: Consul and Consul Pusher

▷ Service Discovery - give ability for services to register themselves and to discover other services via a DNS or HTTP interface. Register external services such as SaaS providers as well.

▷ Health Checking - can provide health checks associated with a given service ("is the webserver returning 200 OK"), or with the local node ("is memory utilization below 90%"). This information can be used by an operator to monitor cluster health, and it is used by the service discovery components to route traffic away from unhealthy hosts.

▷ Multi Datacenter - scales to multiple datacenters out of the box with no complicated configuration This means users of Consul do not have to worry about building additional layers of abstraction to grow to multiple regions.

▷ Key/Value Storage - hierarchical key/value store for any number of purposes, including dynamic configuration, feature flagging, coordination, leader election, and more. The simple HTTP API makes it easy to use.

Components

Page 5: Consul and Consul Pusher

Architecture

Page 6: Consul and Consul Pusher

Architecture #2

Agent - long running daemon on every member of the Consul cluster. The agent is able to run in either client or server mode.

Server nodes are responsible for running the consensus protocol and storing the cluster state. The client nodes are mostly stateless and rely heavily on the server nodes.

All agents can run the DNS or HTTP interfaces, and are responsible for running checks and keeping services in sync. The agent maintains membership information, registers services, runs checks, responds to queries, and more. The agent must run on every node that is part of a Consul cluster.

▷ Client - agent that forwards all RPCs to a server.▷ Server - agent with an expanded set of responsibilities like

maintaining cluster state, responding to RPC queries, forwarding queries to leaders or remote data centers.

Page 7: Consul and Consul Pusher

REST endpoints

■ kv - key value■ agent - api for dealing with agent■ catalog - dealing with data center catalog■ health - show health checks■ sessions - group operations and manage consistent view■ events - fire fast gossip based events■ acl - setup access control lists and security for Consul■ status - check status

Each endpoint provides operations which either take JSON, request params and deliver JSON responses.

Page 8: Consul and Consul Pusher

REST for kv storageManages updates, deletes, fetches of individual keys or key prefixes.

▷ curl http://localhost:8500/v1/kv/dev?recurse => fetches all keys recursively▷ curl http://localhost:8500/v1/kv/dev/db.username => fetches single key (#1)

Output #1

[{

"CreateIndex": 316,

"ModifyIndex": 316,

"LockIndex": 0,

"Key": "dev/db.username",

"Flags": 0,

"Value": "QlRfQ01T"

}]

CreateIndex - internal index value that represents when the entry was created

ModifyIndex - last index that modified this key.

LockIndex - how many times this key has successfully been acquired in a lock.

Key - full path of the entry.

Flags - unsigned integer that can be attached to each entry. Clients can choose to use this however makes sense for their application.

Value - Base64-encoded blob of data. Note that values cannot be larger than 512kB.

Page 9: Consul and Consul Pusher

Libraries

1. https://github.com/kdlan/jconsul

Has support for Spring and fallback strategy to file when key does not exist in Consul, but it is not maintained since April 2015

2. http://www.cfg4j.org

Has fallback strategy in case of failure. Moreover it has ReloadStrategy and PeriodicalReloadStrategy. Still maintained and developed. Probably some effort needed to make it work within Spring, but not huge.

3. https://github.com/Magnetme/consultant

Good library with minimal set of available settings options.

Page 10: Consul and Consul Pusher

Demo of Consul Pusher

Consul Pusher from https://github.com/cfg4j/cfg4j-pusher which is a Spring Boot app that pushes values from configuration files (YAML, properties, etc.) to Consul KVs

I extended above project and pushed on GitHub by adding :

➢ Extract profile from file name (#1) or path otherwise (#2)

Ad.1. /projectName/properties/dev/database.properties

/projectName/properties/qa/database.properties

Ad.2. /projectName/properties/dev.properties

/projectName/properties/qa.properties

➢ Command line input using Apache Commons CLI

java -jar consul-pusher.jar -profile dev -key db.driver.class -value org.h2.Driver

PS. Need a change property in app called mode (file -> interactive)

GitHub => https://github.com/GarciaPL/ConsulPusher

Page 11: Consul and Consul Pusher

Demo of Consuldocker run -d -p 8400:8400 -p 8500:8500 -p 8600:53/udp --name alfa -h alfa -v /tmp/consul:/data progrium/consul -data-dir="/tmp/consul" -server -bootstrap-expect 3 -ui-dir /ui

JOIN_IP="$(docker inspect -f '{{.NetworkSettings.IPAddress}}' alfa)"

docker run -d --name bravo -h bravo progrium/consul -data-dir="/tmp/consul" -server -join $JOIN_IPdocker run -d --name charlie -h charlie progrium/consul -data-dir="/tmp/consul" -server -join $JOIN_IP

1) -data-dir used for persistence of KV’s, /ui used for enabling Web UI2) -bootstrap-expect 3 specifies amount of servers needed for starting electing leader3) Alfa will expose ports like 8400 (RPC), 8500 (HTTP) and 8600 (DNS)

Page 12: Consul and Consul Pusher

Thanks!Any questions ?

Thank you http://www.slidescarnival.com for layout!

References :1. https://www.consul.io/intro/index.html2. https://www.consul.io/docs/agent/http/kv.html3. https://hub.docker.com/r/progrium/consul/4. http://www.mammatustech.com/Microservice-Servi

ce-Discovery-with-Consul5. https://launchpad.net/~bcandrea/+archive/ubuntu/

consul6. https://imaginea.gitbooks.io/consul-devops-handbo

ok/content/7. https://github.com/cfg4j/cfg4j-pusher8. https://github.com/GarciaPL/ConsulPusher