Simon Laws – Apache Flink Cluster Deployment on Docker and Docker-Compose

IBM Confidential 1

Apache Flink Cluster Deployment

On Docker and Docker-Compose

Simon Laws

[email protected]

© 2014 IBM Corporation

What We’re Going To Build

2

Windows 7 IBM Bluemix

VirtualBox 4.3.18

Ubuntu 14.04

Docker 4.3.18

Cygwin

ssh

Flink

Jobmanager

container

Flink

Taskmanager

container

Cloudfoundry

Client

Docker

Flink

image

Flink

Jobmanager

container

Flink

Taskmanager

container

Flink

image


Docker

3

docker.com


Docker

4

docker.com/

whatisdocker


IBM Bluemix

Sign up for a free

account and experiment

5

bluemix.net


IBM Bluemix Signup

Sign up for a free

account and experiment

Various options for

academics, startups etc.

Prices here for when

you go over free

allocation

https://console.eu-

gb.bluemix.net/pricing/

6

https://console.ng.bluemix.net/registration/


Apache Flink and Docker

7

github.com/apache/

flink/tree/master/

flink-contrib/

docker-flink


Install Ubuntu Using VirtualBox

8


VirtualBox 4.3.18

Ubuntu 14.04

Docker 4.3.18

Cygwin

ssh

Flink

Jobmanager

container

Flink

Taskmanager

container

Cloudfoundry

Client

Docker

Flink

image

Flink

Jobmanager

container

Flink

Taskmanager

container

Flink

image


Setting Up The Host Operating System

If you’re on Linux you’re probably good to go

If you’re on Windows or OSX you need to run Linux on a VM As an alternative you could do this on a VM in the new VM service in Bluemix. It’s in Beta at the moment.

Get VirtualBox

Create a Linux VM (I used Ubuntu)

Install Docker

Download Flink Source

Try a Flink example

9


Get VirtualBox

10

virtualbox.org


Get Linux Distro (I went for Ubuntu 14.04)

11

releases.ubuntu.com/14.04


Create VM from ISO in VirtualBox (and set port forwarding)

12

End result is that I can now SSH into the instance of

Ubuntu running inside VitualBox. Everything for this

point on happens inside this Ubuntu VM

To start with make sure that Ubuntu is up to date

> sudo su -

> apt-get update

sudo su -

apt-get updatesudo su -

apt-get update


Install Docker and Flink

13


VirtualBox 4.3.18

Ubuntu 14.04

Docker 4.3.18

Cygwin

ssh

Flink

Jobmanager

container

Flink

Taskmanager

container

Cloudfoundry

Client

Docker

Flink

image

Flink

Jobmanager

container

Flink

Taskmanager

container

Flink

image


Install Docker and Docker-Compose

14

github.com/apache/

flink/tree/master/

flink-contrib/

docker-flink


Download Flink Source

15

git clone https://github.com/apache/flink.git

cd flink/flink-contrib/docker-flinkThe base Docker image

source

Builds the “base” and “flink”

images

The flink Docker image

source

Docker-Compose

configuration

https://github.com/apache/flink.git


Docker Images and Containers

16

SourceDockerImage

DockerContainer

docker build docker run

What Images

Do I have?

docker images

What Containers

Do I have running?

Push elsewhere (Docker Hub)

To share with others

ssh or

docker exec

docker psDockerfile


Dockerfiles – The Recipe for a Docker Image

17


Build Docker Images

18


VirtualBox 4.3.18

Ubuntu 14.04

Docker 4.3.18

Cygwin

ssh

Flink

Jobmanager

container

Flink

Taskmanager

container

Cloudfoundry

Client

Docker

Flink

image

Flink

Jobmanager

container

Flink

Taskmanager

container

Flink

image


Building The Images From the Docker Files

19

./build.sh

If you look in the script it use the “docker build” command to build a “base” image and a

“flink” image

This failed for me on the first attempt with the base build reporting

chpasswd: (user root) pam_chauthtok() failed, error:

System error

chpasswd: (line 1, user root) password not changed

It turned out I had an old version of Docker so I updated to latest and it worked


The Built Images

20


Run Docker Containers

21


VirtualBox 4.3.18

Ubuntu 14.04

Docker 4.3.18

Cygwin

ssh

Flink

Jobmanager

container

Flink

Taskmanager

container

Cloudfoundry

Client

Docker

Flink

image

Flink

Jobmanager

container

Flink

Taskmanager

container

Flink

image


Running Linked Images Manually (To See What’s Going On)

22

docker run -d -p 48080:8080 -p 48081:8081 --expose 6123 --name

jobmanager flink /usr/local/flink/bin/config-flink.sh jobmanager

docker run -d --expose 6121 --expose 6122 --link jobmanager --name

taskmanager1 flink

The job manager

Add a task manager

This failed at first with a strange “Exec format exception”. I made a local fix by removing

the comment from the top of the flink/config-flink.sh file leaving

#!/bin/bash

As the first line


Flink up and Running

23


Running a Test

24

I had previously uploaded flink-java-examples-0.9.1-WordCount.jar from my local Windows install of Flink


Accessing Containers with Docker Exec

25


Making Things Simpler With Docker-Compose

26

docker-compose up –d

docker-compose scale taskmanager=3

docker-compose kill


Connecting To Bluemix

27


VirtualBox 4.3.18

Ubuntu 14.04

Docker 4.3.18

Cygwin

ssh

Flink

Jobmanager

container

Flink

Taskmanager

container

Cloudfoundry

Client

Docker

Flink

image

Flink

Jobmanager

container

Flink

Taskmanager

container

Flink

image


Running the Images On Bluemix

28

Log in to Bluemix:1.cf login

Run the IBM Containers cf CLI plug-in.2.cf ic login

Create a Dockerfile in the root directory of your app source files. Learn more about Dockerfiles.

Build an image from your Dockerfile. The command returns an image ID.3. docker build -t image_name

Tag the image with your private namespace in the IBM Containers registry.4. docker tag image_name registry.stage1.ng.bluemix.net/sluk/image_name:image_tag

The image name is optional. If it is not specified, the image is tagged with latest.

Push this image to the IBM Containers registry:5. docker push registry.stage1.ng.bluemix.net/sluk/image_name:image_tag

You can create a container from this image in the Bluemix Catalog, or with the following command:6. cf ic run --name container_name

registry.stage1.ng.bluemix.net/sluk/image_name:image_tag

http://docs.docker.com/reference/builder/


Install The Cloud Foundary Client Software

29

In the Ubuntu VM we have been using to run Flink using Docker get and install the Cloud

Foundary client tool

wget --output-document=cfcli.deb

https://cli.run.pivotal.io/stable?release=debian64&version=6.12.4&source=githu

b-rel

dpkg -i cfcli.deb

In the Ubuntu VM we have been using to run Flink using Docker get

and install the Cloud Foundary client

cf install-plugin https://static-ice.ng.bluemix.net/ibm-containers-linux_x64

https://cli.run.pivotal.io/stable?release=debian64&version=6.12.4&source=github-rel

https://static-ice.ng.bluemix.net/ibm-containers-linux_x64


Log Into Bluemix – Web User Interface

30


Log Into Bluemix – Command Line Using CF Tool

31


Log Into Bluemix Containers – Command Line Using CF Tool

32


Push Image Up To Bluemix

33


VirtualBox 4.3.18

Ubuntu 14.04

Docker 4.3.18

Cygwin

ssh

Flink

Jobmanager

container

Flink

Taskmanager

container

Cloudfoundry

Client

Docker

Flink

image

Flink

Jobmanager

container

Flink

Taskmanager

container

Flink

image


Tag and Push Up Our Images

34

docker tag flink registry.ng.bluemix.net/sluk/flink

docker push registry.ng.bluemix.net/sluk/flink


List Images Now Up On Bluemix

35

cf ic images


Run Containers On Bluemix

36


VirtualBox 4.3.18

Ubuntu 14.04

Docker 4.3.18

Cygwin

ssh

Flink

Jobmanager

container

Flink

Taskmanager

container

Cloudfoundry

Client

Docker

Flink

image

Flink

Jobmanager

container

Flink

Taskmanager

container

Flink

image


Creating Containers in Bluemix via User Interface

37


Creating Containers in Bluemix via CF Command Line

38

Option 1) Leverage this Cloud Foundry IBM Containers plugin without affecting the local docker environment:

Example Usage:cf ic ps

cf ic images

Option 2) Leverage the docker CLI directly. In this shell, override local docker environment to connect to IBM

Containers by setting these variables,

export DOCKER_HOST=tcp://containers-api.ng.bluemix.net:8443

export DOCKER_CERT_PATH=/root/.ice/certs

export DOCKER_TLS_VERIFY=1

Example Usage:docker ps

docker images

© 2014 IBM Corporation39

Create jobmanager Container in Bluemix via CF Command Line

cf ic run -p 22:22 -p 8080:8080 -p 8081:8081 -p 6123:6123 --expose

6123 --name jobmanager registry.ng.bluemix.net/sluk/flink

/usr/local/flink/bin/config-flink.sh jobmanager

Assign a public IP

so you can get

to the console

© 2014 IBM Corporation40

Create taskmanager Container in Bluemix via CF Command Line

cf ic run -p 22:22 -p 6121:6121 -p 6122:6122 --expose 6121 --

expose 6122 --name taskmanager1 --link jobmanager:jobmanager

registry.ng.bluemix.net/sluk/flink /usr/local/flink/bin/config-

flink.sh taskmanager


Running The Flink Cluster

41


QUESTIONS?

42

Simon Laws – Apache Flink Cluster Deployment on Docker and Docker-Compose

Technology

Transcript of Simon Laws – Apache Flink Cluster Deployment on Docker and Docker-Compose