ContainerCF Documentation - Read the Docs · 1.2.1 Mesos (Marathon) ... 2 Develop ContainerCF and...

ContainerCF DocumentationRelease 0.1.0

Matt Johnson

Jan 14, 2020

Contents

1 Project Info and Run ContainerCF 31.1 Getting Started (Info) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Getting Started (Deploy) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.2.1 Mesos (Marathon) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.2.2 Kubernetes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.3 Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.3.1 Direct coupling of Monit failures to container failures . . . . . . . . . . . . . . . . . . . . . 51.3.2 Central dependancy on NATS for each DEA . . . . . . . . . . . . . . . . . . . . . . . . . . 6

1.4 FAQs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61.4.1 Why did you guys create this? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61.4.2 Where is the Code? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71.4.3 Is this ready for production? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.5 Licenses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2 Develop ContainerCF and Pipeline 92.1 Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.1.1 Initial Development Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.1.2 Pipeline Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.1.3 Repository Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102.1.4 Setting up your own pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.1.5 Public Images and CF versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.1.6 An IMPORTANT note on CF upstream changes . . . . . . . . . . . . . . . . . . . . . . . . 112.1.7 Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.2 Mesos Dev Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.2.1 Pre-Requisite Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.2.2 Mesos nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.2.3 Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3 Demo Video (Marathon) 15

4 Repository Layout 174.1 Consume: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174.2 Develop: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

5 Credits 19

6 Search 21

i

6.1 License . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

ii

ContainerCF Documentation, Release 0.1.0

Welcome to the home of Container.CF!

We are a containerized CloudFoundry build, designed to bring the container and cloudfoundry ecosystems closertogether.

For background information and our reasons behind the project, please see this blog post: http://blogs.cisco.com/cloud/containerized-cloud-foundry-is-key-element-for-cloud-native

For a quick demo on Marathon, see the video below.

Contents 1

http://blogs.cisco.com/cloud/containerized-cloud-foundry-is-key-element-for-cloud-nativehttp://blogs.cisco.com/cloud/containerized-cloud-foundry-is-key-element-for-cloud-native


2 Contents

CHAPTER 1

Project Info and Run ContainerCF

1.1 Getting Started (Info)

The idea behind ContainerCF is for each component of cloudfoundry to be built into docker container’s instead ofBOSH releases deployed virtual machines.

This removes the need to pre-specify a BOSH manifest and IP addresses/pools (which are very IaaS-centric). Instead,ContainerCF takes a different approach to configuration, each component registers itself with a state store (in our case,Consul), this provides two important benefits;

• 1. Components of the system can find eachother.

For example, each component can find the IP of NATS by simply looking up `/cf/jobs/nats/containerip` in the state store.

• 2. Initial ‘bootstrap’ configuration.

Per-deployment configuration, such as default admin username and password (or the CF systemDNS name etc) can all be pre-populated into the state store to dynamically configure a ContainerCFcluster on startup. This allows us to provide pre-built docker images for each CF component, whichconfigure themselves for your specific environment on startup.

Therefore, ALL deployment types below need a Consul cluster to talk to, and (obviously) need to know where thatconsul cluster is. The examples below start up a single node consul cluster for testing as part of the deployment scripts.

Once running, each containerized component also watches relevant parts of the configuration keystore for changes;so if the cloudcontroller API moves for any reason, dependant components will reconfigure with the new reachabilitydetails of the API component. We’re using ConfD with Consul as a backing store to acheive this.

For health management, we expose the state of Monit up to the container ochestrator in use; therefore if a monitjob (internal CF process/subprocess) is failing or unavailable, the container orchestrator will mark the container asunhealthy and destroy it, bringing up a new instance in it’s place (which once again will be configured for reachabilityto other components via consul.)

One other important note, ContainerCF has not been designed to track TCP/UDP usage for each CloudFoundry com-ponent, therefore we require a container infrastructure that gives each container it’s own IP address within the cluster.

3


Most Container infrastructure tools now do this by default, using overlays or container networking tools such as Calico,Weave, Flannel, etc. However, please bare this in mind when considering multi-host container infrastructures.

This info section was designed to provide an overview of how ContainerCF is designed to work. For more informationon how this is all put together, or to dig into the build process yourself, see the development / build pipeline section ofthe docs.

1.2 Getting Started (Deploy)

In the ‘run’ directory, you’ll find pre-created scripts and .json manifests for Marathon and Kubernetes deployments.As Marathon is our build/test environment, these are the most tested/validated at this time.

These scripts consume the public pre-built images for each containerized CF component. For more information on thebuild process see the build/modify section of the docs.

Each deployment type is based around a single `cf_parameters` file, where we expect you to provide the boot-strap information needed for your new ContainerCF deployment, along with any information the container infrastruc-ture you have chosen needs. The deployment process follows this path:

• Fill out `cf_parameters` file.

• Start Consul for service discovery (scripted).

• Tell CF components where Consul is (scripted).

• Start each CF component container from the public ContainerCF docker repository (scripted).

1.2.1 Mesos (Marathon)

For Marathon, cf_parameters needs to know the IP of one of your slave/worker nodes and also the address of themarathon API. These are purely so that the demo deploy scripts work, you could always use the marathon `.json`files directly in other ways if you wished.

cd into the `run/marathon` directory and you should see the following;

MATJOHN2-M-90LM (matjohn2) ~/coderepo/ContainerCF/run/marathon $> lltotal 24drwxr-xr-x 17 matjohn2 staff 578B 14 Nov 13:59 apps/-rw-r--r-- 1 matjohn2 staff 2.0K 14 Nov 15:32 cf_parameters-rw-r--r-- 1 matjohn2 staff 195B 14 Nov 14:02 deploy_cf-rw-r--r-- 1 matjohn2 staff 176B 14 Nov 14:08 deploy_service_appsdrwxr-xr-x 3 matjohn2 staff 102B 14 Nov 14:06 services/-rw-r--r-- 1 matjohn2 staff 3.0K 14 Nov 13:57 ssl.pem-rw-r--r-- 1 matjohn2 staff 1.0K 14 Nov 13:58 update_cf-rw-r--r-- 1 matjohn2 staff 399B 14 Nov 14:07 update_service_apps

Read through the comments in the `cf_parameters` file and update the entries to match your infrastructure.Once done, the `./deploy_service_apps` command will setup a single instance consul cluster. This will bedeployed to the slave/worker host you provided the IP for (so that we know where it is).

Following this, `./deploy_cf` will populate the new consul KV store with our CF cluster settings (basically allthe keys from the `cf_parameters` file) then deploy each CF component (all the `.json` files within the `./app` directory.) If you look at the deploy scripts, you’ll noticed we’re using JQ to combine the JSON files into onelarge file to post to Marathon, we’re also then replacing placeholders for Consul’s IP/Reachability information withthe IP you provided of the marathon worker node.

4 Chapter 1. Project Info and Run ContainerCF


Progress can then be tracked in Marathon and also componets will be seen registering in the consul keystore (theres aUI for consul on HTTP://\protect\T1\textdollarHOST_IP:8500) under the `/cf/jobs` keyspace.

Once all components are started, you should be able to target the CF API using the usual CF client binary. During thedeployment, some components may (and are designed) to die/restart in marathon as all the components come onlineand discover/reconfigure eachother via Consul.

1.2.2 Kubernetes

Kubernetes has higher level traffic management primatives (such as services, DNS lookup etc) than Marathon, so thedeployment scripts (and the information we need from the user in the `cf_parameters` file) is reduced.

cd into the `run/kubernetes` directory and you should see the following;

MATJOHN2-M-90LM (matjohn2) ~/coderepo/ContainerCF/run/kubernetes $> lltotal 16drwxr-xr-x 17 matjohn2 staff 578B 8 Nov 17:47 apps/-rwxr-xr-x 1 matjohn2 staff 454B 8 Nov 17:59 bootstrap-ccf-k8s.sh-rw-r--r-- 1 matjohn2 staff 1.3K 14 Nov 13:41 cf_parameters-rwxr-xr-x 1 matjohn2 staff 1.6K 9 Nov 15:50 deploy-ccf-k8s.shdrwxr-xr-x 4 matjohn2 staff 136B 8 Nov 18:14 services/-rw-r--r-- 1 matjohn2 staff 3.0K 8 Nov 17:47 ssl.pem

Like the Marathon example, configure the relevant settings in the `cf_parameters` file for your deployment, thenrun `./bootstrap-ccf-k8s.sh`.

You must make sure your ‘kubectl’ command is configured to use the cluster you want to deploy against, our demodeployment scripts just run kubectl commands.

Much like the Marathon example, this will deploy a single Consul node as our service discovery cluster, but will alsoassign a service (load balancer) to the consul node, so that we can discover consul’s IP from other components viaDNS. (ccf_consul_srv.ccf). All the demo deployment scripts here create use the namespace `ccf` within kubernetes,which will be created by the `./bootstrap-ccf-k8s.sh` script if it does not exist.

Once done, running `./deploy-ccf-k8s.sh` will push the values from `cf_parameters` to our new con-sul keystore, then schedule each of the CF components as a pod within kubernetes. For the pod descriptions of eachcomponent, see `run/kubernetes/apps` directory.

1.3 Known Issues

Here we document some of the current known-issues, along with how we’re thinking of solving them.

1.3.1 Direct coupling of Monit failures to container failures

It was elegant and simple, in keeping with “be as close to the BOSH release pipeline as possible for future upgrades”mantra, we simply listen for any failed monit jobs in each container to work out if the container is healthy. as eachcontainer is still running monit for the jobs within it, just like a BOSH-based VM of the same type (such as API)would be.

However, this does cause potentially more restarts of containers in certain scenarios than necessary due to re-configuration of components via confd/consul.

For example, lets say the loggretator-trafficcontroller container has crashed. A new one will take it’s place,this new one will likley be assigned a different IP than it’s predecessor, and so the consul key `/cf/jobs/loggregator-trafficcontroller/containerip` will be updated with the new IP for the container. De-

1.3. Known Issues 5

HTTP://\protect \T1\textdollar HOST_IP:8500


pendant jobs in other ContainerCF containers will notice this change, reconfigure relevant components (via confd) andmonit restart the affected jobs.

However, the container stack will then notice this monit ‘restarting’ status and potentially mark the container as failed.In this case, this container will be restarted, using the correct data from consul to configure itself as it comes back up.

Half of me is OK with this; (containers are meant to restart / new instance if someone wants to change something. . .right?) However, the other half wonders if we should allow Monit to try a couple of times on a state change (likeBOSH allows), then we surface a failure to the container orchestrator after X time/failures to schedule a replacementinstance.

This would simply require a new script in the build chain of the containers, allowing for the more complex healthchecking, then mesos/kubernetes health checks for each component would change to use this new check logic. Ie, this,for api.json’s Kubernetes POD definition, currently checks if all monit jobs are healthy:

livenessProbe:exec:command: ["/bin/sh", "-c", "!(/var/vcap/bosh/bin/monit summary | tail -n +3 |

→˓grep -v -E '(running|accessible)$' )"]initialDelaySeconds: 20timeoutSeconds: 10

(Taken from `run/kubernetes/apps/api.json`).

This would be changed to query our new script in each container, which implements more complex monit checkinglogic;

livenessProbe:exec:command: ["/bin/sh", "-c", "/var/vcap/bosh/bin/newmonitcheck.sh"]

initialDelaySeconds: 20timeoutSeconds: 10

1.3.2 Central dependancy on NATS for each DEA

When coupled with the issue above, imagine the NATS container dies, a new one will pretty much immediatley beavailable, but re-configuring each DEA via confd/consul could cause each DEA to be marked as failed around thesame time, obviously this is bad. The solution is either the updates to the health check/surfacing process above, orsimply placing a LB/Proxy infront of NATS.

IE, imagine the KV store key `/cf/jobs/nats/containerip` is set to the IP of a simple TCP proxy. If theNATS container dies, sure the proxy will have to send NATS messages to the new container and some TCP sessionswill fail in the meantime, but the IP of NATS (from the view of all DEA’s and other components) never changes,allowing NATS to recover without alerting all components.

1.4 FAQs

1.4.1 Why did you guys create this?

After lots of conversations with customers, who straddle the PaaS (CloudFoundry) and Container(Docker/rkt/mesos/k8’s) ecosystems, we found more and more reasons to try and bring the ecosystems closertogether. Orchestration seemed the primary hurdle.

For a full writeup of our reasoning, see this blog post.1?1 https://blogs.cisco.com/containerized-cf


https://blogs.cisco.com/containerized-cf


1.4.2 Where is the Code?

UPDATE! Everything we have is now in this repo, including docs and a writeup of the pipeline. See the new sectionsof http://container.cf for details.

1.4.3 Is this ready for production?

This is the first release of an effort to containerize CF, built using a pipeline-up approach in order to make it use-able/upgradeable into the future.

It should be considered a preview release.

It works, recovers, converges and we can’t wait to see where the community takes it; We’ve documented a couple ofissues on the ‘Known Issues’ page which you should consider and test before jumping in with any valued workloads.

Also, for versioning, please see the CF version note at the bottom of the development page.

1.5 Licenses

ContainerCF’s build pipeline code is licensed under the Apache License, Version 2.0, the text of which is reproducedbelow.

However, please be mindful that we package OpenSource cloudfoundry, which contains many components and there-fore the licences of each component should be considered where necessary.

1.5. Licenses 7

http://container.cf

CHAPTER 2

Develop ContainerCF and Pipeline

2.1 Development

Here we cover the development side of the project. You’ll find info on the build process and repo structure below.

2.1.1 Initial Development Goals

We tried to design to the following goals, these inform a lot of the decisions made, why theres more scripting than‘extra helper applications’ and why the pipeline is designed the way it is.

A version of CloudFoundry which is completley containerized, and that follows the design rules below:

1. Do not make a ‘once in time’ PoC. The build pipeline should be able to consume newer CloudFoundry versions.

2. Produce Generic Images. All ‘administrator’ configuration should be done at runtime, users shouldn’t have togenerate their own ContainerCF images.

3. Use the existing CloudFoundry community output (cf-release, etc) for pipeline input, we don’t want to bemaintaining a fork of CF components.

4. Opinionated decisions on which configuration options (see 2.) to surface in the initial release. (Ie, we’ve madesome decisions on backend settings for you to keep runtime configuration simple).

2.1.2 Pipeline Overview

The build pipeline is the “meat” of this project. Creating hacked docker images one time that created a cloudfoundrycluster would have been much easier. We wanted to create something which could be re-used and improved withnewer versions of CF (See rule 1). In keeping with the CF community, http://concourse.ci has been used to build theCI pipeline, with all scripts, data etc being stored in this repository (http://github.com/ciscocloud/containercf).

You can see the repository layout below, everything is pipeline-related apart from the `run` directory, which con-tains pre-written manifests and scripts to deploy the publicly hosted pipeline output (the containers) to marathon orkubernetes container infrastructure.

9

http://concourse.cihttp://github.com/ciscocloud/containercf


The great thing about concourse pipelines is theres only one file to read to get an overview of the flow. See `./concourse/pipeline.yml`. Our pipeline works in the following way:

1. Create a base Container Image with relevant pre-requisites for all containerized components.

2. Use this base image to build up specific container images for each ‘CF component’ listed in the `./components` directory.

3. Configure the CF components from a static CF deployment manifest (`./components/cf-manifest/manifests/deploy.yml`) Using named placeholders for information we don’t know (as the images aregeneric and not tied to a specific deployment).

4. Inject runtime scripts which replace the IP address of this ‘host’ (container) at runtime wherever needed. (Forexample, all the ‘Bind’ / ‘Listen’ configuration file lines usually statically configured by BOSH when a VM isbuilt) Example `./components/api/scripts/run.sh`.

5. Add ConfD templates which replace the ‘named placeholders’ from 3. at runtime with values from the Consulkeystore. (For example, line 136 of the current `deploy.yml` CF manifest would normally need the DNSdomain for the CF deployment. Instead, we have `placeholder-f622194a.example.com`, whichwill be replaced by the KV store key `/cf/system_domain` for each deployment by ConfD. Monit is stillused to control the components within the containers (indeed, the exact same monit configurations and binariesand /var/vcap folder structure pulled from cf-release).

6. Push the newly generated containers to a DockerHub style docker registry (we use bintray, so the pipeline isconfigured for this. Others will work).

7. Use the newly generated images to deploy to a ‘dev’ marathon cluster sitting side by side the pipeline workers.

8. Run CATS against the ContainerCF deployment now running in the ‘dev’ marathon cluster.

9. If successful, re-deploy to a ‘stable’ marathon cluster (and optionally push the images to a different dockerregistry/tag).

If you have read the “getting started” section for running the existing images our pipeline has produced, you’ll befarmiliar with the necessary settings in `./run/marathon/cf_parameters`. Hopefully you can now see whythese are the parameters surfaced to the user for runtime configuration and customization; Answer: They are the oneswe’ve templated out in the CF manifest and written ConfD templates for in points 3 and 5 above.

We chose these parameters as the “minimum needed customization” for both security and operational use (such assetting your own DNS domains). It should be fairly simple to see how you’d surface another setting if you wished too.

A note on ‘unsurfaced’ Keys

Like many bosh manifests, there are placeholder keys (such as the internal JWT key and component to componentpasswords) which we’ve hardcoded in the `deploy.yml` manifest and not surfaced to run-time configuration viathe process above. Theres no reason NOT to surface these, we just haven’t in this release. If you build your ownpipeline to customize the images, you could also just change them directly in the `deploy.yml` file, but we weregoing for public, re-usable generic images. (so surfacing them would be better ;) ).

2.1.3 Repository Layout

Consume:

`run` - Scripts and Manifests to deploy current ContainerCF images.

10 Chapter 2. Develop ContainerCF and Pipeline


Develop:

`concourse` - Concourse CI pipeline Configuration and Parameters.

`components` - Artifacts consumed by concourse for each CF component.

`environments` - Configuration for mesos clusters used during pipeline and testing.

`scripts` - Used by pipeline/concourse to push newly generated images to environments.

`cats` - Configuration and scripts for running the CF CATS test against environments.

`bosh_concourse_deployment0.yml` - Our BOSH manifest for the concourse environment (AWS).

2.1.4 Setting up your own pipeline

To setup your own pipeline (brief overview):

1. Fork this ContainerCF repo.

2. Change the `./concourse/pipeline.yml` git resources (bottom of file) from `ciscocloud/containercf` to your own repo.

3. In the same file, change the docker image store resources (bottom of file) from`shippedrepos-docker-sitd` to your own docker hub or bintray account.

4. Provide your credentials for relevant resources into `./concourse/credentials.yml`.

5. Set the parameters in `./environments/dev/cf_parameters` to match your own Mesos/Marathoncluster (reachable from your Concourse deployment).

6. Make sure the username/password/dns domain you’ve set for your dev deployment in `cf_parameters` isreflected in `./cats/dev/integration_config.json` otherwise you’r CATS tests wont be able tofind your newly deployed ContainerCF or log in.

7. Commit all these changes to your new fork (as Concourse reads everything from git.) If you need to make thefork private due to keys, Mesos environment data etc, this is supported, see the private_key commented sectionsin `pipeline.yml` and the commented out SSH key section in `credentials.yml`

8. Deploy the pipeline (`pipeline.yml`) to your Concourse, godspeed!

2.1.5 Public Images and CF versions

The public images currently referenced in the `./run` section are the output of our initial development pipeline.They have some embedded keys for internal components. See “A note on unsurfaced keys” above. They work, greatfor testing, but if you’re interested in consuming ContainerCF seriously or developing on it, you’ll want your ownpipeline.

Our images currently produce a CloudFoundry at version 224. (This is controlled through which cf-release the pipelinechecks out in `./components/cf-release/Dockerfile`) If you produce newer generic images through thisprocess, by all means shout via twitter @mattdashj or @containercf if you’d like them publicly hosting to use with our`./run` scripts.

2.1.6 An IMPORTANT note on CF upstream changes

As mentioned in the original FAQ, we’re an extremely small team with day jobs. Keen followers of the CF communitywill have noticed the recent ‘sub-componentization’ of larger cf-release sections or components into other repositoriesand the breakup of applications into smaller pieces within CF’s core. While the pipeline could be configured (pretty

2.1. Development 11


simply) to support this new structure, we currently dont have the cycles to do this; so at a certain CF version, thepipeline will break as it won’t be fetching all needed components from cf-release (and/or, there will be new componentsthat the version of CF depends on which we wont have built a container for).

The idea of this being open sourced is our work can be consumed/improved without waiting on us if you have theurge/need to move past this change sooner.

2.1.7 Feedback

By all means shout with thoughts / suggestions / info to @containercf or @mattdashj We’re really hoping this is usefulto the community, if even as a base or lessons learnt for other projects!

Keep on hacking!

2.2 Mesos Dev Cluster

Provided here for reference, the following instructions were used to check our initial Mesos/Marathon environments,which were tied into our Concourse pipeline. A lot of this information is now old (~12 months) and Mesos deploymentsare likley easier, however it is provided here for reference.

OpenDCOS is a very easily install (in comparison) however only run’s on CentOS, which causes issues with theversion of CloudFoundry we have so far run through the pipeline (same as the historical CentOS Stemcell issuesrunning CF).

2.2.1 Pre-Requisite Instructions

This guide assumes you already have an OpenSource Mesos cluster with Marathon version >= 0.14.0 ready to go.

You will need at least 20Gb of available RAM and 8 available CPUs across your Mesos cluster in order to run CloudFoundry comfortably. If you’re planning on pushing more than a few test apps then we recommend you allocate moreRAM and increase the number of DEAs or Diego cells (see cf_parameters instructions later in this README).

We also assume that those machines are running Ubuntu 15.04 (Vivid) or newer. We have not tested for anything else,therefore if you are not running Vivid we recommend that you upgrade before you begin. (Skip this step if you arealready running >= 15.04.)

$ apt-get update && apt-get install update-manager-core $ vi /etc/update-manager/release-upgrades Prompt=normal$ do-release-upgrade n.b. the above step takes a very long time.

2.2.2 Mesos nodes

You can install Cloud Foundry into a Mesos cluster of any size, from a single node upwards. However some compo-nents need a fixed IP address. For the rest of this guide we’ll refer to the slave node that runs these components as themain node. You can elect any of your Mesos slave nodes to be your main node, just be consistent once you’ve pickedone!

Installation

Docker (all nodes)

Mesos (all nodes)



Ensure Mesos knows to use the correct containerizers and has a sufficiently long registration timeout for startingDocker containers.

Flannel (all nodes)

Cloud Foundry in Docker dev cluster uses a Flannel overlay network to communicate across Mesos nodes.

Etcd (main node only)

Necessary to store subnet information for Flannel.

Consul (main node only)

Cloud Foundry jobs make use of confd backed by Consul to enable dynamic provisioning across your Mesos cluster.

$ wget -O /tmp/consul.zip https://releases.hashicorp.com/consul/0.6.0/consul_0.6.0_→˓linux_amd64.zip$ sudo unzip /tmp/consul.zip -d /usr/local/bin/$ rm /tmp/consul.zip

Grub (all nodes)

2.2.3 Troubleshooting

The following items were FAQ’s in our initial Mesos/Marathon build pipeline environment.

TCP or HTTP errors on login

cf api or cf login gives a 503, 502 or 404 error, or Error performing request: Get http://api.: dial tcp:80: getsockopt: connection refused This is most likely because a CF component is not ready yet. Givethings a bit more time to stabilise and try again. If the error persists for longer than 10 minutes or then the fastestsolution is to delete and redeploy.

cf push hangs indefinitely

The API and DEAs are still wiring themselves together. Quit the push, delete the app, wait a little and try again.

Stack not found error

CFoundry::StackNotFound: 250003: The stack could not be found: The requested app stack cflinuxfs2 is not availableon this system The DEAs have not registered correctly with the API. Restart the API job using Marathon.

Instances information unavailable error

Could not fetch instance count: Server error, status code: 503, error code: 220002, message: Instances informationunavailable: getaddrinfo: Name or service not known This can happen when pushing apps to a Diego-enabled CloudFoundry. Components have not finished starting yet. Wait a little and try pushing again.

cf push reports an error dialing loggregator

Loggregator has not yet registered with the router. Wait a little and this should happen.

2.2. Mesos Dev Cluster 13

http://api


No application health status is visible

The HM9000 component polls the API every 6 minutes. The first check of a new CF deployment may fail as compo-nents wire themselves together. Health status should appear within 6 minutes.

Marathon won’t delete my Cloud Foundry!

We’ve seen issues where Docker refuses to kill containers, and so Marathon gets stuck with a never-ending deletion.To work around this you can sudo service docker restart on the node with the immortal container.

• Changelog2

2 https://github.com/CiscoCloud/containercf/blob/master/CHANGELOG.md


https://github.com/CiscoCloud/containercf/blob/master/CHANGELOG.md

CHAPTER 3

Demo Video (Marathon)

15


16 Chapter 3. Demo Video (Marathon)

CHAPTER 4

Repository Layout

4.1 Consume:

`run` - Scripts and Manifests to deploy current ContainerCF images.

4.2 Develop:

`concourse` - Concourse CI pipeline Configuration and Parameters.

`components` - Artifacts consumed by concourse for each CF component.

`environments` - Configuration for mesos clusters used during pipeline and testing.

`scripts` - Used by pipeline/concourse to push newly generated images to environments.

`cats` - Configuration and scripts for running the CF CATS test against environments.

`bosh_concourse_deployment0.yml` - Our BOSH manifest for the concourse environment (AWS).

17


18 Chapter 4. Repository Layout

CHAPTER 5

Credits

A Huge amount of credit goes to Gareth and Claudia (Github: @spikymonky and @callisto13) for their time, effortand awesomeness on this project and Colin Humphreys (@hatofmonkeys) for his guidance. You’ll find all these lovelypeople working over at Pivotal LDN nowadays.

The work also stood heavily the shoulders of Tommy Hughes’ work on ‘Running a PaaS in Docker’ (https://www.ctl.io/developers/blog/post/running-a-paas-in-docker/). This single-container CF in Docker implementation, providedour starting point and validated the base for what we wanted to acheive.

Finally, special hanks to Cisco for sponsoring the initial PoC and Development of ContainerCF.

19

https://www.ctl.io/developers/blog/post/running-a-paas-in-docker/https://www.ctl.io/developers/blog/post/running-a-paas-in-docker/


20 Chapter 5. Credits

CHAPTER 6

Search

• genindex

• search

6.1 License

Licensed under the Apache License, Version 2.03 (the “License”).

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an“AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See theLicense for the specific language governing permissions and limitations under the License.

3 http://www.apache.org/licenses/LICENSE-2.0

21

http://www.apache.org/licenses/LICENSE-2.0

Project Info and Run ContainerCFGetting Started (Info)Getting Started (Deploy)Mesos (Marathon)Kubernetes

Known IssuesDirect coupling of Monit failures to container failuresCentral dependancy on NATS for each DEA

FAQsWhy did you guys create this?Where is the Code?Is this ready for production?

Licenses

Develop ContainerCF and PipelineDevelopmentInitial Development GoalsPipeline OverviewRepository LayoutSetting up your own pipelinePublic Images and CF versionsAn IMPORTANT note on CF upstream changesFeedback

Mesos Dev ClusterPre-Requisite InstructionsMesos nodesTroubleshooting

Demo Video (Marathon)Repository LayoutConsume:Develop:

CreditsSearchLicense

ContainerCF Documentation - Read the Docs · 1.2.1 Mesos (Marathon) ... 2 Develop ContainerCF and...

Documents

Transcript of ContainerCF Documentation - Read the Docs · 1.2.1 Mesos (Marathon) ... 2 Develop ContainerCF and...