How to Secure Your Kubernetes Cluster

by Stanislav Kolenkin, Senior DevOps.

Introduction

2

Kubernetes, or k8s (k, 8 characters, s...get it?), or “kube”  is an open source system for managing containerized applications across multiple hosts, providing basic mechanisms for deployment, maintenance, and scaling of applications.

Why containers and Kubernetes?

3

Kubernetes makes containers easier to manager• Scheduling• Lifecycle management• Scaling• Load balancing

Kubernetes has security features built in• Logging and monitoring• Debugging and introspection• Identity and authorization

Why containers and Kubernetes?

4

Running containers allows you to adopt a fundamentally different security model.

How is securing a container different than a VM?

5

Surface of attack Minimalist host OS and limits the surface of an attack.Resource isolation Host resources are separated using namespaces and

cgroups.Permissions Access controls are for app privileges and shared

resources.Lifetime Containers have a shorter average lifetime.

Container security

6

Privilege escalation Unpatched vulnerability DDoS

kubernetes API compromise

Zero day exploit in common library

Node compromise and exploit

Container security

7

Application security Are my application secure?Infrastructure security Software supply chain Container runtime security

Is my infrastructure secure for developing containers?

Is my container image secure build and deploy?

Is my container secure to run?

Platform security Is my (cloud provider’s) infrastructure secure?

Kubernetes

8

Chenxi Wang, Jane Bond Project

Dr. Chenxi Wang is the founder of the Jane Bond Project, a cybersecurity strategy consulting firm. A well known strategist, speaker, and writer in the cybersecurity industry. Dr. Wang also serves as Vice Chair on the Board of Directors for the OWASP foundation

Kubernetes is so new thatlots of practitioners don’tknow what securitycontrols come with it.

So one of the first things to do isstudy up on what controls are thereand use them to strengthen yoursecurity posture

– Chenxi Wang, Jane Bond Project

9

Kubernetes

10

Kubernetes is one part of your security store

Kubernetes architecture

11

Your security journey

12

Managed Kubernetes Services

13

Managed Kubernetes Services: GKE

14

Managed Kubernetes Services

15

GKE AKS EKSNetwork Policy Calico Self-deployed

(kube-router) Self-deployed

(Calico)RBAC Yes Yes YesMgmt via (vendor) CLI Complete Complete MinimalUpgrades Auto/On-demand On-demand UnclearMulti AZ Yes Partial Yes

Protecting Cluster Metadata

Yes

Getting started

16

Restrict access to kubectl1

Access to kubectl

18

Kubernetes: Enforce bastion model via firewall rules in GCE

19

- name: kubernetes-allow-ssh-to-controller-api type: compute.v1.firewall properties: allowed: - IPProtocol: tcp ports: ["6443"] sourceTags: [ "ssh-bastion" ] sourceRanges: ["10.10.0.0/16"] targetTags: [ "k8s-controller" ] network: global/network/default

Kubernetes Engine: Authorized networks

20

Authorized networks are a way of specifying a restricted range of IP addresses that are permitted to access your container cluster's Kubernetes master endpoint.

GKE uses both Transport Layer Security (TLS) and authentication to provide secure access to your container cluster's Kubernetes master endpoint from the public internet. This provides you the flexibility to administer your cluster from anywhere; however, you might want to further restrict access to a set of IP addresses that you control. You can set this restriction by specifying an authorized network.

List up to 20 authorized networks to connect to the master over HTTPs.

Use RBAC2

RBAC

22

In Kubernetes, Identity and User management are not integrated in the platform and should be managed by external IAM platforms like Keycloak, Active Directory, Google’s IAM, etc. However, authentication and authorization are handled by Kubernetes.

The RBAC model in Kubernetes is based on three elements:● Roles: definition of the permissions for each Kubernetes resource type● Subjects: users (human or machine users) or groups of users● RoleBindings: definition of what Subjects have which Roles

Role-based access control

23

Use a Network Policy3

Network Policy

25

Network Policies is a new Kubernetes feature to configure how groups of pods are allowed to communicate with each other and other network endpoints. In other words, it creates firewalls between pods running on a Kubernetes cluster. This guide is meant to explain the unwritten parts of Kubernetes Network Policies.

By default, Kubernetes does not restrict traffic between pods running inside the cluster. This means any pod can connect to any other pod as there are no firewalls controlling the intra-cluster traffic.

Network Policy

26

Network Policies give you a way to declaratively configure which pods are allowed to connect to each other. These policies can be detailed: you can specify which namespaces are allowed to communicate, or more specifically you can choose which port numbers to enforce each policy on.

NetworkPolicy

27

Kubernetes Engine: Set a Network Policy

28

Create cluster with a network policy using the flag: --enable-network-policyUse a Network Policy as normalExample: restricting traffic to a pod

Introduction

29

We can configure the following Network policy scenarios:

• DENY all traffic to an application• LIMIT traffic to an application• ALLOW all traffic to an application• DENY all non-whitelisted traffic to a namespace• DENY all traffic from other namespaces (LIMIT access to the current namespace)• ALLOW traffic to an application from all namespaces• ALLOW all traffic from a namespace• ALLOW traffic from some pods in another namespace

Introduction

30

We can configure the following Network policy scenarios:

• ALLOW traffic from external clients• ALLOW traffic only to a port of an application• ALLOW traffic from apps using multiple selectors• DENY egress traffic from an application• DENY all non-whitelisted traffic from a namespace• LIMIT egress traffic to the cluster (DENY external egress traffic)

Introduction

31

We could configure soon the following Network policy scenarios:

• LIMIT egress traffic from an application to some pods• ALLOW traffic only to Pods in a namespace

Disable the kubernetes dashboard

4

Kubernetes dashboard

33

Why disable the dashboard? Tesla hack

34

• Hackers accessed the Kubernetes console, which was not password protected

• Console contained privileged cloud account credentials

• Used credentials to access resources and mine cryptocurrency

https://redlock.io/blog/cryptojacking-tesla

Remove the dashboard’s admin access

35

1. Don’t deploy it2. Reduce its permissions3. Reduce its permissions, and add authentication

Disable Kubernetes dashboard

36

The Kubernetes dashboard does not have admin access in GKE 1.7+ clusters, and is disabled by default in GKE 1.10+ clusters.

Use Google Console to perform the same tasks you would use the Kubernetes dashboards for.

Disable default service account token in pods

5

Default service account token in pods

38

Kubernetes: already disabled

39

Service account disabled by default (as RBAC is enabled) for Kubernetes 1.8+ clusters.

apiVersion: v1kind: ServiceAccountmetadata: name: build-robot automountServiceAccountToken: false

…

Kubernetes: already disabled

40

The default service account has no privileges if ABAC is disabled.Attribute-based access control (ABAC) defines an access control paradigm

whereby access rights are granted to users through the use of policies which combine attributes together.

RBAC is enabled by default (and ABAC is disabled) for GKE 1.8+ clusters.

Use pod Security Policy6

Linux security constructs

42

AppArmor is a Linux kernel security module that supplements the standard Linux user and group based permissions to confine programs to a limited set of resources. AppArmor can be configured for any application to reduce its potential attack surface and provide greater in-depth defense. It is configured through profiles tuned to whitelist the access needed by a specific program or container, such as Linux capabilities, network access, file permissions, etc. Each profile can be run in either enforcing mode, which blocks access to disallowed resources, or complain mode, which only reports violations.

Linux security constructs

43

Seccomp

Secure Computing Mode (seccomp) is a kernel feature that allows you to filter system calls to the kernel from a container. The combination of restricted and allowed calls are arranged in profiles, and you can pass different profiles to different containers. Seccomp provides more fine-grained control than capabilities, giving an attacker a limited number of syscalls from the container.

The default seccomp profile for docker is a JSON file and can be viewed here: https://github.com/docker/docker/blob/master/profiles/seccomp/default.json. It blocks 44 system calls out of more than 300 available. Making the list stricter would be a trade-off with application compatibility.

Linux security constructs

44

AppArmor• Linux Security Module that lets you restrict your program’s actions• Controls and audits various process actions, e.g., file functions like read, write,

execute

seccomp• Filters a process’ syscalls to limit what syscalls the process allows• Puts applications in `secure` state with whitelist of allowed syscalls• Docker seccomp default denies ~150 uncommon or potentially unsafe

syscalls

Security context vs security policy

45

PodSecurityPolicy (PSP)

46

To use, enable the admission controller for Pod Security Policykube-apiserver --enable-admission-plugins=PodSecurityPolicy

The ‘restricted’ policy provides a good starting point.

PodSecurityPolicy (PSP)

47

apiVersion: policy/v1beta1kind: PodSecurityPolicymetadata: name: restrictedspec: privileged: false # Don't allow privileged pods! allowPrivilegeEscalation: false # Don’t allow privilegedescalation: runAsUser: # Require the container to run without rootprivileges: rule: ‘MustRunAsNonRoot’ supplementalGroups: # Forbid adding to the root group rule: ‘MustRunAs’ ranges:

- min: 1 max: 65535

Hardening your cluster

48

Hardening your cluster

49

• Enable Protecting Cluster Metadata• Using clair or other solutions for scanning images for known vulnerabilities • Verify binaries that are deployed• Consider sandboxing - gVisor• Limit the identity used by pods - OSS Proposal• Use a service mesh for authentication/encryption - Istio

Protecting Cluster Metadata7

Protecting Cluster Metadata

51

Google Kubernetes Engine uses instance metadata to configure node VMs, but some of this metadata can be used to attack a cluster, and so should be concealed from workloads running on the cluster that don't need access to it.

This page explains how to use GKE's metadata concealment feature to protect only potentially sensitive system metadata from user workloads running on your cluster.

Protecting Cluster Metadata

52

Instance metadata is data about your instance that you can use to configure or manage the running instance. Instance metadata is divided into categories. AWS: http://169.254.169.254, GCP: http://metadata.google.internal

GKE: Protecting cloud metadata

53

Beginning with Kubernetes version 1.9.3, you can enable metadata concealment to prevent user Pods from accessing certain VM metadata for your cluster's nodes, such as Kubelet credentials and VM instance information. Specifically, metadata concealment protects access to kube-env (which contains Kubelet credentials) and the VM's instance identity token.

Metadata concealment firewalls traffic from user Pods (Pods not running on HostNetwork) to the cluster metadata server, only allowing safe queries. The firewall prevents user Pods from using Kubelet credentials for privilege escalation attacks, or from using VM identity for instance escalation attacks.

GKE: Protecting cloud metadata

54

Limitations● Metadata concealment only protects access to kube-env and the node's

instance identity token● Metadata concealment does not restrict access to the node's service account● Metadata concealment does not restrict access to other related instance

metadata

To create a cluster with metadata concealment enabled, you need add the following parameter:

--workload-metadata-from-node=SECURE

Protecting cloud metadata

55

• Metadata-proxyhttps://github.com/GoogleCloudPlatform/k8s-metadata-proxy

• Configure NetworkPolicy to prevent access metadata API• kube2iam

https://github.com/jtblin/kube2iam

Vault8

Secrets and Configmaps

57

Objects of type secret are intended to hold sensitive information, such as passwords, OAuth tokens, and ssh keys. Putting this information in a secret is safer and more flexible than putting it verbatim in a pod definition or in a docker image. Secrets use Base64 for encrypt and decrypt.

ConfigMaps allow you to decouple configuration artifacts from image content to keep containerized applications portable.

Secrets and Configmaps

58

Dynamic secrets on Kubernetes pods using Vault

59

Dynamic secrets on Kubernetes pods using Vault

60

Summary:

• Vault provides a secure solution for sharing and distributing secrets• Some challenges that require customization for your needs• Stakes are extremely high and need to prepare accordingly • Kubernetes and Vault are excellent tools and can work together well, but

proper integration is a non-trivial task.

Use service Mesh9

Mesh network

62

A mesh network is a local network topology in which the infrastructure nodes connect directly, dynamically and non-hierarchically to as many other nodes as possible and cooperate with one another to efficiently route data from/to clients.

Istio

63

Istio gives:• Automatic load balancing for HTTP, gRPC, WebSocket, and TCP traffic.• Fine-grained control of traffic behavior with rich routing rules, retries,

failovers, and fault injection.• A pluggable policy layer and configuration API supporting access controls,

rate limits and quotas.• Automatic metrics, logs, and traces for all traffic within a cluster, including

cluster ingress and egress.• Secure service-to-service authentication with strong identity assertions

between services in a cluster.

Istio

64

Istio

65

GKE ( for k8s version 1.7.6 +) uses k8s network policies to manage which pods and services can communicate . This is relatively straightforward to configure. Istio also has network policies but they’re not the k8s policies you know and love so what’s the difference and why both? This post explains it well so I won’t repeat the reasoning here but this table summarises the differences and why you need both.

Istio

66

Audit9

KubeATF

68

The Kubernetes Ansible Testing Framework is a tool meant to spin up, test, save results, and spin down Kubernetes clusters using the various installation tools/kits/templates readily available in a human and CI/CD friendly way.

Test Cases:1. Kubernetes Installation Tool Testing - If you are a maintainer of a

Kubernetes installation tool, this provides a mechanism to create a repeatable process for creating test clusters to improve consistency and reduce time spent during development.

2. Kubernetes Release Testing - If you are running the e2e (end to end) testing suite on various releases/configurations, this tool may help you.

KubeATF

69

3. Kubernetes Application Testing - If you are deploying an application or a helm chart inside Kubernetes and you are concerned with ensuring compatibility across clouds and versions of Kubernetes, this tool may help you.

4. Kubernetes Security Testing - If you are assessing multiple installation tools for their security posture using things like CIS Benchmarking checks, this tool can help automate the testing process.

Kubeaudit

70

Kubeaudit is a command line tool to audit Kubernetes clusters for various different security concerns: run the container as a non-root user, use a read only root filesystem, drop scary capabilities, don't add new ones, don't run privileged.

Kubeaudit

71

Kubeaudit has multiple checks:● Audit all● Audit security context○ Audit readOnlyRootFilesystem○ Audit runAsNonRoot○ Audit allowPrivilegeEscalation○ Audit privileged○ Audit capabilities

● Audit image● Audit Service Accounts● Audit network policies● Audit resources

K8Guard

72

K8Guard is an auditing system for Kubernetes clusters. It monitors different entities on your cluster for possible violations. K8Guard notifies the violators and then takes action on them. It also provides metrics and dashboards about violations in the cluster through Prometheus.

NeuVector

73

The tools are in the form of scripts which implement tests for the 100+ recommendations in the Kubernetes 1.6 CIS Benchmark for security. They are available here at https://github.com/neuvector/kubernetes-cis-benchmark. There are different security recommendations for the Kubernetes master node and for the worker node, so there are two separate scripts.In general the recommendations focus on auditing key security areas of

NeuVector

74

Kubernetes, including:● Use of privileged containers● API server authentication and authorization● Kubelet authentication● etcd security● Data security, e.g. files, secrets● Certificate management● Security for pods.

TwistLock

75

Twistlock is the industry’s most complete, automated and scalable container cybersecurity platform. From precise, full-lifecycle vulnerability and compliance management to application-tailored runtime defense and cloud native firewalls, Twistlock secures your containers and modern applications against the next generation of threats across the entire application lifecycle.

Why GKE security?

76

• Google Kubernetes Engine provides best practices with security by default.There will always be app-level hardening and tuning to do.

• Google Kubernetes Engine patches you to the latest version, automatically.This keeps you up to date with security patches and with new features.

• Google Kubernetes Engine provides best of Google Cloud securityWith integrations with IAM, Audit Logging, VPC, and more.

Conclusion

77

• Many possibilities to make your cluster more secure• Each with it’s benefits and drawbacks• Generally - Security means (hard) work• Depends on your use Case what to use• Play around with and get a feeling (in a playground)• Monitoring • Logging

Thank you for your attention!Questions?

CONTACTS:

Email: stas.kolenkin@gmail.com

Skype: stas.kolenkin