Cilium – Kernel Native Security & DDOS

Mitigation for Microservices with BPF

Cynthia ThomasTechnology Evangelist, Covalent

@_techcet_

Gordon is back!And he got a job at Lego.

Robot Competition

● Local marketing launches an Robot competition for Danish residents

● Upcoming deadline means tons of uploaded media

Robot Competition

User Data MongoDB

Data store

Inventory

Image UploadService

Web Front-End

KafkaBrokerKafkaBrokerKafkaBroker Image

processingImage processing

Image processing

ZookeeperZookeeperZookeeper

End-User

Admintask

Audit

Gordon’s Task List● Deploy Cilium HTTP-aware security for microservices

● Isolate Kafka resources for old & new services

● DDOS mitigation via XDP/BPF

Gordon’s 1st TaskUpdate HTTP Security for Microservices

Application Architectures

Delivery Frequency

Operational Complexity

Single Server App

Yearly

Low

Evolution of Application Design & Delivery Frequency

Application Architectures

Delivery Frequency

Operational Complexity

Single Server App

Yearly

Low

3-Tier App

Monthly

Moderate

Evolution of Application Design & Delivery Frequency

Application Architectures

Delivery Frequency

Operational Complexity

Single Server App

Yearly

Low

Distributed Microservices

10-100 x’s / day

Extreme

3-Tier App

Monthly

Moderate

Evolution of Application Design & Delivery Frequency

The world still runs on iptablesmatching IPs and ports:

$ iptables -A INPUT -p tcp -s 15.15.15.3 --dport 80 \-m conntrack --ctstate NEW -j ACCEPT

Robot Competition

User Data MongoDB

Data store

Inventory

Image UploadService

Web Front-End

KafkaBroker

Image processing

Zookeeper

End-User

Admintask

Audit

Example: HTTP calls

API

Web Front-End

Security for Microservices

Image UploadService

GET /image

POST /image

GET /image/flagged

PUT /image/id

APIPOST /image

Web Front-End

Security for Microservices

Image UploadService

GET /image

GET /image/flagged

PUT /image/id

POST /image

L3/L4

APIPOST /image

Web Front-End

Security for Microservices

iptables -s 10.1.1.1-p tcp --dport 80-j ACCEPT

Image UploadService

GET /image

POST /image

GET /image/flagged

PUT /image/id

L3/L4

APIPOST /image

Web Front-End

Security for Microservices

iptables -s 10.1.1.1-p tcp --dport 80-j ACCEPT

exposed

exposed

exposed

Image UploadService

GET /image

POST /image

GET /image/flagged

PUT /image/id

Network Security has not evolved for microservices…

…until now.

APIPOST /image

Web Front-End

Cilium API-Aware Security

Image UploadService

GET /image

GET /image/flagged

PUT /image/id

POST /image

L7

APIPOST /image

Web Front-End

Cilium API-Aware Security

Image UploadService

GET /image

POST /image

GET /image/flagged

PUT /image/id

FROM Web Front-EndALLOW POST /image

Under the Hood: BPFLinux

Superpowers

Unleashed

BPF instruction setstruct bpf_insn prog[] = {

BPF_MOV64_REG(BPF_REG_6, BPF_REG_1), BPF_LD_ABS(BPF_B, ETH_HLEN + offsetof(struct iphdr, protocol) /* R0 =

ip->proto */), BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_0, -4), /* *(u32 *)(fp - 4) = r0

*/ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), /* r2 = fp - 4 */BPF_LD_MAP_FD(BPF_REG_1, map_fd), BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2), BPF_MOV64_IMM(BPF_REG_1, 1), /* r1 = 1 */BPF_RAW_INSN(BPF_STX | BPF_XADD | BPF_DW, BPF_REG_0, BPF_REG_1, 0, 0),

/* xadd r0 += r1 */BPF_MOV64_IMM(BPF_REG_0, 0), /* r0 = 0 */BPF_EXIT_INSN(),

};

Reference: https://git.kernel.org/pub/scm/linux/kernel/git/ast/bpf.git/tree/samples/bpf/sock_example.c

Cilium

CiliumAgent

Userspace

Kernelspace

Cilium Architecture

CiliumAgent

API Calls

Userspace

Kernelspace

Plugins

Cilium Architecture

CiliumAgent

BPF

API Calls

BPF Code & Maps

Userspace

Kernelspace

Plugins

Cilium Architecture

CiliumAgent

BPF

API Calls

BPF Code & Maps

Userspace

Kernelspace

Plugins

Cilium Architecture

CiliumAgent

Extremely scalable, highly customized

network filtering, load-balancing, and

monitoring

BPF BPF

API Calls

BPF Code & Maps

Userspace

Kernelspace

Plugins

Cilium Architecture

CiliumAgent

BPF BPF

API Calls

BPF Code & Maps

Packets

Userspace

Kernelspace

Plugins

Cilium Architecture

CiliumAgent

BPF

BPF BPF

API Calls

BPF Code & Maps

Packets

Userspace

Kernelspace

Plugins

Cilium Architecture

CiliumAgent

BPF

BPF BPF

API Calls

BPF Code & Maps

Packets

Userspace

Kernelspace

Plugins

Cilium Architecture

CiliumAgent

BPF

BPF BPF

API Calls

BPF Code & Maps

Packets

Userspace

Kernelspace

CLI,Monitoring,

Policies

Plugins

Cilium Architecture

CiliumAgent

Gordon’s 2nd TaskIsolate Kafka Topics

Kafka?Used for building real-time pipelines and streaming apps.

- Horizontally scalable

- Fault-tolerant

- “Wicked fast”

Defined by its own protocol.

1/3 of all Fortune 500 companies use Kafka[1]

Kafka ConceptsTopic 1 Producers

Kafka Broker

Topic 1

Topic N

Topic 1 Consumer Group A

Topic 1 Consumer Group B

Security for Microservices

Gordon worries about open Kafka topics

The API is too open!How do I bring on

new services?

Robot Competition

User Data MongoDB

Data store

Inventory

Image UploadService

Web Front-End

KafkaBrokerKafkaBrokerKafkaBroker Image

processingImage processing

Image processing

ZookeeperZookeeperZookeeper

End-User

Admintask

Audit

Example: Kafka calls

Kafka.API

Imageprocessing

Security for Microservices

Kafka Broker

Fetch inventory

Fetch image

Fetch userid

Fetch analytics

Produce inventory

Produce image

Produce userid

Produce analytics

Kafka.API

Fetch image

Imageprocessing

Security for Microservices

Kafka Broker

Fetch inventory

Fetch image

Produce userid

Fetch userid

L3/L4

Kafka.API

Fetch image

Imageprocessing

Security for Microservices

iptables -s 10.1.1.1-p tcp --dport 9092-j ACCEPT

Kafka Broker

Fetch inventory

Fetch image

Produce userid

Fetch userid

L3/L4

Kafka.API

Fetch image

Imageprocessing

Security for Microservices

iptables -s 10.1.1.1-p tcp --dport 9092-j ACCEPT

exposed

exposed

exposed

Kafka Broker

Fetch inventory

Fetch image

Produce userid

Fetch userid

Kafka.API

Fetch image

Imageprocessing

Security for Microservices

Kafka Broker

Fetch inventory

Fetch image

Produce userid

Fetch userid

L7

Kafka.API

Fetch image

Imageprocessing

Security for Microservices

Kafka Broker

Fetch inventory

Fetch image

Produce userid

Fetch userid

FROM Image processingALLOW Fetch image

Demo!

Get real with Gordon

Demo: Kafka API Filtering

app1-producer

app1-consumer

app2-producer

app2-consumer

kafka

What just happened?

L7

Kafka.API

Produce imageprocessing

app1-producer

Demo: Kafka API Filtering

Kafka Broker

Produce imageprocessing

Produce userdata

FROM app1-producerALLOW Produce imageprocessing

BPF

BPF BPF

API Calls

BPF Code & Maps

Packets

Userspace

Kernelspace

CLI,Monitoring,

Policies

Plugins

Sidecar Proxy

CiliumAgent

SidecarProxy

BPF

BPF BPF

API Calls

BPF Code & Maps

Packets

Userspace

Kernelspace

CLI,Monitoring,

Policies

Plugins

In-Kernel Kafka Parsing

CiliumAgent

kernel proxyBPF

Gordon’s 3rd TaskDDOS Mitigation

Robot Competition● Local marketing launches

competition for Danish residents only

● DDOS anticipated from other countries and anti-Lego activists

Robot Competition

User Data MongoDB

Data store

Inventory

Image UploadService

Web Front-End

KafkaBroker

Image processing

Zookeeper

End-User

Admintask

Audit

Example: DDOS attack!

BPF

BPF BPF

API Calls

BPF Code & Maps

Packets

Userspace

Kernelspace

CLI,Monitoring,

Policies

Plugins

XDP/BPF handled in NIC driver

CiliumAgent

XDP

Source: https://www.netdevconf.org/2.1/slides/apr6/zhou-netdev-xdp-2017.pdf

Facebook published BPF/XDP numbersfor L3/L4 LB at Netdev 2.1

BPF/XDP throughput

IPVS throughput

BPF with XDP Setup

pktgen attack:~11.6 Mppsrandomly in 10.0.0.0/8

legit traffic:netperf tests on 10.192.1.0/24

Blacklist 16M rules

All /32s in 10.0.0.0/8

BPF with XDP for DDoS mitigationMetric iptables / ipset XDPDDoS rate [packets/s] 11.6M 11.6MDrop rate [packets/s] 7.1M 11.6MTime to load rules [time] 3 min 20 sec 31 secLatency under load [ms] 2.3ms 0.1msThroughput under DDoS [Gbit/s] 0.014 6.5

Requests/s under DDoS [kReq/s] 0.28 82.8

Sender: Send 64B packets as fast as possible è Receiver: Drop as fast as possible

Source: Daniel Borkmann’s presentation:http://schd.ws/hosted_files/ossna2017/da/BPFandXDP.pdf

● Cilium deployment for microservices successfully secured HTTP traffic

● Kafka resources were isolated to protect existing services from new ones

●Mitigated DDOS attacks via XDP/BPF

Gordon’s Summary

Cilium Project Status• Cilium v0.12 release in October

• Docker, Kubernetes, and Mesos integration

• Looking for feedback and contributions

Take Action!

• Getting Started Using Docker: docs.cilium.io/

• Join our Slack community!

• Check out the project website for more details: https://www.cilium.io/

Please ★us on

GitHub

Thank You!

github.com/cilium/cilium

cilium.io

@ciliumproject