Kafka Security

© Hortonworks Inc. 2014

Kafka Security

SSL, Kerberos & Authorization


Who Are We?

Sriharsha ChintalapaniApache Kafka Committer

Apache Storm Committer & PMC

Parth BrahmbhattApache Kafka Contributor

Apache Storm Committer & PMC


Kafka Security

• SSL ( wire encryption)

• SASL ( Kerberos )

• Authorizer (Topic/Host/User level Authorization)


SSL


Kafka Security – SSL

• Kafka networking

• A TCP server listening for incoming connections

• Uses Non-blocking network I/O

• When a client connects to a server it opens a socket channel on

server side and hands it over selector.

• Selector gets polled in a loop. It will wake up whenever there are

connections ready with data to be read or write.

• Long – living connections , once established it will be used to

read/write data until client closed or an exception occurs.



• Kafka networking



• Kafka SSL / SASL requirements

• No User-level API changes to clients

• Retain length-encoded Kafka protocols

• Client must authenticate before sending/receiving requests

• Kafka Channel

• Instead of using socket channel, we added KafkaChannel

which consists a TransportLayer, Authenticator.



• TransportLayer

• Handles network level byte transfers

• PlaintextTransportLayer

• SSLTransportLayer

• Authenticator

• A pluggable interface for authentication implementations

• SaslAuthenticator – Provides SASL handshake and

authenticated user.



KafkaChannel

TransportLayer

Authenticator

Kafka Server

handshake

authenticate



• SSL - Handshake

• Kafka Server configures with Keystore and Truststore

• Kafka Client also needs a truststore with Kafka Server

certificate added to the truststore.

• Keystore configuration on client side is optional unless user wants

client side authentication.



• KafkaChannel

• Before write or read application data , checks if the

channel.ready()

• A channel is ready if its established a connection and

authenticated. No-OP of PlaintextTransportLayer

• If a channel is not ready it goes through channel.prepare()

which internally calls transportLayer.handshake()




• Before sending any application data, both client and server

needs to go though SSL handshake

• SSLTransportLayer uses SSLEngine to establish a non-

blocking handshake.

• SSLEngine provides a state machine to go through several

steps of SSLhandshake




• SocketChannel read

• Returns encrypted data

• Decrypts the data and returns the length of the data from Kafka protocols

• SocketChannel Write

• Writes encrypted data onto channel

• Regular socketChannel returns length of the data written to socket.

• Incase of SSL since we encrypt the data we can’t return exact length written to

socket which will be more than actual data

• Its important to keep track length of data written to network. This signifies if we

successfully written data to the network or not and move on to next request.



• Principal Builder

• SSLTransportLayer gives hostname as authenticated user

• X509Certificate has lot more information about a client

identity.

• PrincipalBuilder provides interface to plug in a custom

PrincipalBuilder that has access to X509Certificate and can

construct a user string out of it.

• Authenticator can use this custom principal to add ACLs



• Performance Impact

• Decrease in throughput by 20%.

• Latency increased by 30 %

• KAFKA-2481 (Ben Stopford) has more details



• listeners=SSL://host.name:port

• ssl.keystore.location

• ssl.keystore.password

• ssl.key.password

• ssl.truststore.location

• ssl.truststore.password

• security.inter.broker.protocol (optional)


SASL/ Kerberos


Kafka Security – SASL

• Simple Authentication and Security Layer, or SASL

• Provides flexibility in using Login Mechanisms

• One can use Kerberos , LDAP or simple passwords to authenticate.

• JAAS Login

• Before client & server can handshake , they need to authenticate with Kerberos or other Identity Provider.

• JAAS provides a pluggable way of providing user credentials. One can easily add LDAP or other mechanism just by changing a config file.



• JAAS Config file

KafkaServer {

com.sun.security.auth.module.Krb5LoginModule required

useKeyTab=true

storeKey=true

serviceName="kafka"

keyTab="/vagrant/keytabs/kafka1.keytab"

principal="kafka/[email protected]";

};

KafkaConfig {


useKeyTab=true

storeKey=true

serviceName="kafka"

keyTab="/vagrant/keytabs/client1.keytab"

principal=”client/[email protected]";

};



• SASL Authenticator• Uses configured login credentials of JAAS config.

• Non-blocking handshake to establish clients identity

• Once handshake established , Kerberos principal name will be the authenticated user.

• Can be layered with SSL for wire encryption or Plaintext incase of wire encryption not needed.• SASL can provide encryption but it has huge performance penalties



Client Broker

Connection

Mechanism list

Selected Mechanism & sasl data

Evaluate and Response

Sasl data

Client Authenticated



• Pass JAAS config file as jvm parameter

• -Djava.security.auth.login.config


Kafka Security – Resources

• SSL

• https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka

• SASL

• https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61326390

• Vagrant Setup

• SASL

• https://github.com/harshach/kafka-vagrant/tree/master/

https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61326390

https://github.com/harshach/kafka-vagrant/tree/sasl/


Authorization


Authorizer

• Controls who can do what

• Pluggable

• Acl based approach


Acl

• Alice is Allowed to Read from Orders-topic from Host-1

Principal Permission Operation Resource Host

Alice Allow Read Orders Host-1


Principal

• PrincipalType:Name

• Supported types: User and Group

• Extensible so users can add their own types

• Wild Card User:*


Operation

• Read, Write, Create, Delete, Alter, Describe,

ClusterAction, All

• Each API as an Operation VS Classification that maps to

APIs.


Resource

• ResourceType:ResourceName

• Topic, Cluster and ConsumerGroup

• Wild card resource ResourceType:*


Permissions

• Allow and Deny

• Anyone without an explicit Allow ACL is denied

• Then why do we have Deny?

• Deny works as negation

• Deny takes precedence over Allow Acls


Hosts

• Why provide this granularity?

• Allows authorizer to provide firewall type security even in

non secure environment.

• * as Wild card.


Configuration

• Authorizer class

• Super users

• Authorizer properties

• Default behavior for resources with no ACLs


SimpleAclAuthorizer

• Out of box authorizer implementation.

• Stores all of its ACLs in zookeeper.

• In built ACL cache to avoid performance penalty.

• Provides authorizer audit log.


CLI

• Add, Remove and List acls

• Convenience options:

--producer and --consumer.


Ranger Policy


Ranger Auditing


Ranger ACL management Audit


Unsecure zookeeper


Zookeeper

• Kafka’s metadata store

• Has its own security mechanism that supports SASL and

MD5-DIGEST for establishing identity and ACL based

authorization

• Create , Delete directly interacts with zookeeper


Securing zookeeper

• Acl on zk nodes: user:cdrwa

• Zookeeper.set.acl

• ZkSecurityMigrator script

• Credit where its due: Flavio Junqueira


Client JAAS

Client {


useKeyTab=true

storeKey=true

serviceName="zookeeper"

keyTab="/vagrant/keytabs/kafka.keytab"

principal="kafka/[email protected]";

};


Future

• KIP-4: Move everything to server side, no direct

interactions with zookeeper

• Group Support (PR already available)

• Pluggable Auditor


Summary

• SSL for wire encryption

• Sasl for authentication

• Authorization

• Secure Zookeeper

Thanks to the community for participation.

Kafka Security

Software

Transcript of Kafka Security