Kafka Security

download Kafka Security

of 45

  • date post

    22-Jan-2018
  • Category

    Software

  • view

    1.669
  • download

    1

Embed Size (px)

Transcript of Kafka Security

  1. 1. Page1 Hortonworks Inc. 2014 Kafka Security SSL, Kerberos & Authorization
  2. 2. Page2 Hortonworks Inc. 2014
  3. 3. Page3 Hortonworks Inc. 2014 Who Are We? Sriharsha Chintalapani Apache Kafka Committer Apache Storm Committer & PMC Parth Brahmbhatt Apache Kafka Contributor Apache Storm Committer & PMC
  4. 4. Page4 Hortonworks Inc. 2014 Kafka Security SSL ( wire encryption) SASL ( Kerberos ) Authorizer (Topic/Host/User level Authorization)
  5. 5. Page5 Hortonworks Inc. 2014 SSL
  6. 6. Page6 Hortonworks Inc. 2014 Kafka Security SSL Kafka networking A TCP server listening for incoming connections Uses Non-blocking network I/O When a client connects to a server it opens a socket channel on server side and hands it over selector. Selector gets polled in a loop. It will wake up whenever there are connections ready with data to be read or write. Long living connections , once established it will be used to read/write data until client closed or an exception occurs.
  7. 7. Page7 Hortonworks Inc. 2014 Kafka Security SSL Kafka networking
  8. 8. Page8 Hortonworks Inc. 2014 Kafka Security SSL Kafka SSL / SASL requirements No User-level API changes to clients Retain length-encoded Kafka protocols Client must authenticate before sending/receiving requests Kafka Channel Instead of using socket channel, we added KafkaChannel which consists a TransportLayer, Authenticator.
  9. 9. Page9 Hortonworks Inc. 2014 Kafka Security SSL TransportLayer Handles network level byte transfers PlaintextTransportLayer SSLTransportLayer Authenticator A pluggable interface for authentication implementations SaslAuthenticator Provides SASL handshake and authenticated user.
  10. 10. Page10 Hortonworks Inc. 2014 Kafka Security SSL KafkaChannel TransportLayer Authenticator Kafka Server handshake authenticate
  11. 11. Page11 Hortonworks Inc. 2014 Kafka Security SSL SSL - Handshake Kafka Server configures with Keystore and Truststore Kafka Client also needs a truststore with Kafka Server certificate added to the truststore. Keystore configuration on client side is optional unless user wants client side authentication.
  12. 12. Page12 Hortonworks Inc. 2014 Kafka Security SSL KafkaChannel Before write or read application data , checks if the channel.ready() A channel is ready if its established a connection and authenticated. No-OP of PlaintextTransportLayer If a channel is not ready it goes through channel.prepare() which internally calls transportLayer.handshake()
  13. 13. Page13 Hortonworks Inc. 2014 Kafka Security SSL SSLTransportLayer Before sending any application data, both client and server needs to go though SSL handshake SSLTransportLayer uses SSLEngine to establish a non- blocking handshake. SSLEngine provides a state machine to go through several steps of SSLhandshake
  14. 14. Page14 Hortonworks Inc. 2014 Kafka Security SSL
  15. 15. Page15 Hortonworks Inc. 2014 Kafka Security SSL SSLTransportLayer SocketChannel read Returns encrypted data Decrypts the data and returns the length of the data from Kafka protocols SocketChannel Write Writes encrypted data onto channel Regular socketChannel returns length of the data written to socket. Incase of SSL since we encrypt the data we cant return exact length written to socket which will be more than actual data Its important to keep track length of data written to network. This signifies if we successfully written data to the network or not and move on to next request.
  16. 16. Page16 Hortonworks Inc. 2014 Kafka Security SSL Principal Builder SSLTransportLayer gives hostname as authenticated user X509Certificate has lot more information about a client identity. PrincipalBuilder provides interface to plug in a custom PrincipalBuilder that has access to X509Certificate and can construct a user string out of it. Authenticator can use this custom principal to add ACLs
  17. 17. Page17 Hortonworks Inc. 2014 Kafka Security SSL Performance Impact Decrease in throughput by 20%. Latency increased by 30 % KAFKA-2481 (Ben Stopford) has more details
  18. 18. Page18 Hortonworks Inc. 2014 Kafka Security SSL listeners=SSL://host.name:port ssl.keystore.location ssl.keystore.password ssl.key.password ssl.truststore.location ssl.truststore.password security.inter.broker.protocol (optional)
  19. 19. Page19 Hortonworks Inc. 2014 SASL/ Kerberos
  20. 20. Page20 Hortonworks Inc. 2014 Kafka Security SASL Simple Authentication and Security Layer, or SASL Provides flexibility in using Login Mechanisms One can use Kerberos , LDAP or simple passwords to authenticate. JAAS Login Before client & server can handshake , they need to authenticate with Kerberos or other Identity Provider. JAAS provides a pluggable way of providing user credentials. One can easily add LDAP or other mechanism just by changing a config file.
  21. 21. Page21 Hortonworks Inc. 2014 Kafka Security SASL JAAS Config file KafkaServer { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true serviceName="kafka" keyTab="/vagrant/keytabs/kafka1.keytab" principal="kafka/host@EXAMPLE.COM"; }; KafkaConfig { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true serviceName="kafka" keyTab="/vagrant/keytabs/client1.keytab" principal=client/host@EXAMPLE.COM"; };
  22. 22. Page22 Hortonworks Inc. 2014 Kafka Security SASL SASL Authenticator Uses configured login credentials of JAAS config. Non-blocking handshake to establish clients identity Once handshake established , Kerberos principal name will be the authenticated user. Can be layered with SSL for wire encryption or Plaintext incase of wire encryption not needed. SASL can provide encryption but it has huge performance penalties
  23. 23. Page23 Hortonworks Inc. 2014 Kafka Security SASL Client Broker Connection Mechanism list Selected Mechanism & sasl data Evaluate and Response Sasl data Client Authenticated
  24. 24. Page24 Hortonworks Inc. 2014 Kafka Security SASL Pass JAAS config file as jvm parameter -Djava.security.auth.login.config
  25. 25. Page25 Hortonworks Inc. 2014 Kafka Security Resources SSL https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka SASL https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61326390 Vagrant Setup SASL https://github.com/harshach/kafka-vagrant/tree/master/
  26. 26. Page26 Hortonworks Inc. 2014 Authorization
  27. 27. Page27 Hortonworks Inc. 2014 Authorizer Controls who can do what Pluggable Acl based approach
  28. 28. Page28 Hortonworks Inc. 2014 Acl Alice is Allowed to Read from Orders-topic from Host-1 Principal Permission Operation Resource Host Alice Allow Read Orders Host-1
  29. 29. Page29 Hortonworks Inc. 2014 Principal PrincipalType:Name Supported types: User and Group Extensible so users can add their own types Wild Card User:*
  30. 30. Page30 Hortonworks Inc. 2014 Operation Read, Write, Create, Delete, Alter, Describe, ClusterAction, All Each API as an Operation VS Classification that maps to APIs.
  31. 31. Page31 Hortonworks Inc. 2014 Resource ResourceType:ResourceName Topic, Cluster and ConsumerGroup Wild card resource ResourceType:*
  32. 32. Page32 Hortonworks Inc. 2014 Permissions Allow and Deny Anyone without an explicit Allow ACL is denied Then why do we have Deny? Deny works as negation Deny takes precedence over Allow Acls
  33. 33. Page33 Hortonworks Inc. 2014 Hosts Why provide this granularity? Allows authorizer to provide firewall type security even in non secure environment. * as Wild card.
  34. 34. Page34 Hortonworks Inc. 2014 Configuration Authorizer class Super users Authorizer properties Default behavior for resources with no ACLs
  35. 35. Page35 Hortonworks Inc. 2014 SimpleAclAuthorizer Out of box authorizer implementation. Stores all of its ACLs in zookeeper. In built ACL cache to avoid performance penalty. Provides authorizer audit log.
  36. 36. Page38 Hortonworks Inc. 2014 CLI Add, Remove and List acls Convenience options: --producer and --consumer.
  37. 37. Page39 Hortonworks Inc. 2014 Ranger Policy
  38. 38. Page40 Hortonworks Inc. 2014 Ranger Auditing
  39. 39. Page41 Hortonworks Inc. 2014 Ranger ACL management Audit
  40. 40. Page42 Hortonworks Inc. 2014 Unsecure zookeeper
  41. 41. Page43 Hortonworks Inc. 2014 Zookeeper Kafkas metadata store Has its own security mechanism that supports SASL and MD5-DIGEST for establishing identity and ACL based authorization Create , Delete directly interacts with zookeeper
  42. 42. Page44 Hortonworks Inc. 2014 Securing zookeeper Acl on zk nodes: user:cdrwa Zookeeper.set.acl ZkSecurityMigrator script Credit where its due: Flavio Junqueira
  43. 43. Page45 Hortonworks Inc. 2014 Client JAAS Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true serviceName="zookeeper" keyTab="/vagrant/keytabs/kafka.keytab" principal="kafka/kafka@WITZEND.COM"; };
  44. 44. Page46 Hortonworks Inc. 2014 Future KIP-4: Move everything to server side, no direct interactions with zookeeper Group Support (PR already available) Pluggable Auditor
  45. 45. Page47 Hortonworks Inc. 2014 Summary SSL for wire encryption Sasl for authentication Authorization Secure Zookeeper Thanks to the community for participation.