1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Troubleshooting KerberosIn Hadoop :Taming the BeastDataWorks Summit
Sept 2017
2 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Author Profile
Vipin Rathor
Sr. Product Specialist (HDP Security)
Contributed to Kerberos, Apache Zeppelin, Apache Atlas
[email protected] / @VipinRathor46
3 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Agenda
• Why Kerberos?
• Where is Kerberos used across the Hadoop Stack?
• What is Kerberos & how does it work
• Realms, Principals and Keytabs
• Systematic Approach to Kerberos Nirvana
• Tools available in Hadoop
• Native Kerberos Tools / Debug Options
• Kerberos Checklist
• Most Common Kerberos Error Messages (& their meanings)
4 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Why Kerberos?
• Universal Authentication mechanism for Hadoop stack
• Integrates with enterprise user management (e.g. Active Directory)
Solves:
• How can parts of a cluster trust each other(NameNodes, DataNodes, YARN, HBase, ZooKeeper...)
• How can users trust the system?
• How can the system trust users?
• Foundation for: how can users delegate rights to applications?
• Without Kerberos: your cluster has NO security
Hadoop clusters are some of the largest Kerberos systems ever!!
5 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Where is Kerberos used across the Hadoop Stack?
• Ubiquitous End-User / Hadoop Service Authentication mechanism
• Hadoop DelegationToken (Delegated authentication to NameNode)
• != Kerberos Tickets
• Bootstrapped with Kerberos authentication token
• HTTP Authentication
• Using SPNEGO (RFC 4559)
• Via Browsers / cURL (curl --negotiate)
• RPC Authentication
• Using Simple Authentication & Security Layer aka SASL (!= SSL)
• Java API Based Kerberos login
• Using JGSS / JAAS
• GSS-API (RFC 2743)
6 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
What is Kerberos
• Open source, Developed by MIT
• Password is NEVER transmitted over wire
• Central trusted authority – Key Distribution Center (KDC)
• Symmetric key (common shared key)
• Flavors:• MIT Kerberos
• Active Directory
• Heimdal Kerberos (OS X)
7 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
How does Kerberos work
End User
- Does kinit (1 & 2)
- Runs HDFS
command (3 - 6)
Hadoop NameNode
- Starts up with nn.service.keytab
- Verifies user and gives access to
HDFS
KDC
- Provisions user keys and
service keytabs (e.g.
nn.service.keytab)
- Provides TGT and TGS
8 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Realms, Principals and Keytabs
• Realm
• User Principal• E.g. [email protected]
• ken/[email protected]
• ken/[email protected]
• Service Principal• E.g. HTTP/[email protected]
• nn/[email protected]
• dn/[email protected]
• dn/[email protected]
• Keytabs• Service keytabs (for service)
• Headless keytabs (for user)
9 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Systematic Approach to Kerberos Nirvana
• Identify the involved parties (user, service, keytabs, nodes)
• Identify the stage where Kerberos is failing
• Based on stage & error message, narrow down between client-side or service-side issue
• Check & verify configurations for correctness using the appropriate tools
• Repeat as necessary
10 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Kerberos Tools Available in Hadoop
• Kdiag
• Runs a series of diagnostic checks & gives suggestions
• hadoop org.apache.hadoop.security.KDiag
11 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Kerberos Tools Available in Hadoop (cntd..)
• HadoopKerberosName
• Checks Auth_to_local rules (Kerberos Principal to Unix user name conversion)
• hadoop org.apache.hadoop.security.HadoopKerberosName
12 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Native Kerberos Tools / Debug Options
• via command line• kinit
• klist -eaf / klist –kte
• kvno
• kdestroy
• export KRB5_TRACE=/tmp/krb5-curl.outcurl -ivL --negotiate -u: "http://namenode-host:50070/webhdfs/v1/?op=LISTSTATUS"
• via debug messages• export HADOOP_JAAS_DEBUG=true
• export HADOOP_ROOT_LOGGER=DEBUG,console
• via Java library• -Dsun.security.krb5.debug=true
• -Dsun.security.spnego.debug=true
• export OPTS=“$OPTS -Dsun.security.krb5.debug=true”
13 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Kerberos Checklist
• FQDN
• Name Resolution
• If DNS is configured, then check reverse lookup
• Date/Time sync (< 5 minutes)
• Configuration file - /etc/krb5.conf
• Principal Names
• Stale Keytabs (via kvno)
• Credential Cache location (JAAS config)
• Which Java suite, JCE policy
• KDC log file - /var/log/kerberos/krb5kdc.log
14 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Most Common Kerberos Error Messages (& their meaning)
• <unknown client> for <unknown service>
• Decrypt Integrity Check Failed
• AES256 EncType not supported
• Clock skew too great
• Kerberos service principal not found in the database
• Client not found in the database
• No valid initial credential found
15 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
References
• http://web.mit.edu/kerberos/
• http://www.kerberos.org/software/tutorial.html
• https://github.com/steveloughran/kerberos_and_hadoop
Top Related