Cloudera HDFS kerberized failure: GSSException No Valid credentials

hi hadooper,

This is the problem that you will be facing once you have enabled kerberos on a cloudera server.

Here are the log of the hadoop looks like:

2016-12-01 20:28:21,650 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 for port 8022: readAndProcess from client 172.31.6.120 threw exception [javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Encryption type AES256 CTS mode with HMAC SHA1-96 is not supported/enabled)]]
2016-12-01 20:28:23,457 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 for port 8022: readAndProcess from client 172.31.0.159 threw exception [javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Encryption type AES256 CTS mode with HMAC SHA1-96 is not supported/enabled)]]
2016-12-01 20:28:23,627 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 for port 8022: readAndProcess from client 172.31.0.158 threw exception [javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Encryption type AES256 CTS mode with HMAC SHA1-96 is not supported/enabled)]]

The quick remedy on this will be applying the Java Cryptograhy Extension (JCE) on all the nodes in the cluster.

Here is the step to apply the JCE jar files.

1. Download the tarball which contents the following jar files.

US_export_policy.jar
local_policy.jar

2. Copying them onto each nodes, and overwrite the existing jar files.

/usr/java/jdk1.7.0_67-cloudera/jre/lib/security/US_export_policy.jar
/usr/java/jdk1.7.0_67-cloudera/jre/lib/security/local_policy.jar

3. Make sure, the permission and ownership of the files are retained.

4. Restart hadoop HDFS services.

5. Verify the log file, if there are the same logs appears as before: /var/log/hadoop-hdfs/*

6. Verify your kerberized HDFS is working properly.

[root@ip-172-31-0-157 ~]# kinit cloudera-scm/admin
Password for cloudera-scm/admin@EXAMPLE.COM:
[root@ip-172-31-0-157 ~]# hadoop fs -ls /
Found 1 items
drwxrwxrwt   - hdfs supergroup          0 2016-11-30 21:59 /tmp

7. If you wish to increase the verbosity of the output, you can always export the environment e.g.

export HADOOP_OPTS="-Dsun.security.krb5.debug=true"

8. If you wish to renew the token ticket

kinit -R

9. The krb client configuration /etc/krb5.conf is also important to specific the type e.g. otherwise, you will have this type of errors.

[root@ip-172-31-11-158 197-hdfs-NAMENODE]# cat /etc/krb5.conf
[libdefaults]
default_realm = EXAMPLE.COM
dns_lookup_kdc = false
dns_lookup_realm = false
ticket_lifetime = 86400
renew_lifetime = 604800
forwardable = true
default_tgs_enctypes = rc4-hmac
default_tkt_enctypes = rc4-hmac
permitted_enctypes = rc4-hmac
udp_preference_limit = 1
kdc_timeout = 3000
[realms]
EXAMPLE.COM = {
kdc = ip-172-31-25-156.ap-southeast-1.compute.internal
admin_server = ip-172-31-25-156.ap-southeast-1.compute.internal
}

[root@ip-172-31-25-156 ~]# hadoop fs -ls /
Java config name: null Native config name: /etc/krb5.conf
Loaded from native config
>>>KinitOptions cache name is /tmp/krb5cc_0
>>>DEBUG  client principal is hiuy@EXAMPLE.COM
>>>DEBUG server principal is krbtgt/EXAMPLE.COM@EXAMPLE.COM
>>>DEBUG key type: 18
>>>DEBUG auth time: Fri Dec 02 03:42:32 EST 2016
>>>DEBUG start time: Fri Dec 02 03:42:32 EST 2016
>>>DEBUG end time: Sat Dec 03 03:42:32 EST 2016
>>>DEBUG renew_till time: Fri Dec 02 03:42:32 EST 2016
>>> CCacheInputStream: readFlags()  FORWARDABLE; RENEWABLE; INITIAL;
>>>DEBUG  client principal is hiuy@EXAMPLE.COM
>>>DEBUG server principal is X-CACHECONF:/krb5_ccache_conf_data/fast_avail/krbtgt/EXAMPLE.COM@EXAMPLE.COM
>>>DEBUG key type: 0
>>>DEBUG auth time: Wed Dec 31 19:00:00 EST 1969
>>>DEBUG start time: null
>>>DEBUG end time: Wed Dec 31 19:00:00 EST 1969
>>>DEBUG renew_till time: null
>>> CCacheInputStream: readFlags()
>>> unsupported key type found the default TGT: 18
16/12/02 04:23:11 WARN security.UserGroupInformation: PriviledgedActionException as:root (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
16/12/02 04:23:11 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
16/12/02 04:23:11 WARN security.UserGroupInformation: PriviledgedActionException as:root (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]