HDFS delegation token cache

The problem

Error text is the following:

HDFS_DELEGATION_TOKEN can't be found in cache

The problem can appear in Hadoop’s NodeManager logs.

Usually it means that NodeManager is trying to use an expired / not renewed HDFS delegation token.

For example, you can face this error while app log aggregation process. The timeline is:

• Your application pass HDFS delegation token to the NodeManager through the ContainerLaunchContext class, because NodeManager needs to localize container resources.
• NodeManager uses the same HDFS delegation token to aggregate the logs. It transfers all app log files from nodes to HDFS.
• You didn’t start containers on some nodes for a long time, so token on corresponding NodeManagers is going to expire.
• When you kill your application, the log aggregation process was triggered, but some NodeManagers are going to give you HDFS_DELEGATION_TOKEN error because they don’t have a valid token.

Solution

To solve this error, your application must renew that token itself, which it previously pass to NodeManager through the ContainerLaunchContext.

Important notice! The token is going to expire and can’t be renewed after about 7 days. You don’t need to do something anymore, because NodeManager is going to make its own token.

By the way, you need to ensure that you are using a Hadoop version >= 2.6.0. In older versions, there is a bug, because of which NodeManager is aren’t going to make its own token: https://issues.apache.org/jira/browse/YARN-2704.