Spark permission denied error

November 8, 2022

What's inside this article ⌄

Spark accesscontrolexception permission
Spark HDFS permission denied
Fix Spark access denied error
Spark permission error solution

You may face this error:

AccessControlException: Permission denied: user=root, access=WRITE, inode="/user"

Full exception text is:

org.apache.spark.SparkContext  - Error initializing SparkContext.
org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x
	at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:279)

First, you need to check that user from which your Spark working on, has home folder under /user like that: /user/John_Doe.

If you already have that folder, try to grant access to /user dir. If you are unable to grant access to that directory or it undesirable in your environment, you can set another directory for Spark to work with:

Add this to your run command: --conf spark.yarn.stagingDir=hdfs://yourcluster/another/hdfs/path
After creation of JavaSparkContext, set: javaSparkContext.setCheckpointDir("/tmp") or path like that.