Access remote HDFS Java

Remote HDFS Configuration

Suppose we have our Java app running on Hadoop clusterA, and we want to access remote HDFS based on Hadoop clusterB. Let’s see how we can do it:

Configuration conf = new Configuration();
conf.addResource(new Path("/path/to/core-site-clusterB.xml"));
conf.addResource(new Path("/path/to/hdfs-site-clusterB.xml"));
FileSystem fileSystem = FileSystem.get(conf);

You need to go to clusterB and gather core-site.xml and hdfs-site.xml from there (default location for Cloudera is /etc/hadoop/conf) and put near your app running in clusterA.

Pay attention to these points:

• we are specifying both core-site.xml and hdfs-site.xml, not just one of them
• we are sending Path object to addResource() method, not just ordinary String!

Troubleshooting

java.lang.IllegalArgumentException: Wrong FS: hdfs://clusterA/your/path, expected: hdfs://clusterB

See “pay attention” section above.

java.lang.IllegalArgumentException: java.net.UnknownHostException: clusterA

See “pay attention” section above.

java.lang.IllegalArgumentException: Wrong FS: hdfs://clustername:8030/your/path, expected: hdfs://clustername

Remove port from a path you want to access: use hdfs://clustername/your/path instead of hdfs://clustername:8030/your/path.