3

I have a Spark image running in a Docker container. I want to access the results saved by Spark in HDFS using WebHDFS from the host machine outside the container. For this I am using the OPEN API which has a redirect before serving the file contents. I am forwarding all ports using -P flag while running the docker image.

I am able to execute other operations like Listing a Directory using the exposed high port on the host machine. But for reading a file the REST call fails as there is a HTTP Redirect.

Any ideas how I can solve this?

Nitin
  • 7,187
  • 6
  • 31
  • 36
  • can you elaborate, or offer a reproducer? – user2915097 Jul 01 '15 at 05:13
  • Adding to the post above: When we run Spark in Docker, we can access the Hadoop UI (generally on port 50070) from host machine by exposing that port to some high port on host machine. This is true for any REST APIs by WebHDFS. For reading a file using HDFS Rest end points, I am using OPEN API from WebHDFS. There is a HTTP Redirect when using OPEN API. How can we force the HTTP redirect to use the new high port after redirect? – Nitin Jul 01 '15 at 17:18

0 Answers0