1

I configured the HDFS Connector for Object storage as described here: https://docs.us-phoenix-1.oraclecloud.com/Content/API/SDKDocs/hdfsconnector.htm#troubleshooting

When I am running distcp with the following command:

hadoop distcp -libjars /usr/lib/oci/oci-hdfs-full-2.7.2.0.jar hdfs://<namenode>:8020/some/folder/in/hdfs oci://bucket@namespace

I am getting an error:

tools.DistCp: Exception encountered java.lang.IllegalArgumentException: Does not contain a valid host:port authority: bucket@namespace

Since it is working on a different cluster, I think it must be configuration problem but I can't find the reason.

Are there any suggestion on how to fix this?

Joe
  • 2,500
  • 1
  • 14
  • 12
Matthias
  • 11
  • 1
  • 2

1 Answers1

2

Out of curiosity, did you actually use exactly the text oci://bucket@namespace? bucket and namespace are supposed to be placeholders.

You can find both if you log into the OCI console (e.g. https://console.us-phoenix-1.oraclecloud.com), go to Storage -> Object Storage, and then click on a bucket name (or, if you don't have a bucket, create one).

On the bucket details page, you can find the bucket name (in big font), and the namespace name.

OCI Bucket Details Screenshot]

In this screenshot, for example, the bucket name is "MyTestBucket" and the namespace name is "mynamespace".

Therefore, the OCI URL should be: oci://MyTestBucket@mynamespace

hadoop distcp -libjars /usr/lib/oci/oci-hdfs-full-2.7.2.0.jar hdfs://<namenode>:8020/some/folder/in/hdfs oci://MyTestBucket@mynamespace

Please forgive me if I misunderstood your problem.

Mathias Ricken
  • 156
  • 3
  • Thanks for the answer. Obviously, bucket@namespace is not what I am really using on the system. The combination between bucket@namespace seems not to matter at all for the because I am getting the same result with an existing and non-existing connection. – Matthias May 31 '18 at 05:54
  • My apologies. Searching for the error message, I'm finding two references to problems with underscores. - https://community.hortonworks.com/questions/181776/zkgc-does-not-contain-a-valid-host.html - https://community.hortonworks.com/questions/96359/javalangillegalargumentexception-does-not-contain.html Does your namespace or bucket name by any chance contain an underscore? – Mathias Ricken Jun 11 '18 at 16:47
  • Do you have a call stack from the exception? – Mathias Ricken Jun 11 '18 at 20:51
  • No underscores in the namespace or bucketname. We can reproduce the issue quite nicely. It is related to a kerberized HDFS. We also opened an issue here on Github: https://github.com/oracle/oci-hdfs-connector/issues/4 – Matthias Jun 12 '18 at 18:23
  • Thank you, Matthias. Sumit on Github is trying to get in touch with you to test a fix. – Mathias Ricken Jun 13 '18 at 16:04