1

I am running a data collector (3.10.0), connected to Control Hub (3.8?). All on-premises. While trying to run a pipeline, I get the following error. The pipeline takes a local file and uploads it to hdfs.

"Pipeline status: RUNNING_ERROR: HADOOPFS_13 - Error while writing to HDFS: java.io.IOException: org,apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)"

I have followed the online documentation to enable Kerberos via CDH 6.2 ("Enabling Kerberos with Cloudera Manager" - https://streamsets.com/documentation/...).

I can knit as the sdc user using the sdc keytab (from root) and can list HDFS, touchz a file etc. Running the pipeline results in the error above. Kerberos authentication is checked in the pipeline config. I am also using a custom Configuration Files Directory value, as it had issues finding the hive-site.xml (I have links in that directory).

Any suggestions?

metadaddy
  • 4,234
  • 1
  • 22
  • 46
Chompers
  • 11
  • 2
  • Your link doesn't lead to any documentation about enabling Kerberos, which makes this look a bit like spam. Please [edit] your question to show the actual tutorial that you used, though it would be far more preferable to simply list the steps that you took here in your question. Links can get outdated quickly, which leaves everyone guessing as to what you actually did or didn't do. The error message is quite generic as it only means you have no ticket. Does the user have the right to get a ticket in the KDC for example? – Secespitus Sep 16 '19 at 13:11
  • Also, please copy error messages when possible. You have a typo in yours Failed to fin**a** any Kerberos tgt". Such typos could on more difficult places make it hard for others to search for your error messages. – Secespitus Sep 16 '19 at 13:11
  • The error message without the typo: “Pipeline status: RUNNING_ERROR: HADOOPFS_13 - Error while writing to HDFS: java.io.IOException: org,apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)" – Chompers Sep 16 '19 at 14:19
  • Apologies, that link seems to have disappeared (“spam?”) but anyway, here's the steps to enable Kerberos: 1. In Cloudera Manager, select the StreamSets service and then click Configuration. 2. Select Enable Kerberos Client. In the Cloudera Manager home page, click Administration > Security. 3. Click Kerberos Credentials. 4. Click Generate Missing Credentials. 5. Restart Data Collector. 6. Configure the stage. The sdc user is a system user, and isn’t in KDC. CDH manages the kerberos ticket. – Chompers Sep 16 '19 at 14:23
  • Sites on the network quite regularly receive spam and a link whose target is not what is described in the text surrounding it and only leading to the landing page of a product as indicated by the "READY TO LEARN MORE? \[SCHEDULE DEMO\] \[FREE TRIAL\]" at the end looks like product placement. The markdown for the link also indicated that you didn't write the complete link. Something like [this link](https://streamsets.com/documentation/datacollector/latest/help/datacollector/UserGuide/Configuration/DCConfig.html#task_azc_wgy_w5) seems to be more what you wanted. Please [edit] your question. – Secespitus Sep 16 '19 at 14:52

0 Answers0