0

I`m getting this error when trying to load data from Data Fusion to Salesforce:

java.lang.RuntimeException: There was issue communicating with Salesforce
    at io.cdap.plugin.salesforce.plugin.sink.batch.SalesforceOutputFormat.getRecordWriter(SalesforceOutputFormat.java:53) ~[1599122485492-0/:na]
    at org.apache.spark.internal.io.HadoopMapReduceWriteConfigUtil.initWriter(SparkHadoopWriter.scala:350) ~[spark-core_2.11-2.3.4.jar:2.3.4]
    at org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:120) ~[spark-core_2.11-2.3.4.jar:2.3.4]
    at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:83) ~[spark-core_2.11-2.3.4.jar:2.3.4]
    at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:78) ~[spark-core_2.11-2.3.4.jar:2.3.4]
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) ~[spark-core_2.11-2.3.4.jar:2.3.4]
    at org.apache.spark.scheduler.Task.run(Task.scala:109) ~[spark-core_2.11-2.3.4.jar:2.3.4]
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345) ~[spark-core_2.11-2.3.4.jar:2.3.4]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_252]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_252]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_252]
Caused by: com.sforce.async.AsyncApiException: InvalidJob : Invalid job id: null
    at com.sforce.async.BulkConnection.parseAndThrowException(BulkConnection.java:182) ~[na:na]
    at com.sforce.async.BulkConnection.doHttpGet(BulkConnection.java:753) ~[na:na]
    at com.sforce.async.BulkConnection.getJobStatus(BulkConnection.java:769) ~[na:na]
    at com.sforce.async.BulkConnection.getJobStatus(BulkConnection.java:760) ~[na:na]
    at io.cdap.plugin.salesforce.plugin.sink.batch.SalesforceRecordWriter.<init>(SalesforceRecordWriter.java:69) ~[1599122485492-0/:na]
    at io.cdap.plugin.salesforce.plugin.sink.batch.SalesforceOutputFormat.getRecordWriter(SalesforceOutputFormat.java:51) ~[1599122485492-0/:na]
    ... 10 common frames omitted
2020-09-03 08:41:28,595 - WARN  [task-result-getter-0:o.a.s.ThrowableSerializationWrapper@192] - Task exception could not be deserialized
java.lang.ClassNotFoundException: Class not found in all delegated ClassLoaders: com.sforce.async.AsyncApiException
    at io.cdap.cdap.common.lang.CombineClassLoader.findClass(CombineClassLoader.java:96) ~[na:na]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[na:1.8.0_252]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[na:1.8.0_252]
    at io.cdap.cdap.common.lang.WeakReferenceDelegatorClassLoader.findClass(WeakReferenceDelegatorClassLoader.java:58) ~[na:na]

What does this error mean? I have assured the input fields match with the SObject definition.

Sophie Sepp
  • 1
  • "Invalid job id" sounds like the bulk api job failed. Skim through the bulk api concepts in https://trailhead.salesforce.com/en/content/learn/modules/api_basics/api_basics_bulk. Check in Setup -> Bulk... if the job was submitted OK, maybe there's a better error message. Has it managed to log in to SF OK? – eyescream Sep 03 '20 at 14:50
  • The job appears in the "Monitor Bulk Data Load Jobs" page, but the status is still "Open" and there is no error message shown. – Sophie Sepp Sep 04 '20 at 06:53
  • Then it might be that Java library is buggy? SF needs instruction that file upload is finished (because you might have decided to load multiple files with up to 10K record chunks, SF has no way of knowing) and then it can start processing. In that link I posted it's mentioned as "Close the Job". – eyescream Sep 04 '20 at 09:41
  • Thanks for the link. I`m not using workbench, though. I use a Data Fusion pipeline that loads, transforms the data and then uses the Salesforce sink to send it. – Sophie Sepp Sep 04 '20 at 10:32

1 Answers1

1

I looked at the stacktrace, and I think I might know what the problem is.

The property mapred.salesforce.job.id is undefined, and when the code executes, it goes to grab the value of this key, and as it is not defined, the job errors out. I think you need to set the mapred.salesforce.job.id flag as a runtime property. To do that in Data Fusion, do the following:

  1. In the Pipeline Studio, navigate to the detail page of the pipeline you would like to configure.
  2. Click on the drop-down next to the Run button.
  3. Set your desired cluster properties, prefixing all property names with system.profile.properties.. For our case, I think the name would be system.profile.properties.mapred:mapred.salesforce.job.id and the value is a number you'd want to use as your ID
TylerH
  • 20,799
  • 66
  • 75
  • 101
Omar I
  • 51
  • 1
  • 2
  • Thank you for your answer. Is there a specific format you have to use for the id or a fixed size? I tried with the following runtime argument and I`m still getting the same error. system.profile.properties.mapred:mapred.salesforce.job.id 123444556781234567 – Sophie Sepp Sep 21 '20 at 09:30
  • Can you pass in the argument as a string? Here's the line in the code that obtains the info: [`String jobId = conf.get(SalesforceSinkConstants.CONFIG_JOB_ID);`](https://github.com/data-integrations/salesforce/blob/f2dfcc850df96567fbdd5352d53bc6a54ba5ae96/src/main/java/io/cdap/plugin/salesforce/plugin/sink/batch/SalesforceOutputFormat.java#L97) – Omar I Sep 21 '20 at 12:25
  • I passed in the argument as a string, it still gives the same error mentioned above. Moreover, I could see in the logs that even before passing a job id in the runtime arguments, one got created automatically: "Started Salesforce job with jobId='7503O000004ORtCQAW'." – Sophie Sepp Sep 21 '20 at 15:07
  • Hey Sophie - I shared this issue internally and we found a bug in the Salesforce plugin. Thanks for pointing this out! Here is a tracking link to the bug: https://issues.cask.co/browse/PLUGIN-406 – Omar I Sep 23 '20 at 14:10
  • 1
    Hey @SophieSepp - a fix has been deployed for the plugin on the Hub. It is now v.1.2.1! – Omar I Sep 28 '20 at 13:54