0

I am working with a complex Google Cloud Dataprep process. In order to make it work properly, I designed different modules and then linked them (with referenced datasets).

If I execute those modules separately they work properly, but when executing them together it gives an error and never starts working.

Creating the Dataflow job failed unexpectedly.

Do you know if there is a limitation in the number of recipes in Dataprep or the number of jobs/transformations in Dataflow?

JSDBroughton
  • 3,966
  • 4
  • 32
  • 52
VCast
  • 1
  • 1
  • What error does it give? – Pablo Feb 18 '19 at 19:21
  • the error it gives is: "Creating the Dataflow job failed unexpectedly" – VCast Feb 19 '19 at 15:41
  • It's a vague error that I have certainly received when trying to run multiple data prep jobs concurrently - I think the stack driver trace said "maximum workers from pool exceeded" or something like that. In that instance, I re-ran whichever jobs failed. Where you have your 'modules' daisy-chained in separate Flows but the results are referenced by subsequent jobs I was under the impression that it compiled it into a single Dataflow job and therefore wouldn't encounter the error for the same reason. Check the Console for the specific error, it gives more detail than Dataprep itself. – JSDBroughton Mar 06 '19 at 11:12

0 Answers0