2

Alright team...Dataprep running into BigQuery. I cannot for the life of me find out why I have the ZONE_RESOURCE_POOL_EXHAUSTED issue for the past 5 hours. The night before, everything was going great, but today, I am having some serious issues.

Can anyone give any insight into how to change the resource pool for Dataflow jobs with regard to Dataprep? I can't even get a basic column transform to push through.

Looking forward to anyone helping me with this because honestly, this issue one of those "just change this and maybe that will fix it and if not, maybe a few weeks and it'll work".

Here is the issue in screenshot: https://i.stack.imgur.com/Qi4Dg.png

UPDATE:

I believe some of my issue may deal with GCP Compute incident 18012 espcially since it's a us-central based issue for creation of instances.

cloudstrife
  • 21
  • 3
  • I am having the same issue. Error in Dataprep: Insufficient workers for Dataflow Dataflow was unable to deploy suffficient workers to run your job. Before re-running, please verify your quota or contact your project administrator. I ran a successful job on: Nov 9, 2018, 11:51:44 AM Same flow, same recipe failed on: Nov 9, 2018, 6:00:28 PM – abridges Nov 11 '18 at 18:03
  • Dataflow log error message: Startup of the worker pool in zone us-central1-b failed to bring up any of the desired 1 workers. ZONE_RESOURCE_POOL_EXHAUSTED: The zone 'projects/fake-project/zones/us-central1-b' does not have enough resources available to fulfill the request. Try a different zone, or try again later. – abridges Nov 11 '18 at 18:18
  • I've been having the exact same problem. Last pipeline to pass was on Thursday night, since then everything I run fails for that reason. I've ran the underlying dataflow job and it passed in another region, so that could be a clue. – IdoL Nov 11 '18 at 19:47
  • @IdoL can you give any insight into steps to do that? Having difficulty finding steps to change the zone... – cloudstrife Nov 12 '18 at 04:35
  • 2
    Use a --region or --zone flag when you start the job. https://cloud.google.com/dataflow/pipelines/specifying-exec-params – FridayPush Nov 12 '18 at 05:24
  • 2
    You also need to find the template to run on dataflow - see instructions here: https://cloud.google.com/dataprep/docs/html/Run-Job-on-Cloud-Dataflow_99745844 – IdoL Nov 12 '18 at 10:54
  • My issue seems to be resolved. Just ran a successful job. Same flow and recipe that was failing on November 9th. Start time Nov 12, 2018, 5:13:34 PM EST. – abridges Nov 12 '18 at 22:46
  • 3
    Seems the outage has been fixed. Thanks to everyone involved! – cloudstrife Nov 13 '18 at 06:52

1 Answers1

0

The incident you mentioned was actually resolved on November 5th and was only affecting the us-central1-a zone. Seeing that your question was posted on November 10th and other users in the comments got the error in the us-central1-b zone, the error is not related to the incident you linked.

As the error message suggests, this is a resource availability issue. These scenarios are rare and are usually resolved quickly. If this ever happens in the future, using Compute Engine instances in other regions/zones will solve the issue. To do so using Dataprep, as mentioned in the comment, after the job is launched from Dataprep, you can re-run the job from Dataflow while specifying the region/zone you would like to run the job in.

Atybzz
  • 327
  • 3
  • 14