0

I'm getting a StackOverflowError on my Beam workers due to running out the thread stack, and because it's deep within the running of a SqlTransform it's not straightforward to reduce the number of calls being made.

Is it possible to change the JVM thread stack size for my workers, either through Google Cloud Dataflow or Beam's own pipeline options?

wrp
  • 99
  • 6
  • As chamikara mentioned in his answer, sounds difficult, maybe you can try using a [high-memory machine type](https://cloud.google.com/compute/docs/machine-types#memory-optimized_machine_type_family), but I'm not pretty sure that it works since Dataflow is managed service is complicated to configured that kind of parameters. – Enrique Zetina Dec 04 '19 at 22:50
  • Is it possible to try and reproduce using the DirectRunner? – Reza Rokni Dec 05 '19 at 03:23

1 Answers1

0

I don't think there's an easy way to do this.

  • If this is an issue of stack trace being purged by Cloud Logging, may be it might be possible to catch the exception yourself and inspect that instead of just logging it.
  • If this is an issue of default stack trace depth set by the JVM not being enough, I don't think there's a way to update this for Dataflow today unfortunately.
chamikara
  • 1,896
  • 1
  • 9
  • 6