6

I am sporadically getting the following errors:

W Refusing to split at '\x00\x00\x00\x15\xbc\x19)b\x00\x01': proposed split position is out of range ['\x00\x00\x00\x15\x00\xff\x00\xff\x00\xff\x00\xff\x00\x01', '\x00\x00\x00\x15\xbc\x19)b\x00\x01'). Position of last group processed was '\x00\x00\x00\x15\xbc\x19)a\x00\x01'.

When it happens, the error is logged every so often and the job never seems to end. Although it seems that it did actually complete the job otherwise.

In the last instance I am using 10 workers and have auto scaling disabled. I am using the Python implementation of Apache Beam.

de1
  • 2,986
  • 1
  • 15
  • 32

1 Answers1

8

This is not an error, it's part of normal operation of a pipeline. We should probably reduce its logging level to INFO and rephrase it, because it very frequently confuses people.

This message (rather obscurely) signals that Dataflow is trying to apply dynamic rebalancing, but there's no work that can be further subdivided.

I.e. your job is stuck doing something non-parallelizable on a small number of workers, while other workers are staying idle. To investigate this further, one would need to look at the code of your job and the Dataflow job id.

jkff
  • 17,623
  • 5
  • 53
  • 85
  • Thank you for that, is there any way to see what exactly is stuck? (e.g. something like a thread dump) The pipeline also runs external programs. – de1 Jan 30 '18 at 09:12
  • Not in Beam Python currently, as far as I know. Beam Java has a feature for detecting stuckness. In Python you can try just adding a bunch of logging and looking at it in Stackdriver. – jkff Jan 30 '18 at 17:36
  • Okay, I was afraid you were going to say that. Thank you. – de1 Jan 30 '18 at 19:51