0

i found a lot of strange error logs related to GCP Dataflow job inside "logName": "...shuffler" and "...shuffler-startup"

The dataflow job works without any problem and there is no error inside "logName": "...worker"

Here are the full error lines with INFO level in "logName": "...shuffler-startup"

E0627 19:09:03.584041 487 curl_http_transport.cc:411] Call to url=<SUBSCRIPTION_NAME>:pull failed with curl status UNKNOWN: . curl=16

E0624 12:56:32.971646 195 curl_http_transport.cc:411] Call to url=<JOB_ID>/workItems:reportStatus failed with curl status UNKNOWN: . curl=16

E0613 09:23:08.731521 205 curl_http_transport.cc:410] Call to url=<JOB_ID>/workItems:reportStatus failed with curl status UNKNOWN: . curl=65: necessary data rewind wasn't possible

E0616 17:11:53.604693 481 curl_http_transport.cc:410] Call to url=<SUBSCRIPTION_NAME>:pull failed with curl status DEADLINE_EXCEEDED: Timed out. curl=28: Operation timed out after 120000 milliseconds with 0 bytes received

E0529 02:48:14.860606 55 curl_http_transport.cc:410] Call to url=<SUBSCRIPTION_NAME>:pull failed with curl status UNKNOWN: . curl=56: BoringSSL SSL_read: SSL_ERROR_SYSCALL, errno 104

E0627 19:51:37.833993 38041 curl_http_transport.cc:411] Call to url=<SUBSCRIPTION_NAME>:acknowledge failed with curl status UNKNOWN: . curl=16

and

here are the examples of error lines with ERROR level in "logName": "...shuffler"

Call to url=<SUBSCRIPTION_NAME>:pull failed with curl status UNKNOWN: . curl=56: BoringSSL SSL_read: Unknown error 104, errno 104

Call to url=<SUBSCRIPTION_NAME>:pull failed with curl status UNKNOWN: . curl=65: necessary data rewind wasn't possible

Call to url=<SUBSCRIPTION_NAME>:pull failed with curl status UNKNOWN: . curl=16

Saeed Mohtasham
  • 1,693
  • 16
  • 27
  • What does this Dataflow job aim to perform? Do these errors permanently occur for each job execution? Is there a longer time in processing, or other measurable impact? – Nick_Kh Jun 29 '20 at 10:38

1 Answers1

1

Apache Beam, and most distributed data processing libraries and engines may sometimes trigger errors. This is because these systems are complex, and involve hundreds of machines, and thousands of threads (or more!).

If your pipeline is succeeding, then you should not worry. These engines and libraries are designed to withstand and recover from these sorts of errors.

This particular error was a temporary issue with the Dataflow service.

If this is caused your pipeline to fail, please try running it again now, as the issue should have been mitigated by now.

Pablo
  • 10,425
  • 1
  • 44
  • 67