Apache Beam Error : Offset X has not been allocated yet

Question

I am seeing below error in my dataflow pipeline through GCP cloud logging.

Error message from worker: java.io.IOException: Failed to advance reader of source: name: "projects/test-env/locations/europe-west9/sessions/kadbckjdabcbdacbakdbckandjlcnaldjncljad/streams/HBKJBNJNLJNKBjhbvjhbjhBJVJVJH"

    org.apache.beam.runners.dataflow.worker.WorkerCustomSources$BoundedReaderIterator.advance(WorkerCustomSources.java:625)
    org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation$SynchronizedReaderIterator.advance(ReadOperation.java:425)
    org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:211)
    org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:169)
    org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:83)
    org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:420)
    org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:389)
    org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:314)
    org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:140)
    org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:120)
    org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:107)
    java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    java.base/java.lang.Thread.run(Thread.java:833)
Caused by: com.google.api.gax.rpc.FailedPreconditionException: io.grpc.StatusRuntimeException: FAILED_PRECONDITION: there was an error operating on 'projects/test-env/locations/europe-west9/sessions/kadbckjdabcbdacbakdbckandjlcnaldjncljad/streams/HBKJBNJNLJNKBjhbvjhbjhBJVJVJH': offset 1366 has not been allocated yet
    com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:57)
    com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72)
    com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60)
    com.google.api.gax.grpc.ExceptionResponseObserver.onErrorImpl(ExceptionResponseObserver.java:82)
    com.google.api.gax.rpc.StateCheckingResponseObserver.onError(StateCheckingResponseObserver.java:86)
    com.google.api.gax.grpc.GrpcDirectStreamController$ResponseObserverAdapter.onClose(GrpcDirectStreamController.java:149)
    io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
    io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
    io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
    com.google.api.gax.grpc.ChannelPool$ReleasingClientCall$1.onClose(ChannelPool.java:455)
    io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
    io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
    io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
    io.grpc.census.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:802)
    io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
    io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
    io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
    io.grpc.census.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:428)
    io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)
    io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
    io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)
    io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)
    io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
    ... 3 more
    Suppressed: java.lang.RuntimeException: Asynchronous task failed
        at com.google.api.gax.rpc.ServerStreamIterator.hasNext(ServerStreamIterator.java:105)
        at org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageStreamSource$BigQueryStorageStreamReader.readNextRecord(BigQueryStorageStreamSource.java:211)
        at org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageStreamSource$BigQueryStorageStreamReader.advance(BigQueryStorageStreamSource.java:206)
        at org.apache.beam.runners.dataflow.worker.WorkerCustomSources$BoundedReaderIterator.advance(WorkerCustomSources.java:622)
        at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation$SynchronizedReaderIterator.advance(ReadOperation.java:425)
        at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:211)
        at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:169)
        at org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:83)
        at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:420)
        at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:389)
        at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:314)
        at org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:140)
        at org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:120)
        at org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:107)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        ... 3 more
Caused by: io.grpc.StatusRuntimeException: FAILED_PRECONDITION: there was an error operating on 'projects/test-env/locations/europe-west9/sessions/kadbckjdabcbdacbakdbckandjlcnaldjncljad/streams/HBKJBNJNLJNKBjhbvjhbjhBJVJVJH': offset 1366 has not been allocated yet
    io.grpc.Status.asRuntimeException(Status.java:535)
    ... 22 more

Although pipeline succeed and does the job as required but wondering if this error has to do something which can't be seen easily like some missing records.

Or Is this just a transient error that we can ignore ?

My Job details

Running dataflow job in Java 17
SDK version 2.39.0

Can you try [latest version 2.40.0](https://beam.apache.org/get-started/downloads/) I hope it will help you ? — Arpita Shrivastava, Jun 19 '23 at 13:49
With latest version I have this issue https://stackoverflow.com/questions/76491501/apache-beam-version-upgrade-fails-the-etl-pipeline — SRJ, Jun 19 '23 at 14:22

score 0 · Answer 1 · answered Jun 19 '23 at 14:22

Are you using a batch job and using BigQueryIO Storage Write API?

You would probably be better off using method FILE_LOADS, as it can get a similar performance, and there's no cost to load data on BigQuery, see https://cloud.google.com/bigquery/pricing.

If you really want to use Storage Write API, I would suggest using the latest Apache Beam (2.48.0), as there were multiple fixes related to performance and consistency.

See https://cloud.google.com/dataflow/docs/support/sdk-version-support-status:

Warning: In versions 2.35.0 to 2.46.0, when using the STORAGE_WRITE_API method with a BigQuery sink, your batch pipelines might have issues with data consistency due to the handling of unrelated transient errors. For more information, see the Apache Beam GitHub issue #26521.

Apache Beam Error : Offset X has not been allocated yet

My Job details

1 Answers1