0

I'm working on simple beam dataflow in JAVA on google cloud platform. I've tested locally and the pipeline is running well.

When i deploy on dataflow, i got this looping error :

{
  insertId: "10mkpdlb7i"
  labels: {4}
  logName: "projects/myproject/logs/dataflow.googleapis.com%2Fjob-message"
  receiveTimestamp: "2022-11-14T09:09:08.216192037Z"
  resource: {
    labels: {
      job_id: "2022-11-14_01_20_11-8382597753945077942"
      job_name: "production-validation-pipeline"
      project_id: "3456543"
      region: "europe-west1"
      step_id: ""
    }
    type: "dataflow_step"
  }
  severity: "ERROR"
  textPayload: "Workflow failed."
  timestamp: "2022-11-14T09:09:07.793137682Z"
}

My dataflow :

  • read from kafka
  • filter message -> keep some of them based on value in the message
  • convert to pubsub message
  • push to pubsub

Thanks in advance for your help

shinjie
  • 1
  • 1
  • 1
    Can you also check the job logs, workers logs and diagnostics tab from `Dataflow` GUI if there are more details on the current error ? – Mazlum Tosun Nov 14 '22 at 15:57
  • I found this log as Internal Issue (cb8641f3e58ae6e4): 63963027:24112 : { insertId: "1aw44twb80" labels: {4} logName: "projects/myproject/logs/dataflow.googleapis.com%2Fjob-message" receiveTimestamp: "2022-11-14T14:43:09.109573279Z" resource: { labels: { job_id: "2022-11-14_06_36_49-2382052343911558560" job_name: "production-validation-pipeline" project_id: "855947445485" region: "europe-west1" step_id: "" } type: "dataflow_step" } severity: "WARNING" textPayload: "Internal Issue (cb8641f3e58ae6e4): 63963027:24112" timestamp: "2022-11-14T14:43:08.459520930Z" } – shinjie Nov 14 '22 at 17:35
  • Did you got any luck with this issue - I am facing the same issue "Internal Issue (c27629d85683fe1b): 63963027:24112" – Siddharth Mar 23 '23 at 15:13

0 Answers0