0

I am trying to deploy ETL pipeline on AWS, Pipeline architecture consist of step function to manage the whole pipeline, it consists of 10 parallel (independent) Fargate (pipeline) invoked using map in step function and a Lambda (Error Notifier),

I have used python 3.7 as base image to build my docker image, pipeline is successfully deployed, but it sometimes randomly fails with container exit code 139.

After further research, I found out that it is sigsegv error (same as error code 11 of linux kernel), related invalid memory access

I am not sure how to rectify this error

KISHAN DHRANGADHARIYA
  • 80
  • 7
  • What's running inside your containers? –  Aug 02 '21 at 14:14
  • I am running a ETL application in python: requirement file: google-cloud-bigquery==2.6.2 google-cloud-bigquery-datatransfer==3.0.0 Google-cloud-pubsub==2.6.1 google-cloud-storage==1.35.0 google-oauth2-tool==0.0.3 google-cloud-logging==2.5.0 traceback2==1.4.0 openpyxl==3.0.6 numpy==1.19.4 pandas==1.2.0 fastparquet==0.5.0 pyarrow==2.0.0 python-dateutil==2.8.1 requests==2.25.1 bigquery_schema_generator==1.4 google-cloud-secret-manager==2.6.0 boto3==1.18.7 – KISHAN DHRANGADHARIYA Aug 02 '21 at 15:15
  • @Hcaertnit this service would fetch data from s3, process it, store it to gcs and ingest it to BQ using lambda is creating bottle neck in pipeline, because of time limit imposed on it. To rectify that we are deploying pipeline in fargate – KISHAN DHRANGADHARIYA Aug 03 '21 at 04:17

1 Answers1

-2

[UPDATE]

Recently found out that problem was, image was built x86_84 mac machine, and was deployed on x86_84 ubuntu system, because of that we got SIGTERM error (error code:7 for linux, and error code: 139 for docker), just redeployed the images AWS codebuild, and everything worked perfectly fine from then on.

KISHAN DHRANGADHARIYA
  • 80
  • 7
  • Which dependencies? – Luke May 30 '22 at 14:29
  • removed pandas library – KISHAN DHRANGADHARIYA Jun 01 '22 at 03:37
  • Thanks. What is is about that library that made it incompatible with ECS fargate? Do you know? – Luke Jun 01 '22 at 14:01
  • Recently found out that problem was, image was built intel mac machine, and was deployed on x86_84 ubuntu system, because we got SIGTERM error (error code:7 for linux, and error code: 139 for docker), just redeployed the images AWS codebuild, and everything worked perfectly fine. – KISHAN DHRANGADHARIYA Jun 26 '22 at 08:40
  • Thank you for that. I have recently upgraded from the .net 3.1 to .net 6 docker images and this has stopped any docker containers exiting for no reason. – Luke Jun 28 '22 at 09:26