Gitlab CI PIP AWSCLI Avoid Install Again

Question

I'm working with GitLab CE, Runners Docker and AWS ECS for Deployment. We created a script that do what we need but we separate the stages and jobs for Development.

The problem is that we need to run this scripts to connect to AWS and let us to register containers and deploy our resources:

services:
  - docker:dind
before_script:
  - apk add build-base python3-dev python3 libffi-dev libressl-dev bash git gettext curl
  - apk add py3-pip
  - pip install six awscli awsebcli
  - $(aws ecr get-login --no-include-email --region "${AWS_DEFAULT_REGION}")
  - IMAGE_TAG="$(echo $CI_COMMIT_SHA | head -c 8)"

The problem is that the script runs every time with the Job, this will not be a problem is the scripts avoid the reinstallation of the dependencies:

So we need to know if we can avoid this behavior in order to run the script only once becase every job take so long on finish for this.

Our complete script .gitlab-ci.yml:

image: docker:latest

stages:
  - build
  - tag
  - push
  - ecs
  - deploy

variables:
  REPOSITORY_URL: OUR_REPO
  REGION: OUR_REGION
  TASK_DEFINITION_NAME: task
  CLUSTER_NAME: default
  SERVICE_NAME: service

services:
  - docker:dind
before_script:
  - apk add build-base python3-dev python3 libffi-dev libressl-dev bash git gettext curl
  - apk add py3-pip
  - pip install six awscli awsebcli
  - $(aws ecr get-login --no-include-email --region "${AWS_DEFAULT_REGION}")
  - IMAGE_TAG="$(echo $CI_COMMIT_SHA | head -c 8)"

build:
  stage: build
  tags:
    - deployment
  script:
      - docker build -t $REPOSITORY_URL:latest .
  only:
      - refactor/ca

tag_image:
  stage: tag
  tags:
    - deployment
  script:
      - docker tag $REPOSITORY_URL:latest $REPOSITORY_URL:$IMAGE_TAG
  only:
      - refactor/ca

push_image:
  stage: push
  tags:
    - deployment
  script:
      - docker push $REPOSITORY_URL:$IMAGE_TAG
  only:
      - refactor/ca

task_definition:
  stage: ecs
  tags:
    - deployment
  script:
      - TASK_DEFINITION=$(aws ecs describe-task-definition --task-definition "$TASK_DEFINITION_NAME" --region "${REGION}")
      - NEW_CONTAINER_DEFINTIION=$(echo $TASK_DEFINITION | python3 $CI_PROJECT_DIR/update_task_definition_image.py $REPOSITORY_URL:$IMAGE_TAG)
      - echo "Registering new container definition..."
      - aws ecs register-task-definition --region "${REGION}" --family "${TASK_DEFINITION_NAME}" --container-definitions "${NEW_CONTAINER_DEFINTIION}"
  only:
      - refactor/ca

register_definition:
  stage: ecs
  tags:
    - deployment
  script:
      - aws ecs register-task-definition --region "${REGION}" --family "${TASK_DEFINITION_NAME}" --container-definitions "${NEW_CONTAINER_DEFINTIION}"
  only:
      - refactor/ca

deployment:
  stage: deploy
  tags:
    - deployment
  script:
      - aws ecs update-service --region "${REGION}" --cluster "${CLUSTER_NAME}" --service "${SERVICE_NAME}"  --task-definition "${TASK_DEFINITION_NAME}"
  only:
      - refactor/ca

you mean that `before_script` runs with every stage build, tag, push, etc.. and you want to avoid this behavior? — Miguel Trejo, Oct 17 '20 at 01:47
Hi, I mean that if we separate the dock tag, push etc per stage the before_script always is running, it will be no problem if the dependencies of python and pip didn't install but it does. So we wan't to install only once pip and run aws login once. — Ulises, Oct 17 '20 at 01:51
I've found a similar question [here](https://forum.gitlab.com/t/before-script-repeated-for-each-job-should-it-not-be-execute-once-only/20786). For the docker stages it woud be easy to rewrite the before_script, for example, `push_image` and `tag_image` just need `IMAGE_TAG` so you could set for this stages before_script: - IMAGE_TAG="$(echo $CI_COMMIT_SHA | head -c 8)" and for `build` stage something like before_script: - echo "Build stage" — Miguel Trejo, Oct 17 '20 at 02:09
is there a problem if `task_definition:` and `deployment` go on the same job? — Miguel Trejo, Oct 17 '20 at 02:20
Not problem at all, just ugly haha all the lines together will work fine but we just wanted to separate by stage. — Ulises, Oct 17 '20 at 02:34

score 0 · Answer 1 · answered Oct 17 '20 at 02:55

One workaround will be to define before_script only for the stage that runs aws ecs register-task-definition and aws ecs update-service.

Additionally, as push and ecs stages access the IMAGE_TAG variable it is convenient to store it on an build.env artifact so that this stages can access it by specifiying a dependency on the image_tag stage.

image: docker:latest
    
stages:
  - build
  - tag
  - push
  - ecs
    
variables:
  REPOSITORY_URL: OUR_REPO
  REGION: OUR_REGION
  TASK_DEFINITION_NAME: task
  CLUSTER_NAME: default
  SERVICE_NAME: service
    
services:
  - docker:dind
    
build:
  stage: build
  tags:
    - deployment
  script:
    - docker build -t $REPOSITORY_URL:latest .
  only:
    - refactor/ca
    
tag_image:
  stage: tag
  tags:
    - deployment
  script:
    - IMAGE_TAG="$(echo $CI_COMMIT_SHA | head -c 8)"
    - echo "IMAGE_TAG=$IMAGE_TAG" >> build.env 
    - docker tag $REPOSITORY_URL:latest $REPOSITORY_URL:$IMAGE_TAG
  only:
    - refactor/ca
  artifacts:
    reports:
      dotenv: build.env 
    
push_image:
  stage: push
  tags:
    - deployment
  script:
    - docker push $REPOSITORY_URL:$IMAGE_TAG
  only:
    - refactor/ca
  dependencies:
    - tag_image

task_definition:
  stage: ecs
  tags:
    - deployment
  before_script:
    - apk add build-base python3-dev python3 libffi-dev libressl-dev bash git gettext curl
    - apk add py3-pip
    - pip install six awscli awsebcli
    - $(aws ecr get-login --no-include-email --region "${AWS_DEFAULT_REGION}")
  script:
    - TASK_DEFINITION=$(aws ecs describe-task-definition --task-definition "$TASK_DEFINITION_NAME" --region "${REGION}")
    - NEW_CONTAINER_DEFINTIION=$(echo $TASK_DEFINITION | python3 $CI_PROJECT_DIR/update_task_definition_image.py $REPOSITORY_URL:$IMAGE_TAG)
    - echo "Registering new container definition..."
    - aws ecs register-task-definition --region "${REGION}" --family "${TASK_DEFINITION_NAME}" --container-definitions "${NEW_CONTAINER_DEFINTIION}"
    - aws ecs update-service --region "${REGION}" --cluster "${CLUSTER_NAME}" --service "${SERVICE_NAME}"  --task-definition "${TASK_DEFINITION_NAME}"
  only:
    - refactor/ca
  dependencies:
    - tag_image

So, I just use the aws/aws-cli image and install docker inside with dind service :D — Ulises, Nov 03 '20 at 22:13

Gitlab CI PIP AWSCLI Avoid Install Again

1 Answers1