0

We are having some services running in docker containers on an EC2 machine (Linux based). These containers are deployed using CircleCI.

Following is the circle CI config.yml

version: 2
jobs:
  deploy:
    docker:
      - image: docker:18.06.0-ce-git
    working_directory: ~/app
    steps:
      - checkout
      - setup_remote_docker
      - run:
          name: Install awscli
          command: |
            apk add --no-cache py-pip
            pip install awscli==1.15.72
      - run:
          name: Build Docker image
          command: |
            docker build -t resource-etl .
      - run:
          name: Push image to ECR
          command: |
            eval $(aws ecr get-login --no-include-email --region $AWS_REGION)
            docker tag ${CIRCLE_PROJECT_REPONAME}:latest ${ECR_REPOSITORY_BASE_URI}/${CIRCLE_PROJECT_REPONAME}:latest
            docker push ${ECR_REPOSITORY_BASE_URI}/${CIRCLE_PROJECT_REPONAME}:latest
      - run:
          name: Deploy container to EC2
          command: |
            sh .circleci/deploy.sh
workflows:
  version: 2
  deploy:
    jobs:
      - deploy
          # filters:
          #   branches:
          #     only: master

Following is the deploy.sh script that configures the log path. The logs need to be stored in a folder /vol/log/{service-name}/{service-name}.log on the EC2 instance.

#!/bin/bash
    
LOG_PATH="/vol/log/${CIRCLE_PROJECT_REPONAME}"

ssh -o "StrictHostKeyChecking no" ${EC2_USERNAME}@${EC2_HOST} << EOF
    # setup logs
    sudo mkdir -p ${LOG_PATH}
    sudo chmod 777 ${LOG_PATH}

    # login and pull image from ECR
    eval $(aws ecr get-login --no-include-email --region $AWS_REGION)
    docker pull ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${CIRCLE_PROJECT_REPONAME}:latest

    # remove old container by name
    docker ps -a -q -f name=${CIRCLE_PROJECT_REPONAME} | xargs --no-run-if-empty docker rm -f

    docker run \
        --detach \
        -e AWS_S3_ACCESS_KEY_ID="${AWS_S3_ACCESS_KEY_ID}" \
        -e AWS_S3_SECRET_ACCESS_KEY="${AWS_S3_SECRET_ACCESS_KEY}" \
        -e AWS_SQS_ACCESS_KEY_ID="${AWS_SQS_ACCESS_KEY_ID}" \
        -e AWS_SQS_SECRET_ACCESS_KEY="${AWS_SQS_SECRET_ACCESS_KEY}" \
        -e AWS_SQS_REGION="${AWS_SQS_REGION}" \
        -e AWS_SQS_QUEUE_NAME="${AWS_SQS_QUEUE_NAME}" \
        -e SLACK_WEBHOOK_URL="${SLACK_WEBHOOK_URL}" \
        -e SLACK_SERVICE_NAME="${SLACK_SERVICE_NAME}" \
        -e TEAMS_WEBHOOK_URL="${TEAMS_WEBHOOK_URL}" \
        -e TEAMS_SERVICE_NAME="${TEAMS_SERVICE_NAME}" \
        -e RDB_API_URL="${RDB_API_URL}" \
        -e CASSANDRA_API_URL="${CASSANDRA_API_URL}" \
        -e ENVIRONMENT="${ENVIRONMENT}" \
        -e TO_BE_PROCESSED_PREFIX="${TO_BE_PROCESSED_PREFIX}" \
        -e PROCESSED_PREFIX="${PROCESSED_PREFIX}" \
        -e REQUIRES_ATTENTION_PREFIX="${REQUIRES_ATTENTION_PREFIX}" \
        -e GENERATED_JSON_PREFIX="${GENERATED_JSON_PREFIX}" \
        -v ${LOG_PATH}:/app/log \
        --log-driver none \
        --name ${CIRCLE_PROJECT_REPONAME} \
        ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${CIRCLE_PROJECT_REPONAME}:latest

    # delete dangling images
    docker images -q -f dangling=true | xargs --no-run-if-empty docker rmi
    # delete dangling volumes
    docker volume ls -q -f dangling=true | xargs --no-run-if-empty docker volume rm
EOF

These are all python services and we use the logging module to add logs throughout the code. The environment variables used in the above deploy.sh file are all set in the Circle CI environment variables.

On checking the container configuration using docker inspect, I see that the LogPath is empty. However, the LogPath is being configured by the script above. These exact same files had been working before and we could see all the logs by doing:

cat /vol/log/{service-name}/{service-name}.log

I don't know what has changed suddenly and we are unable to see any logs.

I am not very familiar docker, so I would really appreciate some help on this.

I have tried removing the flag, --log-driver none from the deploy.sh script. This way logs are being displayed using the docker logs command but not in the log file.

dhilmathy
  • 2,800
  • 2
  • 21
  • 29
ShwetaJ
  • 462
  • 1
  • 8
  • 32

0 Answers0