We are having some services running in docker containers on an EC2 machine (Linux based). These containers are deployed using CircleCI.
Following is the circle CI config.yml
version: 2
jobs:
deploy:
docker:
- image: docker:18.06.0-ce-git
working_directory: ~/app
steps:
- checkout
- setup_remote_docker
- run:
name: Install awscli
command: |
apk add --no-cache py-pip
pip install awscli==1.15.72
- run:
name: Build Docker image
command: |
docker build -t resource-etl .
- run:
name: Push image to ECR
command: |
eval $(aws ecr get-login --no-include-email --region $AWS_REGION)
docker tag ${CIRCLE_PROJECT_REPONAME}:latest ${ECR_REPOSITORY_BASE_URI}/${CIRCLE_PROJECT_REPONAME}:latest
docker push ${ECR_REPOSITORY_BASE_URI}/${CIRCLE_PROJECT_REPONAME}:latest
- run:
name: Deploy container to EC2
command: |
sh .circleci/deploy.sh
workflows:
version: 2
deploy:
jobs:
- deploy
# filters:
# branches:
# only: master
Following is the deploy.sh script that configures the log path. The logs need to be stored in a folder /vol/log/{service-name}/{service-name}.log on the EC2 instance.
#!/bin/bash
LOG_PATH="/vol/log/${CIRCLE_PROJECT_REPONAME}"
ssh -o "StrictHostKeyChecking no" ${EC2_USERNAME}@${EC2_HOST} << EOF
# setup logs
sudo mkdir -p ${LOG_PATH}
sudo chmod 777 ${LOG_PATH}
# login and pull image from ECR
eval $(aws ecr get-login --no-include-email --region $AWS_REGION)
docker pull ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${CIRCLE_PROJECT_REPONAME}:latest
# remove old container by name
docker ps -a -q -f name=${CIRCLE_PROJECT_REPONAME} | xargs --no-run-if-empty docker rm -f
docker run \
--detach \
-e AWS_S3_ACCESS_KEY_ID="${AWS_S3_ACCESS_KEY_ID}" \
-e AWS_S3_SECRET_ACCESS_KEY="${AWS_S3_SECRET_ACCESS_KEY}" \
-e AWS_SQS_ACCESS_KEY_ID="${AWS_SQS_ACCESS_KEY_ID}" \
-e AWS_SQS_SECRET_ACCESS_KEY="${AWS_SQS_SECRET_ACCESS_KEY}" \
-e AWS_SQS_REGION="${AWS_SQS_REGION}" \
-e AWS_SQS_QUEUE_NAME="${AWS_SQS_QUEUE_NAME}" \
-e SLACK_WEBHOOK_URL="${SLACK_WEBHOOK_URL}" \
-e SLACK_SERVICE_NAME="${SLACK_SERVICE_NAME}" \
-e TEAMS_WEBHOOK_URL="${TEAMS_WEBHOOK_URL}" \
-e TEAMS_SERVICE_NAME="${TEAMS_SERVICE_NAME}" \
-e RDB_API_URL="${RDB_API_URL}" \
-e CASSANDRA_API_URL="${CASSANDRA_API_URL}" \
-e ENVIRONMENT="${ENVIRONMENT}" \
-e TO_BE_PROCESSED_PREFIX="${TO_BE_PROCESSED_PREFIX}" \
-e PROCESSED_PREFIX="${PROCESSED_PREFIX}" \
-e REQUIRES_ATTENTION_PREFIX="${REQUIRES_ATTENTION_PREFIX}" \
-e GENERATED_JSON_PREFIX="${GENERATED_JSON_PREFIX}" \
-v ${LOG_PATH}:/app/log \
--log-driver none \
--name ${CIRCLE_PROJECT_REPONAME} \
${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${CIRCLE_PROJECT_REPONAME}:latest
# delete dangling images
docker images -q -f dangling=true | xargs --no-run-if-empty docker rmi
# delete dangling volumes
docker volume ls -q -f dangling=true | xargs --no-run-if-empty docker volume rm
EOF
These are all python services and we use the logging module to add logs throughout the code.
The environment variables used in the above deploy.sh
file are all set in the Circle CI environment variables.
On checking the container configuration using docker inspect,
I see that the LogPath
is empty. However, the LogPath
is being configured by the script above. These exact same files had been working before and we could see all the logs by doing:
cat /vol/log/{service-name}/{service-name}.log
I don't know what has changed suddenly and we are unable to see any logs.
I am not very familiar docker, so I would really appreciate some help on this.
I have tried removing the flag, --log-driver none
from the deploy.sh
script. This way logs are being displayed using the docker logs command but not in the log file.