6

We're using Ansible to roll out security updates on all our EC2 instances which are running stateful services such as databases, search engines etc. This works fine.

I'm wondering what is the best way to make security updates on the ECS instances (which are running stateless web applications in Docker containers). Due to automatic scaling there is a lot of dynamics in the number of instances and their IP addresses. Ansible uses a hardcoded list of IP addresses (the hosts file), so it seems to not really fit the purpose.

Is it even a good idea to update these instances or should we rather tear them down and spawn new ones every once in a while?

Any best practices from DevOps folks out there?

Update:

I found out that Ansible supports dynamic inventory. There is a script that fetches information about the hosts from AWS and generates a dynamic inventory for Ansible, this works fine.

However, one problem remains. Whenever there is a new host that I had not connected to before, the following message is displayed and must be confirmed manually. 

The authenticity of host '10.0.1.247 (10.0.1.247)' can't be established.
ECDSA key fingerprint is SHA256:GSogs6P6CzbOzLm9ByWsXkfz7/2A4qwj4PDvczApS/I.
Are you sure you want to continue connecting (yes/no)? yes

This is very annoying as I want to implement a fully automated update mechanism. Is there a solution for this problem?

Bastian Voigt
  • 165
  • 9
  • For an ASG, you shouldn't be using a static inventory file. Instead, use the EC2 inventory plugin or a custom one to get the IPs of your active ASG instances. – EEAA Dec 15 '16 at 12:41
  • Thanks, I am checking out this plugin now. Is it possible to combine the dynamic EC2 inventory with a static inventory? Or will I need to build two different playbooks? – Bastian Voigt Dec 15 '16 at 13:15
  • That I'm not sure about. – EEAA Dec 15 '16 at 13:16
  • 2
    Static and dynamic inventories can be combined, just point `ansible` to a directory containing both scripts and `ini` files. – dawud Dec 15 '16 at 13:30

2 Answers2

4

Upgrading a container in place is a complete anti-pattern. It is cumbersome, as you need to upgrade each one of them and possibly commit each one of them.

Instead, update the image used to spawn your containers. This is normally done by having a section on your Dockerfile to ensure the image is up to date, so the process of patching is basically rebuilding the image. As an example:

FROM centos:7.2.1511
MAINTAINER Jane Doe <j.doe@foo.com>

RUN yum update -y && \
    yum install -y \
      bar \
      foo && \
    yum clean all
# The rest of your Dockerfile

Rolling out the new image, though, is where I find ECS lacking. You need to build an strategy yourself to ensure no downtime.

It is also a best practice to have a service scanning the images in your registry for vulnerabilities.

Patching the host OS (which might require a reboot) without downtime, is yet another area where I found ECS lacking orchestration capabilities, as in, nothing out of the box is built into the product other than ELB balancing between nodes.

Other container platforms (such as OpenShift) have the ability of evacuating containers (pods,since this is kubernetes) from nodes, and scheduling them elsewhere. The load balancer is aware of this changes, ensuring zero downtime. Additionally,the OS typically used with OpenShift has an improved patching mechanism based on RPM OSTree that greatly reduces the complexity of patchingthe host operating system.

dawud
  • 15,096
  • 3
  • 42
  • 61
3

Whenever there is a new host that I had not connected to before, the following message is displayed and must be confirmed manually. [ ] Is there a solution for this problem?

Modify the ssh_connection of the ansible.cfg, so that it contains -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null arguments.

For example:

[ssh_connection]
ssh_args = -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ControlMaster=auto -o ControlPersist=60s
techraf
  • 4,243
  • 8
  • 29
  • 44