How to spin up a new EC2 instance within a gitlab CI/CD pipeline

Question

I have a scenario where I need to spin up a new EC2 instance and deploy docker image inside the ec2 and run some tests. After all the tests have been executed I need to remove the ec2 instance. How can I do this using gitlab ci/cd. I am pretty new to this does anyone know if this is something achievable using gitlab?

Answer 1

You can use gitlab auto scaling with EC2 instances https://docs.gitlab.com/runner/configuration/runner_autoscale_aws/ . If you have no EC2 instances that you can use as a runner manager create t4g.nano(cheap) and configure it accordingly to instructions in the link.

Based on your runner configuration EC2 instance will die after either IdleTime(seconds) or MaxBuilds(number).

  ...
  [runners.machine]
    IdleCount = 1
    IdleTime = 1800
    MaxBuilds = 100
    MachineDriver = "amazonec2"
    ...
    ]
    [[runners.machine.autoscaling]]
      Periods = ["* * 9-17 * * mon-fri *"]
      IdleCount = 50
      IdleTime = 3600
      Timezone = "UTC"
    [[runners.machine.autoscaling]]
      Periods = ["* * * * * sat,sun *"]
      IdleCount = 5
      IdleTime = 60
      Timezone = "UTC"
    ...

In around 2-3 minutes your pipeline will spin the EC2 instance. Remember to use proper tag for your runner manager it will spin that instance for you and run you job in it.

Below start of a pipeline inside EC2 instance for job with kaniko image. Instance will be terminated according to schedule assigned in your config.toml(above):

• from Monday to Friday from 9am till 5pm after 3600 seconds
• from Saturday to Sunday 60 seconds
• any other time after 1800 seconds.

Answer 2

One way would be to let the instance terminate itself through its shutdown behavior. Thus, you launch the instance with the shutdown behavior of Terminate.

Then, once the instance does its job it simply calls a regular command to shutdown and it will terminate automatically. For linux this is usually:

sudo shutdown -h now