2

I'm using Gitlab shared runner with Docker (current runner version: 10.0.2, docker storage driver: overlay2), running on AWS t2.small instance. I started experiencing issues with builds slowing down after some time (it's hard to say when exactly they become slow) - they take ~10x more time to finish than before. After killing the instance problem disappears for a while and after some time it slows down again.

Things I already checked:

  • CPU usage on machine is around 20% the whole time
  • RAM usage is around 1,5 G during the heaviest build
  • IOPS on EBS are not exhausting all Burst Balance (e.g. right now burst balance is around 80%)
  • Download speed

What else might be causing this ?

Just in case, jobs that are running on this runner are mostly yarn install and yarn build of a medium-sized front-end React application.

pgrzesik
  • 131
  • 3
  • did you ever find a solution to this? – mfink Dec 19 '18 at 21:43
  • Circling back to it right now I believe that the problem might've been with too small instance/exhausted instance credits, however I'm pretty sure I also checked that. As of recently, on bigger instance of EC2 and newer Gitlab runner, problem does not appear. I would recommend looking closely at CPU Credits, monitoring CPU usage and upgrading runner version. – pgrzesik Dec 20 '18 at 12:47
  • With what workloads are you experiencing issues ? – pgrzesik Dec 20 '18 at 12:47
  • I thought it might be cpu credits, but i'm on a t3 which is unlimited by default. I ended up upgrade gitlab runner, which didn't fix it. What ultimately resolve it was a hard stop and start (reboot didn't do it). sigh. – mfink Dec 20 '18 at 13:11

1 Answers1

1

Had the same issue, pipelines timed out after an hour while usually taking ~8mins. Stopped the EKS node EC2 instance in AWS Console (t3.small). (the one that the runner was created on) The Autoscaling group took over, brought up a new node and killed the old one. The pipelines came back to normal speed. Will be looking to increase instance size in the future.

AyCaramba
  • 11
  • 2