The problem:
My Jenkins instance is set up to use Docker in a "cloud" configuration (i.e., not using the docker directive in a pipeline script, but instead spinning up containers as traditional agents on-demand. Configured under https://your.jenkins.url/configureClouds).
I have two agents set up with the same label, both limited to one instance each. The agents are identical except for different names, and different volumes attached for their workspace.
When I trigger two builds, the second build hangs, waiting on the first agent, even though both agent templates have the same label. The same thing happens if I use different labels and specify to use either label in my pipeline script.
If I keep distinct labels for both templates and allow two executors for each, and limit the docker host to only two executors total, Jenkins will still spawn two builds using the same template (which fails for reasons I'll expand on below).
How can I get Jenkins to spawn builds on agents round-robin style (preferably while using a single label)?
Other options I've tried:
If I try to run concurrent builds with just one agent template in the docker cloud config, the builds will use the same exact workspace in the shared volume, which breaks the builds. Jenkins sees each build as running on a different agent, so it doesn't spawn a second workspace like it would when running concurrent builds on a single agent.
Builds work just fine with a single agent template if I don't try to share the workspace (i.e. I don't mount a volume, and no data in the container persists), but then I'm wasting a ton of network and disk IO cloning the git repo every time.
I could use the
dockerdirective in my pipeline script (which I've done previously) which will solve this problem, but then I'm limited to a single docker host. I have multiple hosts available and ultimately I want to leverage all of them for builds. The "cloud" option seems like the easiest way to do so.Alternatively, I could spin up persistent containers on the hosts outside of Jenkins and set them up as "typical" agents that way (so, they wouldn't be created on-demand). But then I have to babysit them. Docker Compose with
restart: alwayswill help, but I'd rather let Jenkins manage everything for me if possible. This is still my backup plan as I know this will work how I expect it to.I also tried setting up a second copy of my docker cloud in Jenkins, and it seems like no matter what, jenkins will only spin up a single container, even if a job is using a different label from the label used by the currently spawned container. I.E., a container for
label_Ais spun up, and a job seekinglabel_Bis spawned. But Jenkins won't spawn a container forlabel_Beven if it's targeting a different cloud instance.
