2

We have a cloud full of self-hosted Azure Agents running on custom AMIs. In some cases, I have some cleanup operations which I'd really like to do either before or after a job runs on the machine, but I don't want the developer waiting for the job to wait either at the beginning or the end of the job (which holds up other stages).

What I'd really like is to have the Azure Agent itself say "after this job finishes, I will run a set of custom scripts that will prepare for the next job, and I won't accept more work until that set of scripts is done".

In a pinch, maybe just a "cooldown" setting would work -- wait 30 seconds before accepting another job. (Then at least a job could trigger some background work before finishing.)

Has anyone had experience with this, or knows of a workable solution?

Elliot Nelson
  • 11,371
  • 3
  • 30
  • 44

3 Answers3

3

I suggest three solutions

  1. Create another pipeline to run the clean up tasks on agents - you can also add demand for specific agents (See here https://learn.microsoft.com/en-us/azure/devops/pipelines/process/demands?view=azure-devops&tabs=yaml) by Agent.Name -equals [Your Agent Name]. You can set frequency to minutes, hours, as you like using cron pattern. As while this pipeline will be running and taking up the agent, the agent being cleaned will not be available for other jobs. Do note that you can trigger this pipeline from another pipeline, but if both are using the same agents - they can just get deadlocked.

  2. Create a template containing scripts tasks having all clean up logic and use it at the end of every job (which you have discounted).

  3. Rather than using static VM's for agent hosting, use Azure scaleset for Self hosted agents - everytime agents are scaled down they are gone and when scaled up they start fresh. This saves a lot of money in terms of sitting agents not doing anything when no one is working. We use this option and went away from static agents. We have also used packer to create the VM image/vhd overnight to update it with patches, softwares required, and docker images cached. ref: https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/scale-set-agents?view=azure-devops

scorpio
  • 1,587
  • 2
  • 15
  • 27
  • Thanks for these suggestions! The issue with (2) and (3) is that neither of them give me what I want, which is the ability to take ~2 minutes of setup out of the pipeline duration. (1) would work, although is a bit tricky with Azure. In Jenkins, i would trigger a new job with high priority on the same self-label (agent name); in Azure it seems I can only target the pool. If there's 25 agents in the pool, I don't think I can target "this agent". But it gives me something to play with! – Elliot Nelson Jan 30 '22 at 19:30
  • 1
    You can target individual agent as well in azure DevOps by adding demand of agent name. – scorpio Jan 31 '22 at 01:32
  • Interesting... to support that, I guess the pipeline would need to accept as a Parameter the agent name, and if the Parameter exists, make it a conditional extra Demand on the pool? I'll give it a shot! – Elliot Nelson Jan 31 '22 at 13:19
  • Updated answer to include how to demand for agent with specific name. You are correct you can make the clean up pipeline configurable to pass list of agents as parameter with some default values and iterate the job for them. – scorpio Feb 01 '22 at 14:35
1

For those discovering this question, there's a much better way: run your self-hosted agent with the --once flag, documented here.

You'll need to wrap it in your own bash script, but something like this works:

while :
do
  echo "Performing pre-job setup..."
  
  echo "Waiting for job..."
  ./run.sh --once

  echo "Cleaning up..."

  sleep 2
done
Elliot Nelson
  • 11,371
  • 3
  • 30
  • 44
0

Another option would be to use a ScaleSet VM setup which preps a new agent for each run and discards the VM when the run is done. It can prepare new VMs in the background while the job is running.

And I suspect you could implement your own IMaintenanceProvirer..

https://github.com/microsoft/azure-pipelines-agent/blob/master/src/Agent.Worker/Maintenance/MaintenanceJobExtension.cs#L53

jessehouwing
  • 106,458
  • 22
  • 256
  • 341