Azure DevOps Server 2019 Retention Policy no longer working

Question

Last week we have migrated from TFVC to Git on our ADS 2019.1 server.

On our validation pipeline we have an aggressive retention policy. It is set to Keep for 2 days, 10 good builds with the branch filter * and all checkboxes cleared ADS writes it as: +refs/heads/*

What we see is that since we migrated, none of the Git builds have been deleted. From the last 8 days we now have hundreds of builds with their multi-GB drop.

Is * the right branch filter to use ? This was the default setting entered by ADS, we do indeed want all builds for all branches to be deleted by this policy.

I also tried **, but that made no change, in the build results page i see all build are build off a temporary branch with a 4 digit number, which is probably the pull request number.

How can we ensure the builds are cleaned up as per the retention policies ?

The Build definition is not linked to a Release pipeline. The builds are triggered from the Pull Requests created by developers. The Pull requests are marked as Completed.

Another build definition, one with a CI trigger that runs after every commit, does still seem to delete builds based on the retention policy defined.

Picture of our TFS Retention policy settings. Master builds are cleaned up, pull request builds are not

List of the oldest builds in TFS after manual cleanup last week, these builds are now 4-5 days old, over the 2 day limit of the retention policy. Also you can see there is no retain lock behind them.

Overview of the deleted builds. You can see 'master' builds in there that have been deleted by the retention policy.

Shows the master builds are deleted after the 2 day retention policy setting

Do you use release pipelines? What is the retention on your release pipelines. If you have selected to keep artifacts on the release pipelines, it will use the release retention and not the build retention. — Matt, Jan 29 '20 at 20:55
Hi @Matt, no we do not use Release pipelines for this definition, i have added more details to the original post — Nico, Feb 04 '20 at 08:42
Why not remove all branch filters if you want this to apply to all. — jessehouwing, Feb 04 '20 at 10:28
That is because TFS enforces you to have one "You must add at least one branch filter." — Nico, Feb 04 '20 at 12:53

Christian Stadelmann · Accepted Answer · 2020-02-20T13:47:21.950

2

You may also try:

Tab "Retention"
Select a policy (in your case: "Keep for 2 days, […]")
Under "Branch filters", add another one
Type: "Include", Branch specification: "refs/pull/*"

Explaination: If you open a terminal in the build agent's source folder (named s) and run git branch --all, you will see a lot of branches like refs/pull/1234 for each pull request (e.g. 1234) there. Also, if you compare your branch specification * to the path shown on the left (+refs/heads/*) you see that it does in fact not match all branches…

edited Feb 20 '20 at 13:47

answered Feb 20 '20 at 13:16

Christian Stadelmann

516
4
14

Thank you for the suggestion @Christian. I have made this change yesterday and the provided filter of ADS seems promising. However today i see no deleted builds yet. The oldest build is from 2/18/2020, 7:08:59 AM Which should be well over 2 days ago. I will let it run like this over the weekend and see what happens. My filter now shows: +refs/heads/*,+refs/pull/* – Nico Feb 21 '20 at 13:47
Thanks again Christian ! your suggestion seems to have fixed the issue "The build was deleted by the retention policy because it finished more than 2 days ago" – Nico Feb 23 '20 at 11:23
Sadly, it still does not fix the retention for CI builds, only for pull request builds. – Christian Stadelmann Feb 24 '20 at 09:23
CI builds are cleaned up just fine by the default "*" filter. So for pull request builds i now add filter * and refs/pull/* and for CI builds i can leave it to * – Nico Feb 24 '20 at 09:47
For me, CI builds are *not* cleaned up by the `*` filter. – Christian Stadelmann Feb 25 '20 at 10:20
I do still see issues with other builds, those queued manually are also not cleaned up consistantly. all master builds are cleaned, most feature branches are not cleaned, though some are.I see now that all Deleted builds are failed builds. Any successful build is apparently not deleted by the policy.. again no Release is linked, we do have another build that is queued inside the definition – Nico Feb 26 '20 at 11:21
1

I have the same issue. I have like 20 build pipelines (no release pipelines) and on all of them the retention policies (automatic cleanup) doesn't work very well. Yesterday I added the in this post suggested filter `refs/pull/*`, and so at least the pr builds are now getting deleted (tested this on a single pipeline). But there's still a build left over, which was manually triggered and from a feature branch, which should have been deleted. Do you guys have found a solution for that? Does maybe the ADS 2020 Update solve this problem? – roli09 Dec 31 '20 at 15:36
2

I think I found the issue: The description of *Minimum to Keep* states, that the number applies to **each branch**. So if you have a build of the `main` branch and a build of a feature `branch`, both are not getting deleted, because they are the last builds of these branches. – roli09 Jan 02 '21 at 12:11

score 0 · Answer 2 · answered Jan 30 '20 at 06:35

0

Azure DevOps Server 2019 Retention Policy no longer working

Agree with Matt. If the artifacts of the release pipeline is Build, the build pipeline will use the release retention instead of the build retention:

So, we need to check the release retention, Project Settings->Release retention:

You could check this info from the document Build and release retention policies-Q&A.

Besides, if the build to be retained indefinitely, the retention policy will no longer apply.

Hope this helps.

answered Jan 30 '20 at 06:35

Leo Liu

71,098
10
114
135

Thank you for the comments, i am aware of the basic principles of ADS. This definition is only linked to the pull request, there is no release definition linked to this build definition. Also the builds are not marked as retain indefinitely. So the only link it has is to the pull request. But i assume we do not need to delete the pull requests ourselves. – Nico Jan 30 '20 at 08:15
@Nico, Thanks for you reply. So, those reserved pipelines come from pull requests? – Leo Liu Jan 30 '20 at 08:20
Correct, "all" the builds in this definition have been queued from a pull request. It could be related to the branch filter option, but as the default is * i assume this must be working. – Nico Jan 30 '20 at 08:28
I found a build definition that has migrated to Git a bit longer ago. Here i found something interesting. This Definition is also configured to delete all builds other than a few days with branch filter *. Here also no builds are deleted, but in the deleted tab i do see deleted builds from December. Which are deleted with reason "The build was deleted by the retention policy because it finished more than 30 days ago." 30 days is the value set under Maximum retention. So its falling back on that setting. So i really suspect this has something to do with the Branch filter. – Nico Jan 30 '20 at 10:01
@Nico, Confirm some info. The branch filter you mean is the branch filters of Enable continuous integration under the trigger tab? You can provide some screenshots in your question, this will help us better understand your question – Leo Liu Feb 03 '20 at 10:52
There is also a branch filter under the Retention tab, see screenshot in my original post – Nico Feb 03 '20 at 14:20

score 0 · Answer 3 · answered Aug 05 '21 at 19:06

The retention policy UI is extremely misleading, because * is not really * - it's refs/heads/*. When the UI (or perhaps the API) saves the 'branch specification' it implicitly adds +refs/heads/ to whatever you enter. main becomes +refs/heads/main, mybranch/* becomes +refs/heads/mybranch/*.

You can see this by looking at the JSON in the RetentionPolicy column in [Build].[tbl_Definition] (assuming default collection DB name):

SELECT [DefinitionName],[RetentionPolicy]
FROM [Tfs_DefaultCollection].[Build].[tbl_Definition]

The default build retention policy, which will never match pull request branches, is stored as:

[
  {
    "branches": [
      "+refs/heads/*"
    ],
    "artifacts": [],
    "artifactTypesToDelete": [
      "FilePath",
      "SymbolStore"
    ],
    "daysToKeep": 10,
    "minimumToKeep": 2,
    "deleteBuildRecord": true,
    "deleteTestResults": true
  }
]

The TFS Job Agent's build retention policy job calls the stored procedure [Build].[prc_GetBuildsForRetention] once per build definition to get a list of potential builds to clean up. Builds that are retained by a release are already excluded from the query results.

You can try it on your own with a query like this (update the definitionId and maxFinishTime, and possibly the dataspaceId and partitionId):

EXEC [Tfs_DefaultCollection].[Build].[prc_GetBuildsForRetention]
    @partitionId = 1,
    @dataspaceId = 22,
    @definitionId = 164,
    @minFinishTime = '0001-01-01',
    @maxFinishTime = '2021-08-03',
    @maxBuilds = 1000

You'll see results similar to this:

To include pull requests in your retention policy, add a new entry and type refs/pull/* into the branch specification box. That will persist into the JSON as +refs/pull/*. You'll probably want to set 'days to keep' low (I use 1), and 'minimum to keep' low (I use 0). Our PR builds are set to expire in a few hours, so there's little benefit in keeping them longer.

For future reference, the build retention policy job code is located in <installFolder>\Application Tier\TFSJobAgent\Plugins\Microsoft.TeamFoundation.Build2.Server.Extensions.dll in the Microsoft.TeamFoundation.Build2.Server.Extensions.BuildRetentionPolicyJob class.

Azure DevOps Server 2019 Retention Policy no longer working

3 Answers3

Linked