0

As I keep uploading updated versions of my cookbooks to chef server I have to wonder if there is a limit to the number of versions of a cookbook chef server can store.

Is this something I should even be concerned about or is it a none issue? Disk space is obviously not what I'm worried about here.

red888
  • 27,709
  • 55
  • 204
  • 392

3 Answers3

4

I don't think there's going to be a good answer to this question. I'd say that you don't want to design your workflow around this problem and definitely don't want to be deleting cookbooks every time you upload.

Periodic garbage collection (once or twice a year) is probably a good idea, but its going to depend deeply on how you structure your cookbooks and their dependencies and how much you 'stress' the dep solver (gecode or molinillo) that you're using.

If you're not seeing slow dep solving, then there's no need to do it. If you are seeing slow dep solving then you should strongly consider doing it.

If you have a large site with dozens of people contributing to hundreds of cookbooks, and you haven't garbage collected old, out of date cookbook versions for years and some of your popular cookbooks have hundreds or thousands of versions stored on the server... Probably be useful to think about it.

If you're just setting out and its mostly your team and a few dozen cookbooks and at most you have a few dozen versions, then its less likely to be an issue.

If you're somewhere in between... maybe?

ADDED (Oct 8 2016): Another issue to consider is that with lots of cookbook versions sometimes the depsolver becomes "overly clever" in picking extremely old cookbook versions that you've forgotten about but which allow it to satsify all its constraints. I just recently ran into this with two cookbooks that had conflicting constraints (dovecot wanted ohai ~> 3.0 while chef_nginx wanted ohai ~> 4.0) to solve this problem the depsolver helpfully downgraded my wrapper cookbook to a version that was so old that I was using uw-imapd instead of dovecot, which removed the ohai constraint from dovecot and made the depsolver happy, while making me quite sad because Ubuntu hasn't even shipped uw-imapd for years now. One way to deal with this is to be more a lot more explicit about constraining cookbooks to the latest versions with pins. You can also simply dump those unused versions out of your chef-server, though, so that they're not available for depsolving.

lamont
  • 3,854
  • 1
  • 20
  • 26
  • 1
    Also depselector/gecode at least favors the most recent version, so if you had 100,000 cookbook versions on your server, but the solutions all picked the latest, then all those old versions shouldn't cause any problems. Without knowing how complicated your graphs are its hard to say how long your dep solving will take, and the best way to measure how complicated your graphs are is by how long your dep solving takes. – lamont Aug 30 '16 at 07:15
0

A less technical answer could come from looking at your question from the "single responsibility principle point of view".

What I mean is: in any good system, each component should have one clear responsibility.

Now you can ask yourself: is it the responsibility of your chef server to act as source code management tool?

What I mean is: there are tools like git, SVN, .. you name it that solely exist in order to allow version management on text files.

Do you want to make your chef server do that, too; or would it be a better strategy to do the versioning of your "config text files" using a different infrastructure?!

GhostCat
  • 137,827
  • 25
  • 176
  • 248
  • OK, so whats the workflow for uploading an updated cookbook then? Delete old cookbook and upload new one? Do Chef users periodically manually purge old cookbook versions? It seems like chef server wants to keep all of the cookbook versions as that is the default behavior. – red888 Aug 29 '16 at 19:33
  • I am not talking about "other" chef users. I simply wanted to point out a dimension that the (hopefully more technical) answers that you (hopefully) see soon ... might miss to point out. – GhostCat Aug 29 '16 at 19:35
  • That's an idea, but chef-server doesn't act as a source code management, you can update a cookbook file without bumping it's version, and chef-server will only serve the latest version of the file with no (official) way to access previous version, so the responsibility of the chef-server is already clear. The server can't guess if you want to keep some version of a cookbook or not, so it has no auto-cleaning mechanism, it's up to the team to know what is deemed obsolete. cc @red888 (this is not an answer at all, just a remark it's too quick to consider the chef-server as a SCM) – Tensibai Aug 30 '16 at 07:58
0

So it depends on your workflow. Chef's ecosystem has two version constraint solvers in use: Chef Server always uses gecode (aka depsolver), Policyfiles always use molinillo, and Berkshelf can use either depending on configuration. Gecode can start to choke when you hit a few thousand overall versions floating around but it depends heavily on how good you are about well formed dependency version constraints. Molinillo is a lot better about this, but that means either using berks apply to lock down all your environments or using Policyfiles (i.e. don't let Chef Server try to run a constraint solution against the full universe).

coderanger
  • 52,400
  • 4
  • 52
  • 75