19

I'm curious to how people are deploying FreeBSD's ports in their environment. I assume that most people using FreeBSD are indeed using Ports (and often portupgrade for upgrading with binaries). I am however interested in how you have this setup, as I'm not satisfied with how things work in recent versions. I'm now running FreeBSD 9.0 and are having issues.

I've set things up as follows:

  • /usr/ports is shared via NFS from one node (with nightly 'portsnap fetch update').
  • Each node mounts /usr/ports with read-write
  • I've set "WRKDIRPREFIX=/usr/tmp" in /etc/make.conf on all nodes
  • I've configured the Portsnap to use a local index by adding the following to /usr/local/etc/pkgtools.conf:

ENV['LOCALINDICES'] ||= '/var/db'

ENV['PORTS_INDEX'] ||= ENV['LOCALINDICES'] + '/INDEX.local'

I can successfully run portupgrade -p package to build a package and then portupgrade -P package to install the binary on the other nodes.

Yet, sometime I receive the following issue: /var/db/INDEX.local:23265:dbm_store failed

I cannot think of any other optimizations I can do to the system, as the index now resides locally, and the only thing really exported is the ports-tree and nothing is ever written to there from the nodes.

Stefan Lasiewski
  • 23,667
  • 41
  • 132
  • 186
vpetersson
  • 861
  • 1
  • 11
  • 22
  • One option would be to have a full local ports-tree on every node (and perhaps just mount 'distfiles' and 'packages'), but this feels like a massive waste of space (and not to mention a lot of unnecessary updates). – vpetersson Feb 28 '12 at 21:31
  • vpeterson : This is a question that needs to get asked (I'm blocked on this right now. And +5 votes and 3 stars suggests we are not alone). Perhaps clean up this question and state some specific problems you are trying to solve. (FWIW, someone voted to close your question. Personally I would very much like to see this, or a similar, question saved). – Stefan Lasiewski Mar 06 '12 at 01:38
  • I'm not sure how to make the question more clear. I also don't think that @voretaq7 actually answers the questions, but rather suggests an alternative method. The question is really about what the topic suggests -- how are people deploying FreeBSD's ports in a multi-node environment. – vpetersson Mar 06 '12 at 14:10

4 Answers4

13

I have never been fully satisfied with the ports system in a large environment -- It always seems like you need to apply some external management to it in order to make it work well.

My best tips (in order of ascending preference, "worst" solution to "best" solution):

If you're building on each host, don't.
If you must, don't do it over NFS with read-write mounts like you describe: You can usually trust the ports to Do The Right Thing and not stomp on the ports tree if you provide alternate work directories, but it's always better to be safe than sorry: Run a local CVS/csup mirror and csup all your hosts from that box, then build locally as you would if they were individual machines.
Yes, I know this means having more disk space on the hosts and an extra step. It's also almost guaranteed to be problem-free.
Caveat: You probably want to sync the package configuration files (rsync or similar) from a designated "configuration host" to ensure consistency on each machine (you can even rsync the whole ports tree if you want, rather than using csup on each node).

Use a Build Host, create packages, and install those.
A much better solution than building on each individual machine: Use a build host to create packages, and point your tools at those packages.
This means keeping a build host around for every architecture you run (or cross-compiling), but it's ultimately nicer for your target machines (no large compile jobs, a guarantee of consistency)

Use a configuration/system management tool.
This is the solution I wound up with -- I build a standard server image and deploy it around my environment using radmind. You can do similar things with Puppet or Chef. This has all the advantages of using a build host (consistency, less load on the individual servers), and adds the benefit of configuration management.

Caveat: This only works really well if your machines are "identical" -- That is you can install the same set of ports on all of them. It can work if you have varying sets of ports, but that substantially increases the administrative overhead.

Disclaimer: I'm the port maintainer for sysutils/radmind. Yeah, I like it that much that I adopted it.

All of this is based on my experience managing various-sized FreeBSD environments (ranging from 1-2 machines to over 100). Configuration/System Management tools that push and maintain a standardized image are really the best way to handle this in my experience.

voretaq7
  • 79,879
  • 17
  • 130
  • 214
  • Good pointers. I have played with Puppet a bit in the past, and love it on Ubuntu. Yet, I'm not sure how well it plays with FreeBSD. I'm yet to try that out. Regardless, even with Puppet, I think it calls on Portupgrade, which takes us back to square one. I don't see any other way it could work, as it would then need to do pkg_delete/pkg_add or 'pkg_add -f' which wouldn't be a good idea. As far as the hardware goes -- they're all identical since they run in a public cloud (KVM/Qemu). – vpetersson Feb 28 '12 at 20:10
  • If your hardware and baseline software configurations are identical I would suggest something like radmind, managing the whole system image. Puppet and Chef are great tools, however as you pointed out they call the underlying OS binaries, which leaves you back at using a build host and distributing packages. radmind avoids this by taking over management at the filesystem level ("If it's not what's supposed to be here, replace or remove it") rather than trying to be a surrogate sysadmin ("Run these commands/change these files for me to configure the box"). – voretaq7 Feb 28 '12 at 20:20
  • Well, the systems have identical hardware, but not identical configurations. I will have to look into Radmind but I'm not sure if that is the best approach. Using the built-in tools *should* work IMHO, which is why I'm reaching out to the community to see if I've missed anything obvious. I can hardly be the only one trying to do this. – vpetersson Feb 28 '12 at 20:29
  • The built-in tools definitely *DO* work, they just require a lot of help (build servers, local distribution of packages, etc.) -- They are really geared toward managing one machine, and don't scale as well as they could. Note that I left out the option of [rolling your own freebsd-update server](http://www.freebsd.org/doc/en/articles/freebsd-update-server/index.html) -- I've never tried this for more than just the base system, but it's ***theoretically*** possible. I just stuck to the stuff I know works :) – voretaq7 Feb 28 '12 at 20:35
  • Yeah, that's an interesting idea actually. I'm just not sure if it would work with distributing ports-packages without much modifications. I'm really curious to how sysadmins of large systems manage this, as there are plenty of large deployments of FreeBSD. Do all of them roll their own solution? If so, that feels pretty odd and not very Open Source-ish. – vpetersson Feb 28 '12 at 21:27
  • @voretaq7: _point your tools at those packages._ Can you point to any articles on how to do this, so that I'm not rebuilding the wheel? – Stefan Lasiewski Mar 13 '12 at 00:35
  • @StefanLasiewski `pkg_add` accepts `-r` to fetch packages from a remote location, see the man page, specifically the section on `PACKAGEROOT` / `PACKAGESITE` environment variables. `portupgrade` and other pkgtools-based systems have similar options (see `pkgtools.conf`) – voretaq7 Mar 13 '12 at 02:27
  • @voretaq7: Yes, that's in the manual. I'm looking for What's the directory layout? Should I use FTP or HTTP? What are some recommended practices? When should I run `make config-recursive`? What other arguments This information is spread all over docs.freebsd.org and forums.freebsd.org, the Porters Handbook, etc. but it's difficult to find the simple steps needed for maintaining our own repository. – Stefan Lasiewski Mar 13 '12 at 19:58
6

Strange that nobody mentioned ports-mgmt/tinderbox:

Tinderbox is a package building system for FreeBSD ports, based on official Portbuild scripts used on pointyhat building cluster. Tinderbox was written by Joe Marcus Clarke.

You can define multiple jails (base system versions) and multiple portstrees. The combination of jail and portstree is called a build. A Tinderbox jail is not what is understood as a jail in FreeBSD, it is in fact a given world in a chroot. Tinderbox supports automatic tracking of dependencies and only rebuilds packages that changed since last run. Tinderbox has support for email notification of failed builds. Tinderbox also integrates well with ccache.

Tinderbox is designed to easily provide package sets of ports you need, for platforms and architectures you need. Tinderbox is also excellent tool for testing new ports and port upgrades, especially for testing dependencies and packing lists. It's also useful for testing ports on various releases of FreeBSD, since you can run FreeBSD 6.X world as a jail on FreeBSD 7.X/8.X host.

Also switching to pkgng greatly simplifies package deployments.
Check it out on github: https://github.com/pkgng/pkgng

SaveTheRbtz
  • 5,691
  • 4
  • 32
  • 45
  • 1
    While that certainly might be handy for building the actual packages in a diverse environment (multiple versions, architectures etc), it doesn't really solve the issue of deploying the packages. – vpetersson Mar 06 '12 at 14:12
  • Tinderbox makes the packages available over HTTP, so you can combine this with the comments on voretaq7's answer to get your deployment solution (e.g. set `PACKAGEROOT` / `PACKAGESITE` and use radmind or Puppet/Chef). – James O'Gorman Mar 13 '12 at 09:04
  • Yes, but building and distributing packages isn't the problem. I can use portupgrade (-p) to build the package and distribute them via NFS (with or without a ports-tree). The problem is that this model still requires either a) a complete ports-tree locally, or b) a ports-tree exported via NFS, which brings us back to problem at hand. – vpetersson Mar 13 '12 at 13:32
  • What do you mean by "complete ports tree locally"? It's only needed on the tinderbox server, not each individual server. Thereafter you just distribute the binary packages, in whatever way you choose. – James O'Gorman Mar 14 '12 at 23:17
  • @JamesO'Gorman You're right. Yet, Tinderbox doesn't really change the equation. Assuming a homogeneous environment, one might as well use 'portupgrade' to build the package. Let's assume that you do ship the package to the node. How would you then install it? You can't use pkg_add to install an upgrade (at least not without forcing). Hence you're left with portupgrade (afaik), which does require a full ports tree (either exported or local) or did I miss something? – vpetersson Mar 15 '12 at 19:47
  • 2
    Portupgrade will do exactly what you'd have to do if you were building from source or using binary packages - `pkg_delete` must be run first and then install the new version. OpenBSD has handled this better by including an upgrade option in `pkg_add`. Not sure about Portupgrade but portmaster can work just using INDEX and not a full ports tree. – James O'Gorman Mar 15 '12 at 21:42
  • 1
    Right, but pkg_delete is hardly a reasonable approach. Let's say you want to upgrade Ruby, Python or any other package that is a prerequisite for a large number of other packages. pkg_delete would then require you to delete all of the dependencies, which is hardly an option for a production system. Portupgrade does a *far* better job with this, but again, it doesn't seem to scale. – vpetersson Mar 17 '12 at 19:53
  • Did you look into `portupgrade` code? The only it's magic I aware of is moving shared libraries to `/usr/local/lib/compat/pkg` – SaveTheRbtz Mar 17 '12 at 22:34
  • When using PACKAGESITE with portmanager, should I just provide the packages, should I also provide other files such as INDEX, UPGRADING, etc. – Stefan Lasiewski Mar 18 '12 at 17:04
  • @vpetersson This is a bit old now but I'll reply anyway: pkg_delete doesn't require you to delete all dependencies - you can use -f to force delete, ignoring the fact that things depend on it. This allows you to pkg_add the new version. In fact this is exactly what the ports tree and all the utilities (portupgrade/portmaster etc.) do. You can then use a portmaster incantation to fixup the dependencies if required. – James O'Gorman Mar 29 '12 at 22:50
4

I have managed 100+ FreeBSD servers by simply sharing /usr read-only over well-tuned NFS, moving the package databases from /var to /usr and symlinking to them (not strictly necessary but enables pkg_info and such). There may have been one or two other files that needed to be moved in one direction or the other and symlinked but the whole setup took me about an hour to figure out. It worked very, very well. Had I run into scaling issues I would have added additional NFS servers and split the workload but it never came up. Performance was never an issue for me (in fact it was great) but I suppose you could put the NFS server's /usr (or a copy thereof) on an md.

fuzz
  • 98
  • 1
  • Sharing the built package files over NFS (which is what it sounds like you're talking about?) is certainly another reasonable approach - you can then use something like puppet (or even homegrown SSH-and-shell scripts) to install/upgrade packages from the NFS share. – voretaq7 Dec 01 '12 at 15:17
1

It appears as nobody got a good solution to this unfortunately. Most likely this is due to limitations in the underlaying tools.

Here's what I came up with: I scrapped the idea of having the entire ports-tree exported. Instead, I gave in and put a full ports-tree on each node. I then mounted 'packages' over NFS (to enable distribution of packages).

I'm also intending to utilize a caching proxy (probably Squid) to speed up the portnap process. I wrote a short post about how to setup this on my blog.

References:

vpetersson
  • 861
  • 1
  • 11
  • 22