1

I have a couple of hosts which are basically a playground for developers. On these hosts, each of them has a directory under /tmp where he is free to do all he wants - store files, write logs etc.

Of course, the logs are to be rotated, or else the disc will be 100% full in a week. The files can be plenty, but I've dealt with it with paths like /tmp/[a-e]*/* and so on and lived happily for a while, but as they try new cool stuff on the machine logrotate rules grow ugly and unmanageable, and it's getting more difficult to understand which files hit the glob. Also, logrotate would segfault if asked to rotate a socket.

I don't feel like trying to enforce some naming policies in that environment, I think it's going to take quite a lot of time and get people annoyed and still would fail at some point.

And I still need to manage the logs, not just rm the dirs at night.

So is it a good idea in circumstances like these to write a script which would handle these temporary files? I prefer sticking with standard utilities whenever possible, but here I think logrotate is getting less and less manageable.

And probably someone heard of some logrotate alternatives which would work well in such an environment? I don't need emailing logs or some other advanced features, so theoretically some well commented find | xargs would do.

P.S. I do have a log aggregator but this stuff is not going to touch my little cute logstash machine.

Roman Grazhdan
  • 334
  • 3
  • 15
  • If there is something about the directory or filenames that lets you know which are logs? Using that to get a list of processes that have such a file in such a location open could be used to create a logrotate rule from a generic boilerplate that would apply to it. – Brian Jun 02 '14 at 21:03
  • An interesting approach! I'll probably play with it, but I'm not sure it would detect all the files. On the other hand, yes, I can use a template for logrotate rules, that might work. – Roman Grazhdan Jun 02 '14 at 21:05
  • Wouldn't disk quptas be useful in this situation ? – user9517 Jun 02 '14 at 21:05
  • Don't think so. It's not like I want to set some hard limit. Quite often some of them might need to write a large amount of data for a day or two, and it wouldn't affect the rest of developers. Managing quotas would be a hassle. – Roman Grazhdan Jun 02 '14 at 21:08

2 Answers2

5

In development environments like this I was a fan of setting up a specific directory that folks needed to symlink their log directories into, usually with a forced naming convention like 'username_branch' or 'username-ProductInTesting' I'd then logrotate all the symlinked directories.

The couple times that folks forgot to add the symlink and the disk got filled up the other devops folks got pretty steamed at them.. So I was off the hook.

Tim Brigham
  • 15,545
  • 10
  • 75
  • 115
2

Since these are developer machines, have you asked them to manage this stuff themselves? Give them the tools and teach them the process, and then it's on them.

Alternatively, if you can't fob this off on them, can you require them to alert you when they add a new log directory so you don't have to find the new ones?

Both of these approaches come down to "Communicate between teams and force accountability", you'll note. How do you handle this in prod? I hope the same way, right?

mfinni
  • 36,144
  • 4
  • 53
  • 86
  • No, production is production and no such things are allowed there. I'm afraid that in this case I can't do much with communication, and it's not because I dislike communication. – Roman Grazhdan Jun 02 '14 at 21:02
  • I meant, when the dev team works with you to push new code/products to production machines, presumably some info is passed along about location and retention requirements for log files. Assuming such, follow the same process in dev, even if it's more frequent. – mfinni Jun 02 '14 at 21:09
  • It might be part of solution. – Roman Grazhdan Jun 03 '14 at 03:48