I'm running about 50 mariadb docker containers on a given host. Due to the un-controllable nature of the application, we sometimes get queries that cause mariadb to write up to 10GB of data to the tmp directory (normal) or more than 10GB (not normal and should be stopped). I can't fix this in the application. Instead, I need to ensure that if more than 10GB is written to tmp it causes mysql to receive an "out of disk space error".
I'm using overlay2 - so if I don't mount a volume for /tmp, then /tmp is part of the container and is inefficient to write to and worse, it will fill up the overlay2 partition and cause all containers to fail. Not a great solution.
I could create a single 50GB partition and bind mount it in all containers - which is better. But it still means that if one container fills the /tmp directory, none of the containers have any /tmp directory space to write to.
I could make each /tmp directory a tmpfs. But I need at least 10GB and just don't have that much RAM to spare.
I could create an individual 10GB partition for each container, but that's 500GB and I don't have that much disk space so spare.
Ideally I'd like to create a single 50GB partition shared amongst all the container, but with each container only allowed to write 10GB to it (that way it would take 5 containers exceeding their limit to affect the entire server).
However, I'm not sure how to do that. All the containers run as the same user, so quota's aren't that helpful.
Or maybe there's another way I'm not thinking of?
