-4

I need to provide the ability to create/read/write/delete files and set quotas for a large number of users of a web-service. These users do not exist as Linux users so I can't apply quotas. The creation of real users just to perform quotas seems like a bad idea. Therefore, I decided to create a per-user file and mount it as a filesystem.

  1. What is the consequence of mounting of several tens of thousands of FS?
  2. Is there a limit on the total number of mounted FS?
Scott Pack
  • 14,907
  • 10
  • 53
  • 83
niXman
  • 137
  • 1
  • 6
  • 2
    Moar detail! NFS? What version of Linux? Which distribution? What is serving/providing the backend storage? – ewwhite Aug 26 '12 at 15:54
  • FS, in this case, it is formatted and mounted file as loop. OS - linux-2.6.32. Distribution - Ubuntu. "What is serving/providing the backend storage?" - I do not understand the question. – niXman Aug 26 '12 at 16:01
  • 1
    [If you have to ask about various operating system limits, you're probably doing something wrong.](http://blogs.msdn.com/b/oldnewthing/archive/2007/03/01/1775759.aspx) – Zoredache Aug 27 '12 at 05:44

3 Answers3

6

First off, mounting that many filesystems seems like a really bad idea - there must be a better solution to whatever problem you have?

Anyway, to answer your second question: an answer on a previous serverfault question seems to indicate that the limit is about 1 million mount points (2^20) per file system type for recent kernels, and you can mount at most 256 different types of filesystem.

For older kernels (pre 2.6) the limit is 256 mount points per filesystem.

Edit in response to the comments, I propose the following alternative solution:

Use XFS, which allows you to use project quota, a form of directory quota.

Create a separate XFS file system which will hold all webservice user data. Mount it at e.g. /mnt/myWebService. Then, for each webservice user, create a project directory (eg /mnt/myWebService/username1 etc) and set quota accordingly.

For instructions on how to set up project directories and quota, you can for example check out this blog entry or the RHEL XFS Quota Management page

Community
  • 1
brain99
  • 1,792
  • 12
  • 19
  • 1
    I need to provide the ability to create/read/write/delete files and set quotas for a large number of users of a web-service. But these users are not real users of Linux, so I can `t apply quotas. And the creation of real users just to perform quotas - I think is even more a bad idea. Therefore, I decided to create a per-user file and mount it as a filesystem. – niXman Aug 26 '12 at 16:14
  • [Pure-FTPd](http://www.pureftpd.org/project/pure-ftpd) supports virtual user quotas (nb files, total filesize). Perhaps you could implement the storage for your webservice through ftp? – brain99 Aug 26 '12 at 16:22
  • Interaction web-interface with Pure-FTPd seems also not the best solution... – niXman Aug 26 '12 at 16:29
  • Of course I don't know what your webservice does and how it uses the filesystem, but couldn't you roll your own system? Keep a database of nb of files and total filesize for your users, and update this each time you add/modify/delete a file? This might work well for e.g. a Dropbox-like service where users explicitly perform actions on files; of course, if file storage is a mere by-product of some totally different service you offer this may be more difficult to implement. Without more detail it's hard to offer solutions... – brain99 Aug 26 '12 at 16:35
  • I had the idea to use the database as a file store. But the difficulty is that these files should be processed by a program which takes files. But the permanent extraction and update of the records in the database is also not a good idea. That's why I focused on the variant with mounting of files as a file system, as I think this is the best possible option. – niXman Aug 26 '12 at 17:03
  • 1
    Well, perhaps you could use XFS? It has [project quota](http://xfs.org/index.php/XFS_FAQ#Q:_Quota:_What.27s_project_quota.3F), which allow you to set quota for different directories at the filesystem level without requiring a separate linux user (or a separate filesystem) per webapp user. – brain99 Aug 27 '12 at 03:58
2

Note that you don't have to create users (as in add them to /etc/passwd or any user database you want), you can set ownership of a file to any uid without it. Just make sure those uids don't overlap with local (or remote) users that may run processes on that server.

However, to be able to change the owner of a file, you need superuser privileges, which may be problematic for your web application.

Note that when you're using ZFS, you get one mount point for every snapshot, and it's not uncommon to have thousands of them, which isn't too much of a problem.

sch
  • 560
  • 4
  • 13
1

There's no need to create a zillion filesystems for this.

The easy way to do this is to create a directory for each user (which you already would have to do) and then check how much space the files in the directory use up each time a user tries to upload something. If the upload would cause them to exceed their limit, reject the upload. This is the way every other web application in the known universe does it.

P.S. You've scattered the details of what you're doing throughout several comments; you should add them to the question instead so that people can find them (and perhaps reverse your downvotes).

Michael Hampton
  • 244,070
  • 43
  • 506
  • 972