collective.xsendfile, ZODB blobs and UNIX file permissions

Question

I am currently trying to configure collective.xsendfile, Apache mod_xsendfile and Plone 4.

Apparently the Apache process does not see blobstrage files on the file system because they contain permissions:

ls -lh var/blobstorage/0x00/0x00/0x00/0x00/0x00/0x18/0xd5/0x19/0x038ea09d0eddc611.blob -r-------- 1 plone plone 1006K May 28 15:30 var/blobstorage/0x00/0x00/0x00/0x00/0x00/0x18/0xd5/0x19/0x038ea09d0eddc611.blob

How do I configure blobstorage to give additional permissions, so that Apache could access these files?

Martijn Pieters · Accepted Answer · 2011-05-29T20:45:55.020

The modes with which the blobstorage writes it's directories and files is hardcoded in ZODB.blob. Specifically, the standard ZODB.blob.FileSystemHelper class creates secure directories (only readable and writable for the current user) by default.

You could provide your own implementation of FileSystemHelper that would either make this configurable, or just sets the directory modes to 0750, and then patch ZODB.blob.BlobStorageMixin to use your class instead of the default:

import os
from ZODB import utils
from ZODB.blob import FilesystemHelper, BlobStorageMixin
from ZODB.blob import log, LAYOUT_MARKER

class GroupReadableFilesystemHelper(FilesystemHelper):
    def create(self):
        if not os.path.exists(self.base_dir):
            os.makedirs(self.base_dir, 0750)
            log("Blob directory '%s' does not exist. "
                "Created new directory." % self.base_dir)
        if not os.path.exists(self.temp_dir):
            os.makedirs(self.temp_dir, 0750)
            log("Blob temporary directory '%s' does not exist. "
                "Created new directory." % self.temp_dir)

        if not os.path.exists(os.path.join(self.base_dir, LAYOUT_MARKER)):
            layout_marker = open(
                os.path.join(self.base_dir, LAYOUT_MARKER), 'wb')
            layout_marker.write(self.layout_name)
        else:
            layout = open(os.path.join(self.base_dir, LAYOUT_MARKER), 'rb'
                          ).read().strip()
            if layout != self.layout_name:
                raise ValueError(
                    "Directory layout `%s` selected for blob directory %s, but "
                    "marker found for layout `%s`" %
                    (self.layout_name, self.base_dir, layout))

    def isSecure(self, path):
        """Ensure that (POSIX) path mode bits are 0750."""
        return (os.stat(path).st_mode & 027) == 0

    def getPathForOID(self, oid, create=False):
        """Given an OID, return the path on the filesystem where
        the blob data relating to that OID is stored.

        If the create flag is given, the path is also created if it didn't
        exist already.

        """
        # OIDs are numbers and sometimes passed around as integers. For our
        # computations we rely on the 64-bit packed string representation.
        if isinstance(oid, int):
            oid = utils.p64(oid)

        path = self.layout.oid_to_path(oid)
        path = os.path.join(self.base_dir, path)

        if create and not os.path.exists(path):
            try:
                os.makedirs(path, 0750)
            except OSError:
                # We might have lost a race.  If so, the directory
                # must exist now
                assert os.path.exists(path)
        return path


def _blob_init_groupread(self, blob_dir, layout='automatic'):
    self.fshelper = GroupReadableFilesystemHelper(blob_dir, layout)
    self.fshelper.create()
    self.fshelper.checkSecure()
    self.dirty_oids = []

BlobStorageMixin._blob_init = _blob_init_groupread

Quite a hand-full, you may want to make this a feature request for ZODB3 :-)

In ZEO setup I should patch ZEO server, right? Did some testing with a server and a client and looks like monkey-patch on the client side was not run (tried upload a file and nothing happened). — Mikko Ohtamaa, May 29 '11 at 20:05
@Mikko: In a ZEO setup, you really want to patch both the ZEO server and the clients. Presumably you have `shared-blob-dir` set to True, in which case both the clients and the server will be working with the blob data directory. In any case, you may first want to test this on a stand-alone setup. Also, an existing blob datadir structure will have to have it's permissions updated (chmod -R g+r), as this patch will not alter existing directories, only newly created directories. — Martijn Pieters, May 29 '11 at 20:24
Thanks Martjin. Here is work-in-progress: https://github.com/miohtama/collective.xsendfile — Mikko Ohtamaa, May 29 '11 at 22:54
@Mikko: I'd love to see benchmarks to show X-SendFile is actually going to be any faster. I am somewhat sceptical, since serving a blob file is very, *very* efficient in Zope. The original request thread is closed, and the file streaming is handled entirely by the underlying async polling implementation, Zope / Plone doesn't have to do anything anymore once the blob file has been located. Apache and nginx will do exactly the same. — Martijn Pieters, May 30 '11 at 06:12
@Mikko: there already is an open ticket for this in the ZODB tracker: https://bugs.launchpad.net/zodb/+bug/683751 — Martijn Pieters, May 30 '11 at 09:56
@Martijn: X-SendFile can be good if you have multiple servers and don't use a shared blob dir. If your database and frontend web server are on the same host, or the frontend web server can get readonly access to the blob data, you avoid duplicating the entire blob data in the blob caches on the other servers. Setting up read-only access to the blob data might be easier than setting up read-write access. — , Jun 03 '11 at 21:32
@Hanno: If you do not use a shared bob dir, you *cause* the data to be shared between all the servers in the blob cache. Only with shared set to True do you prevent this scenario, in which case you are back to the original premise: you save the overhead of the ProxyPass setup only. — Martijn Pieters, Jun 04 '11 at 09:52
@MJ: No :) Most of the application servers won't download any of the blob data to the cache. It's only the blobs that are newly created that will be in there. And you can run the blob cache cleanup to reduce that size. Quite often a user requesting an image, will cause the appserver to first get the blob from the central ZEO server, stream it over the ZEO connection to the local cache, store it on disk and then sent it back over the internal network through the proxies to finally go to the user. In this setup you can save two network roundtrips and a disk copy by sending it directly. — , Jun 06 '11 at 21:12
@Hanno: that's the operation of the non-shared blob setup; in a shared-blob setup the client does not stream it over a ZEO connection. :-) — Martijn Pieters, Jun 06 '11 at 21:21

score 2 · Answer 2 · answered Jun 17 '11 at 08:20

While setting up a backup routine for a ZOPE/ZEO setup, I ran into the same problem with blob permissions.

After trying to apply the monkey patch that Mikko wrote (which is not that easy) i came up with a "real" patch to solve the problem.

The patch suggested by Martijn is not complete, it still does not set the right mode on blob files.

So here's my solution:

1.) Create a patch containing:

Index: ZODB/blob.py
===================================================================
--- ZODB/blob.py    (Revision 121959)
+++ ZODB/blob.py    (Arbeitskopie)
@@ -337,11 +337,11 @@

     def create(self):
         if not os.path.exists(self.base_dir):
-            os.makedirs(self.base_dir, 0700)
+            os.makedirs(self.base_dir, 0750)
             log("Blob directory '%s' does not exist. "
                 "Created new directory." % self.base_dir)
         if not os.path.exists(self.temp_dir):
-            os.makedirs(self.temp_dir, 0700)
+            os.makedirs(self.temp_dir, 0750)
             log("Blob temporary directory '%s' does not exist. "
                 "Created new directory." % self.temp_dir)

@@ -359,8 +359,8 @@
                     (self.layout_name, self.base_dir, layout))

     def isSecure(self, path):
-        """Ensure that (POSIX) path mode bits are 0700."""
-        return (os.stat(path).st_mode & 077) == 0
+        """Ensure that (POSIX) path mode bits are 0750."""
+        return (os.stat(path).st_mode & 027) == 0

     def checkSecure(self):
         if not self.isSecure(self.base_dir):
@@ -385,7 +385,7 @@

         if create and not os.path.exists(path):
             try:
-                os.makedirs(path, 0700)
+                os.makedirs(path, 0750)
             except OSError:
                 # We might have lost a race.  If so, the directory
                 # must exist now
@@ -891,7 +891,7 @@
             file2.close()
         remove_committed(f1)
     if chmod:
-        os.chmod(f2, stat.S_IREAD)
+        os.chmod(f2, stat.S_IRUSR | stat.S_IRGRP)

 if sys.platform == 'win32':
     # On Windows, you can't remove read-only files, so make the

You can also take a look at the patch here -> http://pastebin.com/wNLYyXvw

2.) Store the patch under name 'blob.patch' in your buildout root directory

3.) Extend your buildout configuration:

parts += 
    patchblob
    postinstall

[patchblob]
recipe = collective.recipe.patch
egg = ZODB3
patches = blob.patch

[postinstall]
recipe = plone.recipe.command
command = 
    chmod -R g+r ${buildout:directory}/var
    find ${buildout:directory}/var -type d | xargs chmod g+x
update-command = ${:command}

The postinstall sections sets desired group read permissions on already existing blobs. Note, also execute permission must be given to the blob folders, that group can enter the directories.

I've tested this patch with ZODB 3.10.2 and 3.10.3.

As Martijn suggested, this should be configurable and part of the ZODB directly.

You really do not want to do a direct patch on an egg. Use a monkeypatch instead. — Martijn Pieters, Jun 17 '11 at 08:32

collective.xsendfile, ZODB blobs and UNIX file permissions

2 Answers2