Use GridFS on the local disk with C++

Question

I'm looking for a solution that let's me store some nested filestructure (directories, files) in a single file. I thought about simple archives like tarballs, but due to many operations it's not appropriate.

Than I thought about JSON and store binary files into it, as base64 encoded strings. Then I stumbled upon BSON and its limitation to 16MB per file. Then again found GridFS as a wrapper that splits larger files in to smaller chunks.

That's perfect for my aims, because 16MB chunks would fit perfectly.

Because of the decentral alignment of my project I can't use a central DB like Mongo. Is it possible to use GridFS for a single local file like storage.bsonor something, where I can store, well, my files?

GridFs is part of MongoDb. Why not just use the file system? It stores files very well. :) — WiredPrairie, Feb 14 '14 at 01:29
This is a revolutionary idea. Thank you for opening my eyes :) But no, my situation depends on a one-file-solution. — Chris Pillen, Feb 14 '14 at 09:38

score 1 · Accepted Answer · answered Feb 14 '14 at 12:53

GridFs uses the storage system of MongoDb documents to store small chunks (256K) of files. It's not an aspect of MongoDb that can be split off to be used independently.

MongoDb also, by default, creates data storage files that are 2GB. So, even when using MongoDb, the files wouldn't be stored in a single file. They would be split into multiple data files. File splits like that are not uncommon in databases.

Also, BSON is not limited to 16MB as a storage format. The 16MB limit is imposed by the MongoDb engine. BSON documents could nearly store 2GB of data per specification.

And there is a stand-alone implementation in C++: https://github.com/jbenet/bson-cpp — Chris Pillen, Feb 20 '14 at 14:07

Use GridFS on the local disk with C++

1 Answers1