How can I append data to an Object in OpenStack Swift Object Storage?

Question

currently want to stream realtime IoT data into OpenStack Swift, but it seems there is no support for doing this - also the API seems to support only atomic pushes.

Does anyone have any other experience or any workaround for that?

Thanks a lot!

Answer 1

Consider Dynamic Large Objects (DLO) and Static Large Object (SLO) manifests. You can use a number of smaller objects to construct a large object via the concept of segmentation. From OpenStack Large Object Support, “Segments of the larger object are uploaded and a special manifest file is created that, when downloaded, sends all the segments concatenated as a single object.” OpenStack Swift Object Storage does not support appending to a object per se, however it does provide facility for concatenation of existing objects dynamically and statically.

For example, you can define a filename pattern rawdata_ and establish a DLO manifest within your Object Storage container. Whenever you issue a get for this manifest, OpenStack swift will automatically concatenate all objects matching the defined filename pattern and return it as a single result. This manifest object can also be loaded via the Bluemix Spark service notebooks (e.g. swift://...) to facilitate analytics on the data.

While manifests are primarily used to support > 5 Gb objects, I've found them quite useful to slice and dice objects to construct different amounts of raw data or to handle batch processes that continually add raw data to a given container.

I've written a blog post on Object Storage manifests for the Bluemix service @ http://blog.ibmjstart.net/2016/04/14/e-pluribus-unum-creating-openstack-manifest-objects-in-ibm-bluemix-object-storage/ that may also help shed more light on their usefulness.

Hope this helps and good luck.

Answer 2

Currently no support in OpenStack Swift for appending to objects. There is this blueprint in the community: https://blueprints.launchpad.net/swift/+spec/object-append.