append new msgpack data to end of file

Question

Is it valid to append objects to msgpack data without decoding it first?

I.e something like that:

encode data1
pack data1
write packed data1 to file

Then,

pack data2
append packed data2 to end of file

or should it be:

read file
msgunpack file -> data1
create data1_2 (including data1,data2)
pack data1_2
write packed data1_2 to file

I have the same question, but for python. – Diego Pamio Sep 13 '17 at 03:36 — Diego Pamio, Sep 13 '17 at 03:36

score 1 · Accepted Answer · edited May 23 '17 at 10:33

1

You can just append to the file. Look at How to unpack a msgpack file? for my sample code for reading an writing, you will see that I am appending as well, and this works.

edited May 23 '17 at 10:33

Community

1
1

answered Aug 14 '15 at 10:20

Alex

871
7
23

1

So if you have an array encoded in msgpack, you can extend it just by adding the new msgpack data to the end of the file? – noɥʇʎԀʎzɐɹƆ Nov 23 '16 at 00:30

score 0 · Answer 2 · answered Apr 22 '21 at 16:08

Expanding upon Alex's answer, since the spec still doesn't elucidate this, and I wasn't sure how to translate that paradigm to other libraries. I'm not sure if this is part of the official spec, but it's the least surprising way to implement, and it makes intuitive sense. It's extremely useful if you want an append-only data format.

You ought to be able to simply concatenate any packed msgpack objects. Your library should provide some kind of Unpacker class, which takes a byte stream comprising one or more concatenated msgpack objects, and emits a stream of unpacked structures. If your library lacks this feature, you probably have to implement it yourself.

Here's what that looks like in python's msgpack package:

from io import BytesIO
import msgpack as mp

buf = BytesIO()
buf.write(packer.pack(1))
buf.write(packer.pack(2))
buf.write(packer.pack('foo'))
buf.write(packer.pack(True))
buf.write(packer.pack({"spam":"eggs"}))

print('file contents: {}'.format(buf.getvalue()))
buf.seek(0)

unpacker = mp.Unpacker(buf)
for i, obj in enumerate(unpacker):
    print('{}: {} {}'.format(i, repr(obj), type(obj)))

Out:

file contents: b'\x01\x02\xa3foo\xc3\x81\xa4spam\xa4eggs'
0: 1 <class 'int'>
1: 2 <class 'int'>
2: 'foo' <class 'str'>
3: True <class 'bool'>
4: {'spam': 'eggs'} <class 'dict'>

If you wish to implement your own, the python fallback Unpacker is quite readable.

append new msgpack data to end of file

2 Answers2