Creating meta-data for binary chunks for sending via WebRTC datachannel

Question

I have a datachannel connection between two browsers, and would like to break a file into chunks and send them to/from the clients.

I can read the file and break it up into chunks just fine. However I need a way for the receiving client to know

1. which file the chunk of data relates to (unique identifier).

2. which place the chunk applies to in reconstruction (index number).

When transferring binary data in the browser, it seems the entire payload must be binary. So I can't, for example, create a JSON object with the above properties, and have a data property with the actual binary chunk.

I guess I need to wrap the file chunk into a secondary binary blob which contains the identifier and index. The receiving client would then decode the first, wrapper, chunk to check the meta-data, then handle the actual file chunk based on that information.

How can I do this in the browser? I have done a lot of google searching but can't seem to find any information on this, so wonder if I'm perhaps overlooking something which can help ease this process?

Answer 1

You have to create your own protocol for transfering files.

1. I assume you have a File/Blob object. You probably also use split() method to get chunks.

2. You can simply use a Uint8Array to transfer data.

   1. Create a protocol that satisfies your needs, for example:

      • 1 byte: Package type (255 possible package types)
      • 2 bytes: Length of data (2^16 bytes ~ 64KB of data per chunk)
      • n bytes: <Data>

   2. Send an initial package (e.g. type 0x01)

      • Data contains some information (all or some):
        • Total length of blob/file
        • file type
        • chunk size
        • number of chunks
        • filename
        • ...

   3. Send Chunks of data (e.g. type 0x02)

      • You should use at least two bytes for a sequence number
      • Data follows afterwards (no length needed because you know the total length)

Note: If transfering multiple files, you should add a id or something else.

On the receiver side you can wait for the initial package and and create a new Uint8Array with length of the total file. Afterwards you can use set() to add received data at the chunk position (offset = 0-based-chunk-number*chunk-size). When all chunks are received, you can create the Blob.

Answer 2

In addition to @Robert's very good answer, you can use channel.send(blob) (at least in Firefox<->Firefox). Eventually this should work in Chrome as well.

Answer 3

If it is a simple matter of multiple files, you could just create a new data channel for each new file.

Each channel will take care of it's own buffering, sequence etc.

Something like:

chan = peerCon.createDataChannel("/somedir/somefile", props);

then break your file into <64k chunks and chan.send() them in sequence.

The receiving side can get the label and use it to save the file appropriately

peerCon.ondatachannel = function(channel) {
     console.log("New file " + channel.label);
     channel.onmessage = function(

etc.

P.S. If you really must use a file system protocol over a single channel (say because you want random access behaviour) don't invent a new one, use one that already exists and is tested - I'm fond of 9p from inferno/plan9