Javascript reading local file to UInt8Array fast

Question

How to read local binary file to UInt8Array fast. in below code

function readAllBytesAsUInt8Array(path) {
    var req = new XMLHttpRequest();
    req.open("GET", path, false);
    req.overrideMimeType("text/plain; charset=binary-data");
    req.send(null);
    if (req.status !== 200) {
        console.log("error");
        return null;
    }
    var text = req.responseText;
    var resultArray = new Uint8Array(text.length);
    for(var i = 0; i < text.length;i++){
    resultArray[i] = (text[i].charCodeAt() & 255) & 255;
    }
    return resultArray.buffer;
 }

var text = req.responseText; is executed less than a second,meanwhile this part

var resultArray = new Uint8Array(text.length);
for(var i = 0; i < text.length;i++){
    resultArray[i] = (text[i].charCodeAt() & 255) & 255;
}

takes around 10sec for 50MB of binary file, Is there a way to read binary file to UInt8Array faster ?

just wondering, do you really need to do `var resultArray = new Uint8Array(text.length);`. ? JavaScript doesn't ask for you to predefine the array size for it. just say `resultArray = [];` and fill it just like you fill yours — Dellirium, Sep 21 '17 at 07:48
Why do you not set `.responseType` of `XMLHttpRequest()` to `"arraybuffer"`? — guest271314, Sep 21 '17 at 07:51
it will be more slower as i push each time to the array it will have to reallocate larger memory for every byte appending — user818117, Sep 21 '17 at 07:51
@guest271314 then what is the right way to convert from arraybuffer to uint8array ? var z = new Uint8Array(buffer) ? — user818117, Sep 21 '17 at 07:52
Have you tried using `FileReaderSync()` and transferring the object to avoid copying the object? — guest271314, Sep 21 '17 at 07:52
@ guest271314 it says This interface(FileReaderSync) is only available in workers as it enables synchronous I/O that could potentially block. — user818117, Sep 21 '17 at 07:55

score 3 · Answer 1 · answered Sep 21 '17 at 07:56

3

You can set .responseType of XMLHttpRequest() to "arraybuffer", then pass ArrayBuffer instance to new Uint8Array(). Alternatively you can use fetch() and Response.arrayBuffer() or FileReaderSync() within Worker to transfer the ArrayBuffer to main thread without copying the ArrayBuffer.

req.responseType = "arraybuffer";
let buffer = req.response;
let u = new Uint8Array(buffer);

answered Sep 21 '17 at 07:56

guest271314

1
15
104
177

are `req.response` and `(new Uint8Array(buffer)).buffer` the same thing ? as the method returns `return resultArray.buffer;` – user818117 Sep 21 '17 at 08:03
Yes, `req.response` is an `ArrayBuffer` – guest271314 Sep 21 '17 at 08:06
let u = new Uint8Array(buffer); throws exceptions as Uint8Array() expects first argument as length – user818117 Sep 21 '17 at 10:48
@user818117 No exception is thrown here at either Firefox or Chromium when passing an `ArrayBuffer` to `Uint8Array()`. At which browser was an exception thrown? Can you reproduce the exception at jsfiddle https://jsfiddle.net? – guest271314 Sep 21 '17 at 13:31

score 0 · Answer 2 · answered Sep 21 '17 at 07:48

You can use a TextEncoder! TextEncoder takes a stream of code points as input and emits a stream of bytes.

Here is your new code :)

function readAllBytesAsUInt8Array(path) {
    var req = new XMLHttpRequest();
    req.open("GET", path, false);
    req.overrideMimeType("text/plain; charset=binary-data");
    req.send(null);
    if (req.status !== 200) {
        console.log("error");
        return null;
    }
    var text = req.responseText;
    var encoder = new TextEncoder("utf-8");
    var resultArray = encoder.encode(text);
    return resultArray.buffer;
 }

See how that works, should be a lot faster.

textencoder doesnt work on some browsers like IE,Safari, meanwhile i tested it has the similar performance with above code — user818117, Sep 21 '17 at 07:53

Javascript reading local file to UInt8Array fast

2 Answers2