0

I'm running into out of memory issues when I attempt to decode MessagePack files using the msgpack-lite library in the browser. When run, the browser tab crashes (Chrome shows an 'Awww Snap!' error page). If I run the app with the developer tools open and set some breakpoints, it stops occasionally with a warning of a potential out of memory error. I can then continue execution and the decoding completes successfully. This leads me to believe the garbage collector needs time to run.

I'm wondering if there are any known issues around types of files or ways of interacting with the msgpack-lite library that affect this? Does the organization of file contents affect memory requirements?

In the application I'm using Angular/RXJS and calling msgpack.decode(new Uint8Array()):

this.http.get(url, {responseType: 'arraybuffer'}) .map((response: ArrayBuffer) => BaseService.msgpack.decode(new Uint8Array(response)))

One of the files I'm having problems with is available here: https://file.io/SYxHDb

Any help with this is greatly appreciated!

Brian
  • 563
  • 1
  • 5
  • 15
  • When memory is an issue, you really need to approach the problem differently. You can't decode it all at the same time as that appears to be the issue. Break it apart into smaller bits and decode them individually. Better yet, follow the process of 1) Take an item 2) Decode it 3) Use it 4) Dispose of it. – Slava Knyazev May 21 '18 at 02:10
  • Your link is a 404. – Patrick Roberts May 21 '18 at 02:12
  • The 404 is fixed with a new file – Brian May 21 '18 at 14:28
  • @SlavaKnyazev I'd really like to use streams to parse the file, but msgpack-lite uses Node streams. I don't see how to use it in the browser – Brian May 21 '18 at 14:48
  • You might not need streams. You can just iterate over the array with a loop. – Slava Knyazev May 21 '18 at 14:52
  • The response ArrayBuffer is just the binary encoded message pack file. Can you explain what you mean about iterating over it? – Brian May 21 '18 at 18:15
  • I assume the memory problems stem from the `msgpack.decode()` method call, right? The idea is to iterate over the `Uint8Array` an decode each item individually and only at the end, reassemble all the decoded things. I have no idea what you're working with so I'm stabbing in the dark. – Slava Knyazev May 21 '18 at 18:47
  • Right, but that would require knowing the intimate details of the messagepack format in order to know where it's legal to split the input. At that point I'm pretty much writing my own decoder... – Brian May 22 '18 at 14:27

0 Answers0