10

I am building a file storage for HTML5, and I am using indexedDB as the storage, I ask the files from the server via xmlHttpRequest with the response type as arrayBuffer (for chrome) and blob (for other browsers).

Everything is fine even if the files-collection size is 500MB or more, (hey, it can even reach GB). But I noticed something strange when I add the file to the indexedDB, it will trigger error when the single file exceeds ~120MB, so it is not stored. But when the file is less than 120MB, it will store it.

Notice that it will only have this error when storing a single file > 120MB, for example, an .mp4 file of 200MB will trigger an error, but if I have 5 videos with each of them have a size of 100MB (so the total will be 500MB) it will be all fine.

I would like to know whether this is a limit-rule or some glitch and the two have the same error. I didn't find any documentation about it. I tested it in IE and Chrome.

EDIT: Ok, I got this error apparently in the add or put function of indexedDB when storing the file:

inside the e.target.error.message:

The serialized value is too large (size=140989466 bytes, max=133169152 bytes)

Eldon Lesley
  • 915
  • 6
  • 19
  • 38
  • Why are you storing files in a database? Databases are absolutely not suited for file storage. Especially not for large files. – Cerbrus Jun 23 '14 at 09:49
  • these are dynamic files so cache manifest is not possible. WebSQL is not developed anymore, and filesystem API is only on Chrome – Eldon Lesley Jun 23 '14 at 09:56
  • 6
    @Cerbrus: Presumably this person is storing files in IndexedDB because there is no place to store them other than in IndexedDB (moving data up to a server is not always possible or even desirable). Since Mozilla and the W3C were uninterested in the proposed Chrome HTML5 filesystem, instead advocating use of indexedDB, they should expect developers to attempt to store large amounts of data using IndexedDB, and therefore it should be possible to do so without running into an arbitrary limit that forces said developers to construct a hack on top of IndexedDB to deal with said limits. – Eric Aug 22 '18 at 16:32
  • 1
    We are also facing the same issue. We have to support offline scenario using IndexedDB. We need to make some files available offline hence have to store in IndexedDB. Is this max=133169152 bytes hard coded in Chrome? – Joy George Kunjikkuru Oct 22 '18 at 22:13
  • 1
    This question just came up in my search for the same problem. I need to store a user video capture while offline in idb. I think @Eric has a valid point, with no options for storage, size limitations imposed by idb make programmers do hacky coding to solve rather simple problems. – Philip Attisano May 31 '19 at 13:33

2 Answers2

2

At the time this question was asked, Chrome still didn't support saving Blobs to IndexedDB, it only came the next month.

For anyone facing the same issue nowadays, store Blobs or Files directly, not ArrayBuffers.

Contrarily to ArrayBuffers, saving a Blob to IDB doesn't require to serialize its data, the serialization steps of a Blob are just to make a snapshot of the js object and keep the link to the same underlying byte sequence, which itself is not cloned.
The only limit you should face would be the one of the db itself.

Code taken from Chrome's announcement:

var store = db.transaction(['entries'], 'readwrite').objectStore('entries');

// Store the object  
var req = store.put(blob, 'blob');
req.onerror = function(e) {
    console.log(e);
};
req.onsuccess = function(event) {
    console.log('Successfully stored a blob as Blob.');
};
Kaiido
  • 123,334
  • 13
  • 219
  • 285
0

I think this is an issue with your browser's implementation of IndexedDB. I ran into this same error myself in Firefox, when I tried to store a 100 MB file into a IndexedDB record, but the identical code worked fine in Chrome. It seems different browsers have different implementation quirks and limits.

Personally, I suspect this is a bug in Firefox, since Firefox grants the requested size, but then prevents single-record usage of that entire size, whereas Chrome is more forgiving.

Cerin
  • 60,957
  • 96
  • 316
  • 522