0

Problem:

I'm building an app where I want to display a list of 50k locations. Knowing that the size of geographical point is 16 bytes, I decided to store all of them in a single document.

The location where I store this list is:

db.collection("points").document("geo");

I know that the size of a document is composed of three components:

  1. document name size
  2. The sum of the string size of each field name
  3. The sum of the size of each field value
  4. 32 additional bytes

So in my case, I have:

  1. (6 + 1) + (3 + 1) + 16 = 27
  2. 9 + 1 = 10 as the name of the field is geoPoints
  3. 50,000 * 16 = 800,000
  4. 32 additional bytes

So in my case the total is 800,069 bytes which less than 1,048,576 bytes as it is mentioned in the docs. If I try to add this list to the above document, I get:

INVALID_ARGUMENT: A document cannot be written because it exceeds the maximum size allowed.

However, I'm not able to add less than that. I made some tests and I could only write a number of 40,327 GeoPoints, meaning:

  1. (6 + 1) + (3 + 1) + 16 = 27
  2. 9 + 1 = 10 as the name of the field is geoPoints
  3. 40,327 * 16 = 645,232
  4. 32 additional bytes

So a total of 27 + 10 + 645,232 + 32 = 645,301 bytes, which is much less than what says in the docs.

If I try to write 40,328 instead of 40,327, I get the same error.

Question:

How to store the total of 1,048,576 bytes, as it is mentioned in the docs?

Doug Stevenson
  • 297,357
  • 32
  • 422
  • 441
Pathis Patel
  • 179
  • 1
  • 11
  • Are you sure you are using the right code/list? Are you also sure about the number of objects and types that exist on your list? – Alex Mamo Apr 08 '20 at 19:25
  • @AlexMamo Yes, I'm 100% sure. I tried it several times. Try it your self. – Pathis Patel Apr 09 '20 at 08:14
  • The calculation is incorrect `9 + 1 = 10 as the name of the field is geoPoints` is only for one stored field. You have to do that for every stored field. It's an array so field 0 is named '0' which is one byte. Field 1 is named '1' which is another byte (two total). Field 10 is named '10' which is two bytes, field 100 is named '100' which is three bytes. Etc etc. – Jay Apr 11 '20 at 15:11
  • @Jay There is nowhere specified that. Even so, if I add 1 byte for items from 0-9, 2 bytes for items from 10-99 and so on, it does **not** get closer. please see the last part of my [question](https://stackoverflow.com/questions/61107889/what-is-the-exact-size-of-a-firestore-document). – Pathis Patel Apr 11 '20 at 15:55
  • See this *The sum of the string size of each field name* and then *String sizes are calculated as the number of UTF-8 encoded bytes + 1.* So the calculations are still not correct. If you have 40,327 nodes, the fieldnames themselves will take up 230852 bytes. – Jay Apr 11 '20 at 16:42
  • @Jay It will still be **only** [`876,153`](https://stackoverflow.com/questions/61107889/what-is-the-exact-size-of-a-firestore-document), which is less than the maximum :( There is for sure a problem in their algorithms or/and docs. – Pathis Patel Apr 11 '20 at 17:08

1 Answers1

0

The document also stores a data type for each field. You can see this in the JSON that is returned from the http api. My guess would be that these data type tags also count against the size limit.

Depending on the access rules for your database you can also see this by accessing a document directly via its url:

https://firestore.googleapis.com/v1/projects/PROJECT_NAME/databases/(default)/documents/COLLECTION_NAME/DOCUMENT_ID

{
"name": "projects/PROJECT_NAME/databases/(default)/documents/COLLECTION_NAME/DOCUMENT_ID",
"fields": {
    "Name": {
        "stringValue": "Hobbies"
    },
    "Active": {
        "booleanValue": true
    }
},
"createTime": "2019-12-02T16:03:07.111185Z",
"updateTime": "2020-03-12T13:44:01.827176Z"

}

Gerry St.Pierre
  • 191
  • 1
  • 9
  • Thank you for takeing the time to answer my question. I will try to get an answer to this [question](https://stackoverflow.com/questions/61107889/what-is-the-exact-size-of-a-firestore-document), and I'll get back. – Pathis Patel Apr 08 '20 at 18:49
  • You may be running into another limitation, such as 40K fields indexed. I had some time to play with code and I am able to create a document with 1000 string fields of 1000 characters each. – Gerry St.Pierre Apr 10 '20 at 11:42
  • No answer so far. I've run some tests with only 100 or 200 properties, same result. Cannot write more than 645,301 bytes. – Pathis Patel Apr 10 '20 at 12:27