0

I have a Firestore project where my collection feed has a timestamp field createdDate. As I am querying the feeds based on the created time with other fields, I have created a Composite Index including the timestamp field.

In the Indexing best practices document the following is mentioned.

If you index a field that increases or decreases sequentially between documents in a collection, like a timestamp, then the maximum write rate to the collection is 500 writes per second. If you don't query based on the field with sequential values, you can exempt the field from indexing to bypass this limit.

As I am already using the timestamp field in a Composite Index, I can exempt the field from Automatic Index Settings. Will this bypass this limit of 500 writes per second in the collection or it will still be applied as the Composite Index is using the field. Please help me understand this.

Tapas Mukherjee
  • 2,088
  • 1
  • 27
  • 66

1 Answers1

0

The limit comes from the existence of hotspots in any index in the collection. While Firestore can update all indexes in parallel, its throughput is limited by the slowest index. Removing the hotspot from one index, but leaving it on another, means there is no increase in write throughput for the collection.

Note that this is not a hard-coded limit, but it comes from the physical limitations of having to update clustered files across multiple data centers.

Frank van Puffelen
  • 565,676
  • 79
  • 828
  • 807
  • Thanks for the explanation @Frank. I am almost getting it. Now using a timestamp or increasing numeric values is a very common scenario and a timestamp is used mainly so that we can query with it. So defiantly Indexing needs to be done. Is there any suggesion or best practice for my scenario where collections like Feed will have writes by thousands of users and I need to have a timestamp? – Tapas Mukherjee Aug 26 '20 at 21:53
  • Also, I want to understand when the document says timestamp which is 'increases or decreases sequentially'. A collection like Feed will have a high number of documents created with an increasing timestamp but they are defiantly not sequential or 'monotonically increasing'. – Tapas Mukherjee Aug 26 '20 at 22:43
  • I see most developers overestimating the need for their users to write to the same location/collection, so I'd first look into that if I were you. Thousands of users, sure thing, but thousands **per second**? If you really do, centralize the writes through a proxy that debounces them. Some of the largest apps in the world are using Firestore or its ancestor Datastore for handling *huge* volumes, it's a matter of coming up with an architecture that supports your write and read volume. – Frank van Puffelen Aug 26 '20 at 22:44
  • Thanks for the advice @Frank. I also found this document which I think can help me https://firebase.google.com/docs/firestore/solutions/shard-timestamp – Tapas Mukherjee Aug 26 '20 at 23:30
  • Nice. I didn't know we had a doc for sharded timestamps, or I definitely would've pointed to that straight away. Thanks for sharing! – Frank van Puffelen Aug 26 '20 at 23:33