How can I deal with the firestore whereIn limit of 10 when using a switch stream?

Question

I have the following Stream that takes in a Stream of a group and returns a Stream of it's members. I use a switchmap to get the members from the group snapshot.

But I have the following problem. I use a where query with a whereIn filter. But the problem is that the whereIn can only take in a list that have 10 entries or less according to the firestore documentation

Limitations Note the following limitations for in and array-contains-any:

in and array-contains-any support up to 10 comparison values.

https://firebase.google.com/docs/firestore/query-data/queries#limitations

So I am having some difficulties with dealing with that in this senario.

  Stream<List<UserModel>> groupMembersStream(Stream<GroupModel> groupStream) {
    return groupStream.switchMap(
      (value) => _fireStore
          .collection(APIRoutes.users)
          .where(FieldPath.documentId, whereIn: value.members.keys.toList(growable: false))
          .snapshots()
          .map((snapshot) =>
              snapshot.documents.map((document) => UserModel.fromFirestore(document)).toList(growable: false)),
    );
  }

Because I need the group member id's to start with so I need a switchMap. So I cannot just simply split up the group members list and then do a seperate query for each chunk of 10 id's.

So how do I deal with this?

score 5 · Answer 1 · answered Nov 12 '20 at 21:43

Another possible solution (using Quiver & RxDart packages)

  Future<List<QueryDocumentSnapshot>> listItems(
      List<dynamic> itemIds) async {
    final chunks = partition(itemIds, 10);
    final querySnapshots = await Future.wait(chunks.map((chunk) {
      Query itemsQuery = FirebaseFirestore.instance
          .collection('collection')
          .where("id", whereIn: chunk);
      return itemsQuery.get();
    }).toList());
    return querySnapshots == null
        ? []
        : await Stream.fromIterable(querySnapshots)
            .flatMap((qs) => Stream.fromIterable(qs.docs))
            .toList();
  }

Looks more generic this way (you can even factor out the query using a supplier function).

score 2 · Accepted Answer · answered May 24 '20 at 14:13

2

I ended up doing it like this

  Stream<List<UserModel>> groupMembersStream(Stream<GroupModel> groupStream) {
    return groupStream.switchMap(
      (value) => _chunckSizeGroupMembersStream(value),
    );
  }

  Stream<List<UserModel>> _chunckSizeGroupMembersStream(GroupModel group) {
    final List<List<String>> memberChunks = chunkSizeCollection(group.members.keys.toList(growable: false), 10);
    List<Stream<List<UserModel>>> streams = List<Stream<List<UserModel>>>();
    memberChunks.forEach((chunck) => streams.add(_fireStore
        .collection(APIRoutes.userCollection)
        .where(FieldPath.documentId, whereIn: chunck)
        .snapshots()
        .map((snapshot) =>
            snapshot.documents.map((document) => UserModel.fromFirestore(document)).toList(growable: false))));
    return ZipStream(streams, (value) => value.last);
  }

answered May 24 '20 at 14:13

anonymous-dev

2,897
9
48
112

what is `chunkSizeCollection` ? – Oleksandr Yefymov Jan 17 '21 at 01:50
The Firestore whereIn has a limit of 10 elements. The chunkSizeCollection just splits the array in chunks of 10 elements. – anonymous-dev Jan 17 '21 at 13:06
Mike, I adapted this code and I'm getting a throw The method 'add' was called on null. Receiver: null Tried calling: add(Instance of '_MapStream>'). The .add needs an await...are you not getting that? The problem I'm running into is the await requires a future that shortcircuits the stream. wondering if maybe I'm missing something. – jbryanh Feb 14 '21 at 00:49

How can I deal with the firestore whereIn limit of 10 when using a switch stream?

2 Answers2

Linked