Error while using paged query with Active Directory

Question

I want to make paginated LDAP queries to my Active Directory, and I am using SimplePagedResultsControl of UnboundID SDK. As per my use cases, I have 2 types of filters -

1. Querying users: (&(objectClass=user)(memberof=cn=group1,dc=a,dc=com)(memberof=cn=group2,dc=a,dc=com)(...)...)
2. Querying groups: (&(objectclass=group)(cn=group1)(cn=group2)(...)...)

Simplified code -

   private SearchResponse makeQuery(
    SearchRequest request, ConnectionMetadata metadata) {
      FullLDAPInterface connection = 
         ConnectionConstructor.getConnection(metadata);

      return connection.search(request);
   }

// Calling above method in another class.

   private SearchResponse ldapQuery(String baseDn, LdapScope scope, String ldapFilter, String[] attributes, ASN1OctetString cookie) {
      // metadata consists of server IP, host, port, certificate.
      ConnectionMetadata connectionMetadata = getMetadata();
      SearchRequest request = 
         new SearchRequest(
            baseDn,
            scope,
            ldapFilter,
            attributes
         );

      Control pageControl = 
         new SimplePagedResultsControl(
            100 /*pagesize*/,
            cookie,
            true /*criticality*/ 
         );

       request.addControl(pageControl);
       return makeQuery(request, connectionMetadata);
   }

   public void pagedLdapQuery(String baseDn, LdapScope scope, String ldapFilter, String[] attributes) {
       SearchResponse response = ldapQuery(baseDn, scope, ldapFilter, attributes, new ASN1OctetString(""));
       
        ASN1OctetString cookie = SimplePagedResultsControl.get(response).getCookie();
        while (cookie.getSize() > 0) {
            response = ldapQuery(baseDn, scope, ldapFilter, attributes, new cookie);
            cookie = SimplePagedResultsControl.get(response).getCookie();
        }
}

Have been seeing some strange behaviour where the first page has the expected number of objects, but with a page cookie of 4 bytes only. Generally I have seen it to be several hundred bytes.

On querying for the second page using the same query as before and the 4 byte cookie, the response consists of all the objects that match the filter(which are lot more than page size, but less than 1000 (size limit)). As per my understanding, AD is acting as if the cookie was not provided (I have confirmed that the cookie is valid, but does not work).

My hunch is that this is happening because we are opening a new connection for querying EACH page of the query, and probably the cookies are not valid across connections. But it seems strange that the cookie is not valid across multiple connections, and I have not been able to find any supporting documentation for this.

I have tried setting MaxResultSetsPerConn = 10 (was 0 earlier) and MinResultSets = 4 (was 0 earlier) (Ref). But pagination is still not working correctly for me.

Another observation: The LDAP query does not always return non-paginated responses. It changes with the number of memberof or cn components in the query. For example:
(&(objectClass=user)(memberof=cn=group1,dc=a,dc=com)(memberof=cn=group2,dc=a,dc=com) works correctly, and a cookie of several hundred bytes is returned.

but (&(objectClass=user)(memberof=cn=group,dc=a,dc=com)(memberof=cn=group2,dc=a,dc=com)(memberof=cn=group3,dc=a,dc=com)) behaves as if the page cookie was not provided at all from the second page onwards.

How can I make this work? Are there any other AD settings that can be tuned?