CKQueryOperation working with big batch

Question

I have problem with creating CKQuery operation with big batch of data. My query works with 100 results but after more results query fail, because one thread is bad dispatched or something (libdispatch.dylib`dispatch_group_leave: ) i am lost... any idea?

+ (void) fetchAnswersWithRecordId:(CKRecordID *)recordId completionHandler:(CloudKitCompletionHandler)handler {
NSPredicate *predicate = [NSPredicate predicateWithFormat:@"ANSrecordID == %@", recordId];
CKQuery *query = [[CKQuery alloc] initWithRecordType:ckAnswers predicate:predicate];

CKQueryOperation *operation = [[CKQueryOperation alloc] initWithQuery:query];
CKQueryOperation * __weak weakSelf = operation;
operation.resultsLimit = 300;
NSMutableArray *tmp = [[NSMutableArray alloc] init];


operation.recordFetchedBlock = ^(CKRecord *record) {
    [tmp addObject:record];
};

operation.queryCompletionBlock = ^(CKQueryCursor *cursor, NSError *error) {
    if (!handler)
        return;

    NSArray *array = [NSArray arrayWithArray:tmp];
    if(cursor != nil) {
        CKQueryOperation *newOperation = [[CKQueryOperation alloc] initWithCursor:cursor];
        newOperation.recordFetchedBlock = weakSelf.recordFetchedBlock;
        newOperation.completionBlock = weakSelf.completionBlock;
        [[self publicCloudDatabase] addOperation:newOperation];
    } else {
        NSLog(@"Results: %lu", [array count]);
        dispatch_async(dispatch_get_main_queue(), ^{
            handler(array, error);
        });
    }
};

[[self publicCloudDatabase] addOperation:operation];}

score 2 · Answer 1 · answered May 13 '15 at 03:55

I think your issue lies with the __weak operation and the way you create an operation inside another operation. Here is an example (in swift) of how I do something similar i.e. get additional results, but in a fetch and not a query. Note the use of a instance variable to initialize first time and the use of semi-recursion through GCD dispatch_aync:

private func _fetchRecordChangesFromCloud() {
    if !_fetching {
        // this is the first and only time this code is called in GCD recusion

        // we clean the caches we use to collect the results of the fetch
        // so we can then save the record in the correct order so references can be created
        _fetchedModifiedRecords = []
        _fetchedDeletedRecordIDs = []

        // mark fetching has started
        _fetching = true
    }

    let operation = CKFetchRecordChangesOperation(recordZoneID: _customRecordZoneID, previousServerChangeToken: _serverChangeToken)

    operation.recordChangedBlock = { (record: CKRecord?) in
        if let record = record {
            println("Received record to save: \(record)")
            self._fetchedModifiedRecords.append(record)
        }
    }

    operation.recordWithIDWasDeletedBlock = { (recordID: CKRecordID?) in
        if let recordID = recordID {
            println("Received recordID to delete: \(recordID)")
            self._fetchedDeletedRecordIDs.append(recordID)
        }
    }

    operation.fetchRecordChangesCompletionBlock = {
        (serverChangeToken: CKServerChangeToken?, clientChangeToken: NSData?, error: NSError?) -> Void in

        if let error = error {
            println("Error in fetching record changes: \(error)")
            // try again next sync
            self._fetchAfterNextSuccessfullSync = true
            self._fetching = false
            return
        }

        // fetched records successfuly
        println("fetched records successfuly")

        if let serverChangeToken = serverChangeToken {
            self._serverChangeToken = serverChangeToken
        }

        if operation.moreComing {
            // we need to create another operation object and do it again
            dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0)) {
                self._fetchRecordChangesFromCloud()
            }
        } else {
            // we are finally done

            // process the fetched records
            self._processFetchedRecords()

            // save all changes back to persistent store
            self._saveBackgroundContext()

            // we are done
            self._fetching = false
        }
    }

    self._privateDatabase.addOperation(operation)
}

score 2 · Answer 2 · answered Jul 21 '15 at 12:35

FYI, dispatch_async() is not necessary, this is a memory management issue. The following works properly for a multi-batch fetch:

//
__block CKQueryOperation* enumerateOperationActive = nil;

//
NSPredicate* predicate = [NSPredicate predicateWithValue:TRUE];
CKQuery* query = [[[CKQuery alloc] initWithRecordType:@"myType" predicate:predicate] autorelease];

CKQueryOperation* enumerateOperation = [[[CKQueryOperation alloc] initWithQuery:query] autorelease];

// DEBUG: fetch only 1 record in order to "force" a nested CKQueryOperation cycle
enumerateOperation.resultsLimit = 1;

enumerateOperation.recordFetchedBlock = ^(CKRecord* recordFetched)
    {
        // ...
    };

enumerateOperation.queryCompletionBlock = ^(CKQueryCursor* cursor, NSError* error)
    {
        if (error)
        {
            // ...
        }
        else
        {
            if (cursor)
            {
                CKQueryOperation* enumerateOperationNested = [[[CKQueryOperation alloc] initWithCursor:cursor] autorelease];

                // DEBUG: fetch only 1 record in order to "force" a doubly-nested CKQueryOperation cycle
                enumerateOperationNested.resultsLimit = 1;

                enumerateOperationNested.recordFetchedBlock = /* re-used */ enumerateOperationActive.recordFetchedBlock;
                enumerateOperationNested.queryCompletionBlock = /* re-used */ enumerateOperationActive.queryCompletionBlock;

                // CRITICAL: keep track of the very last (active) operation
                enumerateOperationActive = enumerateOperationNested;
                [database addOperation:enumerateOperationNested];
            }
        }
    };

//

// CRITICAL: keep track of the very last (active) operation
enumerateOperationActive = enumerateOperation;
[database addOperation:enumerateOperation];

NOTE: if you attempt to access (the original) enumerateOperation.queryCompletionBlock instead of (the very last) enumerateOperationActive.queryCompletionBlock, the operation will never complete.

Why `autorelease`? Are you not using ARC? – Mojo66 May 12 '16 at 10:22 — Mojo66, May 12 '16 at 10:22

score 0 · Answer 3 · answered Sep 19 '15 at 21:48

I liked the recursive solution on the main thread. Below is my solution in Objective C. I'm using a class level var: _recordArray, allocated ahead of time.

- (void) readRecords_Resurs: (CKDatabase *) database query: (CKQuery *) query cursor: (CKQueryCursor *) cursor {

    // Send either query or cursor

    CKQueryOperation *operation;
    if (query != nil) operation = [[CKQueryOperation alloc] initWithQuery: query];
    else operation = [[CKQueryOperation alloc] initWithCursor: cursor];
    operation.recordFetchedBlock = ^(CKRecord *record) {
        [_recordArray addObject:record];
    };
    operation.queryCompletionBlock = ^(CKQueryCursor *cursor, NSError *error) {
        if (cursor == nil || error != nil) {
            // Done
            dispatch_async(dispatch_get_main_queue(), ^{ [self readRecordsDone: error == nil ? nil : [error localizedDescription]]; });
        }
        else {
            // Get next batch using cursor
            dispatch_async(dispatch_get_main_queue(), ^{ [self readRecords_Resurs: database query: nil cursor: cursor]; });
        }
    };

    [database addOperation: operation]; // start
}

- (void) readRecordsDone: (NSString *) errorMsg {
}

malhal · Answer 4 · 2015-12-10T10:23:36.167

My solution is a category that uses two operations but both use the same blocks, and you can supply how many results per request you'd like.

@interface CKDatabase (MH)

/* Convenience method for performing a query receiving the results in batches using multiple network calls. Best use max 400 for cursorResultsLimit otherwise server sometimes exceptions telling you to use max 400. Even using CKQueryOperationMaximumResults can cause this exception.  */
- (void)mh_performCursoredQuery:(CKQuery *)query cursorResultsLimit:(int)cursorResultsLimit inZoneWithID:(CKRecordZoneID *)zoneID completionHandler:(void (^)(NSArray /* CKRecord */ *results, NSError *error))completionHandler;

@end

@implementation CKDatabase(MH)

- (void)mh_performCursoredQuery:(CKQuery *)query cursorResultsLimit:(int)cursorResultsLimit inZoneWithID:(CKRecordZoneID *)zoneID completionHandler:(void (^)(NSArray /* CKRecord */ *results, NSError *error))completionHandler{

    //holds all the records received.
    NSMutableArray* records = [NSMutableArray array];

    //this block adds records to the result array
    void (^recordFetchedBlock)(CKRecord *record) = ^void(CKRecord *record) {
        [records addObject:record];
    };

    //this is used to allow the block to call itself recurively without causing a retain cycle.
    void (^queryCompletionBlock)(CKQueryCursor *cursor, NSError *error)
    __block __weak typeof(queryCompletionBlock) weakQueryCompletionBlock;

    weakQueryCompletionBlock = queryCompletionBlock = ^void(CKQueryCursor *cursor, NSError *error) {
        //if any error at all then end with no results. Note its possible that we got some results,
        // and even might have got a cursor. However if there is an error then the cursor doesn't work properly so will just return with no results.
        if(error){
            completionHandler(nil,error);
        }
        else if(cursor){
            CKQueryOperation* cursorOperation = [[CKQueryOperation alloc] initWithCursor:cursor];
            cursorOperation.zoneID = zoneID;
            cursorOperation.resultsLimit = cursorResultsLimit;
            cursorOperation.recordFetchedBlock = recordFetchedBlock;
            cursorOperation.queryCompletionBlock = weakQueryCompletionBlock; // use the weak pointer to prevent retain cycle
            //start the recursive operation and return.
            [self addOperation:cursorOperation];
        }
        else{
            completionHandler(records,nil);
        }
    };

    //start the initial operation.
    CKQueryOperation* queryOperation = [[CKQueryOperation alloc] initWithQuery:query];
    queryOperation.zoneID = zoneID;
    queryOperation.resultsLimit = cursorResultsLimit;
    queryOperation.recordFetchedBlock = recordFetchedBlock;
    queryOperation.queryCompletionBlock = queryCompletionBlock;
    [self addOperation:queryOperation];
}

@end

Not sure, last time I tried swift it didn't have error handling — malhal, Apr 10 '16 at 21:24

CKQueryOperation working with big batch

4 Answers4