efficiently display 100,000 items using Core Data

Question

I am using a NSFetchResultsController to display 100,000 + records in a UITableView. This works but it is SLOW, especially on an iPad 1. It can take 7 seconds to load which is torture for my users.

I'd also like to be able to use sections but this adds at least another 3 seconds onto the laod time.

Here is my NSFetchResultsController:

- (NSFetchedResultsController *)fetchedResultsController {

    if (self.clientsController != nil) {
        return self.clientsController;
    }

    NSFetchRequest *request = [[NSFetchRequest alloc] init];
    NSEntityDescription *entity = [NSEntityDescription entityForName:@"Client" inManagedObjectContext:self.managedObjectContext];
    [request setEntity:entity];
    [request setPredicate:[NSPredicate predicateWithFormat:@"ManufacturerID==%@", self.manufacturerID]];
    [request setFetchBatchSize:25];

    NSSortDescriptor *sort = [[NSSortDescriptor alloc]  initWithKey:@"UDF1" ascending:YES];
    NSSortDescriptor  *sort2= [[NSSortDescriptor alloc] initWithKey:@"Name" ascending:YES];
    [request setSortDescriptors:[NSArray arrayWithObjects:sort, sort2,nil]];

    NSArray *propertiesToFetch = [[NSArray alloc] initWithObjects:@"Name", @"ManufacturerID",@"CustomerNumber",@"City", @"StateProvince",@"PostalCode",@"UDF1",@"UDF2", nil];
    [request setPropertiesToFetch:propertiesToFetch];

    self.clientsController =
    [[NSFetchedResultsController alloc] initWithFetchRequest:request
                                        managedObjectContext:self.managedObjectContext sectionNameKeyPath:nil
                                                   cacheName:nil];

    return self.clientsController;

}

I have an index on ManufacturerID which is used in my NSPredicate. This seems like a pretty basic NSFetchRequest - anything I can do to speed this up? Or have I just hit a limitation? I must be missing something.

Answer 1

First: you can use the NSFetchedResultsController's cache to speed up display after the first fetch. This should quickly go down to a fraction of a second.

Second: you can try to display the only the first screenful and then fetch the rest in the background. I do this in the following way:

• When the view appears, check if you have the first page cache.
• If not, I fetch the first page. You can accomplish this by setting the fetch request's fetchLimit.
  • In case you are using sections, do two quick fetches to determine the first section headers and records.
• Populate a second fetched results controller with your long fetch in a background thread.
  • You can either create a child context and use performBlock: or
  • use dispatch_async().
• Assign the second FRC to the table view and call reloadData.

This worked quite well in one of my recent projects with > 200K records.

Answer 2

I know the answer @Mundi provided is accepted, but I've tried implementing it and ran into problems. Specifically the objects created by the second FRC will be based on the other thread's ManagedObjectContext. Since these objects are not thread safe and belong to their own MOC on the other thread, the solution I found was to fault the objects as they are being loaded. So in cellForRowAtIndexPath I added this line:

NSManagedObject *object = [self.fetchedResultsController objectAtIndexPath:indexPath];
object = (TapCellar *)[self.managedObjectContext existingObjectWithID:[object objectID] error:nil];

Then you have an object for the correct thread you are in. One further caveat is that the changes you make to the objects won't be reflected in the background MOC so you'll have to reconcile them. What I did was make the background MOC a private queue MOC and the foreground one is a child of it like this:

NSPersistentStoreCoordinator *coordinator = [self persistentStoreCoordinator];
if (coordinator != nil) {
  _privateManagedObjectContext = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSPrivateQueueConcurrencyType];
  [_privateManagedObjectContext setPersistentStoreCoordinator:coordinator];

  _managedObjectContext = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSMainQueueConcurrencyType];
  [_managedObjectContext setParentContext:_privateManagedObjectContext];

}

Now when I make changes in the main thread, I can reconcile them easily by doing this:

if ([self.managedObjectContext hasChanges]) {
   [self.managedObjectContext performBlockAndWait:^{
      NSError *error = nil;
      ZAssert([self.managedObjectContext save:&error], @"Error saving MOC: %@\n%@",
             [error localizedDescription], [error userInfo]);
   }];
}

I wait for it's return since I'm going to reload the table data at this point, but you can choose not to wait if you'd like. The process is pretty quick even for 30K+ records since usually only one or two are changed.

Hope this helps those who are stuck with this!