5

I am implementing a custom NSIncrementalStore subclass which uses a relational database for persistent storage. One of the things that I still struggle with is the support for optimistic locking.

(feel free to skip this lengthy description right to my question below)

I analyzed how Core Data's SQLite incremental store approaches this problem by examining SQL logs produced by it and came up with following conclusions:

  • Each entity table in the database has a Z_OPT column which indicates the number of times a particular instance of this entity (row) has been modified, starting from 1 (initial insertion).

  • Each time a managed object is modified, Z_OPT value in its corresponding database row is incremented.

  • The store maintains cache (referred to as row cache in Core Data docs) of NSIncrementalStoreNode instances, each having a version property equal to Z_OPT value returned by previous SELECT or UPDATE SQL query on managed object's row.

  • When a managed object is returned from NSManagedObjectContext (e.g. by executing NSFetchRequest on it), MOC creates snapshot of this object which contains this version number.

  • When the object is modified or deleted, Core Data makes sure that it has not been modified or deleted outside the context by comparing versions of cached row and object snapshot. All of this happen when -save: is called on the context that the object belongs to. If the versions are different then a merge conflict is detected and handled based on set merging policy.

When MOC is being saved, the -newValuesForObjectWithID:withContext:error: method is called for each modified/deleted object which in turn returns NSIncrementalStoreNode with version number. This version is then compared to snapshot's version and if they are different, the save fails with appropriate merge conflicts (at least with default merge policy).

This simple use case works properly with my store since -newValuesForObjectWithID:withContext:error: checks the row cache first which is enough if the object was concurrently modified in other context using the same store instance. If this is the case, then the cache contains updated row with higher version number which is enough to detect a conflict.

But how can I detect than the underlying database has been modified outside my store, possibly by other application or other store instance using the same database file? I know this is an unfrequent edge case but Core Data handles it properly and I would prefer to do the same.

Core Data's store uses SQL queries like these to update/delete object's row:

UPDATE ZFOO SET Z_OPT=Y, (...) WHERE (...) AND Z_OPT=X
DELETE FROM ZFOO WHERE (...) AND Z_OPT=X

where:
X - version number last known to the store (from cache)
Y - new version number

If such a query fails (no rows affected) the row is updated in store's cache and its version compared against the one previously cached.

My question is: how can a custom NSIncrementalStore inform Core Data that optimistic locking failure has occurred for some updated/deleted/locked objects? It is only the store that is able to tell that when it handles NSSaveChangesRequest passed to it its -executeRequest:withContext:error: method.

If the underlying database does not change under the store, then conflicts are detected since Core Data calls -newValuesForObjectWithID:withContext:error: on each modified/deleted/locked object prior to executing save changes request on the store. I was not able to find any way for NSIncrementalStore to inform Core Data that an optimistic locking failure has occurred after it started to handle the save request. Is there some undocumented way to do that? Core Data seems to throw some exception in that case which is then magically translated into failed save request with NSError listing all the conflicts. I am only able to mimic that partly by returning nil from -executeRequest:withContext:error: and creating the error message by myself. I think there must be a way to use the standard Core Data conflict handling mechanism in this scenario as well.

Bartlomiej Hyzy
  • 53
  • 5
  • I haven't really dived into how exactly this is implemented in the CoreData framework. But I do know that when CoreData reports a merge conflict there is also a possibility that the merge conflict will hold the persistentSnapshot object (the object as it now exist in the store). this might happen when the coordinator commit the saved data to the store and find it altered. [see here](https://developer.apple.com/library/mac/#documentation/CoreData/Reference/NSMergeConflict_Class/Reference/Reference.html) – Dan Shelly Apr 05 '13 at 06:34
  • @DanShelly: Thank you for your comment. You're completely right and this is what I'm actually trying to achieve in my incremental store implementation. I am able to detect and produce merge conflicts between managed object context and persistent store's row cache (case #1 in "Overview" section of the doc you linked) but not between the row cache and external storage (rel. database in my case, see case #2). I can detect such a conflict just like CoreData's SQLite store does but I don't know how to report this conflict to MOC that is being saved. – Bartlomiej Hyzy Apr 05 '13 at 09:40

2 Answers2

1

I realize that this is not an answer to you question, but I will try and give you my point of view on CoreData and correlation to Databases:

(1st level cache)
NSPesistentStoreCoordinator + NSPersistentStore == A single connection to the database

(2nd level cache)
NSManagedObjectContext == cache over the connection holding changes

So, to my understanding your issue is that you have multiple connections to your store, each making changes, but you have no central version control over your records. Your store will receive a -executeRequest:withContext:error: with NSSaveRequestType
You will then be responsible to verify that the record versions match, if you find a conflict in the connection level (level 1) you report version mismatch between the context (level 2) and the coordinator.
you need to report version missmatch between your connection (level 1) and your store.
To be able to do this your store must report changes on it across all connections to it (ConnectionManager), or it might offer hooks to changes performed on it.
I'm no SQLite expert, but the SQLite API does have something to offer in that area:
update hook
commit hook
changes
total changes
(I have no experience in setting these kind of hooks, but if CoreData use them it will not show in the debug logs)

you can report these errors by setting the error pointer (NSError**) and setting its internal data to match the one that CoreData coordinator is setting (create merge conflict and set the information in them as needed)

Note that optimistic locking failure will only occur during -executeRequest:withContext:error: (unless you have a rogue connection to the store, one that is not tracked by the manager.
To support this behaviour your manager might need to verify each record as it is committed for a save [huge performance cost] , or use some hooks into the changes recently made to records )

To handle multiple connections to your store you might need to have a shared cache of NSIncrementalStoreNode, keyed by the store url:
static @{
url1 : actualCacheMapping1,
url2 : actualCacheMapping2,
...
}
each connection save to the store will be verified agains the store url actual cache.

Hope this make some sense for you.

Dan Shelly
  • 5,991
  • 2
  • 22
  • 26
  • Thanks for sharing your point of view. While in general I agree with your assumptions, I'm not sure about the 1st level cache you mention. From the tests I performed I conclude that there is no layer in Core Data that manages and monitors multiple connections to the same SQLite databases. Each record is checked for concurrent modifications outside the store by simply comparing its Z_OPT value in DB with the value known to the store (see sample update/delete statements I posted). I was hoping it's possible to avoid filling the NSError manually (like in case #1), but I'll try that, thanks. – Bartlomiej Hyzy Apr 06 '13 at 21:50
  • You were right about the 1st level cache (found no proof of it). I have edited my response. hope this will help :) – Dan Shelly Apr 07 '13 at 04:23
  • Thank you, your answer was indeed helpful. Now I return NSError with manually created NSMergeConflict-s and it works like a charm :) – Bartlomiej Hyzy Apr 08 '13 at 10:12
1

My question is: how can a custom NSIncrementalStore inform Core Data that optimistic locking failure has occurred for some updated/deleted/locked objects? It is only the store that is able to tell that when it handles NSSaveChangesRequest passed to it its -executeRequest:withContext:error: method.

In an NSIncrementalStore, NSIncrementalStoreNodes represent the store snapshots. The version property of the node is the optimistic locking primitive. The persistent store is responsible for detecting optimistic locking failures in at the store level, while the managed object context can detect them higher up. An optimistic locking failure at the store level might happen if the system the store is talking to was changed by something else, and there is a conflict between that system's state and that representation of state in the persistent store. For example, if the store was communicating with a web service and the web service data was changed by another user, etc.

If an optimistic locking failure is detected in your store implementation during a save, your store is responsible for creating NSMergeConflict objects describing it. These will be propagated up by the NSPersistentStoreCoordinator.

[[NSMergeConflict alloc] initWithSource:managedObject newVersion:newVersion oldVersion:oldVersion cachedSnapshot:inMemorySnapshot persistedSnapshot:storedSnapshot];

Snapshot dictionaries should include all modelled attribute property names as keys along with their values. This does not include relationships. For some stores, using the values from the reference objects or NSIncrementalStoreNodes may suffice as long as they only include the modelled attribute property name as keys (and those are easy to get from the entity description).

Once these objects have been created, create an NSError in the NSCocoaErrorDomain with the code NSPersistentStoreSaveConflictsError. The userInfo object should contain the key NSPersistentStoreSaveConflictsErrorKey which should contain an array of the NSMergeConflict objects. Return that from the save request, and the NSPersistentStoreCoordinator will be responsible for finding resolution. Rememeber, you should not generate merge conflicts for conflicts between the state of objects in the NSManagedObjectContext and your store, only for conflicts between whatever in-memory or cached state in your store and where ever the data is kept or persisted (like a web service, or database, etc.)

quellish
  • 21,123
  • 4
  • 76
  • 83