1

I deleted my last question on this topic in order to research the problem further.

I'm working on a content based image retrieval database that indexes the local photo album on iPhone. I've written filters and methods for feature extraction, but as a preprocessing step, I use CIDetector to detect face rectangles and filter those rectangles through a local binary pattern filter and extract histograms of small rectangles.

The process can obviously take a good amount of time for indexing a large photo album, so I've designed it to be asynchronous, with a background thread that runs the filter and extracts features.

The trouble with this is that the only method to access photo album images seems to be via PHAsset. There are two API's that are commonly used with it, both of which require callback blocks. One of them retrieves a UIImage representation, this is not what I want, primarily because I will need to load many of them and memory constraints are tight and I don't intend to show the image at all. It will also return a dictionary of objects, one of which being the URL for the image file itself, but using +[CIImage imageWithContentsOfURL] will return nil and the iOS console will show that permission is denied to the image when using this API.

The other is -[PHAsset requestContentEditingInputWithOptions]. This particular API will force the callback block to be invoked in the main queue, no matter what queue called the API, the callback block is always dispatched into the main queue. Using this API, a CIImage can be successfully be loaded using +[CIImage imageWithContentsOfURL], meaning that the API has effectively given permission to access the file. I can then put the CIImage on a different queue that will perform all index processing steps.

This approach seems to work, but only for a short time. Eventually the app crashes, and all I see is that during earlier processing a message shows up that says Connection to assetsd was interrupted or assetsd died.

If I don't queue the operation on another queue and just let it run on the main queue which the API forces the callback into, everything is fine and that message doesn't show up in the console. But this leaves my main thread constantly busy doing very expensive image processing.

Edit: It occurs to me that I'm probably misusing this API, but it appeared to be the only thing that would give me the data I wanted in the appropriate format (a CIImage that could be used to load pixel data from a file when necessary, during processing).

The API I'm currently using is probably more suited for an interactive image filtering session where the image edit data is used with UI and the service that is managing it keeps a session open with the app.

Joey Carson
  • 2,973
  • 7
  • 36
  • 60
  • Have you tried to create the CIContext with contextWithEAGLContext? – Matías Marquez Oct 26 '15 at 17:40
  • No I haven't. Could you elaborate on why that would be necessary? The code works part of the time, but seems to crash due to the assetsd service that sends the data to the app disconnecting from the process. The EAGLContext would be just for render time. – Joey Carson Oct 26 '15 at 17:45
  • I'm not an expert on the issue, but as far as I understand it, you main thread is blocked by images processing as you said, and the contextWithEAGLContext should be able to send that processing from the CPU to the GPU. Why it crashes I don't know, but it may solve your performance issue. – Matías Marquez Oct 26 '15 at 17:49
  • As far as rendering sub-rectangles goes, I'm already using a custom CIFilter kernel program, which is essentially Apple's GLSL so the rendering is parallelized on GPU already. There's also some manual buffer copies and I'm using OpenCV for histogram computation, so I think it's about as efficient it gets within reason. I think this API is more meant for editing sessions, where the main thread talks back to assetsd. I'm trying to think if I can use PHImageManager and just load the pixel data for each call, I can then store it in tmp and hopefully read it via CIImage with no dependence on API. – Joey Carson Oct 26 '15 at 17:54
  • @JoeyCarson did you ever solve this problem? – luna_ Aug 08 '16 at 19:20
  • @luna_ I did take a sort of workaround approach here to resolve it. Since the crash seemed to occur because I was keeping the image data session opened (or perhaps confusing assetsd because the image data was read on a separate thread than it was passed on), I instead copied the image data to NSTemporaryDirectory. That way the background processing thread can just read from local temp storage. See the code at https://github.com/JoeyCarson/cbir_database_ios/blob/master/CBIRD/CBIRD/PhotoIndexer.m – Joey Carson Aug 08 '16 at 19:35

0 Answers0