Speed up nautilus python-extensions for reading image's Exif

Question

I've written a Nautilus extension which reads picture's metadata (executing exiftool), but when I open folders with many files, it really slows down the file manager and hangs until it finishes reading the file's data.

Is there a way to make Nautilus keep its work while it runs my extension? Perhaps the Exif data could appear gradually in the columns while I go on with my work.

#!/usr/bin/python

# Richiede:
# nautilus-python
# exiftool
# gconf-python

# Versione 0.15

import gobject
import nautilus
from subprocess import Popen, PIPE
from urllib import unquote
import gconf

def getexiftool(filename):
    options = '-fast2 -f -m -q -q -s3 -ExifIFD:DateTimeOriginal -IFD0:Software -ExifIFD:Flash -Composite:ImageSize -IFD0:Model'
    exiftool=Popen(['/usr/bin/exiftool'] + options.split() + [filename],stdout=PIPE,stderr=PIPE)
    #'-Nikon:ShutterCount' non utilizzabile con l'argomento -fast2
    output,errors=exiftool.communicate()
    return output.split('\n')

class ColumnExtension(nautilus.ColumnProvider, nautilus.InfoProvider, gobject.GObject):
    def __init__(self):
        pass

    def get_columns(self):
        return (
            nautilus.Column("NautilusPython::ExifIFD:DateTimeOriginal","ExifIFD:DateTimeOriginal","Data (ExifIFD)","Data di scatto"),
            nautilus.Column("NautilusPython::IFD0:Software","IFD0:Software","Software (IFD0)","Software utilizzato"),
            nautilus.Column("NautilusPython::ExifIFD:Flash","ExifIFD:Flash","Flash (ExifIFD)","Modalit\u00e0 del flash"),
            nautilus.Column("NautilusPython::Composite:ImageSize","Composite:ImageSize","Risoluzione (Exif)","Risoluzione dell'immagine"),
            nautilus.Column("NautilusPython::IFD0:Model","IFD0:Model","Fotocamera (IFD0)","Modello fotocamera"),
            #nautilus.Column("NautilusPython::Nikon:ShutterCount","Nikon:ShutterCount","Contatore scatti (Nikon)","Numero di scatti effettuati dalla macchina a questo file"),
            nautilus.Column("NautilusPython::Mp","Mp","Megapixel (Exif)","Dimensione dell'immagine in megapixel"),
        )

    def update_file_info_full(self, provider, handle, closure, file):
        client = gconf.client_get_default()

        if not client.get_bool('/apps/nautilus/nautilus-metadata/enable'):
            client.set_bool('/apps/nautilus/nautilus-metadata/enable',0)
            return

        if file.get_uri_scheme() != 'file':
            return

        if file.get_mime_type() in ('image/jpeg', 'image/png', 'image/gif', 'image/bmp', 'image/x-nikon-nef', 'image/x-xcf', 'image/vnd.adobe.photoshop'):
            gobject.timeout_add_seconds(1, self.update_exif, provider, handle, closure, file)
            return Nautilus.OperationResult.IN_PROGRESS

        file.add_string_attribute('ExifIFD:DateTimeOriginal','')
        file.add_string_attribute('IFD0:Software','')
        file.add_string_attribute('ExifIFD:Flash','')
        file.add_string_attribute('Composite:ImageSize','')
        file.add_string_attribute('IFD0:Model','')
        file.add_string_attribute('Nikon:ShutterCount','')
        file.add_string_attribute('Mp','')

        return Nautilus.OperationResult.COMPLETE

    def update_exif(self, provider, handle, closure, file):
        filename = unquote(file.get_uri()[7:])

        data = getexiftool(filename)

        file.add_string_attribute('ExifIFD:DateTimeOriginal',data[0].replace(':','-',2))
        file.add_string_attribute('IFD0:Software',data[1])
        file.add_string_attribute('ExifIFD:Flash',data[2])
        file.add_string_attribute('Composite:ImageSize',data[3])
        file.add_string_attribute('IFD0:Model',data[4])
        #file.add_string_attribute('Nikon:ShutterCount',data[5])
        width, height = data[3].split('x')
        mp = float(width) * float(height) / 1000000
        mp = "%.2f" % mp
        file.add_string_attribute('Mp',str(mp) + ' Mp')

        Nautilus.info_provider_update_complete_invoke(closure, provider, handle, Nautilus.OperationResult.COMPLETE)

        return false

gpoo · Accepted Answer · 2012-05-07T01:37:20.247

That happens because you are invoking update_file_info, which is part of the asynchronous IO system of Nautilus. Therefore, it blocks nautilus if the operations are not fast enough.

In your case it is exacerbated because you are calling an external program, and that is an expensive operation. Notice that update_file_info is called once per file. If you have 100 files, then you will call 100 times the external program, and Nautilus will have to wait for each one before processing the next one.

Since nautilus-python 0.7 are available update_file_info_full and cancel_update, which allows you to program async calls. You can check the documentation of Nautilus 0.7 for more details.

It worth to mention this was a limitation of nautilus-python only, which previously did not expose those methods available in C.

EDIT: Added a couple of examples.

The trick is make the process as fast as possible or make it asynchronous.

Example 1: Invoking an external program

Using a simplified version of your code, we make asynchronous using GObject.timeout_add_seconds in update_file_info_full.

from gi.repository import Nautilus, GObject
from urllib import unquote
from subprocess import Popen, PIPE

def getexiftool(filename):
    options = '-fast2 -f -m -q -q -s3 -ExifIFD:DateTimeOriginal'
    exiftool = Popen(['/usr/bin/exiftool'] + options.split() + [filename],
                     stdout=PIPE, stderr=PIPE)
    output, errors = exiftool.communicate()
    return output.split('\n')

class MyExtension(Nautilus.ColumnProvider, Nautilus.InfoProvider, GObject.GObject):
    def __init__(self):
        pass

    def get_columns(self):
        return (
            Nautilus.Column(name='MyExif::DateTime',
                            attribute='Exif:Image:DateTime',
                            label='Date Original',
                            description='Data time original'
            ),
        )

    def update_file_info_full(self, provider, handle, closure, file_info):
        if file_info.get_uri_scheme() != 'file':
            return

        filename = unquote(file_info.get_uri()[7:])
        attr = ''

        if file_info.get_mime_type() in ('image/jpeg', 'image/png'):
            GObject.timeout_add_seconds(1, self.update_exif, 
                                        provider, handle, closure, file_info)
            return Nautilus.OperationResult.IN_PROGRESS

        file_info.add_string_attribute('Exif:Image:DateTime', attr)

        return Nautilus.OperationResult.COMPLETE

    def update_exif(self, provider, handle, closure, file_info):
        filename = unquote(file_info.get_uri()[7:])

        try:
            data = getexiftool(filename)
            attr = data[0]
        except:
            attr = ''

        file_info.add_string_attribute('Exif:Image:DateTime', attr)

        Nautilus.info_provider_update_complete_invoke(closure, provider, 
                               handle, Nautilus.OperationResult.COMPLETE)
        return False

The code above will not block Nautilus, and if the column 'Date Original' is available in the column view, the JPEG and PNG images will show the 'unknown' value, and slowly they will being updated (the subprocess is called after 1 second).

Examples 2: Using a library

Rather than invoking an external program, it could be better to use a library. As the example below:

from gi.repository import Nautilus, GObject
from urllib import unquote
import pyexiv2

class MyExtension(Nautilus.ColumnProvider, Nautilus.InfoProvider, GObject.GObject):
    def __init__(self):
        pass

    def get_columns(self):
        return (
            Nautilus.Column(name='MyExif::DateTime',
                            attribute='Exif:Image:DateTime',
                            label='Date Original',
                            description='Data time original'
            ),
        )

    def update_file_info_full(self, provider, handle, closure, file_info):
        if file_info.get_uri_scheme() != 'file':
            return

        filename = unquote(file_info.get_uri()[7:])
        attr = ''

        if file_info.get_mime_type() in ('image/jpeg', 'image/png'):
            metadata = pyexiv2.ImageMetadata(filename)
            metadata.read()

            try:
                tag = metadata['Exif.Image.DateTime'].value
                attr = tag.strftime('%Y-%m-%d %H:%M')
            except:
                attr = ''

        file_info.add_string_attribute('Exif:Image:DateTime', attr)

        return Nautilus.OperationResult.COMPLETE

Eventually, if the routine is slow you would need to make it asynchronous (maybe using something better than GObject.timeout_add_seconds.

At last but not least, in my examples I used GObject Introspection (typically for Nautilus 3), but it easy to change it to use the module nautilus directly.

Thanks, I'll try later and ley you know about update_file_info_full; I've tried to make C extension, but without success because of the "inexistent" documentation. (Infact there's also another question of mine on StackOverflow about that) — Stefano d'Antonio, May 06 '12 at 10:58
I've tried changing update_file_info to update_file_info_full [Pastebin code](http://pastebin.com/MJ2CAM7x), but it's still as slow as before, probably I didn't understand how does it works. Is there an example somewhere? — Stefano d'Antonio, May 06 '12 at 14:16
You have to make your code asynchronous. Nautilus will not do it for you. update_file_info_full allows you to return Nautilus.OperationResult.IN_PROGRESS (which let Nautilus know the operation has not finished). For every time you return '...IN_PROGRESS' (which might be per file), you need a method that call Nautilus.info_provider_update_complete_invoke (which tell Nautilus the operation is complete). Read the documentation I linked. — gpoo, May 06 '12 at 16:17
By the way, it is recommended to paste the code snippets here rather than using pastebin, which would expire sooner than later and people would lose the context. — gpoo, May 06 '12 at 16:31
Now it's just a matter of "curiousness", because I solved my problem with creating extensions with C and using libexiv (even if it doesn't support raw files...) which is much faster. So I don't want to waste a lot of time understanding how does it works with Python. However I appreciate your help and can check your answer as "accepted". — Stefano d'Antonio, May 06 '12 at 22:39
I added an example that make the call asynchronous. In Python you can also use a library (like *pyexiv2*), which is also faster than invoking external processes. — gpoo, May 07 '12 at 01:38
It is still blocking nautilus, even if I followed your suggestions. I can't use pyexiv2 because it does not support raw files (if I remember well, because I already thought about that, but left the idea because of some missing feature) — Stefano d'Antonio, May 07 '12 at 13:07
The first example should block Nautilus (async and non-async) and the difference was noticeable in a directory with +100 of photos. The async version did not show the column values fast, but Nautilus was always responsive. With respect to the code in the comment, not it is not possible. Edit your question and put it there. — gpoo, May 07 '12 at 15:30
I thought the first example (with timeout_add_seconds, update_file_info_full instead of update_file_info) was to let exiftool work apart from nautilus so it can not block. If the first example is still blocking, what's the difference between this and my previcious code? — Stefano d'Antonio, May 07 '12 at 21:03
Sorry, I meant in the opposite order. The first one should not block. — gpoo, May 07 '12 at 23:12
You are talking about making the calculations asynchronous... would you mind giving a short example illustrating that? Thanks! — Nicolas Raoul, Aug 15 '18 at 01:23

score 1 · Answer 2 · answered Dec 11 '16 at 09:11

The above solution is only partly correct.

Between state changes for file_info metadata, the user should call file_info.invalidate_extension_info() to notify nautilus of the change. Failing to do this could end up with 'unknown' appearing in your columns.

file_info.add_string_attribute('video_width', video_width)
file_info.add_string_attribute('video_height', video_height)
file_info.add_string_attribute('name_suggestion', name_suggestion)   

file_info.invalidate_extension_info()

Nautilus.info_provider_update_complete_invoke(closure, provider, handle, Nautilus.OperationResult.COMPLETE)

Full working example here:

Fully working example

API Documentation

To the downvoter: is this wrong for some reason? If so, can you explain? — Stefano d'Antonio, Dec 11 '16 at 17:23

score 0 · Answer 3 · answered Dec 14 '16 at 21:49

thanks to Dave!

i was looking for a solution to the 'unknown' text in the column for ages

file_info.invalidate_extension_info()

Fixed the issue for me right away :)

Per the api API Documentation

https://projects-old.gnome.org/nautilus-python/documentation/html/class-nautilus-python-file-info.html#method-nautilus-python-file-info--invalidate-extension-info

Nautilus.FileInfo.invalidate_extension_info

def invalidate_extension_info()

Invalidates the information Nautilus has about this file, which causes it to request new information from its Nautilus.InfoProvider providers.

Speed up nautilus python-extensions for reading image's Exif

3 Answers3

Example 1: Invoking an external program

Examples 2: Using a library