Several `with`s in `try`s

Question

I have several possible files which could hold my data; they can be compressed in different ways, so to open them I need to use file(), gzip.GzipFile() and other which also return a file object (supporting the with interface).

I want to try each of them until one succeeds in opening, so I could do something like

try:
  with gzip.GzipFile(fn + '.gz') as f:
    result = process(f)
except (IOError, MaybeSomeGzipExceptions):
  try:
    with xCompressLib.xCompressFile(fn + '.x') as f:
      result = process(f)
  except (IOError, MaybeSomeXCompressExceptions):
    try:
      with file(fn) as f:
        result = process(f)
    except IOError:
      result = "some default value"

which obviously isn't feasible in case I have dozens of possible compression variants. (The nesting will get deeper and deeper, the code always looking very much alike.)

Is there a nicer way to spell this out?

EDIT: If possible I'd like to have the process(f) out of the try/except as well to avoid accidental catching of exceptions raised in the process(f).

Answer 1

Yea, you could put all your variants through a list and try them until one of them works, thus un-nesting your code:

def process_gzip(fn):
    with gzip.GzipFile(fn + '.gz') as f:
        return process(f)

def process_xlib(fn):
    with xCompressLib.xCompressFile(fn + '.x') as f:
        return process(f)

def process_builtin(fn):
    with file(fn) as f:
        return process(f)

process_funcs = [process_gzip, process_xlib, process_builtin]

#processing code:

for process_f in process_funcs:
    try:
        result = process_f(fn)
        break
    except IOError:
        #error reading the file, keep going
        continue
    except:
        #processing error, re-raise the exception
        raise

Or, to reduce amount of code you could make a process_func factory, since they all have the same form:

def make_process_func(constructor, filename_transform):
    with constructor(filename_transform) as f:
        return process(f)

process_funcs = [
    make_process_func(gzip.GzipFile, lambda fn: fn + '.gz'),
    make_process_func(xCompressLib.xCompressFile, lambda fn: fn + '.x'),
    make_process_func(file, lambda fn: fn),
]

Answer 2

I'd write a custom context manager:

from contextlib import contextmanager

filetypes = [('.gz', gzip.GzipFile, (IOError, MaybeSomeGzipExceptions)), 
             ('.x', xCompressLib.xCompressFile, (IOError, MaybeSomeXCompressExceptions))]

@contextmanager
def open_compressed(fn):
    f = None
    try:
        for ext, cls, exs in filetypes:
            try:
                f = cls(fn + ext)
            except exs:
                pass
            else:
                break
        yield f
    finally:
        if f is not None:
            f.close()

with open_compressed(fn) as f:
    result = "some default value" if f is None else process(f)

Or possibly just a function that returns a context manager:

filetypes = [('.gz', gzip.GzipFile, (IOError, MaybeSomeGzipExceptions)), 
             ('.x', xCompressLib.xCompressFile, (IOError, MaybeSomeXCompressExceptions))]

class UnknownCompressionFormat(Exception):
    pass

def open_compressed(fn):
    for ext, cls, exs in filetypes:
        try:
            return cls(fn + ext)
        except exs:
            pass
    raise UnknownCompressionFormat

try:
    with open_compressed(fn) as f:
        result = process(f)
except UnknownCompressionFormat:
    result = "some default value"

Answer 3

Would this work:

extensions = [('.gz', gzip.GzipFile, (IOError, MaybeSomeGzipExceptions)), 
              ('.x', xCompressLib.xCompressFile, (IOError, MaybeSomeXCompressExceptions))] # and other such entries
processed = False
for ext, (compressor, errors) in extensions.iteritems():
    try:
        with compressor(fn+ext) as f:
            try:
                result = process(f)
                processed = True
                break
            except:
                raise
    except errors:
        pass
if not processed:
    result = "some default value"

Hope that helps