-1

Goal

Pass images from a C process to a Python process quickly.

Details

The C process is a command line tool which produces raster images. The Python process is a GUI application which asks to C tool to produce images and then shows these images. So, a user (after his action) waits for the images to be shown in the GUI.

Speed requirements

Currently used system is to write files to the disk by C application and then read back by the Python application. The size of images may differ in pixels (but approx. 1000x1000). The current format is uncompressed, however using the compressed format the transfer was slow too.

Usually, more than one image need to be created and shown, so some parallel approach can be applied too. However, parallelization by itself (as well as compression) hasn't made the transfer quicker.

Platforms and license

The solution should be cross-platform, especially GNU/Linux, MS Windows and Mac OS X.

The project is under GNU GPL 2 or higher, so the solution should be compatible.

Libraries

The solution should not bring new large dependencies such as boost library. Smaller, system or already used libraries are more welcome.

The C application is using cairo library. It is preferred not to add new dependencies unless it is really needed. In extreme case, it would be possible to wrap C code into Python script using ctypes but it is not preferred.

The Python application is using wxPython, ctypes and PIL. New dependencies can be added and new libraries can be (even) included (but less dependencies is better).

wenzeslaus
  • 667
  • 6
  • 13

2 Answers2

0

Given that...

The C process is a command line tool...

...your options are quite limited. I answered a similar question quite recently, which uses pipes.

If you could replace the command line tool with a daemon process, you could avoid the overhead of creating a new process each time, and use something like sockets instead.

Community
  • 1
  • 1
Aya
  • 39,884
  • 6
  • 55
  • 55
  • There was also a very similar question but MS Windows only: http://stackoverflow.com/questions/2536331/most-efficient-way-to-send-images-across-processes – wenzeslaus Apr 30 '13 at 09:11
0

How "slow" that is? If yu are already using libraries - such as PIL and cairo - they do have optimized code for dealing with images. Your solution couldbe a bit optimized by transfeirng fiels in memory, instead of writng then to disk, and then re-reading them - and also , by not using raw-images - a LZ style compression can boost speeds over uncompressed images - I think "pcx" or "tiff" could use such compressions by default. (PNG compression on the other hand is slow).

For using in-memory transfer instead of going trough the disk I think you coulduse the follwoing approach: Create a binary module uising "cython" that will feed the raw image data to a PIL Image object - and call the cython0--reated functions using multiprocessing, maybe with a backport of concurrent.futures

Cython is a requisite to build, but not a requisite to run such moduels- I have to add, since dependencies are a concern.

In any case, I think you should profile your system accuratelly before spending much time on this - I don't think you can achieve much better speed rates (maybe it can get to a 200-300% increase, if the image processing step is light - but if that would reduce your time from 3 seconds to 1 you would not be that better - from your question I infer you want a boost of an order of magnitude). If you are not running your image processing C program in parallel already, you should spawn at least one such process per CPU core, of course.

jsbueno
  • 99,910
  • 10
  • 151
  • 209
  • Although I'm still not sure which way to go, you provided good points. For example, I can confirm that using uncompressed format instead of PNG is a bit better. – wenzeslaus Apr 30 '13 at 09:10