4

We have a python based module which we want to distribute to our customers by creating a compiled copy understandable by linux system(i.e. .so file)

We have evaluated cython which does this quite easily, but we're seeing it's creating as many .so file as .pyx/.py file but we want to make one uber .so file for complete package. We want to do it smartly in a sense that if we add dependency to other module in future, uber compiled file should have all the dependencies.

Any recommendations, how we can do it neatly?

Vardan Gupta
  • 3,505
  • 5
  • 31
  • 40
  • Is it possible to compile these as static libraries (**.a**) instead of dynamic libraries (the end product will still be dynamic)? – kabanus Apr 06 '18 at 10:14
  • 1
    Python only permits one module per `.so`. I would suggest packaging your library as a wheel. Then the number of `.so` files is meaning less. They only have to deal with one file. – Dan D. Apr 06 '18 at 10:17
  • The idea behind creating one uber compiled file was reduce memory footprint, I was checking importing any 3rd party module in python program and running multiple instances of such programs were consuming memory linearly, I wanted to create .so so that Proportionate Set Size(PSS) can be reduced with increasing number of execution of same program. If creating a wheel can do that, I would love to know how we can do that? – Vardan Gupta Apr 06 '18 at 10:22
  • @kabanus how would I able to import .a library to any python program, I know I can import any .so based python module to py program. – Vardan Gupta Apr 06 '18 at 10:29
  • @VardanGupta You can merge **.a** files to an `.so` file with `gcc`. Also, another option would be to merge your code through some script to a single prior to compiling. – kabanus Apr 06 '18 at 10:32
  • Thanks, good to know that! I'm still looking if there is any alternative neat way. – Vardan Gupta Apr 06 '18 at 10:35
  • @DanD.: Python allows only one module per `.so` _name_—you can use multiple links to one file as one module each. – Davis Herring Apr 06 '18 at 13:26

1 Answers1

2

cx_freeze can create re-distributable packages of python modules.

For example:

cxfreeze my_script_using_my_python_module.py --target-dir dist

Whether the python package is compiled or regular python is not really relevant. What is relevant is that your customers will need to have compatible python version, as well as compatible libc/gcc, to run it.

So for the purposes of distributing a python module to third parties, a single .so will not be compatible with everyone. Ever.

cx_freeze bundles the required python interpreter version and python packages so there are no dependencies. It is also cross-platform.

danny
  • 5,140
  • 1
  • 19
  • 31
  • Totally agreeing to your point that .so will never be compatible with every customer but more importantly we're working on to improve memory consumption. To talk about few stats we have a dependency over python repo which on basic usage like import , taking 50MB and we wanted to run 1000+ similar processes,, if we can make distribution shareable, we will definitely boost concurrent execution. Regards cxfreeze, we'll definitely give a shot, if it can help us! Thanks – Vardan Gupta Apr 06 '18 at 15:54
  • 1
    Very much doubt that 50MB is taken up by the python code that you wrote. It is mostly the python interpreter. So that 50MB will be shared among all threads in that interpreter and also among processes on Linux as shared process memory. Either way, cx_freeze will bundle whatever the python script given to it is running. If it is starting 1000 processes, they will all share the same python distribution. – danny Apr 06 '18 at 16:09