2

Background

I'm building an importer to encrypt my code. When compiling, I encrypt the code object in pyc files. When loading, I use this customized importer to decrypt the code object before execution.

Since my code is bundled in zip format, I decided to modify zipimporter from the builtin zipimport package by adding the decrypt logic into it.

I'm using Python 3.7

Problem

I modified the zimpimport.c and make it into C extension. The encrypt & decrypt process works fine but I started to see ImportError & AttributeError. For example:

"""module: test.py"""
class Foo:

    def bar():
        pass

foo = Foo()

// Import Error: can not import name 'foo' from test 
// Attribute Error: foo has no attribute bar

When server starts, this error only occurs during the first few minutes randomly. So I suspect that it's a multithreading problem and it's caused by a thread seeing a partially initialized module object.

What I tried

After checking the origin c code, the implementation for loading a module is:

mod = PyImport_AddModuleObject(fullname);
// ... some code
mod = PyImport_ExecCodeModuleObject(fullname, code, modpath, NULL);

It first add module to sys.modules, then execute some c code and execute the module code at last. If another thread read the sys.modules before it finishes executing module code, it can get the ImportError and AttributeError.

I just copy the original implementation and it doesn't use import lock. I guess it's handled by interpreter on outer level and I have to explicitly use import lock. So I wrap the code block with the import lock as following:

#include "import.h"
_PyImport_AcquireLock();
// wrapped code block
_PyImport_ReleaseLock();

My Question

After adding import lock, the error still happens. I'm not familiar with python c api. Have I used the lock in a correct way? Are there other possible reasons that can cause this error?

Code

To test, put the following files in same folder:

Run python setup.py build_ext --inplace and it will generate zzimporter.so. And it works just like the buit-in zipimport module.

CtheSky
  • 2,484
  • 14
  • 16
  • `test` is always a bad name to use because there's an existing module called `test` so you can end up importing the wrong thing by accident – DavidW Jan 27 '19 at 20:21
  • It's for demonstration purpose only, just want to hide some unnecessary infos. But it's a good advice, thanks. – CtheSky Jan 28 '19 at 02:55
  • @CtheSky: You do certainly seem to be all about hiding things. Do you know that it’s pretty trivial to reconstitute an unencrypted `.pyc` from the resulting module object? – Davis Herring Jan 28 '19 at 05:44
  • @DavisHerring That's true. To prevent that, I use embedding python, so there's no available module object (or at least not very easy to get). And pyinstaller also provides [this option](https://pyinstaller.readthedocs.io/en/stable/usage.html#encrypting-python-bytecode). – CtheSky Jan 30 '19 at 10:02

0 Answers0