How to import lib folder within Modules

Question

I had a GAE app which contains three Modules and a lib folder. When I tried to import the 3rd party library from the lib folder. GAE pops a ImportError.

I could get it to work by symlinking ./lib to ./Module_1/lib and ./Module_2/lib and also creating a appengine_config.py in each of the modules. But doing this seemed really dirty.
Is there a cleaner way to import app_root/lib from module_1 and module_2?

This seemed to be promising(https://cloud.google.com/appengine/docs/python/config/appconfig#Python_app_yaml_Includes), but don't know what to put inside include.yaml.

-- App Root/
    -- Module_1/
       module_1.yaml 
       module_1.py

    -- Module_2/
       module_2.yaml
       module_2.py

    -- lib/ 
       -- cloudstorage/
          ..
       -- 3rd_library_1/
          ..
          ..
       -- 3rd_library_2/
          ..
          ..

    appengine_config.py
    main.py (default module)
    app.yaml(default module)
    queue.yaml
    dispatch.yaml

In module_1.py or module_2.py, when I do

    import cloudstorage as gcs

It complains

    ImportError: No module named cloudstorage

However, when it's being imported within main.py, it works fine.

In the appengine_config.py:

import os
import sys

# Add ./lib to sys path
sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'lib'))

Also tried to print sys.path from main.py:

sys.path in main.py :
[ 
  '/base/data/home/apps/s~my-app/2.381942946570489905', 
  '/base/data/home/apps/s~my-app/2.381942946570489905/lib', 
  ...
  ...
 ]

sys.path in module_1.py:

 [
  '/base/data/home/apps/s~my-app/module_1:2.381942955973772449',      
  '/base/data/home/runtimes/python27/python27_dist/lib/python27.zip', 
   ...
   ...
  ]

Only folder containing *.yaml file will be uploaded on GAE, so your `lib` folder should be inside of your module. Make a symlink. — Dmytro Sadovnychyi, Feb 02 '15 at 05:48

Paul Liang · Accepted Answer · 2015-02-04T11:16:40.177

Credit goes to Google Cloud Platform Tech Solutions Representative Adam:

Modules documentations may not be explicitly stated, but the folder 'Module1', 'Module2' as well as the default module actually run inside separate Python virtual environments on separate instances and need to be self contained. They cannot 'see' any directories above them which exist on the local filesystem, and 'default.py' can't see anything in each of the module directories. The whole folder tree isn't copied to each module instance.

He suggested that instead of making symlinks, just copy ./lib to each of the modules.

I do not like the idea very much.

Firstly, these modules share some base class, duplicating them is really an anti-pattern.

Secondly, copying the lib folders everywhere corrupts unit tests as nose will try to run all unit tests it can run, also because it's a pain to explicitly exclude the directories.

At the end of the day, I wrote a makefile to help deployment / testing easier...

# Create simlinks before deployment.
deploy: mksimlnks
    appcfg.py --oauth2 update $(CURDIR)/app.yaml
    appcfg.py --oauth2 update $(CURDIR)/MODULE_1/module_1.yaml
    appcfg.py --oauth2 update $(CURDIR)/MODULE_2/module_2.yaml
    appcfg.py --oauth2 update_queues $(CURDIR)

mksimlnks:
    ln -s $(CURDIR)/lib $(CURDIR)/MODULE_1/lib
    ln -s $(CURDIR)/lib $(CURDIR)/MODULE_2/lib

# Need to remove symlinks before unittest
# or unit test will explode.
test: rmsimlnks
    nosetests --exclude-dir=lib --with-gae -w $(CURDIR) --with-coverage --cover-html

# Remove all symlinks
rmsimlnks:
    rm -rf $(shell find * -type l)

# remove symlinks and other stuff
clean: rmsimlnks
    rm -f $(shell find * -name *.pyc)
    rm -f $(shell find * -name .DS_Store)
    rm -f .coverage
    rm -rf $(CURDIR)/cover

How to import lib folder within Modules

1 Answers1

Linked