35

I have a python project and I want to be able to install it using something like python setup.py install so that the installation automatically creates a systemd service.

I'm having some trouble, most probably setting the paths or imports correctly.

My environment:

  • Ubuntu 15.04
  • Python 2.7 (although it would be great to make it work in py3 too).

Project Structure:

+ top-folder
  + super_project
    + folder1
      __init__.py
      file1.py
    + folder2
      __init__.py
      file2.py
    __init__.py
    main.py
  setup.py
  setup.cfg

setup.py:

from setuptools.command.install import install
from setuptools import setup, find_packages
import subprocess
import os


class CustomInstallCommand(install):

  def run(self):
    install.run(self)
    current_dir_path = os.path.dirname(os.path.realpath(__file__))
    create_service_script_path = os.path.join(current_dir_path, 'super_project', 'install_scripts', 'create_service.sh')
    subprocess.check_output([create_service_script_path])

setup(
  name='super-project',
  author='Myself',
  version='0.0.1',
  description='My Description',
  packages=find_packages(exclude=['contrib', 'docs']),
  # this will create the /usr/local/bin/super-project entrypoint script
  entry_points={
    'console_scripts': [
      'super-project = super_project.main:main'
    ]
  },
  cmdclass={'install': CustomInstallCommand}
)

main.py

from super_project.folder1.file1 import Class1
from super_project.folder2.file2 import Class2
import logging


def main():
  logging.info('Executing super-project...')
  (...)
  logging.info('super-project execution finished.')

if __name__ == '__main__':
  main()

setup.cfg

[bdist_wheel]
universal=1

create_service.sh (more or less):

SYSTEMD_SCRIPT_DIR=$( cd  $(dirname "${BASH_SOURCE:=$0}") && pwd)
cp -f "$SYSTEMD_SCRIPT_DIR/super-project.service" /lib/systemd/system
chown root:root /lib/systemd/system/super-project.service

systemctl daemon-reload
systemctl enable super-project.service

super-project.service

[Unit]
Description=Super Description

[Service]
Type=simple
ExecStart=/usr/local/bin/super-service
Restart=always

[Install]
WantedBy=multi-user.target

The installation of the package generates the following output:

$ sudo python setup.py install --record files.txt
running install
running build
running build_py
copying super_project/main.py - build/lib.linux-x86_64-2.7/super_project
running install_lib
copying build/lib.linux-x86_64-2.7/super_project/__init__.py - /usr/local/lib/python2.7/dist-packages/super_project
copying build/lib.linux-x86_64-2.7/super_project/main.py - /usr/local/lib/python2.7/dist-packages/super_project
copying build/lib.linux-x86_64-2.7/super_project/db/__init__.py - /usr/local/lib/python2.7/dist-packages/super_project/db
copying build/lib.linux-x86_64-2.7/super_project/db/db_gateway.py - /usr/local/lib/python2.7/dist-packages/super_project/db
(...)
byte-compiling /usr/local/lib/python2.7/dist-packages/super_project/__init__.py to
__init__.pyc
byte-compiling /usr/local/lib/python2.7/dist-packages/super_project/main.py to
main.pyc
byte-compiling /usr/local/lib/python2.7/dist-packages/super_project/db/__init__.py to
__init__.pyc
byte-compiling /usr/local/lib/python2.7/dist-packages/super_project/db/db_gateway.py
to db_gateway.pyc
(...)
running install_egg_info
running egg_info
writing requirements to super_project.egg-info/requires.txt
writing super_project.egg-info/PKG-INFO
writing top-level names to super_project.egg-info/top_level.txt
writing dependency_links to super_project.egg-info/dependency_links.txt
writing entry points to super_project.egg-info/entry_points.txt
reading manifest file 'super_project.egg-info/SOURCES.txt'
writing manifest file 'super_project.egg-info/SOURCES.txt'
Copying super_project.egg-info to /usr/local/lib/python2.7/dist-packages/super_project-0.0.1.egg-info
running install_scripts
Installing ai-scenario-qa script to /usr/local/bin
writing list of installed files to 'files.txt'

The super-project file is created in /usr/local/bin:

#!/usr/bin/python
# EASY-INSTALL-ENTRY-SCRIPT: 'super-project==0.0.1','console_scripts','super-project'
__requires__ = 'super-project==0.0.1'
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.exit(
        load_entry_point('super-project==0.0.1', 'console_scripts', 'super-project')()
    )

The installation seems successful, although:

$ systemctl status super-project.service
● super-project.service
   Loaded: not-found (Reason: No such file or directory)
   Active: inactive (dead)

The error I can see in /var/log/syslog:

 Feb 16 20:48:34  systemd[1]: Starting  Super Description...
 Feb 16 20:48:34  super-project[22517]: Traceback (most recent call last):
 Feb 16 20:48:34  super-project[22517]: File "/usr/local/bin/super-project", line 9, in <module
 Feb 16 20:48:34  super-project[22517]: load_entry_point('super-project==0.0.1', 'console_scripts', 'super-project')()
 Feb 16 20:48:34  super-project[22517]: File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 521, in load_entry_point
 Feb 16 20:48:34  super-project[22517]: return get_distribution(dist).load_entry_point(group, name)
 Feb 16 20:48:34  super-project[22517]: File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2632, in load_entry_point
 Feb 16 20:48:34  super-project[22517]: return ep.load()
 Feb 16 20:48:34  super-project[22517]: File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2312, in load
 Feb 16 20:48:34  super-project[22517]: return self.resolve()
 Feb 16 20:48:34  super-project[22517]: File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2318, in resolve
 Feb 16 20:48:34  super-project[22517]: module = __import__(self.module_name, fromlist=['__name__'], level=0)
 Feb 16 20:48:34  super-project[22517]: ImportError: No module named main
 Feb 16 20:48:34  systemd[1]: super-project.service: main process exited, code=exited, status=1/FLURE
 Feb 16 20:48:34  systemd[1]: Unit super-project.service entered fled state.
 Feb 16 20:48:34  systemd[1]: super-project.service failed.
 Feb 16 20:48:34  systemd[1]: super-project.service holdoff time over, scheduling restart.
 Feb 16 20:48:34  systemd[1]: start request repeated too quickly for super-project.service
 Feb 16 20:48:34  systemd[1]: Failed to start Super Description.
 Feb 16 20:48:34  systemd[1]: Unit super-project.service entered fled state.
 Feb 16 20:48:34  systemd[1]: super-project.service failed.

As can be seen, the module main cannot be found. This is the main problem.

When changing code/conf, I remove the super-project/service as follows:

$ sudo systemctl disable super-project.service
$ sudo rm -f /lib/systemd/system/super-project.service
$ sudo systemctl daemon-reload
$ su
# cat files.txt | xargs rm -r

On the other hand:

  • If I execute $ super-project from /usr/local/bin/, the script starts correctly (no import exception) but the configuration files cannot be read (most probably because of relative/absolute path issues).
  • If I execute $ super-project from top-folder (folder containing the project code/files) the script runs perfectly

What am I missing? I've spend a lot of time searching what the issue might be. It seems that the package is correctly set up in the dist-packages directory and all the service files are correctly created once the setup is executed.

I've read things about using from __future__ import absolute_import, but I'm not sure if I have to add that to my main.py (it does not work) or to all the files in my project.

pylover
  • 7,670
  • 8
  • 51
  • 73
newlog
  • 1,050
  • 1
  • 11
  • 23
  • Try logging sys.path from the script and compare it to whatever it is when starting manually. – ther Feb 28 '16 at 19:32
  • 11
    I suggest different approach; Python packages themselves should stay neutral as possible and do not force any systemd manipulation which imply root priviledges during `setup.py` run. What happens if user wants to install your package to a virtualenv for development? Instead, I suggest your application ships with a separate script in `bin` which can do systemd installation: https://packaging.python.org/en/latest/distributing/#entry-points – Mikko Ohtamaa Feb 28 '16 at 19:43
  • 6
    Alternatively build a package for your operating system (`deb`, `rpm`, etc.) as that's the right place to put operating system specific installation commands. – Mikko Ohtamaa Feb 28 '16 at 19:44
  • Or use something like ansible to provision your machine, install python package and install service config. – irqed Mar 01 '16 at 16:12
  • Wonder if [this](http://mechanix-tips.blogspot.com/2014/03/importerror-entry-point-consolescripts.html) could help you with the `ImportError`? – augurar Mar 02 '16 at 07:49
  • @ther The sys.path is the same if I run it from any directory. I cannot see the sys.path when the service is started because it does not even get to execute main.py – newlog Mar 07 '16 at 19:41
  • @augurar I've changed the permissions (to 777, just testing) of everything related to my module inside the dist-packages directory, but when restarting the service, the import error still occurred. – newlog Mar 07 '16 at 19:43
  • @MikkoOhtamaa I'm already using the entry_points feature to call the main module and it does not work. I think that what you are suggesting is the same I'm doing by overriding the run() method of the setuptools install class. In any case, I guess that at some point I'll have to try using a deb package. But it pisses me off not knowing what is going on :) – newlog Mar 07 '16 at 19:47
  • Is there any reason you want to use setup.py to do this as opposed to using a Makefile? – James Mchugh Nov 14 '19 at 14:12

2 Answers2

1

You get an ImportError, because the module in question is not in sys.path or not accessible, because of some file system permissions.
Here's a script to check file system permissions of a given distribution, group and name.

chk_perm.py

from pkg_resources import get_distribution
import os
import sys

dist, group, name = sys.argv[1:]
dist = get_distribution(dist)
location = dist.location
einfo = dist.get_entry_info(group, name)
if not einfo:
    print('No such group "{}" or name "{}"'.format(group, name))
    sys.exit(1)
m_name = einfo.module_name
path = format(os.path.join(location, *m_name.split('.')))
path = path if os.access(path, os.F_OK) else '{}.py'.format(path)
print('If path "{}" exists: {}'.format(path, os.access(path, os.F_OK) if path.endswith('.py') else True))
print('If path "{}" readable: {}'.format(path, os.access(path, os.R_OK)))

Test;

$ python chk_perm.py setuptools console_scripts easy_install
If path "lib/python2.7/site-packages/setuptools/command/easy_install.py" exists: True
If path "lib/python2.7/site-packages/setuptools/command/easy_install.py" readable: True

$ foo
Traceback (most recent call last):
  File "bin/foo", line 9, in <module>
    load_entry_point('mypkg==0.0.4', 'console_scripts', 'foo')()
  File "lib/python2.7/site-packages/pkg_resources/__init__.py", line 549, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "lib/python2.7/site-packages/pkg_resources/__init__.py", line 2542, in load_entry_point
    return ep.load()
  File "lib/python2.7/site-packages/pkg_resources/__init__.py", line 2202, in load
    return self.resolve()
  File "lib/python2.7/site-packages/pkg_resources/__init__.py", line 2208, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
ImportError: No module named main

$ python chk_perm.py mypkg console_scripts foo
If path "lib/python2.7/site-packages/pkg/main.py" exists: True
If path "lib/python2.7/site-packages/pkg/main.py" readable: False

$ ls -l lib/python2.7/site-packages/pkg/main.py 
-rw-rw---- 1 root root 104 Mar  6 22:52 lib/python2.7/site-packages/pkg/main.py

$ sudo chmod o+r lib/python2.7/site-packages/pkg/main.py
$ ls -l lib/python2.7/site-packages/pkg/main.py 
-rw-rw-r-- 1 root root 104 Mar  6 22:52 lib/python2.7/site-packages/pkg/main.py

$ python chk_perm.py mypkg console_scripts foo
If path "lib/python2.7/site-packages/pkg/main.py" exists: True
If path "lib/python2.7/site-packages/pkg/main.py" readable: True

$ foo
App is running
Nizam Mohamed
  • 8,751
  • 24
  • 32
  • The main.py in my dist-packages directory has the following permissions (-rw-rw-r-), so I guess this is not the problem. Thanks though. – newlog Mar 07 '16 at 18:53
  • 1
    @newlog parent directory of main.py must have execute bit set for directory traversal. – Nizam Mohamed Mar 08 '16 at 05:41
  • @Newlog importing the `main` module within the package might help. Try adding `from . import main` in `pkg/__init__.py`. – Nizam Mohamed Mar 08 '16 at 06:17
0

Be sure that you application ca be executed from other directories, this seems like a classic case where you assume that current directory is where the start script exists.

It has nothing to do with with systemd. Also try to execute the start command from outside your login shell (your .profile is not loaded by services).

sorin
  • 161,544
  • 178
  • 535
  • 806