0

Attempting to self-host readthedocs internally and had very little experience with Django.

I've followed all the steps on readthedocs (created virtualenv in /usr/share/ owned by me with group ownership www-data) for the development webserver and that works fine.

Have taken the following additional steps to setup wsgi and apache to selfhost:

  • installed additional dependencies

    • sudo apt-get install libapache2-mod-wsgi libpq-dev redis-server
    • cd /usr/share/readthedocs/
    • pip install psycopg2 redis django-redis-cache django-redis-sessions
    • pip install --upgrade hiredis

  • configured redis to listen on socket and relaxed permissions to 777

apache was configured as a vhost with:

<VirtualHost *:80>
    ServerName docs.vagrant

    WSGIScriptAlias / /usr/share/readthedocs/checkouts/readthedocs.org/readthedocs/wsgi.py
    WSGIProcessGroup www-data
    WSGIDaemonProcess www-data \
    python-path=/usr/share/readthedocs/checkouts/readthedocs.org/readthedocs/:/usr/share/readthedocs/lib/python2.7/site-packages/:/usr/share/readthedocs/:/usr/share/readthedocs/checkouts/readthedocs.org/:/usr/share/readthedocs/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/

    <Directory /usr/share/readthedocs/checkouts/readthedocs.org/readthedocs/>
        <Files wsgi.py>
        Require all granted
        </Files>
    </Directory>
</VirtualHost>

and the traffic seems to be going to the backend correctly. However RTD complains about a missing attribute REDIS. Apache error log output :

[wsgi:info] [pid 19595] mod_wsgi (pid=19595): Starting process 'www-data' with uid=33, gid=33 and threads=15.
[wsgi:info] [pid 19595] mod_wsgi (pid=19595): Initializing Python.
[wsgi:info] [pid 19595] mod_wsgi (pid=19595): Attach interpreter ''.
[wsgi:info] [pid 19596] mod_wsgi (pid=19596): Attach interpreter ''.
[wsgi:info] [pid 19597] mod_wsgi (pid=19597): Attach interpreter ''.
[wsgi:info] [pid 19595] mod_wsgi (pid=19595): Adding '/usr/share/readthedocs/checkouts/readthedocs.org/readthedocs/' to path.
[wsgi:info] [pid 19595] mod_wsgi (pid=19595): Adding '/usr/share/readthedocs/lib/python2.7/site-packages/' to path.
[wsgi:info] [pid 19595] mod_wsgi (pid=19595): Adding '/usr/share/readthedocs/' to path.
[wsgi:info] [pid 19595] mod_wsgi (pid=19595): Adding '/usr/share/readthedocs/checkouts/readthedocs.org/' to path.
[wsgi:info] [pid 19595] mod_wsgi (pid=19595): Adding '/usr/share/readthedocs/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/' to path.
[wsgi:info] [pid 19595] mod_wsgi (pid=19595): Create interpreter 'docs.vagrant:3380|'.
[wsgi:info] [pid 19595] mod_wsgi (pid=19595): Adding '/usr/share/readthedocs/checkouts/readthedocs.org/readthedocs/' to path.
[wsgi:info] [pid 19595] mod_wsgi (pid=19595): Adding '/usr/share/readthedocs/lib/python2.7/site-packages/' to path.
[wsgi:info] [pid 19595] mod_wsgi (pid=19595): Adding '/usr/share/readthedocs/' to path.
[wsgi:info] [pid 19595] mod_wsgi (pid=19595): Adding '/usr/share/readthedocs/checkouts/readthedocs.org/' to path.
[wsgi:info] [pid 19595] mod_wsgi (pid=19595): Adding '/usr/share/readthedocs/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/' to path.
[wsgi:info] [pid 19595] [remote 10.0.2.2:512] mod_wsgi (pid=19595, process='www-data', application='docs.vagrant:3380|'): Loading WSGI script '/usr/share/readthedocs/checkouts/readthedocs.org/readthedocs/wsgi.py'.
[wsgi:info] [pid 19624] mod_wsgi (pid=19624): Initializing Python.
[wsgi:info] [pid 19624] mod_wsgi (pid=19624): Attach interpreter ''.
[wsgi:error] [pid 19595] [09/Sep/2015 10:30:01] WARNING [readthedocs.api.client:16] SLUMBER_USERNAME/PASSWORD settings are not set
[wsgi:error] [pid 19595] [remote 10.0.2.2:512] mod_wsgi (pid=19595): Exception occurred processing WSGI script '/usr/share/readthedocs/checkouts/readthedocs.org/readthedocs/wsgi.py'.
[wsgi:error] [pid 19595] [remote 10.0.2.2:512] Traceback (most recent call last):
[wsgi:error] [pid 19595] [remote 10.0.2.2:512]   File "/usr/share/readthedocs/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 170, in __call__
[wsgi:error] [pid 19595] [remote 10.0.2.2:512]     self.load_middleware()
[wsgi:error] [pid 19595] [remote 10.0.2.2:512]   File "/usr/share/readthedocs/lib/python2.7/site-packages/django/core/handlers/base.py", line 52, in load_middleware
[wsgi:error] [pid 19595] [remote 10.0.2.2:512]     mw_instance = mw_class()
[wsgi:error] [pid 19595] [remote 10.0.2.2:512]   File "/usr/share/readthedocs/lib/python2.7/site-packages/django/middleware/locale.py", line 24, in __init__
[wsgi:error] [pid 19595] [remote 10.0.2.2:512]     for url_pattern in get_resolver(None).url_patterns:
[wsgi:error] [pid 19595] [remote 10.0.2.2:512]   File "/usr/share/readthedocs/lib/python2.7/site-packages/django/core/urlresolvers.py", line 402, in url_patterns
[wsgi:error] [pid 19595] [remote 10.0.2.2:512]     patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
[wsgi:error] [pid 19595] [remote 10.0.2.2:512]   File "/usr/share/readthedocs/lib/python2.7/site-packages/django/core/urlresolvers.py", line 396, in urlconf_module
[wsgi:error] [pid 19595] [remote 10.0.2.2:512]     self._urlconf_module = import_module(self.urlconf_name)
[wsgi:error] [pid 19595] [remote 10.0.2.2:512]   File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
[wsgi:error] [pid 19595] [remote 10.0.2.2:512]     __import__(name)
[wsgi:error] [pid 19595] [remote 10.0.2.2:512]   File "/usr/share/readthedocs/checkouts/readthedocs.org/readthedocs/urls.py", line 8, in <module>
[wsgi:error] [pid 19595] [remote 10.0.2.2:512]     from readthedocs.api.base import (ProjectResource, UserResource,
[wsgi:error] [pid 19595] [remote 10.0.2.2:512]   File "/usr/share/readthedocs/checkouts/readthedocs.org/readthedocs/api/base.py", line 28, in <module>
[wsgi:error] [pid 19595] [remote 10.0.2.2:512]     redis_client = redis.Redis(**settings.REDIS)
[wsgi:error] [pid 19595] [remote 10.0.2.2:512]   File "/usr/share/readthedocs/lib/python2.7/site-packages/django/conf/__init__.py", line 49, in __getattr__
[wsgi:error] [pid 19595] [remote 10.0.2.2:512]     return getattr(self._wrapped, name)
[wsgi:error] [pid 19595] [remote 10.0.2.2:512] AttributeError: 'Settings' object has no attribute 'REDIS'
[wsgi:error] [pid 19595] [remote 10.0.2.2:51327] mod_wsgi (pid=19595): Exception occurred processing WSGI script '/usr/share/readthedocs/checkouts/readthedocs.org/readthedocs/wsgi.py'.
[wsgi:error] [pid 19595] [remote 10.0.2.2:51327] Traceback (most recent call last):
[wsgi:error] [pid 19595] [remote 10.0.2.2:51327]   File "/usr/share/readthedocs/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 170, in __call__
[wsgi:error] [pid 19595] [remote 10.0.2.2:51327]     self.load_middleware()
[wsgi:error] [pid 19595] [remote 10.0.2.2:51327]   File "/usr/share/readthedocs/lib/python2.7/site-packages/django/core/handlers/base.py", line 52, in load_middleware
[wsgi:error] [pid 19595] [remote 10.0.2.2:51327]     mw_instance = mw_class()
[wsgi:error] [pid 19595] [remote 10.0.2.2:51327]   File "/usr/share/readthedocs/lib/python2.7/site-packages/django/middleware/locale.py", line 24, in __init__
[wsgi:error] [pid 19595] [remote 10.0.2.2:51327]     for url_pattern in get_resolver(None).url_patterns:
[wsgi:error] [pid 19595] [remote 10.0.2.2:51327]   File "/usr/share/readthedocs/lib/python2.7/site-packages/django/core/urlresolvers.py", line 402, in url_patterns
[wsgi:error] [pid 19595] [remote 10.0.2.2:51327]     patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
[wsgi:error] [pid 19595] [remote 10.0.2.2:51327]   File "/usr/share/readthedocs/lib/python2.7/site-packages/django/core/urlresolvers.py", line 396, in urlconf_module
[wsgi:error] [pid 19595] [remote 10.0.2.2:51327]     self._urlconf_module = import_module(self.urlconf_name)
[wsgi:error] [pid 19595] [remote 10.0.2.2:51327]   File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
[wsgi:error] [pid 19595] [remote 10.0.2.2:51327]     __import__(name)
[wsgi:error] [pid 19595] [remote 10.0.2.2:51327]   File "/usr/share/readthedocs/checkouts/readthedocs.org/readthedocs/urls.py", line 8, in <module>
[wsgi:error] [pid 19595] [remote 10.0.2.2:51327]     from readthedocs.api.base import (ProjectResource, UserResource,
[wsgi:error] [pid 19595] [remote 10.0.2.2:51327]   File "/usr/share/readthedocs/checkouts/readthedocs.org/readthedocs/api/base.py", line 28, in <module>
[wsgi:error] [pid 19595] [remote 10.0.2.2:51327]     redis_client = redis.Redis(**settings.REDIS)
[wsgi:error] [pid 19595] [remote 10.0.2.2:51327]   File "/usr/share/readthedocs/lib/python2.7/site-packages/django/conf/__init__.py", line 49, in __getattr__
[wsgi:error] [pid 19595] [remote 10.0.2.2:51327]     return getattr(self._wrapped, name)
[wsgi:error] [pid 19595] [remote 10.0.2.2:51327] AttributeError: 'Settings' object has no attribute 'REDIS'

I realise I will also need elasticsearch on and properly configured but I will cross that bridge when I get that far.

UPDATE: Issuing the command grep -i redis /usr/share/readthedocs/checkouts/readthedocs.org/readthedocs/settings/* returned 

__init__.py:        'BACKEND': 'redis_cache.RedisCache',
__init__.py:            'PARSER_CLASS': 'redis.connection.HiredisParser'
onebox.py:REDIS = {
onebox.py:BROKER_URL = 'redis://localhost:6379/0'
onebox.py:CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'
onebox.py:        'BACKEND': 'redis_cache.cache.RedisCache',
onebox.py:            'CLIENT_CLASS': 'redis_cache.client.DefaultClient',
postgres.py:        'BACKEND': 'redis_cache.RedisCache',
postgres.py:            'PARSER_CLASS': 'redis.connection.HiredisParser'
sqlite.py:REDIS = {
sqlite.py:BROKER_URL = 'redis://localhost:6379/0'
sqlite.py:CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'

so I've reconfigured redis to no longer use the socket and go back to bind/port but the issue persists (with the same stacktrace).

Alasdair
  • 566
  • 1
  • 4
  • 16
  • does it work with the dev server? – e4c5 Sep 10 '15 at 08:55
  • yes the dev server, as invoked by `./manage.py runserver` works, as stated above in the second sentence '... and that works fine.' – Alasdair Sep 10 '15 at 12:27
  • Sorry missed that bit. This part of the stacktrace ' redis.Redis(**settings.REDIS)' says that it's trying to open a connection to redis using parameters you are supposed to have in your settings.py is your settings file different between production and dev? – e4c5 Sep 10 '15 at 12:48
  • Thanks for your input. the settings is a directory `/usr/share/readthedocs/checkouts/readthedocs.org/readthedocs/settings` with the files `base.py base.pyc docker.py __init__.py __init__.pyc onebox.py postgres.py sqlite.py sqlite.pyc test.py` grepping those files for redis i noticed in `onebox` and `sqlite` it set the broker url to `redis://localhost:6379` so I enabled redis back on the port instead of the socket and restarted but the issue remains with the stacktrace being generated identical to before. `redis-cli ping` does respond with `PONG` – Alasdair Sep 10 '15 at 14:41

1 Answers1

0

Okay got past this issue now. Had to add:

/usr/share/readthedocs/checkouts/readthedocs.org/readthedocs/settings/

to the apache python-path.

and:

REDIS = { 
    'host': 'localhost',
    'port': 6379,
    'db': 0,
}

to the postgres.py in settings (symlink from init.py)

Now to sort the rest of the missing dependency errors.

Alasdair
  • 566
  • 1
  • 4
  • 16