1

My Django development website with background processing via redis works locally, but doesn't work on heroku. My requirements.txt contains 'redis==2.7.2' and 'rq==0.3.2' and I spin up web=1 and worker=1 on heroku. I have the 'redistogo:nano' add-on. The instructions for verifying that it works on the heroku console don't work: there's no 'redis.set()' or 'redis.get()'. So I tried the code from Running Heroku background tasks with only 1 web dyno and 0 worker dynos as follows. In settings.py:

import os
import os.path
import socket
import sys
import urlparse
from rq import Queue
from worker import conn

redis_q = Queue(connection=conn)
...

Here's worker.py:

import os
import redis
from rq import Worker, Queue, Connection
listen = ['high', 'default', 'low']
redis_url = os.getenv('REDISTOGO_URL', 'redis://localhost:6379')
conn = redis.from_url(redis_url)
if __name__ == '__main__':
    with Connection(conn):
        worker = Worker(map(Queue, listen))
        worker.work()

Here are fragments of the view doing the background call:

from upload import *  # where process_upload_file() lives
...
    catchall = {...}
    catchall_string = str(catchall)
    redis_q.enqueue(process_upload_file, catchall_string)
...

And upload.py has:

def process_upload_file(catchall_string):
    ...

Why doesn't this work?

(For extra credit, if somebody can give me a better way to pass the arguments as a string, it would be appreciated.)

Procfile:

web: gunicorn exim.wsgi -b "0.0.0.0:$PORT"
worker: python worker.py

Here are the heroku logs from the latest attempt after setting up an S3 bucket and pointing heroku to it. It couldn't find either 'http://s3.amazonaws.com/mycompany/sample.csv' (shown here) or 'http://mycompany.s3.amazonaws.com/sample.csv'.

    0:46:31+00:00 app[worker.1]: [2013-02-09 00:46] INFO: worker: 
    2013-02-09T00:46:44+00:00 heroku[web.1]: Starting process with command `gunicorn myapp.wsgi -b "0.0.0.0:27999"`
    2013-02-09T00:46:45+00:00 app[web.1]: 2013-02-09 00:46:45 [2] [INFO] Starting gunicorn 0.15.0
    2013-02-09T00:46:45+00:00 app[web.1]: 2013-02-09 00:46:45 [2] [INFO] Listening at: http://0.0.0.0:27999 (2)
    2013-02-09T00:46:45+00:00 app[web.1]: 2013-02-09 00:46:45 [2] [INFO] Using worker: sync
    2013-02-09T00:46:45+00:00 app[web.1]: 2013-02-09 00:46:45 [5] [INFO] Booting worker with pid: 5
    2013-02-09T00:46:50+00:00 heroku[web.1]: State changed from starting to up
    2013-02-09T00:47:15+00:00 heroku[router]: at=info method=GET path=/favicon.ico host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=3ms service=3132ms status=404 bytes=30
    2013-02-09T00:47:15+00:00 heroku[router]: at=info method=GET path=/upl/upload75/ host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=6ms service=980ms status=200 bytes=958
    2013-02-09T00:47:15+00:00 heroku[router]: at=info method=GET path=/favicon.ico host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=2ms service=16ms status=404 bytes=30
    2013-02-09T00:47:21+00:00 heroku[router]: at=info method=POST path=/upl/upload75/ host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=4ms service=244ms status=302 bytes=5
    2013-02-09T00:47:22+00:00 heroku[router]: at=info method=GET path=/upl/upload77/ host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=2ms service=162ms status=200 bytes=44024
    2013-02-09T00:47:22+00:00 heroku[router]: at=info method=GET path=/static/css/bootstrap.min.css host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=1ms connect=3ms service=26ms status=200 bytes=95438
    2013-02-09T00:47:22+00:00 heroku[router]: at=info method=GET path=/static/css/style.css host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=2ms service=15ms status=200 bytes=515
    2013-02-09T00:47:22+00:00 heroku[router]: at=info method=GET path=/static/js/jquery-1.7.2.min.js host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=3ms service=17ms status=200 bytes=94840
    2013-02-09T00:47:22+00:00 heroku[router]: at=info method=GET path=/static/js/scripts.js host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=1ms connect=6ms service=5ms status=200 bytes=509
    2013-02-09T00:47:22+00:00 heroku[router]: at=info method=GET path=/static/js/bootstrap.min.js host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=2ms service=7ms status=200 bytes=22323
    2013-02-09T00:47:22+00:00 heroku[router]: at=info method=GET path=/static/snippets/footer.html host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=2ms service=16ms status=200 bytes=537
    2013-02-09T00:47:22+00:00 heroku[router]: at=info method=GET path=/static/img/groovepaper.png host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=1ms service=11ms status=200 bytes=74596
    2013-02-09T00:47:22+00:00 heroku[router]: at=info method=GET path=/static/font/fontawesome-webfont.woff host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=3ms service=12ms status=200 bytes=41752
    2013-02-09T00:47:22+00:00 heroku[router]: at=info method=GET path=/favicon.ico host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=1ms service=15ms status=404 bytes=30
    2013-02-09T00:47:37+00:00 heroku[router]: at=info method=POST path=/upl/upload77/ host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=2ms service=182ms status=200 bytes=98
    2013-02-09T00:47:37+00:00 app[worker.1]: [2013-02-09 00:47] INFO: worker: default: upload.process_upload_file("{'server_file': u'/app/myapp/myapp/uploads/d27/org20130208194721.csv', 'client_file': u'aorg.csv', 'myco_id': 27, 'user_email': u'me.myself@mycompany.com', 'ignore_header_line': True, 'content_type': 'org', 'column_choices': {u'org_name': 0, u'addr1': 1, u'country_code': 8}}") (eb79ed7e-cff8-4071-a0c8-b1d4b1bd834f)
    2013-02-09T00:47:37+00:00 heroku[router]: at=info method=GET path=/favicon.ico host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=1ms service=9ms status=404 bytes=30
    2013-02-09T00:47:42+00:00 app[worker.1]: [2013-02-09 00:47] ERROR: horse: IOError: [Errno 2] No such file or directory: 'http://s3.amazonaws.com/mycompany/sample.csv'
    2013-02-09T00:47:42+00:00 app[worker.1]:     rv = job.perform()
    2013-02-09T00:47:42+00:00 app[worker.1]:   File "/app/refactor.py", line 72, in read_uploaded_file
    2013-02-09T00:47:42+00:00 app[worker.1]:     allrows = csv.reader(open(filepath, 'rU'), dialect='excel')
    2013-02-09T00:47:42+00:00 app[worker.1]: IOError: [Errno 2] No such file or directory: 'http://s3.amazonaws.com/mycompany/sample.csv'
    2013-02-09T00:47:42+00:00 app[worker.1]:   File "/app/refactor.py", line 41, in read_csv
    2013-02-09T00:47:42+00:00 app[worker.1]: [2013-02-09 00:47] WARNING: horse: Moving job to failed queue.
    2013-02-09T00:47:42+00:00 app[worker.1]: Traceback (most recent call last):
    2013-02-09T00:47:42+00:00 app[worker.1]:     output, nrows, error = acceptable_upload_filetypes[ext](filepath, **kwargs)
    2013-02-09T00:47:42+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/rq/worker.py", line 393, in perform_job
    2013-02-09T00:47:42+00:00 app[worker.1]: 
    2013-02-09T00:47:42+00:00 app[worker.1]: [2013-02-09 00:47] DEBUG: horse: Invoking exception handler <bound method Worker.move_to_failed_queue of <rq.worker.Worker object at 0x12cb310>>
    2013-02-09T00:47:42+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/rq/job.py", line 328, in perform
    2013-02-09T00:47:42+00:00 app[worker.1]:     self._result = self.func(*self.args, **self.kwargs)
    2013-02-09T00:47:42+00:00 app[worker.1]:   File "/app/upload.py", line 248, in process_upload_file
    2013-02-09T00:47:42+00:00 app[worker.1]:     all_rows, nrows, error = read_uploaded_file(server_file, maxrows=maxrows)
    2013-02-09T00:47:42+00:00 app[worker.1]: [2013-02-09 00:47] INFO: worker: 
    2013-02-09T00:47:42+00:00 app[worker.1]: [2013-02-09 00:47] INFO: worker: *** Listening on high, default, low...
    (venv)me@host:~/myapp$
Community
  • 1
  • 1
user2044195
  • 11
  • 3
  • It would help folk if you supplied your Procfile, as well as a description of "doesn't work". What doesn't work? What do the logs say? Perhaps add them to your question (`heroku logs`) – Jon Mountjoy Feb 06 '13 at 11:11
  • When I went to try it again, to get logs to post, I got errors I hadn't seen before. It was missing the environment variable DJANGO_SETTINGS_MODULE. I fixed that with 'heroku config:add DJANGO_SETTINGS_MODULE=exim.settings' Then the worker couldn't find the uploaded file. I may need to use AWS S3 to store the file if it's not where the worker can see it. Still looking into it. – user2044195 Feb 06 '13 at 23:29
  • Specifically, IOError: [Errno 2] No such file or directory: u'/app/exim/exim/uploads/d27/pro20130206164202.xls' – user2044195 Feb 06 '13 at 23:39
  • I found this in a description of heroku's ephemeral filesystem: "During the dyno’s lifetime its running processes can use the filesystem as a temporary scratchpad, but no files that are written are visible to processes in any other dyno . . ." Since the worker is in another dyno, I'll have to use AWS S3 to store the uploaded files. – user2044195 Feb 07 '13 at 04:12
  • I've created an AWS bucket and set up heroku with the key ID and the key, and I manually uploaded a file to the bucket. Heroku still can't find the file, although at least it's not telling me there's no such bucket and it's not denying access. I've tried 'http://mybucket.s3.amazonaws.com/myfile' and 'http://s3.amazonaws.com/mybucket/myfile'. Here's the heroku logs: – user2044195 Feb 09 '13 at 01:36
  • Well done - looks like you have the Heroku config. Does your app run locally? It looks like you have a bucket permission problem perhaps? – Jon Mountjoy Feb 09 '13 at 14:39
  • My app ran locally with local storage and redis, but not with S3 storage. It can see the bucket but doesn't find the file. I've used aws at the terminal to print the file; it finds it OK that way. – user2044195 Feb 11 '13 at 14:08
  • I did a minimal test: I manually uploaded the file to my bucket, then changed the background process to read that file explicitly. That way I didn't have to change any settings or test that the file uploaded. Then I went through the motions of uploading my user copy of the file and then kicked off the background process. All it had to do was find the file that was already there. It can see the bucket but not the file. – user2044195 Feb 11 '13 at 14:40

0 Answers0