0

I'm trying to run background task processing, redis and rabbitMQ work in separate docker containers


    @dramatiq.actor(store_results=True)
    def count_words(url):
        try:
            response = requests.get(url)
            count = len(response.text.split(" "))
            print(f"There are {count} words at {url!r}.")
        except requests.exceptions.MissingSchema:
            print(f"Message dropped due to invalid url: {url!r}")
    
    result_backend = RedisBackend(host="172.17.0.2", port=6379)
    result_broker = RabbitmqBroker(host="172.17.0.5", port=5672)
    
    result_broker.add_middleware(Results(backend=result_backend))
    
    dramatiq.set_broker(result_broker)
    message = count_words.send('https://github.com/Bogdanp/dramatiq')
    print(message.get_result(block=True))

RabbitMQ:

{"queue_name":"default","actor_name":"count_words","args":["https://github.com/Bogdanp/dramatiq"],"kwargs":{},"options":{},"message_id":"8e10b6ef-dfef-47dc-9f28-c6e07493efe4","message_timestamp":1608877514655}

Redis


    1:C 22 Dec 2020 13:38:15.415 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
    1:M 22 Dec 2020 13:38:15.417 * Running mode=standalone, port=6379.
    1:M 22 Dec 2020 13:38:15.417 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
    1:M 22 Dec 2020 13:38:15.417 # Server initialized
    1:M 22 Dec 2020 13:38:15.417 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
    1:M 22 Dec 2020 13:38:15.417 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled (set to 'madvise' or 'never').
    1:M 25 Dec 2020 10:08:12.274 * Background saving terminated with success
    1:M 26 Dec 2020 19:23:59.445 * 1 changes in 3600 seconds. Saving...
    1:M 26 Dec 2020 19:23:59.660 * Background saving started by pid 24
    24:C 26 Dec 2020 19:23:59.890 * DB saved on disk
    24:C 26 Dec 2020 19:23:59.905 * RDB: 4 MB of memory used by copy-on-write
    1:M 26 Dec 2020 19:23:59.961 * Background saving terminated with success

Error:


    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/dramatiq/message.py", line 147, in get_result
        return backend.get_result(self, block=block, timeout=timeout)
      File "/usr/local/lib/python3.6/dist-packages/dramatiq/results/backends/redis.py", line 81, in get_result
        raise ResultTimeout(message)
    dramatiq.results.errors.ResultTimeout: count_words('https://github.com/Bogdanp/dramatiq')
Максим
  • 1
  • 1
  • How are you launching the containers? Is there any error from the worker process? – David Maze Dec 27 '20 at 12:58
  • @DavidMaze docker run -d --name rabbitmq -p 5672:5672 -p 15672:15672 -p 15674:15674 -p 25672:25672 -p 61613:61613 -v rabbitmq_data:/var/lib/rabbitmq rabbitmq:3.6.14-management and docker run --name redis -d redis No errors during startup – Максим Dec 27 '20 at 14:27
  • If nothing else, the containers probably need to be on the same `docker run --net`. Editing the question to include these details would be more readable than including them in a comment. – David Maze Dec 27 '20 at 16:06
  • @DavidMaze I know their ip addresses, so they don't have to be on the same network – Максим Dec 27 '20 at 16:25
  • Those IP addresses will change as soon as you delete and recreate the containers; I wouldn't rely on them (or even look them up). Both Redis and PostgreSQL are TCP-based so you'll probably get a more affirmative error ("connection refused") if that's the actual problem, though. – David Maze Dec 27 '20 at 16:44

0 Answers0