Celery Flower API result not returning

Question

Calling Flower API's /api/task/apply/* is just hanging without the results returning, even though the task was processed and succeeded according to the Web UI.

Reproducing the error:

# hangs even though task succeeded in logs
curl -d '{"args": [1, 2]}' -H "Content-Type: application/json" -X POST http://0.0.0.0:5000/api/task/apply/tasks.add

Checking the task by id just shows it as pending even though the Flower UI shows it as a success and I also see the task result in the worker logs.

curl -X GET http://0.0.0.0:5000/api/task/result/02fa2656-c92e-44e6-a2af-c04e666e3f89
# {"task-id": "02fa2656-c92e-44e6-a2af-c04e666e3f89", "state": "PENDING"}

I suspect this log line is indicative of the issue: celery-worker_1 | [2018-03-06 15:25:15,160: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None].

There's also a Traceback in the flower side (see logs below) which happens when I hit Ctrl+C to get out of the hanging curl command above.

tasks.py

from celery import Celery

app = Celery('tasks', broker='redis://celery-redis:6379//0', backend='redis://celery-redis:6379//0')


@app.task
def add(a, b):
    return a + b

Logs

celery-worker_1  | [2018-03-06 15:24:36,271: DEBUG/MainProcess] pidbox received method active() [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': 'afe22ec9-5cde-330c-9306-c8dea457eecb'} ticket:48fb973b-51a1-4ad4-9482-04e31b19fb14]
celery-worker_1  | [2018-03-06 15:24:36,282: DEBUG/MainProcess] pidbox received method reserved() [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': 'd9911fb2-c681-3fe2-bdb2-248c14bf8351'} ticket:b69f2b6c-b7d7-4d26-a398-970ef22e4280]
celery-worker_1  | [2018-03-06 15:24:36,288: DEBUG/MainProcess] pidbox received method revoked() [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': '31402263-0a94-3226-89ee-2eb9e0a138b9'} ticket:0a33227a-9b3e-4d9b-bbfd-fd7aa108403b]
celery-worker_1  | [2018-03-06 15:24:36,303: DEBUG/MainProcess] pidbox received method conf(with_defaults=False) [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': 'bb3879ef-1424-3c02-a3eb-6c33faf985d2'} ticket:6ef872d6-d512-4393-bcdb-9ad8982afdd5]
celery-flower_1  | [W 180306 15:24:37 control:44] 'stats' inspect method failed
celery-flower_1  | [W 180306 15:24:37 control:44] 'active_queues' inspect method failed
celery-flower_1  | [W 180306 15:24:37 control:44] 'registered' inspect method failed
celery-flower_1  | [W 180306 15:24:37 control:44] 'scheduled' inspect method failed
celery-flower_1  | [D 180306 15:24:37 app:62] Successfully updated worker cache
celery-worker_1  | [2018-03-06 15:24:40,160: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
celery-worker_1  | [2018-03-06 15:24:40,160: INFO/MainProcess] Events of group {task} enabled by remote.
celery-worker_1  | [2018-03-06 15:24:45,159: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
celery-worker_1  | [2018-03-06 15:24:50,159: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
celery-worker_1  | [2018-03-06 15:24:55,158: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
celery-worker_1  | [2018-03-06 15:25:00,160: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
celery-worker_1  | [2018-03-06 15:25:05,158: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
celery-flower_1  | [D 180306 15:25:08 tasks:126] Invoking a task 'tasks.add' with '[1, 2]' and '{}'
celery-worker_1  | [2018-03-06 15:25:08,153: INFO/MainProcess] Received task: tasks.add[59ad1f38-dfab-4f09-8dfe-772789b3b630]  
celery-worker_1  | [2018-03-06 15:25:08,154: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fef7f6fa0d0> (args:('tasks.add', '59ad1f38-dfab-4f09-8dfe-772789b3b630', {'lang': 'py', 'task': 'tasks.add', 'id': '59ad1f38-dfab-4f09-8dfe-772789b3b630', 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '59ad1f38-dfab-4f09-8dfe-772789b3b630', 'parent_id': None, 'argsrepr': '[1, 2]', 'kwargsrepr': '{}', 'origin': 'gen1@bbc42c108b2f', 'reply_to': 'b64af5c4-8366-3299-bcd7-6eecb9eaf952', 'correlation_id': '59ad1f38-dfab-4f09-8dfe-772789b3b630', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': None}}, b'[[1, 2], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
celery-worker_1  | [2018-03-06 15:25:08,157: DEBUG/MainProcess] Task accepted: tasks.add[59ad1f38-dfab-4f09-8dfe-772789b3b630] pid:8
celery-worker_1  | [2018-03-06 15:25:08,159: INFO/ForkPoolWorker-1] Task tasks.add[59ad1f38-dfab-4f09-8dfe-772789b3b630] succeeded in 0.002012488999753259s: 3
celery-worker_1  | [2018-03-06 15:25:10,159: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
celery-worker_1  | [2018-03-06 15:25:15,160: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
celery-flower_1  | Exception in thread Thread-2:
celery-flower_1  | Traceback (most recent call last):
celery-flower_1  |   File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
celery-flower_1  |     self.run()
celery-flower_1  |   File "/usr/lib/python3.6/threading.py", line 864, in run
celery-flower_1  |     self._target(*self._args, **self._kwargs)
celery-flower_1  |   File "/usr/lib/python3.6/site-packages/flower/api/tasks.py", line 154, in wait_results
celery-flower_1  |     self.finish(response)
celery-flower_1  |   File "/usr/lib/python3.6/site-packages/tornado/web.py", line 1018, in finish
celery-flower_1  |     self.flush(include_footers=True)
celery-flower_1  |   File "/usr/lib/python3.6/site-packages/tornado/web.py", line 973, in flush
celery-flower_1  |     start_line, self._headers, chunk, callback=callback)
celery-flower_1  |   File "/usr/lib/python3.6/site-packages/tornado/http1connection.py", line 400, in write_headers
celery-flower_1  |     future = self._write_future = Future()
celery-flower_1  |   File "/usr/lib/python3.6/asyncio/events.py", line 678, in get_event_loop
celery-flower_1  |     return get_event_loop_policy().get_event_loop()
celery-flower_1  |   File "/usr/lib/python3.6/asyncio/events.py", line 584, in get_event_loop
celery-flower_1  |     % threading.current_thread().name)
celery-flower_1  | RuntimeError: There is no current event loop in thread 'Thread-2'.

Dockerfile

FROM alpine:3.6

# System setup
ENV USER app
RUN addgroup -g 1000 app && adduser -D -G app -u 1000 app
ENV dir /home/app
ENV LC_ALL=en_US.utf-8

# System dependencies
RUN apk add --no-cache python3 python3-dev
RUN python3 -m ensurepip
RUN pip3 install --upgrade pip setuptools
RUN apk add --no-cache openssl-dev \
  gcc \
  musl-dev \
  git

WORKDIR ${dir}

# Update pip
RUN pip3 install --upgrade pip setuptools

# App dependencies
RUN pip3 install celery[redis]==4.1.0 \
  flower==0.9.2

# File upload
COPY tasks.py ${dir}/tasks.py
RUN chown -R ${USER}: ${dir}

# App setup
USER ${USER}

# Flower port
EXPOSE 5000

CMD true

docker-compose.yml

version: '3'
services:
  celery-redis:
    image: redis:4-alpine
  celery-worker:
    build: .
    image: example/worker
    command: celery -A tasks worker --loglevel=DEBUG --concurrency 1
    depends_on:
      - celery-redis
    restart: always
  celery-flower:
    build: .
    image: example/flower
    command: celery flower -A tasks --port=5000 --debug
    depends_on:
      - celery-redis
    ports:
      - '5000:5000'

Itay Granik · Accepted Answer · 2018-03-13T15:54:36.973

1

i had the same issue and i was able to solve it.

in DockerFile add the following :

RUN pip install tornado==4.5.2

4.5.2 is an older version which works fine with apply call , the newest version of tornado is causing the bug.

hope it help , let me know if you got more issues , i am having a running env now.

edited Mar 13 '18 at 15:54

answered Mar 13 '18 at 13:36

Itay Granik

36
2

1

This should be in the comment section – Sandwell Mar 13 '18 at 13:45
1

@Thanks for figuring it out! – Kamil Sindi Mar 13 '18 at 16:26

score 0 · Answer 2 · answered Mar 10 '18 at 07:40

0

You are starting flower with celery flower -A tasks. So flower starts on default port 5555.

You should start flower with flower -A tasks --port=5000 or change ports of flower service to 5555 and send requests from that port.

answered Mar 10 '18 at 07:40

Chillar Anand

27,936
9
119
136

Apologies I had a typo in the question. I was using `--port=5000`. Does the above work for you? – Kamil Sindi Mar 10 '18 at 14:38

Celery Flower API result not returning

2 Answers2