4

This is not a question about how to capture logging on celery workers. Is there any way to capture celery logging on a Producer. What I want is to capture every log that get generated by celery on the Producer when I call task.delay(...) or task.apply_async(...).

EDIT: I don't want to capture worker logs on producer. I want to capture everything that happen in celery from the time of my call to apply_async until the task is sent to the broker.

konoufo
  • 427
  • 5
  • 15

1 Answers1

1

No, there is no way to capture worker logs on the producer. All you get is the exception, if thrown. Any logging is happening on the worker side, so you have to examine logs of that particular worker, or if you use some centralised log system then you have to look for logs from that worker...

Update: seems like you want to capture eventual logging from Celery on the producer (client) side. As far as I know Celery and the underlying transport handling library (Kombu) do not log anything. I could be wrong of course, but I can't remember seeing any logging there and I have read Celery (Kombu not that much to be fair) code many times...

A possible solution is to make Celery workers send logs to some centralised system that your Celery client can access...

DejanLekic
  • 18,787
  • 4
  • 46
  • 77
  • I do not want to capture worker logs. I want to capture producer logs. What I mean is I want to see every log of everything Celery does *while sending* a task . Does this make sense ? – konoufo Aug 09 '19 at 18:43
  • Seems indeed to be the case that no logs are directly produced by either Celery or Kombu. – konoufo Aug 21 '19 at 18:13
  • Celery may not but a custom task (like mine) certainly does. It can log information before apply_sync publishes the task and I landed here with the same question as @konoufo because I would like to use the Celery logger (for consistency) but see the log on the console of my client (Producer in konoufo's words). And I'm stuck for now. – Bernd Wechner Apr 17 '20 at 10:56
  • @BerndWechner the solution is either use the centralized log server OR just use your favorite Message Queue to send over the logs. I'm working on writing a simple module for the latter. – konoufo Apr 18 '20 at 07:11