1

I'm still new to Python and Celery. I've setup Celery app to send all logs to Google Cloud Logging.

It seems that part of the logs are sent as DEBUG, but some as INFO, causing me so much strain when trying to read them...

Example logging handler

LOGGING_GCP = {
'version': 1,
'disable_existing_loggers': True,
'formatters': {
    'standard': {
        'class': 'logging.Formatter',
        'format': '%(message)s | %(module)s:%(lineno)d',
    },
},
'handlers': {
    'gcp': {
        'level': 'DEBUG',
        'class': 'google.cloud.logging.handlers.ContainerEngineHandler',
        'formatter': 'standard',
    },
},
'loggers': {
    '': {
        'handlers': ['gcp'],
        'level': 'DEBUG',
        'propagate': 'False',
    }
}

}

Celery is started with --loglevel DEBUG.

Scheduler config


    CELERYD_HIJACK_ROOT_LOGGER=False,
    CELERYD_REDIRECT_STDOUTS=False,

If I check logs I below as DEBUG:

"tasks.someTaskName sent. id->26108d79-46cd-41dc-a074-925fc2dedff8"
"Task accepted: tasks.someTaskName[6781cc58-d158-45dd-a9a8-5b6c26239a79] pid:8"
"beat: Waking up now."

While these are printed as INFO:

"Received task: tasks.someTaskName[59955295-2dc1-4667-b163-2c4181658dd3] "
"Scheduler: Sending due task someTaskName (tasks.someTaskName)"
"Task tasks.someTaskName[6781cc58-d158-45dd-a9a8-5b6c26239a79] succeeded in 0.0221633310002s: None"

I'd like INFO tasks to be printed as DEBUG so that my actual application INFO logs are clearly separated.

dzh
  • 286
  • 1
  • 19
  • That makes no sense. If you instruct Celery to run with DEBUG log-level you will get both DEBUG and INFO (as well as all other) logging entries. They are easy to separate anyway because they have "DEBUG" and "INFO" words in the logging stream... – DejanLekic Nov 04 '21 at 11:02
  • I see, the flags are to control what gets printed, not at what level. I guess it's just unfortunate that Celery prints INFO messages willy-nilly. Was able to filter them out in GCP logs but it's unnecessary PITA. – dzh Nov 05 '21 at 00:03
  • I do not get it... If log entry says it is an INFO entry, then it is an INFO entry (that particular log-level entry). – DejanLekic Nov 05 '21 at 17:07

1 Answers1

0

You can't change how Celery creates the logs: the Celery code calls log.info(...) for logs that the developers think should be INFO, log.debug(...) for DEBUG, etc.

You can change what your application does with those logs. The loggers section in your config describes how to configure specific loggers: each key is a logger name. The logger names are usually module names, so you can specify them at any level that makes sense.

If you don't want some INFO entries at all, set the logger to WARN for that subset. This will log WARN+ for modules under celery.worker, and INFO+ for other all celery logs:

    'celery': {
        'handlers': ['gcp'],
        'level': 'INFO',
    },
    'celery.worker': {
        'handlers': ['gcp'],
        'level': 'WARN',
    }

If you really don't want celery logs mixed with your application logs, you could use a different handler to send them somewhere else:

    'celery': {
        'handlers': ['gcp-celery'],
        'level': 'DEBUG',
    }
kielni
  • 4,779
  • 24
  • 21