How to hide Celery task id from structlog?

Question

I used Structlog and Celery in my Django application and both work very well, but I would like to prevent task_id from appearing in the log when the worker executes tasks. How can I do that?

The reason is that task_id is a key:value of 36 characters long so it makes log output hard to read.

2020-07-23 14:20:00.052156 [info     ] Update started                   [data.tasks] task_id=b79c9b3b-ae4b-41c6-951a-72b4f19fa2ac
2020-07-23 14:20:01.659316 [info     ] Update complete                  [data.models] exchange=aaa new=0 task_id=b79c9b3b-ae4b-41c6-951a-72b4f19fa2ac time=0.42 update=0
2020-07-23 14:20:01.936658 [info     ] Update complete                  [data.models] exchange=bbbbbb new=0 task_id=b79c9b3b-ae4b-41c6-951a-72b4f19fa2ac time=0.03 update=0
2020-07-23 14:20:02.451733 [info     ] Update complete                  [data.models] exchange=hhh new=0 task_id=b79c9b3b-ae4b-41c6-951a-72b4f19fa2ac time=0.28 update=0

This is how my structlog setup for Celery:

structlog.configure(
    processors=[
        structlog.stdlib.filter_by_level,
        structlog.processors.TimeStamper(fmt="%Y-%m-%d %H:%M:%S.%f"),
        structlog.stdlib.add_logger_name,
        structlog.stdlib.add_log_level,
        structlog.stdlib.PositionalArgumentsFormatter(),
        structlog.processors.StackInfoRenderer(),
        structlog.processors.format_exc_info,
        structlog.processors.UnicodeDecoder(),
        structlog.processors.ExceptionPrettyPrinter(),
        structlog.stdlib.ProcessorFormatter.wrap_for_formatter,
    ],
    context_class=structlog.threadlocal.wrap_dict(dict),
    logger_factory=structlog.stdlib.LoggerFactory(),
    wrapper_class=structlog.stdlib.BoundLogger,
    cache_logger_on_first_use=True,
)


@receiver(signals.modify_context_before_task_publish)
def receiver_modify_context_before_task_publish(sender, signal, context):
    keys_to_keep = {"request_id", "parent_task_id"}
    new_dict = {key_to_keep: context[key_to_keep] for key_to_keep in keys_to_keep if key_to_keep in context}
    context.clear()
    context.update(new_dict)

Thank you

score 1 · Accepted Answer · answered Jul 24 '20 at 03:27

The format of a log line is controlled by the formatter set on the handler of the logger in question. If you're seeing the task ID in the log, that implies that you are hitting the celery task logger (celery.utils.log.get_task_logger). Modify that formatter, and you can configure logging to remove the task ID from the log line. Here is a sample yaml dictionary config you can use with logging.dictConfig to control the format of the celery task logger:

loggers:
  celery.task:
    level: INFO
    propagate: false
    handlers:
      - brief
handlers:
  brief:
    class: logging.StreamHandler
    formatter: brief
    level: DEBUG
    stream: ext://sys.stdout
formatters:
  brief:
    (): logging.Formatter
    format: '{log_color}[{name}] {message}'
    style: '{'

How to hide Celery task id from structlog?

1 Answers1