Why is python structlog logging a redactError every time I add a list as a key word arg?

Question

I'm using python structlog version 19.1.0 (https://www.structlog.org/en/19.1.0/index.html) and in a previous deployment, I used this same version and did not get any redactError messages in my logs. But now, whenever I add key-word args to a log message (especially if the value is a list), it also logs a redactError.

My code:

logger = structlog.get_logger()
logger.debug("My custom event name", email_addresses=["some.email.address@mail.com"])

The log message:

{"event":"My custom event name", "email_addresses":["some.email.address@mail.com"], "level":"info", "msg":"\u003cnil\u003e", "redactError":"1 error occurred:\n\t* error redacting item email_addresses: 1 error occurred:\n\t* error redacting item 0: unknown interface interface {} not redacted\n\n\n\n","request":"b8214486-d270-416b-ba3a-cc36b784c19d", "stream":"stderr", "thread":140500243846912, "time":"2020-07-22T19:28:50Z","timestamp":"2020-07-22T19:28:50.737388Z"}

I have no idea why it's giving me a redactError when it is clearly still able to add the key word arguments in the log message. I've also noticed that if the list is longer than 1, it will add more errors to the redactError string (e.g. "error redacting item <>: unknown interface interface {} not redacted\n" for every single index).

I'm really baffled as to why I'm getting this error. It hasn't interfered with my code's ability to execute, it just really clutters up the logs. Any help is appreciated!

Edit: thanks for some of the comments below. Adding my logging configuration in case that is the issue:

# setting up the logging config
level = getattr(logging, "DEBUG")
formatter = logging.Formatter("%(message)s")
sh = logging.StreamHandler()
sh.setFormatter(formatter)

logger = logging.getLogger()
logger.setLevel(level)
logger.addHandler(sh)

processors = [
    structlog.stdlib.filter_by_level,
    structlog.stdlib.add_log_level,
    structlog.stdlib.add_logger_name,
    structlog.processors.format_exc_info,
    structlog.processors.TimeStamper(fmt='iso', utc=True),
    structlog.processors.StackInfoRenderer(),
    add_thread,
    structlog.processors.JSONRenderer(),
]

structlog.configure(
    processors=processors,
    context_class=structlog.threadlocal.wrap_dict(dict),
    logger_factory=structlog.stdlib.LoggerFactory(),
    wrapper_class=structlog.stdlib.BoundLogger,
    cache_logger_on_first_use=True,
)

I was unable to reproduce this and I don't see anything in structlog about redacting. Is the log message going through some other software before it gets to you? Did you do something not shown to set up or configure the logger object? — Dennis Sparrow, Jul 22 '20 at 23:48
@DennisSparrow, thanks for trying to reproduce the error. I just added my configuration above. Everything looks fine to me, but maybe you'll spot something! — Emily Collins Nielsen, Jul 23 '20 at 13:35

score 1 · Answer 1 · answered Jul 23 '20 at 07:49

1

This error does not come from structlog.

It looks like you have either a structlog processor or a stdlib logging handler that is trying to redact personal identifying information from your log entries and it seems to be confused by the fact that you email addresses field is a list and not a string.

We'd need more information about your logging and structlog configuration to tell more.

answered Jul 23 '20 at 07:49

hynek

3,647
1
18
26

Thanks for confirming that this isn't structlog. I just added my configuration above in case the error is coming from there. Everything looks fine to me, but maybe you'll spot something! – Emily Collins Nielsen Jul 23 '20 at 13:36
It appears the issue is JSONRenderer.... When I change `structlog.processors.JSONRenderer()` to `structlog.processors.KeyValueRenderer()`, the redactError doesn't happen. I don't know why JSONRenderer doesn't like lists, but here we are! – Emily Collins Nielsen Jul 23 '20 at 14:04
I also don't think it's the PII. I tried putting a generic list in as a key word arg, `to_adds=["thing1","thing2"]`, and that also gave me a redactError: `"redactError":"1 error occurred:\n\t* error redacting item to_adds: 2 errors occurred:\n\t* error redacting item 0: unknown interface interface {} not redacted\n\t* error redacting item 1: unknown interface interface {} not redacted\n\n\n\n"` – Emily Collins Nielsen Jul 23 '20 at 14:46
Could it be that something else is injecting handlers/processors? Try https://www.structlog.org/en/stable/api.html#structlog.get_config – hynek Jul 23 '20 at 16:07

score 0 · Answer 2 · answered Jul 23 '20 at 14:47

It appears the issue is structlog.processors.JSONRenderer.... When I change structlog.processors.JSONRenderer() to structlog.processors.KeyValueRenderer(), the redactError doesn't happen. I don't know why JSONRenderer doesn't like lists, but maybe someone else knows.

Why is python structlog logging a redactError every time I add a list as a key word arg?

2 Answers2