0

We observed a memory leak in a code making the following call with a structlog logger in python:

logger.debug("RANGEB obs {}/{}".format(i + 1,obs_cnt), observation=observation)

Details:

  • observation is a dictionary.
  • the program works in a multi-threaded application.

It is very strange since the leak appears around every 600 calls and around 250kB were leaked.

It took us some time to hunt the memory leak and we got it with memory_profiler. Is this possible that we miss a configuration parameter of structlog that will prevent the memory leak ?

Manuel Yguel
  • 198
  • 1
  • 9
  • I am sure community would appreciate more details on this problem. It would help to answer the question, too – Marat Sep 21 '21 at 17:25
  • Also, formatting log messages is not a great idea - pass this onto the [logging module itself](https://docs.python.org/3/library/logging.html#logrecord-attributes) – Marat Sep 21 '21 at 17:28
  • Can you elaborate why formatting log messages is not a great idea ? – Manuel Yguel Sep 22 '21 at 09:19
  • 1
    As Marat said, we’d need to see your config for both structlog and – if used – stdlib logging. String-formatting _and_ structlog doesn’t make a log of sense tho: just use them as key/values: `logger.debug("RANGEB obs", num=i+1, obs_cnt=obs_cnt, observation=observation)` whether you use structlog or stdlib for formatting it is a matter of choice. Your approach has the downside of that the string gets _aways_ rendered which is a runtime overhead you’d rather avoid on debug levels. – hynek Sep 22 '21 at 09:48

0 Answers0