logging.BasicConfig encoding param is not working

Question

I'm getting logging error (UnicodeEncodeError) while trying log and print url which contain not ASCII letter. I specified encoding while setting up logger in my_logging.py

---

my_logging.py

import logging
import sys
from pathlib import Path


def get_logger(filepath: Path) -> None:
    logging.basicConfig(
        level=logging.DEBUG,
        encoding='utf-8',
        format="[{asctime},{msecs:03.0f}]:[{levelname}]:{message}",
        datefmt='%d.%m.%Y %H:%M:%S',
        style='{',
        handlers=[
            logging.FileHandler(filepath, mode='a'),
            logging.StreamHandler(sys.stdout),
        ]
    )

    logging.getLogger('asyncio').setLevel(logging.WARNING)


def log_and_print(msg: str) -> None:
    logging.info(msg)
    print(msg)

Error:

--- Logging error ---
Traceback (most recent call last):
  File "C:\Users\vboxuser\AppData\Local\Programs\Python\Python311\Lib\logging\__init__.py", line 1113, in emit
    stream.write(msg + self.terminator)
  File "C:\Users\vboxuser\AppData\Local\Programs\Python\Python311\Lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode characters in position 156-157: character maps to <undefined>
Call stack:
  File "C:\Users\vboxuser\30.rossiya-airlines\app.py", line 162, in run
    asyncio.run(download_reports_for_month(login=self.login, password=self.password, month=self.month))
  File "C:\Users\vboxuser\AppData\Local\Programs\Python\Python311\Lib\asyncio\runners.py", line 190, in run
    return runner.run(main)
  File "C:\Users\vboxuser\AppData\Local\Programs\Python\Python311\Lib\asyncio\runners.py", line 118, in run
    return self._loop.run_until_complete(task)
  File "C:\Users\vboxuser\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 640, in run_until_complete
    self.run_forever()
  File "C:\Users\vboxuser\AppData\Local\Programs\Python\Python311\Lib\asyncio\windows_events.py", line 321, in run_forever
    super().run_forever()
  File "C:\Users\vboxuser\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 607, in run_forever
    self._run_once()
  File "C:\Users\vboxuser\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 1922, in _run_once
    handle._run()
  File "C:\Users\vboxuser\AppData\Local\Programs\Python\Python311\Lib\asyncio\events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "C:\Users\vboxuser\30.rossiya-airlines\scrapper.py", line 138, in download_reports_for_month
    await _download_reports_for_month(login=login, password=password, month=month)
  File "C:\Users\vboxuser\30.rossiya-airlines\scrapper.py", line 169, in _download_reports_for_month
    log_and_print(f'Start downloading {url=}')
  File "C:\Users\vboxuser\30.rossiya-airlines\my_logging.py", line 27, in log_and_print
    logging.info(msg)
Message: "Start downloading url='https://edu.rossiya-airlines.com/workplan/view_flight_report-1?flight_date=02-08-2023&flight_number=ФВ6321&legnum=1&departure_airport_id=705&arrival_airport_id=307&id_para=1214928'"
Arguments: ()

Line raising error:

log_and_print(f'Start downloading {url=}')

---

I can workaround exceptions by encoding every msg manually:

def log_and_print(msg: str) -> None:
    logging.info(msg.encode('utf-8')
    print(msg)

But then binary prefix appears. I don't like it so much and want to understand why passing encoding='utf-8' in logging.BasicConfig is not working as I expect. What am i missing?

---

Windows 10, Python 3.11.4

Answer 1

The issue you are experiencing is due to the fact that the StreamHandler used to log messages to the console is using the default encoding of your system, which is cp1252 on Windows. This encoding does not support all Unicode characters, which is why you are getting a UnicodeEncodeError when trying to log a message containing non-ASCII characters.

The encoding parameter in the basicConfig function only applies to the FileHandler, which is used to log messages to a file. It does not affect the encoding used by the StreamHandler.

To fix this issue, you can explicitly set the encoding of the StreamHandler to 'utf-8' when creating it. Please refer below:

import logging
import sys
from pathlib import Path

def get_logger(filepath: Path) -> None:
    stream_handler = logging.StreamHandler(sys.stdout)
    stream_handler.setStream(sys.stdout)
    stream_handler.setFormatter(logging.Formatter(fmt="[{asctime},{msecs:03.0f}]:[{levelname}]:{message}", datefmt='%d.%m.%Y %H:%M:%S', style='{'))
    
    logging.basicConfig(
        level=logging.DEBUG,
        format="[{asctime},{msecs:03.0f}]:[{levelname}]:{message}",
        datefmt='%d.%m.%Y %H:%M:%S',
        style='{',
        handlers=[
            logging.FileHandler(filepath, mode='a', encoding='utf-8'),
            stream_handler,
        ]
    )

    logging.getLogger('asyncio').setLevel(logging.WARNING)

def log_and_print(msg: str) -> None:
    logging.info(msg)
    print(msg)

In the above code, we create a StreamHandler object and set its stream to sys.stdout and its formatter to a Formatter object with the desired format. We then pass this StreamHandler object to the handlers parameter of the basicConfig function, along with the FileHandler.