18

On Windows 10 the logging module send this error (using scrapy)

# --- Logging error ---
...
# UnicodeEncodeError: 'charmap' codec can't encode characters in position 175-176: character maps to <undefined>

I have read that I should add encoding='utf-8' but I did not find how I could add it in the code below. EDIT: According to the tuto it isnt needed. 

configure_logging(install_root_handler=False) #override default log settings
logging.basicConfig(
    filename='logfile.log',
    format='%(levelname)s: %(message)s',
    datefmt='%m-%d %H:%M',
    level=logging.INFO #CRITICAL ERROR WARNING  INFO    DEBUG    NOTSET 
)

I found many questions on the topics, but mostly on python 2 (or not related to the logging module). And the logging tutorial don't even talk about utf-8. (Notice that I can print UTF8 characters without any problem. The problem only occur with the logging module)

J. Does
  • 785
  • 3
  • 10
  • 23

1 Answers1

34

I'm not using Scrapy but I ran into the same issue with vanilla Python 3.6 logging of not being able to write UTF-8 characters via logging to a file (console worked just fine).

Based on this comment I added 'utf-8' to my FileHandler initialization. I'm configuring logging via an INI file so it looks like this:

[handler_file]
class = FileHandler
args = (r'log.txt', 'a', 'utf-8')
level = NOTSET
formatter = generic

To use logging.basicConfig() I think that you should be able to do the following:

configure_logging(install_root_handler=False) #override default log settings
logging.basicConfig(
    handlers=[logging.FileHandler('logfile.log', 'w', 'utf-8')],
    format='%(levelname)s: %(message)s',
    datefmt='%m-%d %H:%M',
    level=logging.INFO #CRITICAL ERROR WARNING  INFO    DEBUG    NOTSET 
)
Cukic0d
  • 5,111
  • 2
  • 19
  • 48
smsearcy
  • 513
  • 5
  • 8
  • The answer with `logging.basicConfig()` doesn't work for me. – SomJura Nov 03 '20 at 10:48
  • @SomJura, can you be a little more specific? I just tried it on Python 3.8 and it seemed to work (I did skip the `configure_logging()` call). – smsearcy Nov 06 '20 at 04:28
  • It seems that it didn't work because I was using `import logging`, which is not necessay anymore with newer versions of Python. – SomJura Nov 08 '20 at 12:54
  • Is it possible in windows? Facing the same issue, I have tried the above solution but it's not working. I'm using Python 3.8.10. I tried removing the import statement, 'import logging', but that's causing errors. 'NameError: name 'logging' is not defined' – Abhirami V S Oct 26 '21 at 06:32
  • @Abhi, I initially had this problem in Windows so it should work in Windows. What error are you having? Reviewing the second `logging.basicConfig()` example, I'm not sure what that first line with `configure_logging()` is doing there, what if you delete that? You will need `import logging`. – smsearcy Oct 28 '21 at 18:07
  • @smsearcy the error is resolved by upgrading Python version to 3.9, I saw this https://bugs.python.org/issue37111, and the fix was in Python 3.9. – Abhirami V S Nov 02 '21 at 11:58
  • Oh my god thanks – A-Makeyev Jun 25 '23 at 19:07