Python french locale on Windows 10 not UTF8?

Question

I'm trying to write a multiplateform script which works OK on Mac Os but failed on Windows if I load the french local. I use the following code at startup to load the locale depending on the host OS :

import locale
from sys import platform

if platform == "linux" or platform == "linux2":
    locale.setlocale( locale.LC_ALL, 'fr_FR')
elif platform == "darwin":
    locale.setlocale( locale.LC_ALL, 'fr_FR')
elif platform == "win32":
    locale.setlocale( locale.LC_ALL, 'French_France.1252')

My code also use following libraries :

import pandas as pd
import numpy as np

import matplotlib.pyplot as plt

import time as time
import datetime

The code failed when I display a localised date on a matplotlib figure. The faulty line is :

fig.tight_layout()

And return this error :

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 10: invalid continuation byte

Obviously a problem of encoding.

My computer use windows 10 and my python setup is Python 3.6.4 |Anaconda 4.4.0 (64-bit)| (default, Jan 16 2018, 10:22:32) [MSC v.1900 64 bit (AMD64)]

The full error returned is :

'Periode du lundi 04/12/2017 17:11:41 au lundi 04/12/2017 21:59:59 \n Duree de 4h48m19s'

---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-28-4b9aa2e0100e> in <module>()
     48     fig=trace_et_leq_ln(et)
     49 
---> 50     fig.tight_layout()
     51     fig.subplots_adjust(top=0.85)
     52 

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\figure.py in tight_layout(self, renderer, pad, h_pad, w_pad, rect)
   2028         kwargs = get_tight_layout_figure(
   2029             self, self.axes, subplotspec_list, renderer,
-> 2030             pad=pad, h_pad=h_pad, w_pad=w_pad, rect=rect)
   2031         self.subplots_adjust(**kwargs)
   2032 

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\tight_layout.py in get_tight_layout_figure(fig, axes_list, subplotspec_list, renderer, pad, h_pad, w_pad, rect)
    349                                      subplot_list=subplot_list,
    350                                      ax_bbox_list=ax_bbox_list,
--> 351                                      pad=pad, h_pad=h_pad, w_pad=w_pad)
    352 
    353     if rect is not None:

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\tight_layout.py in auto_adjust_subplotpars(fig, renderer, nrows_ncols, num1num2_list, subplot_list, ax_bbox_list, pad, h_pad, w_pad, rect)
    127             continue
    128 
--> 129         tight_bbox_raw = union([ax.get_tightbbox(renderer) for ax in subplots
    130                                 if ax.get_visible()])
    131         tight_bbox = TransformedBbox(tight_bbox_raw,

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\tight_layout.py in <listcomp>(.0)
    128 
    129         tight_bbox_raw = union([ax.get_tightbbox(renderer) for ax in subplots
--> 130                                 if ax.get_visible()])
    131         tight_bbox = TransformedBbox(tight_bbox_raw,
    132                                      fig.transFigure.inverted())

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\axes\_base.py in get_tightbbox(self, renderer, call_axes_locator)
   4013             bb.append(self._right_title.get_window_extent(renderer))
   4014 
-> 4015         bb_xaxis = self.xaxis.get_tightbbox(renderer)
   4016         if bb_xaxis:
   4017             bb.append(bb_xaxis)

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\axis.py in get_tightbbox(self, renderer)
   1088             return
   1089 
-> 1090         ticks_to_draw = self._update_ticks(renderer)
   1091         ticklabelBoxes, ticklabelBoxes2 = self._get_tick_bboxes(ticks_to_draw,
   1092                                                                 renderer)

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\axis.py in _update_ticks(self, renderer)
    972 
    973         interval = self.get_view_interval()
--> 974         tick_tups = list(self.iter_ticks())
    975         if self._smart_bounds and tick_tups:
    976             # handle inverted limits

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\axis.py in iter_ticks(self)
    919         self.major.formatter.set_locs(majorLocs)
    920         majorLabels = [self.major.formatter(val, i)
--> 921                        for i, val in enumerate(majorLocs)]
    922 
    923         minorLocs = self.minor.locator()

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\axis.py in <listcomp>(.0)
    919         self.major.formatter.set_locs(majorLocs)
    920         majorLabels = [self.major.formatter(val, i)
--> 921                        for i, val in enumerate(majorLocs)]
    922 
    923         minorLocs = self.minor.locator()

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\plotting\_converter.py in __call__(self, x, pos)
   1106         else:
   1107             fmt = self.formatdict.pop(x, '')
-> 1108             return Period(ordinal=int(x), freq=self.freq).strftime(fmt)
   1109 
   1110 

pandas/_libs/period.pyx in pandas._libs.period._Period.strftime()

pandas/_libs/period.pyx in pandas._libs.period.period_format()

pandas/_libs/period.pyx in pandas._libs.period._period_strftime()

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 10: invalid continuation byte

Error in callback <function install_repl_displayhook.<locals>.post_execute at 0x00000213979E4840> (for post_execute):

---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\pyplot.py in post_execute()
    148             def post_execute():
    149                 if matplotlib.is_interactive():
--> 150                     draw_all()
    151 
    152             # IPython >= 2

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\_pylab_helpers.py in draw_all(cls, force)
    148         for f_mgr in cls.get_all_fig_managers():
    149             if force or f_mgr.canvas.figure.stale:
--> 150                 f_mgr.canvas.draw_idle()
    151 
    152 atexit.register(Gcf.destroy_all)

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\backend_bases.py in draw_idle(self, *args, **kwargs)
   2059         if not self._is_idle_drawing:
   2060             with self._idle_draw_cntx():
-> 2061                 self.draw(*args, **kwargs)
   2062 
   2063     def draw_cursor(self, event):

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\backends\backend_agg.py in draw(self)
    428             # if toolbar:
    429             #     toolbar.set_cursor(cursors.WAIT)
--> 430             self.figure.draw(self.renderer)
    431         finally:
    432             # if toolbar:

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\artist.py in draw_wrapper(artist, renderer, *args, **kwargs)
     53                 renderer.start_filter()
     54 
---> 55             return draw(artist, renderer, *args, **kwargs)
     56         finally:
     57             if artist.get_agg_filter() is not None:

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\figure.py in draw(self, renderer)
   1297 
   1298             mimage._draw_list_compositing_images(
-> 1299                 renderer, self, artists, self.suppressComposite)
   1300 
   1301             renderer.close_group('figure')

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\image.py in _draw_list_compositing_images(renderer, parent, artists, suppress_composite)
    136     if not_composite or not has_images:
    137         for a in artists:
--> 138             a.draw(renderer)
    139     else:
    140         # Composite any adjacent images together

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\artist.py in draw_wrapper(artist, renderer, *args, **kwargs)
     53                 renderer.start_filter()
     54 
---> 55             return draw(artist, renderer, *args, **kwargs)
     56         finally:
     57             if artist.get_agg_filter() is not None:

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\axes\_base.py in draw(self, renderer, inframe)
   2435             renderer.stop_rasterizing()
   2436 
-> 2437         mimage._draw_list_compositing_images(renderer, self, artists)
   2438 
   2439         renderer.close_group('axes')

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\image.py in _draw_list_compositing_images(renderer, parent, artists, suppress_composite)
    136     if not_composite or not has_images:
    137         for a in artists:
--> 138             a.draw(renderer)
    139     else:
    140         # Composite any adjacent images together

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\artist.py in draw_wrapper(artist, renderer, *args, **kwargs)
     53                 renderer.start_filter()
     54 
---> 55             return draw(artist, renderer, *args, **kwargs)
     56         finally:
     57             if artist.get_agg_filter() is not None:

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\axis.py in draw(self, renderer, *args, **kwargs)
   1131         renderer.open_group(__name__)
   1132 
-> 1133         ticks_to_draw = self._update_ticks(renderer)
   1134         ticklabelBoxes, ticklabelBoxes2 = self._get_tick_bboxes(ticks_to_draw,
   1135                                                                 renderer)

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\axis.py in _update_ticks(self, renderer)
    972 
    973         interval = self.get_view_interval()
--> 974         tick_tups = list(self.iter_ticks())
    975         if self._smart_bounds and tick_tups:
    976             # handle inverted limits

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\axis.py in iter_ticks(self)
    919         self.major.formatter.set_locs(majorLocs)
    920         majorLabels = [self.major.formatter(val, i)
--> 921                        for i, val in enumerate(majorLocs)]
    922 
    923         minorLocs = self.minor.locator()

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\axis.py in <listcomp>(.0)
    919         self.major.formatter.set_locs(majorLocs)
    920         majorLabels = [self.major.formatter(val, i)
--> 921                        for i, val in enumerate(majorLocs)]
    922 
    923         minorLocs = self.minor.locator()

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\plotting\_converter.py in __call__(self, x, pos)
   1106         else:
   1107             fmt = self.formatdict.pop(x, '')
-> 1108             return Period(ordinal=int(x), freq=self.freq).strftime(fmt)
   1109 
   1110 

pandas/_libs/period.pyx in pandas._libs.period._Period.strftime()

pandas/_libs/period.pyx in pandas._libs.period.period_format()

pandas/_libs/period.pyx in pandas._libs.period._period_strftime()

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 10: invalid continuation byte

---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
~\AppData\Local\Continuum\Anaconda3\lib\site-packages\IPython\core\formatters.py in __call__(self, obj)
    339                 pass
    340             else:
--> 341                 return printer(obj)
    342             # Finally look for special method names
    343             method = get_real_method(obj, self.print_method)

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\IPython\core\pylabtools.py in <lambda>(fig)
    236 
    237     if 'png' in formats:
--> 238         png_formatter.for_type(Figure, lambda fig: print_figure(fig, 'png', **kwargs))
    239     if 'retina' in formats or 'png2x' in formats:
    240         png_formatter.for_type(Figure, lambda fig: retina_figure(fig, **kwargs))

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\IPython\core\pylabtools.py in print_figure(fig, fmt, bbox_inches, **kwargs)
    120 
    121     bytes_io = BytesIO()
--> 122     fig.canvas.print_figure(bytes_io, **kw)
    123     data = bytes_io.getvalue()
    124     if fmt == 'svg':

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\backend_bases.py in print_figure(self, filename, dpi, facecolor, edgecolor, orientation, format, **kwargs)
   2214                     orientation=orientation,
   2215                     dryrun=True,
-> 2216                     **kwargs)
   2217                 renderer = self.figure._cachedRenderer
   2218                 bbox_inches = self.figure.get_tightbbox(renderer)

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\backends\backend_agg.py in print_png(self, filename_or_obj, *args, **kwargs)
    505 
    506     def print_png(self, filename_or_obj, *args, **kwargs):
--> 507         FigureCanvasAgg.draw(self)
    508         renderer = self.get_renderer()
    509         original_dpi = renderer.dpi

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\backends\backend_agg.py in draw(self)
    428             # if toolbar:
    429             #     toolbar.set_cursor(cursors.WAIT)
--> 430             self.figure.draw(self.renderer)
    431         finally:
    432             # if toolbar:

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\artist.py in draw_wrapper(artist, renderer, *args, **kwargs)
     53                 renderer.start_filter()
     54 
---> 55             return draw(artist, renderer, *args, **kwargs)
     56         finally:
     57             if artist.get_agg_filter() is not None:

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\figure.py in draw(self, renderer)
   1297 
   1298             mimage._draw_list_compositing_images(
-> 1299                 renderer, self, artists, self.suppressComposite)
   1300 
   1301             renderer.close_group('figure')

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\image.py in _draw_list_compositing_images(renderer, parent, artists, suppress_composite)
    136     if not_composite or not has_images:
    137         for a in artists:
--> 138             a.draw(renderer)
    139     else:
    140         # Composite any adjacent images together

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\artist.py in draw_wrapper(artist, renderer, *args, **kwargs)
     53                 renderer.start_filter()
     54 
---> 55             return draw(artist, renderer, *args, **kwargs)
     56         finally:
     57             if artist.get_agg_filter() is not None:

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\axes\_base.py in draw(self, renderer, inframe)
   2435             renderer.stop_rasterizing()
   2436 
-> 2437         mimage._draw_list_compositing_images(renderer, self, artists)
   2438 
   2439         renderer.close_group('axes')

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\image.py in _draw_list_compositing_images(renderer, parent, artists, suppress_composite)
    136     if not_composite or not has_images:
    137         for a in artists:
--> 138             a.draw(renderer)
    139     else:
    140         # Composite any adjacent images together

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\artist.py in draw_wrapper(artist, renderer, *args, **kwargs)
     53                 renderer.start_filter()
     54 
---> 55             return draw(artist, renderer, *args, **kwargs)
     56         finally:
     57             if artist.get_agg_filter() is not None:

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\axis.py in draw(self, renderer, *args, **kwargs)
   1131         renderer.open_group(__name__)
   1132 
-> 1133         ticks_to_draw = self._update_ticks(renderer)
   1134         ticklabelBoxes, ticklabelBoxes2 = self._get_tick_bboxes(ticks_to_draw,
   1135                                                                 renderer)

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\axis.py in _update_ticks(self, renderer)
    972 
    973         interval = self.get_view_interval()
--> 974         tick_tups = list(self.iter_ticks())
    975         if self._smart_bounds and tick_tups:
    976             # handle inverted limits

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\axis.py in iter_ticks(self)
    919         self.major.formatter.set_locs(majorLocs)
    920         majorLabels = [self.major.formatter(val, i)
--> 921                        for i, val in enumerate(majorLocs)]
    922 
    923         minorLocs = self.minor.locator()

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\matplotlib\axis.py in <listcomp>(.0)
    919         self.major.formatter.set_locs(majorLocs)
    920         majorLabels = [self.major.formatter(val, i)
--> 921                        for i, val in enumerate(majorLocs)]
    922 
    923         minorLocs = self.minor.locator()

~\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\plotting\_converter.py in __call__(self, x, pos)
   1106         else:
   1107             fmt = self.formatdict.pop(x, '')
-> 1108             return Period(ordinal=int(x), freq=self.freq).strftime(fmt)
   1109 
   1110 

pandas/_libs/period.pyx in pandas._libs.period._Period.strftime()

pandas/_libs/period.pyx in pandas._libs.period.period_format()

pandas/_libs/period.pyx in pandas._libs.period._period_strftime()

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 10: invalid continuation byte

<matplotlib.figure.Figure at 0x2139d1ab630>

And the repr of ther string is :

'Periode du lundi 04/12/2017 17:11:41 au lundi 04/12/2017 21:59:59 \n Duree de 4h48m19s'

After spending too much time on google I can' find any clear answer. Any suggestions ?

Could you also post the line throwing the exception, and also a `repr` of the faulty string? — CristiFati, Feb 18 '18 at 20:54
Repr of the faulty string : 'Periode du lundi 04/12/2017 17:11:41 au lundi 04/12/2017 21:59:59 \n Duree de 4h48m19s' — Adri, Feb 18 '18 at 21:09
I see that *Matplotlib* tries to decode the string as *utf8*, doesn't care about the locale. I am not an expert on *Matplotlib*, maybe you could "tell" it somehow that the date comes in another encoding. Also there's no clear piece of code posted that would trigger this behavior. You should probably edit the question, and add a script that triggers this when being run, instead of scattered lines of code. — CristiFati, Feb 21 '18 at 22:12
Ok thanks I'll try to give you more info, but now I'm on my Windows hardware. — Adri, Feb 21 '18 at 22:25
I see this new feature in Python 3.7 : https://www.python.org/dev/peps/pep-0540/ . This make me think of my bug and this feature may solve it. It looks mike for now Python depends on the encoding of the locale's OS. On Unix's platform it's 100% UTF8 but maybe on windows it's not ? That what I understand when I read : "UTF-8 support is nearly ubiquitous for data read and written by modern platforms. It also has excellent support in Python. The problem is simply that the locale is frequently misconfigured. An obvious solution suggests itself: ignore the locale encoding and use UTF-8." — Adri, Feb 21 '18 at 22:29
and : "When decoding bytes from UTF-8 using the default strict error handler, **Python 3 raises a UnicodeDecodeError on the first undecodable byte.**" — Adri, Feb 21 '18 at 22:29
Not sure (I'd tend to say no). But you can give it a shot, although errors don't magically disappear. — CristiFati, Feb 21 '18 at 22:31

Python french locale on Windows 10 not UTF8?

0 Answers0