How to read utf-8 encoded JSON file (locally/from internet) in a string in Kodi?

Question

I followed Kodi simple video plugin tutorial which works as expected with JSON string embedded into main.py file.

The tutorial refers that JSON string can be obtained by other means from other media (file, internet).

My attempt to read utf-8 JSON file into the string was unsuccessful so far.

Initially I tried the following approach

import json
import codecs
fname = 'c:/temp/iptv/video_data.json'

with open(fname, encoding='utf-8') as f:
    data = f.read()
    VIDEO = json.loads(data)

What produced the following error in log file

2020-05-09 11:11:29.327 T:19428   ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
                                             - NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
                                            Error Type: <type 'exceptions.TypeError'>
                                            Error Contents: 'encoding' is an invalid keyword argument for this function
                                            Traceback (most recent call last):
                                              File "C:\Users\Alex Fox\AppData\Roaming\Kodi\addons\plugin.video.example\main.py", line 27, in <module>
                                                with open(fname, encoding='utf-8') as f:
                                            TypeError: 'encoding' is an invalid keyword argument for this function
                                            -->End of Python script error report<--

I investigated the problem and found that Kodi 18.4 uses python27.dll and I assume that this library is somehow accessed from Kodi.

I substituted the code above with following code snippet into main.py file

import json
import codecs
fname = 'c:/temp/iptv/video_data.json'

with open(fname, 'rb') as f:
    bytes = f.read()
    data = bytes.decode('utf-8')
    VIDEOS = json.loads(data)

On plugin's run it produces following error record in log file

2020-05-09 16:54:17.024 T:5700   ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
                                             - NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
                                            Error Type: <type 'exceptions.UnicodeEncodeError'>
                                            Error Contents: 'ascii' codec can't encode characters in position 1-4: ordinal not in range(128)
                                            Traceback (most recent call last):
                                              File "C:\Users\Alex Fox\AppData\Roaming\Kodi\addons\plugin.video.example\main.py", line 239, in <module>
                                                router(sys.argv[2][1:])
                                              File "C:\Users\Alex Fox\AppData\Roaming\Kodi\addons\plugin.video.example\main.py", line 233, in router
                                                list_categories()
                                              File "C:\Users\Alex Fox\AppData\Roaming\Kodi\addons\plugin.video.animatron\main.py", line 137, in list_categories
                                                url = get_url(action='listing', category=category)
                                              File "C:\Users\Alex Fox\AppData\Roaming\Kodi\addons\plugin.video.animatron\main.py", line 66, in get_url
                                                return '{0}?{1}'.format(_url, urlencode(kwargs))
                                              File "C:\bin\Portable\Kodi\system\python\Lib\urllib.py", line 1343, in urlencode
                                                v = quote_plus(str(v))
                                            UnicodeEncodeError: 'ascii' codec can't encode characters in position 1-4: ordinal not in range(128)
                                            -->End of Python script error report<--

While at the same time following code outside KODI (tested with Python 3.8.2)

import json
import codecs
fname = 'c:/temp/iptv/animation.json'

with open(fname, 'rb') as f:
    bytes = f.read()
    str   = bytes.decode('utf-8')
    VIDEOS = json.loads(str)
    print(json.dumps(VIDEOS, indent=4))
    for key in VIDEOS.keys():
        print(key)

outputs JSON dump and VIDEO dictionary keys properly.

What is a proper way to read utf-8 encoded JSON file/internet into a string in Kodi?

In your Python-2 example, you wrote `bytes.encode`, but it should be `bytes.decode`, like you wrote in your Python-3 example. — lenz, May 09 '20 at 21:26
@lenz -- I've updated the ticket with `decode('utf-8')` which produces an error (included). I tried many variation so far unsuccessfully. — Polar Bear, May 10 '20 at 00:01
None of the lines in the traceback correspond to your code snippet. My guess is that this Kodi tool has inherently bad Unicode support, in particular if it runs Python 2. — lenz, May 10 '20 at 06:53
@lenz -- well, all I could see is *python27.dll* and no other option than try to run plugin and read what appears in the log file. I do not see any other possibility to debug the cause of the problem. We know that other plugins plugins perform assigned task to them. I am not very well versed how internals of Kodi work -- what seems should be easy to implement does not work for unknown to me reason. I attempted to contact the author of the plugin in hope that he may read my message. — Polar Bear, May 10 '20 at 10:27

Polar Bear · Answer 1 · 2020-05-13T08:22:27.960

Solution for obtaining data from web server:

Download and install from zip file script.modules.requtests-2.22.0.zip in Kodi.

Then add into addon.xml file <import addon="script.module.requests" version="2.22.0"/>

<requires>
  <import addon="xbmc.python" version="2.25.0"/>
  <import addon="script.module.requests" version="2.22.0"/>
</requires>

Now in plugin's main.py file insert following code snippet

import json
import requests

url = 'http://iptv.server.com/series.json'

VIDEOS = {}
html = requests.get(url)
DATA = json.loads(html.content)
for title in DATA.keys():
    title_utf8 = title.encode('utf-8')
    VIDEOS[title_utf8] = []
    for episode in DATA[title]:
        episode['name']  = episode['name'].encode('utf-8')
        episode['genre'] = episode['genre'].encode('utf-8')
        VIDEOS[title_utf8].append(episode)

Launch Kodi and test the plugin.

Next code snippet is for reading file from local storage

import json        

fname = 'c:/temp/iptv/animation.json';

VIDEOS = {}

with open(fname,'r') as f:
    str = f.read()
    DATA = json.loads(str)

for title in DATA.keys():
    title_utf8 = title.encode('utf-8')
    VIDEOS[title_utf8] = []
    for episode in DATA[title]:
        episode['name']  = episode['name'].encode('utf-8')
        episode['genre'] = episode['genre'].encode('utf-8')
        VIDEOS[title_utf8].append(episode)

How to read utf-8 encoded JSON file (locally/from internet) in a string in Kodi?

1 Answers1