JSONDecodeError Expecting value: line 1 column 1 (char 0) when receiving via Bluetooth

Question

I am transmitting a JSON payload from an Arduino microcontroller and attempting to receive it using a Python script:

import bluetooth  #pybluez 
import json

sensor_address = "18:D9:31:YY:C7:4A"
socket = bluetooth.BluetoothSocket(bluetooth.RFCOMM)
socket.connect((sensor_address, 1))
buffer = ""

print("Listening...")
while True: 
    data = socket.recv(1024)
    buffer += str(data, encoding='ascii')
    print(buffer) # used to check json payload
    try: 
        data = json.loads(buffer)
        print("Received:", data)
        buffer = ""
    except json.JSONDecodeError as e:
        print(e)
        continue

Examining the value stored in buffer before entering the try statement, I see what appears to be perfectly valid JSON:

{"a_x":957.5195,"a_y":-0.488281,"a_z":315.918,"g_x":-0.625954,"g_y":-1.305344}{"a_x":961.914,"a_y":-1.953125,"a_z":297.8516,"g_x":-2.816794,"g_y":2.572519}{"a_x":964.8437,"a_y":3.417969,"a_z":303.2227,"g_x":-1,"g_y":0.374046}

However the result of my script is only Expecting value: line 1 column 1 (char 0) repeatedly.

Why doesn't the code inside the try statement execute once a complete payload has been received?

My hunch is that at no time does a valid JSON payload appear in buffer, but instead valid payloads appear along with incomplete payloads.

Is it possible to use a regular expression to extract a valid payload from a string?

ukBaz · Accepted Answer · 2023-03-20T22:18:05.267

The data in buffer is not valid JSON so that is why you are seeing the error.

The buffer seems to have the information in the format of a Python dictionary so you could use Python re module to extract the dictionary and then use ast.literal_eval to turn the string in to a Python dictionary.

In the example below I've mocked the reading of the socket as I don't have your device.

import re

pattern = re.compile(r'{.*?}')


class socket:
    """Mock reading data from socket"""
    pointer = 3
    chunk_size = 8
    feed = (b'{"a_x":957.5195,"a_y":-0.488281,"a_z":315.918,"g_x":-0.625954,"g_y":-1.305344}'
            b'{"a_x":961.914,"a_y":-1.953125,"a_z":297.8516,"g_x":-2.816794,"g_y":2.572519}'
            b'{"a_x":964.8437,"a_y":3.417969,"a_z":303.2227,"g_x":-1,"g_y":0.374046}')

    @classmethod
    def recv(cls):
        data = cls.feed[cls.pointer:cls.pointer + cls.chunk_size]
        cls.pointer += cls.chunk_size
        return data


def process_reading(buffer):
    match = re.search(pattern, buffer)
    start_idx, end_idx = match.span()
    reading = literal_eval(buffer[start_idx:end_idx])
    buffer = buffer[end_idx:]
    return buffer, reading


def main():
    buffer = ''
    data = True
    while data:
        data = socket.recv()
        # print("Data:", data)
        buffer += str(data, encoding='ascii')
        print("Buffer contents", buffer)
        if re.search(pattern, buffer):
            buffer, measurement = process_reading(buffer)
            print("\tMeasurement:", measurement.get('g_x'))


if __name__ == '__main__':
    main()

This gave the following output:

Buffer contents _x":957.
Buffer contents _x":957.5195,"a_
Buffer contents _x":957.5195,"a_y":-0.48
Buffer contents _x":957.5195,"a_y":-0.488281,"a_
Buffer contents _x":957.5195,"a_y":-0.488281,"a_z":315.9
Buffer contents _x":957.5195,"a_y":-0.488281,"a_z":315.918,"g_x"
Buffer contents _x":957.5195,"a_y":-0.488281,"a_z":315.918,"g_x":-0.6259
Buffer contents _x":957.5195,"a_y":-0.488281,"a_z":315.918,"g_x":-0.625954,"g_y"
Buffer contents _x":957.5195,"a_y":-0.488281,"a_z":315.918,"g_x":-0.625954,"g_y":-1.3053
Buffer contents _x":957.5195,"a_y":-0.488281,"a_z":315.918,"g_x":-0.625954,"g_y":-1.305344}{"a_x
Buffer contents _x":957.5195,"a_y":-0.488281,"a_z":315.918,"g_x":-0.625954,"g_y":-1.305344}{"a_x":961.91
Buffer contents _x":957.5195,"a_y":-0.488281,"a_z":315.918,"g_x":-0.625954,"g_y":-1.305344}{"a_x":961.914,"a_y":
Buffer contents _x":957.5195,"a_y":-0.488281,"a_z":315.918,"g_x":-0.625954,"g_y":-1.305344}{"a_x":961.914,"a_y":-1.95312
Buffer contents _x":957.5195,"a_y":-0.488281,"a_z":315.918,"g_x":-0.625954,"g_y":-1.305344}{"a_x":961.914,"a_y":-1.953125,"a_z":
Buffer contents _x":957.5195,"a_y":-0.488281,"a_z":315.918,"g_x":-0.625954,"g_y":-1.305344}{"a_x":961.914,"a_y":-1.953125,"a_z":297.8516
Buffer contents _x":957.5195,"a_y":-0.488281,"a_z":315.918,"g_x":-0.625954,"g_y":-1.305344}{"a_x":961.914,"a_y":-1.953125,"a_z":297.8516,"g_x":-
Buffer contents _x":957.5195,"a_y":-0.488281,"a_z":315.918,"g_x":-0.625954,"g_y":-1.305344}{"a_x":961.914,"a_y":-1.953125,"a_z":297.8516,"g_x":-2.816794
Buffer contents _x":957.5195,"a_y":-0.488281,"a_z":315.918,"g_x":-0.625954,"g_y":-1.305344}{"a_x":961.914,"a_y":-1.953125,"a_z":297.8516,"g_x":-2.816794,"g_y":2
Buffer contents _x":957.5195,"a_y":-0.488281,"a_z":315.918,"g_x":-0.625954,"g_y":-1.305344}{"a_x":961.914,"a_y":-1.953125,"a_z":297.8516,"g_x":-2.816794,"g_y":2.572519}
    Measurement: -2.816794
Buffer contents {"a_x":9
Buffer contents {"a_x":964.8437,
Buffer contents {"a_x":964.8437,"a_y":3.
Buffer contents {"a_x":964.8437,"a_y":3.417969,"
Buffer contents {"a_x":964.8437,"a_y":3.417969,"a_z":303
Buffer contents {"a_x":964.8437,"a_y":3.417969,"a_z":303.2227,"g
Buffer contents {"a_x":964.8437,"a_y":3.417969,"a_z":303.2227,"g_x":-1,"
Buffer contents {"a_x":964.8437,"a_y":3.417969,"a_z":303.2227,"g_x":-1,"g_y":0.3
Buffer contents {"a_x":964.8437,"a_y":3.417969,"a_z":303.2227,"g_x":-1,"g_y":0.374046}
    Measurement: -1
Buffer contents

This is such a clean and elegant solution, no doubt someone will find it helpful. Unfortunately, in my case, the `process_reading()` function is never entered as confirmed by placing a print statement at function entry. It seems as though the conditional on `re.match(pattern, buffer)` never evaluates to true. This is piuzzling because printing the contents of buffer as in your solution shows: `Buffer contents: 3,"g_x":-2.015267,"g_y":-0.832061}{"a_x":954.1015,"a_y":-89.84374,"a_z":322.7539,"g_x":1.305344,"g_y":-0.778626}{"a_x":950.6835, ...` — John Harrington, Mar 20 '23 at 21:57
Also, why is the contents of `buffer` not valid JSON? I only ask because I have programmed how the data is serialized and transmitted using the `ArduinoJson` library with an Arduino microcontroller. The library is well documented and highly rated. — John Harrington, Mar 20 '23 at 22:01
I've changed the `match` to `search` as `match` assumes that the buffer starts with a `{`. This way it will recover if there isn't the opening `{`. I said `buffer` isn't valid JSON because it has the `}{` which isn't valid. I'm guessing they are expected to be separate entries. — ukBaz, Mar 20 '23 at 22:19
That did the trick, thank you for explaining why it works and for clarifying the JSON validity. — John Harrington, Mar 20 '23 at 22:34

JSONDecodeError Expecting value: line 1 column 1 (char 0) when receiving via Bluetooth

1 Answers1