How to correctly parse JSON with Unicode escape sequences?

Question

use serde_json::json; // 1.0.66
use std::str;

fn main() {
    let input = "{\"a\": \"b\\u001fc\"}";
    let bytes = input.as_bytes();
    let json: serde_json::Value = serde_json::from_slice(bytes).unwrap();
    for (_k, v) in json.as_object().unwrap() {
        let vec = serde_json::to_vec(v).unwrap();
        let utf8_str = str::from_utf8(&vec).unwrap();
        println!("value: {}", v);
        println!("utf8_str: {}", utf8_str);
        println!("bytes: {:?}", vec);
    }
}

How can the value of object key "a" be transformed into the following string?

b\u{1f}c

I've tried with serde_json and str::from_utf8, but I always get "b\u001fc" as the result. The escaped character sequence is not interpreted correctly. How this can be solved?

score 2 · Accepted Answer · edited Aug 31 '21 at 17:05

The problem is this line:

let vec = serde_json::to_vec(v).unwrap();

From the serde_json docs on to_vec():

Serialize the given data structure as a JSON byte vector.

You are deserializing from JSON, getting the values of the object, serializing them back to JSON and printing that. You don't want to serialize back to JSON, you want to print the "raw" string, so something like this does what you want:

fn main() {
    let input = "{\"a\": \"b\\u001fc\"}";
    let bytes = input.as_bytes();
    let json: serde_json::Value = serde_json::from_slice(bytes).unwrap();
    for (_k, v) in json.as_object().unwrap() {
        let string = v.as_str().unwrap();
        println!("bytes: {:?}", string);
    }
}

Playground

score 2 · Answer 2 · answered Aug 31 '21 at 16:03

I think things are closer to working than you think. Your problem is not that the escape sequence isn't being interpreted correctly, but rather that serde_json::to_vec(v) essentially re-encodes v (which is serde_json::value::Value::String) into a vector of JSON-encoded bytes. This means that it picks up the surrounding quote characters (byte 34) and turns the escape sequence into a literal ['\\', 'u', ...] — because that's how it would look in JSON.

If you want to get the string value out, you can do this:

for (_k, v) in json.as_object().unwrap() {
    if let serde_json::value::Value::String(s) = v {
        println!("{:?}", s);
    }
}

This prints "b\u{1f}c", the Rust string you want.

How to correctly parse JSON with Unicode escape sequences?

2 Answers2

Linked