Need to parse integers in JSON as integers, not floats

Question

First of all let me explain the problem.

I have a stream of JSON records coming into my Golang app. It basically forwards these to a data store (InfluxDB). There are some integer values in the JSON, and also some float values. It is essential that these get forwarded to the data store with the original data type. If they don't, there will be type conflicts and the write operation will fail.

The Ruby JSON parser has no trouble doing this:

require 'json'
obj = { "a" => 123, "b" => 12.3 }
parsed = JSON.parse(obj.to_json)

print parsed["a"].class # => Integer
print parsed["b"].class # => Float

The encoding/json package in Golang, does have some trouble (all numbers are parsed as floats):

package main

import "encoding/json"
import "fmt"

func main () {
  str := "{\"a\":123,\"b\":12.3}"
  var parsed map[string]interface{}
  json.Unmarshal([]byte(str), &parsed)
  for key, val := range parsed {
    switch val.(type) {
    case int:
      fmt.Println("int type: ", key)
    case float64:
      fmt.Println("float type: ", key)
    default:
      fmt.Println("unknown type: ", key)
    }
  }
}

Which prints:

float type:  a
float type:  b

I need a way to parse ints as ints, and floats as floats, in the way the Ruby JSON parser does.

It is not feasible in this case to parse everything as strings and check whether or not there is a decimal. Certain values come in as strings such as "123" and I need to push those as strings.

I do not have structs for the parsed objects, nor is that an option. The golang app doesn't actually care about the schema and just forwards the input as it receives it.

I tried the approach outlined here: Other ways of verifying reflect.Type for int and float64 (using reflect) but it did not accurately identify the int:

t := reflect.TypeOf(parsed["a"])
k := t.Kind()
k == reflect.Int // false
k == reflect.Float64 // true

You should unmarshal into a struct where your fields are properly typed and defined. If you really can't do this, and you're just passing the data on somewhere else, then why are you unmarshaling the JSON at all? — Michael Hampton, Nov 22 '18 at 01:46
@MichaelHampton there is no reason for me to do this, besides this one case of distinguishing int and float. I typecheck each of the values of the JSON objects that they are either float/string/int. The only thing missing is the ability to distinguish int and float. — max pleaner, Nov 22 '18 at 01:48
@MichaelHampton I'm unmarshalling into objects which can be passed to the data store by the client library. — max pleaner, Nov 22 '18 at 01:49
encoding/json will happily unmarshal an int into an int field of a struct. It doesn't really make a lot of sense why you aren't doing this. Are you using the right language? Go might not be right for whatever you're trying to do. — Michael Hampton, Nov 22 '18 at 01:51
I appreciate your help but saying "are you using the right language" is not really helpful. The program is already fully functional except for this one issue of distinguishing int / float in JSON. Defining structs probably isn't as trivial as you imagine because there are 80+ valid struct permutations, and it's completely unnecessary for any purpose other than distinguishing ints/floats — max pleaner, Nov 22 '18 at 01:52
Only 80? I have a Go project with 365 such structs (and adding more regularly). I know exactly how non-trivial it is. But it's a whole lot better than trying to abuse reflection like this. So I speak from experience when I ask if you're using the right language. — Michael Hampton, Nov 22 '18 at 02:06
@MichaelHampton Ok well I appreciate your advice them. Just really wish it was possible and honestly I'd probably rather annotate the input JSON with type hints rather than maintain such a schema, it's just too much work. — max pleaner, Nov 22 '18 at 02:42
Yeah it's a lot of work. But the payoff is not having to worry about entire classes of bugs because type safety. Your situation boils down to: How does the program know if it's an int or a float? If you unmarshal into a struct, it's easy. Otherwise the library has to guess, and it will always create a float. But can you then convert it back to an int? How do _you_ (the programmer) know if it's an int or a float? That's what your spec is for. The struct makes sure your JSON and your spec match. — Michael Hampton, Nov 22 '18 at 03:18

peterSO · Accepted Answer · 2018-11-23T18:40:17.007

For example, Ruby JSON number types using the general Go mechanism for custom JSON values,

package main

import (
    "encoding/json"
    "fmt"
    "strconv"
)

func main() {
    str := `{"a":123,"b":12.3,"c":"123","d":"12.3","e":true}`
    var raw map[string]json.RawMessage
    err := json.Unmarshal([]byte(str), &raw)
    if err != nil {
        panic(err)
    }
    parsed := make(map[string]interface{}, len(raw))
    for key, val := range raw {
        s := string(val)
        i, err := strconv.ParseInt(s, 10, 64)
        if err == nil {
            parsed[key] = i
            continue
        }
        f, err := strconv.ParseFloat(s, 64)
        if err == nil {
            parsed[key] = f
            continue
        }
        var v interface{}
        err = json.Unmarshal(val, &v)
        if err == nil {
            parsed[key] = v
            continue
        }
        parsed[key] = val
    }
    for key, val := range parsed {
        fmt.Printf("%T: %v %v\n", val, key, val)
    }
}

Playground: https://play.golang.org/p/VmG8IZV4CG_Y

Output:

int64: a 123 
float64: b 12.3 
string: c 123 
string: d 12.3 
bool: e true

Another example, Ruby JSON number types using the Go json.Number type,

package main

import (
    "encoding/json"
    "fmt"
    "strings"
)

func main() {
    str := `{"a":123,"b":12.3,"c":"123","d":"12.3","e":true}`
    var parsed map[string]interface{}
    d := json.NewDecoder(strings.NewReader(str))
    d.UseNumber()
    err := d.Decode(&parsed)
    if err != nil {
        panic(err)
    }
    for key, val := range parsed {
        n, ok := val.(json.Number)
        if !ok {
            continue
        }
        if i, err := n.Int64(); err == nil {
            parsed[key] = i
            continue
        }
        if f, err := n.Float64(); err == nil {
            parsed[key] = f
            continue
        }
    }
    for key, val := range parsed {
        fmt.Printf("%T: %v %v\n", val, key, val)
    }
}

Playground: https://play.golang.org/p/Hk_Wb0EM-aY

Output:

int64: a 123
float64: b 12.3
string: c 123
string: d 12.3
bool: e true

A working version of @ShudiptaSharma's suggestion.

I appreciate this answer, i'll try it tomorrow. – max pleaner Nov 22 '18 at 21:49 — max pleaner, Nov 22 '18 at 21:49

Shudipta Sharma · Answer 2 · 2018-11-22T05:49:57.400

There exists a type json.Number in json package which has 3 format functions String() string, Float64() (float64, error) and Int64() (int64, error). We can use them to parse integer and float type.

So, I handle json integer parsing in this way:

package main

import "encoding/json"
import "fmt"
import "strings"
import (
    "reflect"
)

func main () {
    str := "{\"a\":123,\"b\":12.3}"

    var parsed map[string]interface{}
    d := json.NewDecoder(strings.NewReader(str))
    d.UseNumber()
    fmt.Println(d.Decode(&parsed))

    for key, val := range parsed {
        fmt.Println(reflect.TypeOf(val))
        fmt.Printf("decoded to %#v\n", val)
        switch val.(type) {
        case json.Number:
            if n, err := val.(json.Number).Int64(); err == nil {
                fmt.Println("int64 type: ", key, n)
            } else if f, err := val.(json.Number).Float64(); err == nil {
                fmt.Println("float64 type: ", key, f)
            } else {
                fmt.Println("string type: ", key, val)
            }
        default:
            fmt.Println("unknown type: ", key, val)
        }
        fmt.Println("===============")
    }
}

I appreciate this answer, i'll try it tomorrow. – max pleaner Nov 22 '18 at 21:49 — max pleaner, Nov 22 '18 at 21:49

Need to parse integers in JSON as integers, not floats

2 Answers2

Linked