android, when read the file from res/raw, context.getAssets().open(fileName), the content read in are corrupted

Question

on Android, using com.google.gson:gson:2.8.5,

when passing a josnString and when the json is large (noticed for example when string length is 669304), it got

com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: 
Unterminated string at line 2902 column 564904 path $.items.result[10].quoted_status.full_text

or different errors:

the json has array of sub json strings, structured like,

{
      "items": {
        "result": [
          {subJsonString...},
          {subJsonString...},
          ...
        ]
     }
}

and the json string is stored in the res/raw/jsonstring.txt, and read in at runtime before give to Gson fro deserialization.

if reduce the sub json elements in the array (i.e. 10 items or less in the array) it works fine, and the individual json elements string all working fine. But when the array has more items it starts to throw.

update: it seems is problem from reading the json string from res/raw after further look, the log shows the output json string read from the res/raw/jsonString.txt are corrupted, when there are more items. Here after adding the 11th sub item into the json string array, it shows 000000000... at certain point (the 11th item who encountered issue are just a copy from the 1st item, so the string should be fine)

here is the code how the raw string is read, how it logs it

val jsonStr = getJsonFromFile(context, "res/raw/jsoninraw.txt")
returnedModel = gson.fromJson<T>(jsonStr, dataClassType)

......

fun getJsonFromFile(context: Context?, fileName: String): String? {
        val json: String
        try {
            val inputStream: InputStream = context.getAssets().open(fileName)
            val size = inputStream.available()
            val buffer = ByteArray(size)
            inputStream.use { it.read(buffer) }
            json = String(buffer)
        } catch (ioException: IOException) {
            ioException.printStackTrace()
            return null
        }
        return json
            .also {
                logString(it)
            }
    }

fun logString(jsonStr: String) {
    val chunkSize = 512
    var i = 0
    while (i < jsonStr.length) {
        Log.e("+++", jsonStr.substring(i, Math.min(jsonStr.length, i + chunkSize)))
        i += chunkSize
    }
}

what are the better way to deserialize the json string into model?

It seems like the exception is coming because there is a syntax error in your json that you receive and it's not because of the size of it. Try to copy it and validate it there are many json online utilities — DemoDemo, Feb 18 '20 at 16:28
tried just put in sub json (the block {} contains the errorded line), the single one sun json element in the total json array works fine, actually when size is not too big, putting three sub json in that outer json array is still fine. — lannyf, Feb 18 '20 at 16:37
It's hard to tell you where exactly the problem in your json (especially when you have such big json). If I were you I will format the json (there are many online tools for that) and go to line 2902 I think you might forgot to put a , or closing bracket of some sort. — DemoDemo, Feb 18 '20 at 16:51
tried format the json with validation tool, and start at putting only one json element in the json array (works), then copy/paste the same json element as 2nd one in the array, then 3rd, ... it works a 10 items (paste same one 10 times into the json array, size 99853). After that when paste one more into the array (now 11 items in the json array now), it got error: `gson.JsonSyntaxException: EOFException: End of input at line 2619 column 16584 path $.items.result[10].quoted_status.entities.media[0].sizes.medium.w` (total string length now is 115505). So I feel it is string length related. — lannyf, Feb 18 '20 at 19:24
see my update, seems it the json string read from the res/raw/jsonstring.txt through context.getAssets().open(fileName) is corrupted. But the json string is validated and well formatted. — lannyf, Feb 19 '20 at 14:12

score 0 · Answer 1 · answered Feb 19 '20 at 20:29

Found the problem that the ByteArray has 64k limit, so anything beyond is corrupted.

the updated the getJsonFromFile() works.

fun getJsonFromFile(context: Context?, fileName: String?): String? {
        val returnString = StringBuilder()
        var inputStream: InputStream? = null
        var isr: InputStreamReader? = null
        var input: BufferedReader? = null
        try {

            inputStream = if (context == null) {
                val cLoader = this.javaClass.classLoader
                cLoader.getResourceAsStream(fileName)
            } else {
                context.getAssets().open(fileName)
            }

          // either #1: 
          // returnString.append(inputStream?.readBytes()?.toString(Charsets.UTF_8))


         // or #2:            
            inputStream?.let {
                val inputString = inputStream.bufferedReader().use { it.readText() }
                returnString.append(inputString)
            }

// or #3:
//            isr = InputStreamReader(inputStream)
//            input = BufferedReader(isr)
//            var line: String? = ""
//            while (input.readLine().also { line = it } != null) {
//                returnString.append(line)
//            }
        } catch (e: Exception) {
            e.printStackTrace()
            return null
        } finally {
            try {
                isr?.close()
                inputStream?.close()
                input?.close()
            } catch (e2: Exception) {
                e2.message
            }
        }
        return returnString.toString()
    }

android, when read the file from res/raw, context.getAssets().open(fileName), the content read in are corrupted

1 Answers1