1

I have a standard Dialogflow agent, using javascript/node.js webhooks. It works perfectly well in most cases. I have recently encountered a problem which has me at a complete loss. I am currently saving some JSON-objects in conv.data to minimize the external API-calls my webhook have to make. For one specific JSON-object, fetched from an external API using node-fetch, the response I send from my side looks perfectly ordinary. I use firebase and the firebase logs do not show any error messages or any sign that there might be a problem. But I get this error in the Google Actions console:

UnparseableJsonResponse API Version 2: Failed to parse JSON response string with 'INVALID_ARGUMENT' error: "Parsing terminated before end of input. 8,\\"3\\":12},\\"w ^".

And in the stackdriver logs, the received response does not start with the usual

Received response from agent with body: HTTP/1.1 200 OK Server: ... etc

Instead it starts in the middle of the external API-JSON-file

Received response from agent with body: 8,\\"3\\":12},\\"winPercentage\\":1392}}}}, ... etc

This does not happen the first time the agent responds after fetching the JSON from the external API. The second time the agent responds after fetching the JSON, everything crashes regardless of whether the information from the JSON is used by that second call, regardless of anything at all except if the JSON file is overwritten between first and second call. If the file is overwritten the program runs perfectly. So the problem is likely part of storing and/or parsing this specific JSON file. Unfortunately the API I use in this application is not a public one and due to NDAs I cannot give any access to that JSON, so I understand that it is probably impossible for you to help me. I will however give as much information about the JSON as I can, and hope for the best:

  • It is valid according to https://codebeautify.org/jsonvalidator and jsonlint.com
  • It is structured the exact same way as other JSON files from the same API which do not crash the application
  • It is slightly larger that other JSON files from the same API. It has around 340 000 characters, others are around 280-300 000.
  • All JSONs, this as well as those that work, is from a Swedish company, therefore unusual characters like å, ä and ö are likely present.
  • The error message is always the same, except the start of the response is in different places in the JSON file. "8,\\"3\\":12}, ...", "ostPosition\\":2 ...", "3804,\\"startPoints\\":2960 ..." are some examples.

I am extremely grateful for any and all help I might receive, even if it's just what questions I need to ask, or where I might try troubleshooting next.

Prisoner
  • 49,922
  • 7
  • 53
  • 105
24HamstersInATrenchcoat
  • 13
  • 5
  • Are those strings in the JSON? Any indication where in the JSON they may be? Can you at least show the code to illustrate how you're storing it in `conv.data`? – Prisoner Feb 04 '20 at 12:02
  • If you are referring to the strings that start the responses, yes. They are part of the JSON. I cannot find any pattern as to where in the JSON they appear, but it seems to be more often in the last 3rd of the JSON. I do not have enough of a sample size to be able to say for sure though. As to the code: `conv.data.variableNameX = fetch(url)).json()` Edit: where fetch is node-fetch – 24HamstersInATrenchcoat Feb 04 '20 at 12:10
  • After more testing it seems to mainly be the last 5-8 000 characters that are featured in the second reply, though the exact number of character varies. I have tried to eliminate some of the JSON that is not used, but the problem persists. – 24HamstersInATrenchcoat Feb 04 '20 at 12:54
  • Your question about where in the JSON the strings were set me on the right track. Could you post it as an answer, so I can give you proper cred? I will give more information about how I actually solved it in a comment to that answer. – 24HamstersInATrenchcoat Feb 04 '20 at 13:35

1 Answers1

1

I suspect the problem is that the JSON you're trying to save is larger than the buffer size they allocate for conv.data, although I can't find any documentation to say there is some specific limit.

I'd check to see where the strings you're seeing in the error header are located in the JSON and try to keep it well under that limit.

Prisoner
  • 49,922
  • 7
  • 53
  • 105
  • 1
    After Prisoner asked about where in the JSON the error strings were located I was able to eliminate unused data from the JSON and the program started working again. From my testing, and knowing what other info I save in the conversation, I can see that the character limit (which seems to be for the whole conversation object, not just `conv.data`) is probably around 350 000 characters. – 24HamstersInATrenchcoat Feb 04 '20 at 13:50