Webhook call failed. Error: Failed to parse webhook JSON response: Expect message object but got: [Chinese letters]

Question

I'm building my own WebhookClient for dialog flow. My code is the following (using Azure Functions, similar to Firebase Functions):

module.exports = async function(context, req) {
    const agent = new WebhookClient({ request: context.req, response: context.res });

    function welcome(agent) {
        agent.add(`Welcome to my agent!!`);
    }

    let intentMap = new Map();

    intentMap.set("Look up person", welcome);

    agent.handleRequest(intentMap);
}

I tested the query and the response payload looks like this:

{
    "fulfillmentText": "Welcome to my agent!!",
    "outputContexts": []
}

And the headers in the response look like this:

Transfer-Encoding: chunked
Content-Type: application/json; charset=utf-8
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Date: Tue, 11 Dec 2018 18:16:06 GMT

But when I test my bot in dialog flow, it returns the following:

Webhook call failed. Error: Failed to parse webhook JSON response: Expect message object but got: "笀ഀ਀ ∀昀甀氀昀椀氀氀洀攀渀琀吀攀砀琀∀㨀 ∀圀攀氀挀漀洀攀琀漀洀礀愀最攀渀琀℀℀∀Ⰰഀ਀ ∀漀甀琀瀀甀琀䌀漀渀琀攀砀琀猀∀㨀嬀崀ഀ਀紀".

There's Chinese symbols!? Here's a video of me testing it out in DialogFlow: https://i.stack.imgur.com/d1C8d.jpg

Are you using the azure portal editor to write the function? I had the exact same problem. My solution was just to use visual studio to create the function instead (https://learn.microsoft.com/en-us/azure/azure-functions/functions-create-your-first-function-visual-studio) , and using ngrok to create a tunnel to localhost that dialogflow can call as the webhook. — mintsponge, Dec 12 '18 at 11:55
@mintsponge I'm locally writing it in NodeJS and Visual Studio Code. I didn't understand what you meant by the ngrok thing. I don't know what ngrok is. — AskYous, Dec 13 '18 at 19:49
@mintsponge did you run ngrok on your local dev machine or on Azure functions somehow? — AskYous, Dec 14 '18 at 00:20
what url are you using in the fulfilment section of dialogflow? Using an Azure url is what caused the problem for me. Instead I ran the function in VS which deployed it to local host. But since Dialogflow can only access public url’s, ngrok was run (locally) to create a public ngrok url — mintsponge, Dec 16 '18 at 19:19

gplumb · Answer 1 · 2018-12-31T19:15:15.733

I know this should be a comment (as it isn't really an answer), but it's fairly verbose and I didn't want it to get lost in the noise.

I have the same problem using WebAPI on a local machine (using ngrok to tunnel back to Kestrel). A friend of mine has working code (he's hosting in AWS rather than Azure), so I started examining the differences between our responses. I've notice the following:

This occurs with Azure Functions and WebAPI (so it's not that)
The JSON payloads are identical (so it's not that)
Working payload isn't chunked
Working payload doesn't have a content type

As an experiment, I added this code to Startup.cs, in the Configure method:

app.Use(async (context, next) =>
{
    var original = context.Response.Body;
    var memory = new MemoryStream();

    context.Response.Body = memory;
    await next();

    memory.Seek(0, SeekOrigin.Begin);

    if (!context.Response.Headers.ContentLength.HasValue)
    {
        context.Response.Headers.ContentLength = memory.Length;
        context.Response.ContentType = null;
    }

    await memory.CopyToAsync(original);
});

This code disables response chunking, which is now causing a new and slightly more interesting error for me in the google console:

*Webhook call failed. Error: Failed to parse webhook JSON response: com.google.gson.stream.MalformedJsonException: Unterminated object at line 1 column 94 path $.\u0000\\"\u0000f\u0000u\u0000l\u0000f\u0000i\u0000l\u0000l\u0000m\u0000e\u0000n\u0000t\u0000M\u0000e\u0000s\u0000s\u0000a\u0000g\u0000e\u0000s\u0000\\"\u0000.\

I thought this could be encoding at first, so I stashed my JSON as a string and used the various Encoding classes to convert between them, to no avail.

I fired up Postman and called my endpoint (using the same payload as Google) and I can see the whole response payload correctly - it's almost as if Google's end is terminating the stream part-way through reading...

Hopefully, this additional information will help us figure out what's going on!

Update

After some more digging and various server/lambda configs, I spotted this post here: https://github.com/googleapis/google-cloud-dotnet/issues/2258

It turns out that json.net IS the culprit! I guess it's something to do with the formatters on the way out of the pipeline. In order to prove this, I added this hard-coded response to my POST controller and it worked! :)

return new ContentResult()
{
    Content = "{\"fulfillmentText\": null,\"fulfillmentMessages\": [],\"source\": null,\"payload\": {\"google\": {\"expectUserResponse\": false,\"userStorage\": null,\"richResponse\": {\"items\": [{\"simpleResponse\": {\"textToSpeech\": \"Why hello there\",\"ssml\": null,\"displayText\": \"Why hello there\"}}],\"suggestions\": null,\"linkOutSuggestion\": null}}}}",
    ContentType = "application/json",
    StatusCode = 200
};

In order to rule out my proxy (ngrok), I created an Azure Function with a hard-coded JSON payload (which is valid and responds quickly). Sadly, my local machine config is not the issue either as the same issue persists. This also rules out JSON.net as the root cause, so I'm kind of stumped now... — gplumb, Dec 31 '18 at 18:40
It turns out json.net IS the culprit! I've updated my answer above. — gplumb, Dec 31 '18 at 19:15

score -1 · Answer 2 · answered Dec 15 '18 at 14:21

Despite the HTTP header saying the charset is utf-8, that is definitely using the utf-16le character set, and then the receiving side is treating them as utf-16be. Given you're running on Azure, it sounds like there is some configuration you need to make in Azure Functions to represent the output as UTF-8 instead of using UTF-16 strings.

Webhook call failed. Error: Failed to parse webhook JSON response: Expect message object but got: [Chinese letters]

2 Answers2

Update