We are implementing Actions on Google with Dialogflow fulfillment using the newly released Java/Kotlin API.
It's called Speech Bank.
While going through the Account Linking process testing on the smartphone, the user is getting MalformedResponse error preventing the completion of the flow and consequent successful hand-off back to the regular flow.
The logs (detailed below) contain the MalformedResponse: Failed to parse Dialogflow response into AppResponse because of empty speech response message, and the user receives Speech bank isn't responding right now. Try again soon. message on her device.
Here's a bit more details on our setup:
The action is configured for Account Linking utilizing our own OAuth 2 compliant mock infrastructure.
There's a single intent (called RawText) configured in Dialogflow, the rest of the interactions are to be taken care of by own internal application via its web hook.
Here's how the state machine is coded in Java so far:
public class AoGApp extends DialogflowApp {
private final static Logger log = LoggerFactory.getLogger(AoGApp.class);
public static final String GREETING = "GOOGLE_ASSISTANT_WELCOME";
@ForIntent("RawText")
//@ForIntent("actions.intent.MAIN")
public ActionResponse launchRequestHandler(ActionRequest request) {
String userId = request.getAppRequest().getUser().getUserId();
log.info("userId={}",userId);
String queryText = request.getWebhookRequest().getQueryResult().getQueryText();
log.info("queryText={}", queryText);
String speech = null;
ResponseBuilder responseBuilder = getResponseBuilder(request);
if (isBlank(userId) || GREETING.equalsIgnoreCase(queryText)) {
speech = "\nHi. I sense a great banking experience in your future, I see that your account isn't connected. "
+ "I've sent a link to your Google Assistant app that will get you started and set up in just several simple steps. "
+ "Don't worry, I'll be here waiting, just summon me when you're ready.";
responseBuilder.add(
new SignIn()
.setContext(speech));
} else {
speech = "Welcome. You can say hello.";
responseBuilder.add(speech);
}
return responseBuilder.build();
}
@ForIntent("actions.intent.SIGN_IN")
public ActionResponse getSignInStatus(ActionRequest request) {
ResponseBuilder responseBuilder = getResponseBuilder(request);
String text = "Hello from sign-in handler";
responseBuilder.add(text);
log.info(text);
return responseBuilder.build();
}
}
and the associated HttpRequest processing:
@Override
protected void handlePOST(final Request request, final HttpServletResponse response) {
try {
String rawRequest = ControllerUtils.toString(request.getReader());
String jsonResponse = app.handleRequest(rawRequest, getHeadersMap(request)).get();
log.info("Generated response:\n {}", ControllerUtils.prettyPrint(jsonResponse));
response.setContentType(APPLICATION_JSON.getMimeType());
response.getWriter().write(jsonResponse);
} catch (Exception e) {
handleError(response, e);
}
}
public final class ControllerUtils {
private final static Logger log = LoggerFactory.getLogger(ControllerUtils.class);
private static ObjectMapper mapper = new ObjectMapper();
public static String toString(BufferedReader reader) throws Exception {
String rawRequest = reader
.lines()
//.map(e -> e.concat(System.lineSeparator()))
.collect(Collectors.joining(System.lineSeparator()));
log.info("Received AoG Request {}",rawRequest);
return rawRequest;
}
public static String prettyPrint(String json) throws Exception {
return mapper.writerWithDefaultPrettyPrinter().writeValueAsString(mapper.readValue(json, Object.class));
}
public static Map<String, String> getHeadersMap(org.eclipse.jetty.server.Request jettyRequest){
return Collections.list((Enumeration<String>) jettyRequest.getHeaderNames())
.stream()
.collect(Collectors.toMap(
name -> name,
jettyRequest::getHeader));
}
}
As configured above, the OAuth authorization code flow undertakes the normal OAuth 2 steps:
hits
/loginendpoint to supply credentialshits
/tokenendpoint to obtain the token (its value istoken1in the below logs. We have a facility to generate & inject our own tokens, this is a testing environment so we produced thistoken1value which seem to have been successfully incorporated into the subsequent request.)
Below is the detailed screen shot of failed interaction, with the attached log provided by Actions on Google console:
[
{
"textPayload": "Sending request with post data: {\"user\":{\"userId\":\"ABwppHFQHUBr0RrWA_OuL-kK2sxTPUvQtL3D-x2Ydr-7uxLt9zzEFzJrGB-X96d9XY8k9XTJj-RUg9WpzGB9jg\",\"locale\":\"en-US\",\"lastSeen\":\"2019-02-20T21:32:22Z\",\"userStorage\":\"{\\\"data\\\":{}}\"},\"conversation\":{\"conversationId\":\"ABwppHE35s8T6qSdaaCMNiWuMdY7UsvQ3sHbLZJOQkVA4AFD2nhKuqTTvoJTkVh7yU81GCHbvlfZTxmULLd4Ug\",\"type\":\"NEW\"},\"inputs\":[{\"intent\":\"actions.intent.MAIN\",\"rawInputs\":[{\"inputType\":\"VOICE\",\"query\":\"open speech Bank\"}]}],\"surface\":{\"capabilities\":[{\"name\":\"actions.capability.AUDIO_OUTPUT\"},{\"name\":\"actions.capability.MEDIA_RESPONSE_AUDIO\"},{\"name\":\"actions.capability.SCREEN_OUTPUT\"},{\"name\":\"actions.capability.WEB_BROWSER\"}]},\"isInSandbox\":true,\"availableSurfaces\":[{\"capabilities\":[{\"name\":\"actions.capability.AUDIO_OUTPUT\"},{\"name\":\"actions.capability.SCREEN_OUTPUT\"},{\"name\":\"actions.capability.WEB_BROWSER\"}]}]}.",
"insertId": "f9fzrtf3hjgn4",
"resource": {
"type": "assistant_action",
"labels": {
"project_id": "speechbank-e8a15",
"version_id": "",
"action_id": "actions.intent.MAIN"
}
},
"timestamp": "2019-02-21T13:47:56.713587946Z",
"severity": "DEBUG",
"labels": {
"channel": "preview",
"source": "AOG_REQUEST_RESPONSE",
"querystream": "GOOGLE_USER"
},
"logName": "projects/speechbank-e8a15/logs/actions.googleapis.com%2Factions",
"trace": "projects/366800784520/traces/ABwppHE35s8T6qSdaaCMNiWuMdY7UsvQ3sHbLZJOQkVA4AFD2nhKuqTTvoJTkVh7yU81GCHbvlfZTxmULLd4Ug",
"receiveTimestamp": "2019-02-21T13:47:57.205496026Z"
},
{
"textPayload": "Received response from agent with body: HTTP/1.1 200 OK\r\nServer: nginx/1.13.6\r\nDate: Thu, 21 Feb 2019 13:47:57 GMT\r\nContent-Type: application/json;charset=UTF-8\r\nContent-Length: 426\r\nX-Cloud-Trace-Context: d8cb97627afa1d2977b9f567f29598de/11157405402824233090;o=0\r\nGoogle-Actions-API-Version: 2\r\nX-SHARD: shard-2\r\nVia: 1.1 google\r\nAlt-Svc: clear\r\n\r\n{\"conversationToken\":\"[\\\"_actions_on_google\\\"]\",\"expectUserResponse\":true,\"expectedInputs\":[{\"inputPrompt\":{},\"possibleIntents\":[{\"intent\":\"actions.intent.SIGN_IN\",\"inputValueData\":{\"@type\":\"type.googleapis.com/google.actions.v2.SignInValueSpec\"}}]}],\"responseMetadata\":{\"status\":{\"message\":\"Success (200)\"},\"queryMatchInfo\":{\"queryMatched\":true,\"intent\":\"f645f492-f6dc-4e7e-8da6-45711c654ad0\"}},\"userStorage\":\"{\\\"data\\\":{}}\"}.",
"insertId": "f9fzrtf3hjgn5",
"resource": {
"type": "assistant_action",
"labels": {
"version_id": "",
"action_id": "actions.intent.MAIN",
"project_id": "speechbank-e8a15"
}
},
"timestamp": "2019-02-21T13:47:57.190979036Z",
"severity": "DEBUG",
"labels": {
"source": "AOG_REQUEST_RESPONSE",
"querystream": "GOOGLE_USER",
"channel": "preview"
},
"logName": "projects/speechbank-e8a15/logs/actions.googleapis.com%2Factions",
"trace": "projects/366800784520/traces/ABwppHE35s8T6qSdaaCMNiWuMdY7UsvQ3sHbLZJOQkVA4AFD2nhKuqTTvoJTkVh7yU81GCHbvlfZTxmULLd4Ug",
"receiveTimestamp": "2019-02-21T13:47:57.205496026Z"
},
{
"textPayload": "Sending request with post data: {\"user\":{\"userId\":\"ABwppHFQHUBr0RrWA_OuL-kK2sxTPUvQtL3D-x2Ydr-7uxLt9zzEFzJrGB-X96d9XY8k9XTJj-RUg9WpzGB9jg\",\"accessToken\":\"token1\",\"locale\":\"en-US\",\"lastSeen\":\"2019-02-20T21:32:22Z\",\"userStorage\":\"{\\\"data\\\":{}}\"},\"conversation\":{\"conversationId\":\"ABwppHE35s8T6qSdaaCMNiWuMdY7UsvQ3sHbLZJOQkVA4AFD2nhKuqTTvoJTkVh7yU81GCHbvlfZTxmULLd4Ug\",\"type\":\"ACTIVE\",\"conversationToken\":\"[\\\"_actions_on_google\\\"]\"},\"inputs\":[{\"intent\":\"actions.intent.SIGN_IN\",\"rawInputs\":[{}],\"arguments\":[{\"name\":\"SIGN_IN\",\"extension\":{\"@type\":\"type.googleapis.com/google.actions.v2.SignInValue\",\"status\":\"OK\"}},{\"name\":\"text\"}]}],\"surface\":{\"capabilities\":[{\"name\":\"actions.capability.WEB_BROWSER\"},{\"name\":\"actions.capability.SCREEN_OUTPUT\"},{\"name\":\"actions.capability.MEDIA_RESPONSE_AUDIO\"},{\"name\":\"actions.capability.AUDIO_OUTPUT\"}]},\"isInSandbox\":true,\"availableSurfaces\":[{\"capabilities\":[{\"name\":\"actions.capability.WEB_BROWSER\"},{\"name\":\"actions.capability.SCREEN_OUTPUT\"},{\"name\":\"actions.capability.AUDIO_OUTPUT\"}]}]}.",
"insertId": "120k9w1f3jmw55",
"resource": {
"type": "assistant_action",
"labels": {
"version_id": "",
"action_id": "actions.intent.SIGN_IN",
"project_id": "speechbank-e8a15"
}
},
"timestamp": "2019-02-21T13:48:28.768213970Z",
"severity": "DEBUG",
"labels": {
"source": "AOG_REQUEST_RESPONSE",
"querystream": "GOOGLE_USER",
"channel": "preview"
},
"logName": "projects/speechbank-e8a15/logs/actions.googleapis.com%2Factions",
"trace": "projects/366800784520/traces/ABwppHE35s8T6qSdaaCMNiWuMdY7UsvQ3sHbLZJOQkVA4AFD2nhKuqTTvoJTkVh7yU81GCHbvlfZTxmULLd4Ug",
"receiveTimestamp": "2019-02-21T13:48:28.912828815Z"
},
{
"textPayload": "Received response from agent with body: HTTP/1.1 200 OK\r\nServer: nginx/1.13.6\r\nDate: Thu, 21 Feb 2019 13:48:28 GMT\r\nContent-Type: application/json;charset=UTF-8\r\nContent-Length: 570\r\nX-Cloud-Trace-Context: 664d8fdaf9cd3d880d41f11ac2176e0e/16724608154084655134;o=0\r\nGoogle-Actions-API-Version: 2\r\nAssistant-Interaction-Error-Code: -1\r\nAssistant-Interaction-Error-Message: Failed to parse Dialogflow response into AppResponse because of empty speech response\r\nX-SHARD: shard-2\r\nVia: 1.1 google\r\nAlt-Svc: clear\r\n\r\n{\n \"responseMetadata\": {\n \"status\": {\n \"code\": 10,\n \"message\": \"Failed to parse Dialogflow response into AppResponse because of empty speech response\",\n \"details\": [{\n \"@type\": \"type.googleapis.com/google.protobuf.Value\",\n \"value\": \"{\\\"id\\\":\\\"5d4bed8d-c58c-4429-9838-f758d6f335f2\\\",\\\"timestamp\\\":\\\"2019-02-21T13:48:28.806Z\\\",\\\"lang\\\":\\\"en-us\\\",\\\"result\\\":{},\\\"status\\\":{\\\"code\\\":200,\\\"errorType\\\":\\\"success\\\"},\\\"sessionId\\\":\\\"ABwppHE35s8T6qSdaaCMNiWuMdY7UsvQ3sHbLZJOQkVA4AFD2nhKuqTTvoJTkVh7yU81GCHbvlfZTxmULLd4Ug\\\"}\"\n }]\n }\n }\n}.",
"insertId": "120k9w1f3jmw56",
"resource": {
"type": "assistant_action",
"labels": {
"project_id": "speechbank-e8a15",
"version_id": "",
"action_id": "actions.intent.SIGN_IN"
}
},
"timestamp": "2019-02-21T13:48:28.899033790Z",
"severity": "DEBUG",
"labels": {
"channel": "preview",
"source": "AOG_REQUEST_RESPONSE",
"querystream": "GOOGLE_USER"
},
"logName": "projects/speechbank-e8a15/logs/actions.googleapis.com%2Factions",
"trace": "projects/366800784520/traces/ABwppHE35s8T6qSdaaCMNiWuMdY7UsvQ3sHbLZJOQkVA4AFD2nhKuqTTvoJTkVh7yU81GCHbvlfZTxmULLd4Ug",
"receiveTimestamp": "2019-02-21T13:48:28.912828815Z"
},
{
"textPayload": "MalformedResponse: Failed to parse Dialogflow response into AppResponse because of empty speech response",
"insertId": "1b6j2e6f39jvuy",
"resource": {
"type": "assistant_action",
"labels": {
"project_id": "speechbank-e8a15",
"version_id": "",
"action_id": "actions.intent.SIGN_IN"
}
},
"timestamp": "2019-02-21T13:48:28.899403302Z",
"severity": "ERROR",
"labels": {
"channel": "preview",
"source": "JSON_RESPONSE_VALIDATION",
"querystream": "GOOGLE_USER"
},
"logName": "projects/speechbank-e8a15/logs/actions.googleapis.com%2Factions",
"trace": "projects/366800784520/traces/ABwppHE35s8T6qSdaaCMNiWuMdY7UsvQ3sHbLZJOQkVA4AFD2nhKuqTTvoJTkVh7yU81GCHbvlfZTxmULLd4Ug",
"receiveTimestamp": "2019-02-21T13:48:28.914061262Z"
}
]
Based on the above setup description, could anyone please give us a hand figuring out what's causing the MalformedResponse exception and what needs to change to eliminate it.
This exception is so obscure that it brings about a myriad questions and places one at one's wits end as to where to start approaching it. I'll list just a few here and would really appreciate some guidance.
Should there be any correlation between intent names in AoG and Dialogflow? Should they follow any naming convention? Could the cause of the error lie in their somehow being misnamed?
Can the MalformedResponse be interpreted as lack of a particular field on the response? Since Google chose to expose the inner workings of the conversion between different message formats (Dialogflow and AppResponse), is there a listing somewhere of what fields are required on a Dialogflow response?
Is this to imply that even the OAuth messages that are being passed around in this case need to contain some speech?
Initially, the
userIdreceived from Dialogflow seems to always benullbut the query text seems to be populated withGOOGLE_ASSISTANT_WELCOME, so we are starting the account linking flow logic based on the assumption that it isnull. Is that a right assumption to have?Under which circumstances will the
userIdbe initially populated (like in Alexa where it's autogenerated upon enabling a skill for the user) so that theelsecondition above could get triggered?Should the OAuth token issued by the authentication infrastructure and supported by AoG be in any particular format, i.e. OIDC or JWT. Could it be any random string? Is
token1still a valid token in AoG parlance (as it is in Alexa)?Any misconfigured Java intent handler(s)? Which intent name in the response from the AoG account linking flow should we be reacting to?
Is there a catch-all intent name(s), a handler for which can be incorporated into the Java app to facilitate further debugging of the above?
What is meant by "empty speech response", what values are we not providing that are expected and cause the breakage?
Anything that we have configured which should not have been configured?
If it matters at all, here's the log from our webhook:
[java] 02-21-2019 13:47:57 [qtp2056234595-127] INFO domain.lola.user.utils.http.ControllerUtils [toString:30] - Received AoG Request {
[java] "responseId": "0156911c-d7e8-405b-bf8f-f23320c02030",
[java] "queryResult": {
[java] "queryText": "GOOGLE_ASSISTANT_WELCOME",
[java] "parameters": {
[java] "any": ""
[java] },
[java] "allRequiredParamsPresent": true,
[java] "fulfillmentMessages": [{
[java] "text": {
[java] "text": [""]
[java] }
[java] }],
[java] "outputContexts": [{
[java] "name": "projects/speechbank-e8a15/agent/sessions/ABwppHE35s8T6qSdaaCMNiWuMdY7UsvQ3sHbLZJOQkVA4AFD2nhKuqTTvoJTkVh7yU81GCHbvlfZTxmULLd4Ug/contexts/google_assistant_welcome",
[java] "parameters": {
[java] "any.original": "",
[java] "any": ""
[java] }
[java] }, {
[java] "name": "projects/speechbank-e8a15/agent/sessions/ABwppHE35s8T6qSdaaCMNiWuMdY7UsvQ3sHbLZJOQkVA4AFD2nhKuqTTvoJTkVh7yU81GCHbvlfZTxmULLd4Ug/contexts/actions_capability_screen_output",
[java] "parameters": {
[java] "any.original": "",
[java] "any": ""
[java] }
[java] }, {
[java] "name": "projects/speechbank-e8a15/agent/sessions/ABwppHE35s8T6qSdaaCMNiWuMdY7UsvQ3sHbLZJOQkVA4AFD2nhKuqTTvoJTkVh7yU81GCHbvlfZTxmULLd4Ug/contexts/actions_capability_audio_output",
[java] "parameters": {
[java] "any.original": "",
[java] "any": ""
[java] }
[java] }, {
[java] "name": "projects/speechbank-e8a15/agent/sessions/ABwppHE35s8T6qSdaaCMNiWuMdY7UsvQ3sHbLZJOQkVA4AFD2nhKuqTTvoJTkVh7yU81GCHbvlfZTxmULLd4Ug/contexts/google_assistant_input_type_voice",
[java] "parameters": {
[java] "any.original": "",
[java] "any": ""
[java] }
[java] }, {
[java] "name": "projects/speechbank-e8a15/agent/sessions/ABwppHE35s8T6qSdaaCMNiWuMdY7UsvQ3sHbLZJOQkVA4AFD2nhKuqTTvoJTkVh7yU81GCHbvlfZTxmULLd4Ug/contexts/actions_capability_web_browser",
[java] "parameters": {
[java] "any.original": "",
[java] "any": ""
[java] }
[java] }, {
[java] "name": "projects/speechbank-e8a15/agent/sessions/ABwppHE35s8T6qSdaaCMNiWuMdY7UsvQ3sHbLZJOQkVA4AFD2nhKuqTTvoJTkVh7yU81GCHbvlfZTxmULLd4Ug/contexts/actions_capability_media_response_audio",
[java] "parameters": {
[java] "any.original": "",
[java] "any": ""
[java] }
[java] }],
[java] "intent": {
[java] "name": "projects/speechbank-e8a15/agent/intents/f645f492-f6dc-4e7e-8da6-45711c654ad0",
[java] "displayName": "RawText"
[java] },
[java] "intentDetectionConfidence": 1.0,
[java] "languageCode": "en-us"
[java] },
[java] "originalDetectIntentRequest": {
[java] "source": "google",
[java] "version": "2",
[java] "payload": {
[java] "isInSandbox": true,
[java] "surface": {
[java] "capabilities": [{
[java] "name": "actions.capability.AUDIO_OUTPUT"
[java] }, {
[java] "name": "actions.capability.MEDIA_RESPONSE_AUDIO"
[java] }, {
[java] "name": "actions.capability.SCREEN_OUTPUT"
[java] }, {
[java] "name": "actions.capability.WEB_BROWSER"
[java] }]
[java] },
[java] "inputs": [{
[java] "rawInputs": [{
[java] "query": "open speech Bank",
[java] "inputType": "VOICE"
[java] }],
[java] "intent": "actions.intent.MAIN"
[java] }],
[java] "user": {
[java] "userStorage": "{\"data\":{}}",
[java] "lastSeen": "2019-02-20T21:32:22Z",
[java] "locale": "en-US",
[java] "userId": "ABwppHFQHUBr0RrWA_OuL-kK2sxTPUvQtL3D-x2Ydr-7uxLt9zzEFzJrGB-X96d9XY8k9XTJj-RUg9WpzGB9jg"
[java] },
[java] "conversation": {
[java] "conversationId": "ABwppHE35s8T6qSdaaCMNiWuMdY7UsvQ3sHbLZJOQkVA4AFD2nhKuqTTvoJTkVh7yU81GCHbvlfZTxmULLd4Ug",
[java] "type": "NEW"
[java] },
[java] "availableSurfaces": [{
[java] "capabilities": [{
[java] "name": "actions.capability.AUDIO_OUTPUT"
[java] }, {
[java] "name": "actions.capability.SCREEN_OUTPUT"
[java] }, {
[java] "name": "actions.capability.WEB_BROWSER"
[java] }]
[java] }]
[java] }
[java] },
[java] "session": "projects/speechbank-e8a15/agent/sessions/ABwppHE35s8T6qSdaaCMNiWuMdY7UsvQ3sHbLZJOQkVA4AFD2nhKuqTTvoJTkVh7yU81GCHbvlfZTxmULLd4Ug"
[java] }
[java] 02-21-2019 13:47:57 [qtp2056234595-127] INFO domain.lola.user.utils.actionsongoogle.AoGApp [launchRequestHandler:26] - userId=null
[java] 02-21-2019 13:47:57 [qtp2056234595-127] INFO domain.lola.user.utils.actionsongoogle.AoGApp [launchRequestHandler:28] - queryText=GOOGLE_ASSISTANT_WELCOME
[java] 02-21-2019 13:47:57 [qtp2056234595-127] INFO domain.lola.user.utils.actionsongoogle.AoGBotService [handlePOST:103] - Generated response:
[java] {
[java] "outputContexts" : [ {
[java] "lifespanCount" : 99,
[java] "name" : "projects/speechbank-e8a15/agent/sessions/ABwppHE35s8T6qSdaaCMNiWuMdY7UsvQ3sHbLZJOQkVA4AFD2nhKuqTTvoJTkVh7yU81GCHbvlfZTxmULLd4Ug/contexts/_actions_on_google",
[java] "parameters" : {
[java] "data" : "{}"
[java] }
[java] } ],
[java] "payload" : {
[java] "google" : {
[java] "expectUserResponse" : true,
[java] "isSsml" : false,
[java] "systemIntent" : {
[java] "intent" : "actions.intent.SIGN_IN",
[java] "data" : {
[java] "@type" : "type.googleapis.com/google.actions.v2.SignInValueSpec"
[java] }
[java] },
[java] "userStorage" : "{\"data\":{}}"
[java] }
[java] }
[java] }
