How to add live speech to text data for microsoft bot framework(exactly what google speech does)

Question

I am new to microsoft bot framework. I already build a simple chatbot, and when I publish it and deloyed it to webapp channel, it looks like this

where user will select or type text and bot will respond. Now what i need is, I need to add a mic near by send option so that if user clicks on the mic and starts speaking then it should be typed automatically by bot(exactly how google speech does)

Im having bing speech to text api ref key but I dont know how to add and activate the mic functionality in it.

If anyone knows please help me to resolve this

score 3 · Accepted Answer · answered Nov 14 '17 at 08:52

You can leverage Embeddable web chat control for the Microsoft Bot Framework plugin in your web application. You can refer to https://github.com/Microsoft/BotFramework-WebChat/blob/master/samples/speech/index.html for a speech sample.

And generally speaking for some points in code for quick test:

Include style and script files in your web page:

<link href='https://cdn.botframework.com/botframework-webchat/latest/botchat.css' rel="stylesheet"/> ... <script src="https://cdn.botframework.com/botframework-webchat/latest/botchat.js"></script> <script src="https://cdn.botframework.com/botframework-webchat/latest/CognitiveServices.js"></script>

Init speech options:

const speechOptions = {
  speechRecognizer: new CognitiveServices.SpeechRecognizer({ subscriptionKey: 'YOUR_COGNITIVE_SPEECH_API_KEY' }),
  speechSynthesizer: new CognitiveServices.SpeechSynthesizer({
    gender: CognitiveServices.SynthesisGender.Female,
    subscriptionKey: 'YOUR_COGNITIVE_SPEECH_API_KEY',
    voiceName: 'Microsoft Server Speech Text to Speech Voice (en-US, JessaRUS)'
  })
};

Init BotChat in script:

BotChat.App({
  bot: bot,
  locale: params['locale'],
  resize: 'detect',
  // sendTyping: true,    // defaults to false. set to true to send 'typing' activities to bot (and other users) when user is typing
  speechOptions: speechOptions,
  user: user,
  // locale: 'es-es', // override locale to Spanish
  directLine: {
    domain: params['domain'],
    secret: params['s'],
    token: params['t'],
    webSocket: params['webSocket'] && params['webSocket'] === 'true' // defaults to true
  }
}, document.getElementById('BotChatGoesHere'));

thanks @GaryLiu for the response, I tried this but I am getting error as directline.botframework.com/v3/directline/conversations Failed to load resource: the server responded with a status of 403 (Forbidden).. and where I ll find token for directline in botframework? — Sunitha Bist, Nov 14 '17 at 12:44
The speech connection is built on WSS portocol, you need to force to use `https` to request your server. — Gary Liu, Nov 15 '17 at 01:27
@Sana, as there is an upgrade, so the code will be different, please refer to the new sample https://github.com/Microsoft/BotFramework-WebChat/blob/master/samples/06.c.cognitive-services-speech-services-js/index.html — Gary Liu, Feb 21 '19 at 01:56

score 1 · Answer 2 · answered Nov 04 '22 at 14:56

This is the code I used. Create Speech API service and add direct line channel with default site to the bot setup in azure.

<!DOCTYPE html>
<html lang="en-US">
<head>
    <title>Web Chat: Cognitive Services Speech Services using JavaScript</title>
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <!-- Cognitive Services Speech Services adapter is only available in full bundle -->
    <!--
      This CDN points to the latest official release of Web Chat. If you need to test against Web Chat's latest bits, please refer to using Web Chat's latest bits:
      https://github.com/microsoft/BotFramework-WebChat#how-to-test-with-web-chats-latest-bits
    -->
    <script crossorigin="anonymous" src="https://cdn.botframework.com/botframework-webchat/latest/webchat.js"></script>
    <style>
        html,
        body {
            height: 100%;
        }

        body {
            margin: 0;
        }

        #webchat {
            height: 100%;
            width: 100%;
        }
    </style>
</head>
<body>
    <div id="webchat" role="main"></div>
    <script>
        (async function () {
            let authorizationToken;
            let region;
            const speechServicesTokenRes = await fetch(
                'https://<your-speech-service-region-eg-uksouth>.api.cognitive.microsoft.com/sts/v1.0/issueToken',
                { method: 'POST', headers: { 'Ocp-Apim-Subscription-Key': '<your-speech-api-key>' } }
            );

            if (speechServicesTokenRes.status === 200) {
                region = '<your-azure-region-eg-uksouth>',
                    authorizationToken = await speechServicesTokenRes.text()
            } else {
                return (new Error('error!'))
            }

            const webSpeechPonyfillFactory = await window.WebChat.createCognitiveServicesSpeechServicesPonyfillFactory({
                credentials: {
                    authorizationToken: authorizationToken,
                    region: region
                }
            });

            var dl = window.WebChat.createDirectLine({ secret: '<your-directline-default-site-secret-key>' });

            window.WebChat.renderWebChat({
                directLine: dl,
                webSpeechPonyfillFactory: webSpeechPonyfillFactory
            }, document.getElementById('webchat'));

        })().catch(err => console.error(err));

    </script>
</body>
</html>

How to add live speech to text data for microsoft bot framework(exactly what google speech does)

2 Answers2