QNA Maker not returning exact matches

Question

QNA Maker is not returning the Exact matches from the knowledge bases.

So we've been using the botbuilder-ai library to call the QNA maker. However QNA Maker is not returning the exact match which is there in the knowledgebase. However if I check that in the qnamaker.ai portal's test feature it works.

Here is the code that I'm using to extract the results.

const this.qnaRecognizer = new QnAMaker(
                {
                    knowledgeBaseId: keyvault.QnAMakerSecret.knowledgebaseID,
                    endpointKey: keyvault.QnAMakerSecret.qnaAuthKEY,
                    host: keyvault.QnAMakerSecret.qnaEndPointHost,
                },
                {
                    scoreThreshold: QNA_CONFIDENCE_THRESHOLD,
                    top: QNA_NUM_OF_RESULTS,
                }
            );
const qnaResult = await this.qnaRecognizer.getAnswers(step.context);

As expected, it would call the QNA Maker's endpoint and should return results. Instead its returning a blank Array.

Now, I monitored the qnamaker.ai calls and noticed that when they are calling the APIs, they are passing one more parameter which is isTest = true.

Here are the results:

Without isTest = true

URL: https://qnamaker-host.azurewebsites.net/qnamaker/knowledgebases/<kbid>/generateAnswer
Method: POST
Result: {
    "answers": [
        {
            "questions": [],
            "answer": "No good match found in KB.",
            "score": 0,
            "id": -1,
            "source": null,
            "metadata": []
        }
    ],
    "debugInfo": null
}

With isTest = true

URL: https://qnamaker-host.azurewebsites.net/qnamaker/knowledgebases/<kbid>/generateAnswer
Method: POST
Result: {
    "answers": [
        {
            "questions": [
                "Who are you?"
            ],
            "answer": "I am an intelligent bot",
            "score": 100,
            "id": 2,
            "source": "Editorial",
            "metadata": [
                {
                    "name": "_id",
                    "value": "<removed>"
                }
            ],
            "context": {
                "isContextOnly": false,
                "prompts": []
            }
        }
    ],
    "debugInfo": null
}

Now I should be expecting same behaviour without setting isTest = true in this case. Also, in each APIs I cannot pass isTest = true because I'm directly using their library to do this.

Can someone please help in this? Thanks in advance.

Tony Ju · Answer 1 · 2019-05-17T05:09:48.140

1

The most likely reason is that you didn't publish the knowledge base.

With "isTest": true, you are requesting the test knowledge base instead of the published knowledge base. Refer to this document for more details.

Update:

There is a design limitation with QnA Maker when multiple KBs exist in a resource, the test environment is impacted by other KBs. The production environment is isolated from other KBs. This is due to the Azure Search indexes being shared among KBs in the test environment. One work-around is to only have one KB in the resource when using the test environment.

edited May 17 '19 at 05:09

answered May 17 '19 at 01:29

Tony Ju

14,891
3
17
31

@DhrumanBhadeshiya Do you have multiple KBs exist in a resource? If so, see my updated answer. – Tony Ju May 17 '19 at 05:09
I am little bit confused here. What do you mean by production environment & testing environment of QNA Maker? Is there any switch to change that? – Dhruman Bhadeshiya May 17 '19 at 05:20
Also, I'm not using test environment inside my bot. My bot is calling the production environment itself. Although I am not getting the results from production KB. – Dhruman Bhadeshiya May 17 '19 at 05:22
@DhrumanBhadeshiya Do you have multiple KBs in a resource? Yes, if you didn't use IsTest=true, it will calling the production env – Tony Ju May 17 '19 at 05:25
Yes I have multiple KBs in a resource. And I am not using isTest=true from my bots. That I was just debugging. But even after publishing the bot & not using isTest=true, I am not getting the results. – Dhruman Bhadeshiya May 17 '19 at 05:29
@DhrumanBhadeshiya I see. That might be the issue. I encountered the same issue with multiple kbs in a resource. I raised a support ticket to Azure support engineer. The updated answer is the reply from them. This is a known issue. – Tony Ju May 17 '19 at 05:34
@DhrumanBhadeshiya The results in test environment didn't publish to product environment successfully even we click the publish button. – Tony Ju May 17 '19 at 05:38
So you mean to say that QNA Maker cannot handle the multiple knowledge bases? – Dhruman Bhadeshiya May 17 '19 at 10:08
@DhrumanBhadeshiya Yes, this may cause index issue. This is the known issue and will be fixed by the product team. – Tony Ju May 17 '19 at 10:48
Are you sure this is the case? I am not talking about multiple knowledgebase test index. I'm talking about multiple knowledgebase published indices. Is QNA Maker still not able to properly handle published indices? – Dhruman Bhadeshiya May 20 '19 at 10:42
@DhrumanBhadeshiya Yes, I know. I encountered the same issue as yours. Single kb will solve this issue. – Tony Ju May 20 '19 at 12:53
@DhrumanBhadeshiya If you still have any other concerns regarding this, you can raise a support ticket on Azure portal by following this link https://learn.microsoft.com/en-us/azure/azure-supportability/how-to-create-azure-support-request – Tony Ju May 21 '19 at 07:19

QNA Maker not returning exact matches

1 Answers1