0

I am encountering an issue with the LangChain's Self-Critique Chain (Constitutional AI) when using it with AzureOpenAI. While it works perfectly fine with OpenAI, I receive the following error message when using AzureOpenAI:

openai.error.InvalidRequestError: The response was filtered due to the prompt triggering Azure OpenAI’s content management policy. Please modify your prompt and retry. To learn more about our content filtering policies please read our documentation: https://go.microsoft.com/fwlink/?linkid=2198766

Interestingly, this error occurs regardless of the query I provide. Even a simple question about Canada's capital triggers Azure's content filtering. It's worth noting that this issue does not occur when using OpenAI, and it also doesn't occur when using AzureOpenAI without the Self-Critique Chain.

I have tested it with different prompts and principles, but the result is always the same when I use AzureOpenAI: content policy filtering!

hmfGeek
  • 31
  • 2
  • Why the down votes!? Wrong Tags? – hmfGeek May 29 '23 at 02:25
  • Please provide enough code so others can better understand or reproduce the problem. – Community May 29 '23 at 18:06
  • What "LangChain's Self-Critique Chain (Constitutional AI)" is injecting in the prompt? Azure OpenAI has a content safety feature which is clearly documented here: https://learn.microsoft.com/en-US/azure/cognitive-services/openai/concepts/content-filter – Nicolas R May 31 '23 at 10:43

1 Answers1

0

I also faced this issue. Looks like LangChain adds few examples as part of FewShotPrompting to help the model understand the critical parts. And Azure finds those chats offensive and flags it off. There are 2 ways to get over this.

  1. Clear all examples and just rely on the prompt to guide the model to filter out. This can be done using the following code:
constitutional_chain.critique_chain.prompt.examples.clear()
constitutional_chain.revision_chain.prompt.examples.clear()
  1. Replace the existing examples with examples that Azure wont find offended. Remember that each model is a dict with the following keys:
dict_keys(['input_prompt', 'output_from_model', 'critique_request', 'critique', 'revision']

I am going with the first one and planning to add examples to the list as and when I find an offensive response not flagging out.

Hope this helps.

Naresh
  • 3,174
  • 1
  • 17
  • 22