override langchain llm completion call to handle 400 moderated response

Question

Using langchain==0.0.198, langchainplus-sdk==0.0.8, python==3.10

I am working with openAI, Azure.
Currently, the moderation endpoint that was available in the non-Azure openAI is no longer available with Azure openAI, and has been "bundled" with the call to Completion as "content-filtering".

I need to handle the content-filtration myself, since I don't want the llm to respond directly to calls that are flagged with moderation type issues.

How can I override langchain, which is essentially making the Completion call, such that I can inspect the response for such flagged moderation, and handle it myself?
The response is a 400, and the "finish_reason" is "content_filter".

Another main reason to override it is the fact that langchain doesn't seem to handle the 400 well, and crashes later on. This is one git issue I found.

Looking at langchain code, I think I identified the method I would need to override in its definition, or where it is called (I am still looking into it)

If there is any other way of doing that, please let me know.

Thank you!

Answer 1

To override the langchain LLM completion call to handle 400 moderated responses, you can do the following steps:

• Create a custom class that inherits from the LLMCompletion class.

• In the __call__() method of the custom class, override the _make_completion_request() method.

• In the _make_completion_request() method, add code to check the response for the finish_reason field. If the finish_reason field is content_filter, then handle the response yourself.

  from langchain import LLMCompletion
  
      class CustomLLMCompletion(LLMCompletion):
          def __call__(self, prompt, **kwargs):
              response = self._make_completion_request(prompt, **kwargs)
              if response["finish_reason"] == "content_filter":
                  # handle the content-filtered response
              else:
                  # return the response from the completion API
                  return response