How to send OpenAI stream response from Nextjs API to client

Question

I tried openai-streams + nextjs-openai, they only works for Node 18+, however, they failed on Node 17 and lower. I'm restricted to Node 17 and lower as Digital Oceans App Platform currently not supporting Node 18.

I also tried this method which works well on client side, but it exposes the API key. I want to implement within the NextJS API route, but I'm unable to pass the streaming response to the client.

With the code below, I can only get the first chunk of response from the API route, and not able to get the streaming response to have the ChatGPT effect. Please kindly help.

// /api/prompt.js

import { Configuration, OpenAIApi } from "openai";
import { Readable } from "readable-stream";

const configuration = new Configuration({
  apiKey: process.env.NEXT_PUBLIC_OPENAI_API_KEY,
});
const openai = new OpenAIApi(configuration);

export default async function handler(req, res) {
  const completion = await openai.createCompletion(
    {
      model: "text-davinci-003",
      prompt: "tell me a story",
      max_tokens: 500,
      stream: true,
    },
    { responseType: "stream" }
  );

  completion.data.on("data", async (data) => {
    const lines = data
      .toString()
      .split("\n")
      .filter((line) => line.trim() !== "");

    for (const line of lines) {
      const message = line.replace(/^data: /, "");
      if (message === "[DONE]") {
        return;
      }
      try {
        const parsed = JSON.parse(message);
        const string = parsed.choices[0].text;
        Readable.from(string).pipe(res);
      } catch (error) {
        console.error("Could not JSON parse stream message", message, error);
      }
    }
  });

// /components/Completion.js

export default function Completion() {
  const [text, setText] = useState();

  const generate = async () => {
    const response = await fetch("/api/prompt");
    console.log("response: ", response);
    const text = await response.text();
    console.log("text: ", text);
    setText((state) => state + text);
  };
  
  // ... rest
}

score 2 · Answer 1 · edited Jul 13 '23 at 18:42

2

Well you can use StreamingTextResponse of vercel AI sdk

Attached is some example code form their docs

import { Configuration, OpenAIApi } from 'openai-edge'
import { OpenAIStream, StreamingTextResponse } from 'ai'

const config = new Configuration({
  apiKey: process.env.OPENAI_API_KEY
})
const openai = new OpenAIApi(config)

export const runtime = 'edge'

export async function POST(req) {
  const { messages } = await req.json()
  const response = await openai.createChatCompletion({
    model: 'gpt-4',
    stream: true,
    messages
  })
  const stream = OpenAIStream(response)
  return new StreamingTextResponse(stream)
}

Notes:

This requires the Edge Runtime
The library openai-edge requires Node 18+. However, this is expected to be deprecated soon with the support for streaming in the official openai library.

edited Jul 13 '23 at 18:42

Jayant Bhawal

2,044
2
31
32

answered Jul 07 '23 at 07:03

shivam singh

49
5

Answer needs supporting information Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](https://stackoverflow.com/help/how-to-answer). – moken Jul 11 '23 at 12:35
Is it possible to modify the data within this stream? E.g. I wish to return an object with the content (`data.choices[0].message?.content`) and a custom field. My understanding is that due to the fact that `OpenAiStream` formats the response, this is not possible and you need to build a custom stream. – DeveloperRyan Jul 23 '23 at 19:51

score 0 · Answer 2 · answered May 21 '23 at 09:41

You can use the openai-streams/node entrypoint on Node <18, which will return a Node.js Readable instead of a WHATWG ReadableStream. I'll update the docs to be clearer soon.

Node: Consuming streams in Next.js API Route

If you cannot use an Edge runtime or want to consume Node.js streams for another reason, use openai-streams/node:

import type { NextApiRequest, NextApiResponse } from "next";
import { OpenAI } from "openai-streams/node";

export default async function test(_: NextApiRequest, res: NextApiResponse) {
  const stream = await OpenAI("completions", {
    model: "text-davinci-003",
    prompt: "Write a happy sentence.\n\n",
    max_tokens: 25,
  });

  stream.pipe(res);
}

How to send OpenAI stream response from Nextjs API to client

2 Answers2

Node: Consuming streams in Next.js API Route

Linked