0

The following code streams some data from OpenAI's chat completion API that responds using server sent events.

export async function promptStream(
  apiKey: string,
  userId: string,
  prompt: Prompt,
  handleNewChunk: (chunk: string) => Promise<void>
) {
  const req = https.request(
    {
      hostname: "api.openai.com",
      port: 443,
      path: "/v1/chat/completions",
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        Authorization: "Bearer " + apiKey,
      },
    },
    function (res) {
      res.on("data", async (data) => {
        handleNewChunk(data)
        }
      });
      res.on("end", () => {
        console.log("No more data in response.");
      });
    }
  );

  const body = JSON.stringify({
    ...prompt,
    stream: true,
  });

  req.write(body);

  req.end();

}

When I run this function locally using a Firebase functions emulator, it works perfectly fine: https://www.loom.com/share/bfebefe0fc6a40df972ee1be179123db

However, after deploying to Cloud Functions gen2 (which is built on top of Cloud Run) and invoking it, there's a 10-15 second delay, then all of the text comes flooding in at once.

skillshot
  • 90
  • 1
  • 6

1 Answers1

0

My issue was that I needed to block the entire function until the full response is read. Wrapping the entire body of promptStream with a Promise and returning that fixed it for me.

The issue is that my closures in the res.on callback were being marked as background processes. https://cloud.google.com/run/docs/configuring/cpu-allocation

skillshot
  • 90
  • 1
  • 6
  • The `res.on` callback is not a background process. It is a thread that is part of your application and not a separate process. Your code must wait on the promise to complete. As written this post is not an answer. Clearly specify the code changes that are required. Otherwise, your post should be a comment or addition to your original post (question). – John Hanley May 05 '23 at 20:00