I've seen examples using the Vercel ai
library to quickly create a streamable OpenAI response type (e.g. https://stackoverflow.com/a/76634615/5348578), however, I'd like to do data modification prior to sending a response.
Is this possible using the libraries? If not, what is the correct approach?
Currently, I'm attempting to do the following without the use of the aforementioned libraries:
const completionMessage = await openai.createChatCompletion({
model: 'gpt-4',
stream: true,
messages: conversation,
});
const transformStream = new Transform({
transform(chunk, encoding, callback) {
const stringData = chunk.toString();
// Parse the Server-Sent Event
const eventData = stringData.replace('data: ', '');
/*
If `stream` is set, partial message deltas will be sent. Tokens will be sent as data-
only server-sent events as they become available, with the stream terminated by a data:
[DONE] message.
*/
if (eventData === '[DONE]') {
return callback();
}
// Parse the data
const data = JSON.parse(eventData);
const message = data.choices[0].message?.content;
const newData = {
type: 'chat',
data: {
role: 'assistant',
content: message,
},
myProperty: false,
};
this.push(JSON.stringify(newData));
callback();
},
});
const responseStream = completionMessage.body;
if (responseStream === null) {
throw new Error('Response stream is null');
}
const writableStream = new WritableStream({
write(chunk) {
transformStream.write(chunk);
},
close() {
transformStream.end();
},
abort(err) {
transformStream.destroy(err);
},
});
responseStream.pipeTo(writableStream);
return new NextResponse(transformStream);
However, this approach does not seem to work as the transformStream
is not a valid type to assign to NextResponse
.