Node/Typescript: How to close a writestream in the process 'exit' event - red/blue function problem / async infection

Question

In my current FOSS Discord bot project I have this log.ts file which handles the logging for the bot.

It creates multiple fs.WriteStream objects, which write to each log file. There is a section in the code when await log('CLOSE_STREAMS') is called to run the WriteStream#close() functions on each WriteStream, returning a promise. This is used in the process.on('exit') handler to save the log files before we close.

The problem here is that the 'exit' event handler can not schedule any additional work into the event queue.

How could I handle calling the CLOSE_STREAMS in a way where I can run this exit handler as I am expecting?

Function Implementation, simplified

log.ts log('CLOSE_STREAMS')

// Main
export default function log(mode: 'CLOSE_STREAMS'): Promise<void>;
export default function log(mode: 'v' | 'i' | 'w' | 'e', message: any, _bypassStackPrint?: boolean): void;
// eslint-disable-next-line @typescript-eslint/explicit-module-boundary-types
export default function log(mode: 'v' | 'i' | 'w' | 'e' | 'CLOSE_STREAMS', message?: any, _bypassStackPrint = false): void | Promise<void> {
    if (mode === 'CLOSE_STREAMS')
        // Close all of the file streams
        return new Promise((resolve) => {
            errStr.end(() => {
                warnStr.end(() => {
                    allStr.end(() => {
                        resolve();
                    });
                });
            });
        });
    else {

index.ts

This is the way the log is killed in the uncaught exceptions; this would be how I want to do it for the exit event.

// If we get an uncaught exception, close ASAP.
process.on('uncaughtException', async (error) => {
    log('e', 'Killing client...', true);
    client.destroy();
    log('e', 'Client killed.', true);
    log('e', 'Closing databases...', true);
    client.closeDatabases();
    log('e', 'Closed databases.', true);
    log('e', 'An uncaught exception occured!', true);
    log('e', `Error thrown was:`, true);
    error.stack?.split('\n').forEach((item) => {
        log('e', `${item}`, true);
    });
    log('e', 'Stack trace dump:', true);
    let stack = new Error().stack?.split('\n');
    stack?.shift();
    if (!stack) stack = [];

    stack.forEach((item) => {
        log('e', `${item}`, true);
    });
    log('e', 'Process exiting.', true);
    log('e', 'Exit code 5.', true);
    log('e', 'Goodbye!', true);
    await log('CLOSE_STREAMS'); // <<<<<<<<<<<< HERE
    process.exit(5);
});

Answer 1

As you know, you can't reliably use asynchronous operations in the processing of an exit event because the process will exit before they complete. As such, I don't think there's any way to do this reliably with a nodejs stream. Streams have a completely asynchronous API and implementation, including flushing and closing.

In a few Google searches I found a few other people asking for the same thing for the same reasons and there was no solution offered. As best I know, these are the options:

1. Do some hacking on the stream object to add a synchronous flushAndClose() method to your stream. You'd have to work into the internals of the stream to get the buffer and the file handle and do your own synchronous write of any remaining buffer and then do a synchronous close on the file handle. Note, even this has a problem case if there's currently an asynchronous write operation in process.

2. Abandon the built-in stream and just implement your own lightweight logfile interface that makes it easy for you to have both asynchronous writing (for normal use) and a synchronous flushAndClose() operation for emergency shut-down. Note, even this has a problem case if there's currently an asynchronous write operation in process when you want to do the synchronous close.

3. Rather than using process.on('exit', ...) to trigger closing of the log files, go up one level higher in the chain. Whatever it is that triggers closing of your app, put it in an async function that will wait for the log files to be properly closed before calling process.exit() so you get the log files closed when you still have the ability to handle asynchronous operations.

4. Do logging from a different (more stable) process. This process can then message that logging process what it wants logged and that process can manage getting the logging info safely to disk independent of whether the source process shuts down abruptly or not.

Note: Exiting a process will automatically close any open file selectors (the OS takes care of that for you). So, as long as this is an edge case shut-down in some fatal error condition, not a common normal shut-down, then perhaps you don't have a big problem here to really solve.

The worst that could happen is that you might lose some very recently logged lines of data if they hadn't yet been flushed from the stream. Note that streams write data immediately to their descriptor when possible so they don't generally accumulate lots of buffered data. The time when they do buffer data is when a new write to the stream happens, but the previous write is still in operation. Then the data to be written gets buffered until the prior write operation completes. So, data is never left sitting in the buffer with an idle stream. This tends to minimize (though not eliminate) data loss on an immediate shut-down.

If this is a normal, regular shut-down, then you should be able to use option #3 above and reshape how the shut-down occurs so you can use asynchronous code where you want so you can properly shutdown the streams.