Rolling file implementation

Question

I am always curious how a rolling file is implemented in logs.

How would one even start creating a file writing class in any language in order to ensure that the file size is not exceeded.

The only possible solution I can think of is this:

write method:
    size = file size + size of string to write
    if(size > limit)
        close the file writer
        open file reader
        read the file
        close file reader
        open file writer (clears the whole file)
        remove the size from the beginning to accommodate for new string to write
        write the new truncated string
    write the string we received

This seems like a terrible implementation, but I can not think up of anything better.

Specifically I would love to see a solution in java.

EDIT: By remove size from the beginning is, let's say I have 20 byte string (which is the limit), I want to write another 3 byte string, therefore I remove 3 bytes from the beginning, and am left with end 17 bytes, and by appending the new string I have 20 bytes.

In logging frameworks it's a little more complicated because they have different policies for doing the rollover. Download the source code, for example, for `logback` and look at the classes `RollingFileAppender` and `FixedWindowRollingPolicy`, particularly the method `rollover()`. — Sotirios Delimanolis, May 24 '13 at 15:58
Thanks, this is interesting, but what if I want to create something that simply extends PrintWriter for example, and I want it to be as simple and elegant as possible? — Quillion, May 24 '13 at 16:09
What you have posted has the gist of it, but what do you mean by _remove the size from the beginning_? — Sotirios Delimanolis, May 24 '13 at 16:19

Sotirios Delimanolis · Accepted Answer · 2013-05-24T16:18:37.690

Because your question made me look into it, here's an example from the logback logging framework. The RollingfileAppender#rollover() method looks like this:

public void rollover() {
    synchronized (lock) {
        // Note: This method needs to be synchronized because it needs exclusive
        // access while it closes and then re-opens the target file.
        //
        // make sure to close the hereto active log file! Renaming under windows
        // does not work for open files
        this.closeOutputStream();

        try {
            rollingPolicy.rollover(); // this actually does the renaming of files
        } catch (RolloverFailure rf) {
            addWarn("RolloverFailure occurred. Deferring roll-over.");
            // we failed to roll-over, let us not truncate and risk data loss
            this.append = true;
        }

        try {
            // update the currentlyActiveFile           
            currentlyActiveFile = new File(rollingPolicy.getActiveFileName());

            // This will also close the file. This is OK since multiple
            // close operations are safe.
            // COMMENT MINE this also sets the new OutputStream for the new file
            this.openFile(rollingPolicy.getActiveFileName()); 
        } catch (IOException e) {
            addError("setFile(" + fileName + ", false) call failed.", e);
        }
    }
}

As you can see, the logic is pretty similar to what you posted. They close the current OutputStream, perform the rollover, then open a new one (openFile()). Obviously, this is all done in a synchronized block since many threads are using the logger, but only one rollover should occur at a time.

A RollingPolicy is a policy on how to perform a rollover and a TriggeringPolicy is when to perform a rollover. With logback, you usually base these policies on file size or time.

That seems terribly inefficient though no? Do all the logging things use it? Opening and closing a stream is very costly, and once you do hit the ceiling, doing it so often sounds terrible. — Quillion, May 24 '13 at 16:24
Closing one stream? No, not really. Obviously, you wouldn't set the trigger policy to make it open a new file every second (though you could). That's how logging frameworks do it. If you're going to write to a file, you need to open its stream. So, doing a rolling implementation, will require that, no way around it. — Sotirios Delimanolis, May 24 '13 at 17:03
I don´t think it´s that inefficient as usually it´s an sporadic operation. Specifically talking about logs, if it happens too often it could mean that either youre sending too much info to the logs or the log size is way too small, i think. — jambriz, May 24 '13 at 17:08

Rolling file implementation

1 Answers1

Linked