Can multiple processes append to a file using fopen without any concurrency problems?

Question

I have a process opening a file in append mode. In this case it is a log file. Sample code:

int main(int argc, char **argv) {
    FILE *f;
    f = fopen("log.txt", "a");
    fprintf(f, "log entry line");
    fclose(f);
}

Two questions:

If I have multiple processes appending to the same file, will each log line appear distinctly or can they be interlaced as the processes context switch?
Will this write block if lots of processes require access to the file, therefore causing concurrency problems?

I am considering either doing this in its simplest incarnation or using zeromq to pump log entries over pipes to a log collector.

I did consider syslog but I don't really want any platform dependencies on the software.

The default platform is Linux for this btw.

score 11 · Answer 1 · answered Sep 26 '11 at 08:33

11

I don't know about fopen and fprintf but you could open the file using O_APPEND. Then each write will go at the end of the file without a hitch (without getting mixed with another write).

Actually looking in the standard:

The file descriptor associated with the opened stream shall be allocated and opened as if by a call to open() with the following flags:
a or ab          O_WRONLY|O_CREAT|O_APPEND

So I guess it's safe to fprintf from multiple processes as long as the file has been opened with a.

answered Sep 26 '11 at 08:33

cnicutar

178,505
25
365
392

2

You still have to switch the buffering to no or line. With full buffering it may run out of buffer in the middle of the entry and call write with that. – Jan Hudec Sep 26 '11 at 09:11
@Jan Hudec That's right. Yet another reason to use `write(2)` directly. – cnicutar Sep 26 '11 at 09:14

score 8 · Answer 2 · edited Sep 26 '11 at 08:46

8

The standard (for open/write, not fopen/fwrite) states that

If the O_APPEND flag of the file status flags is set, the file offset shall be set to the end of the file prior to each write and no intervening file modification operation shall occur between changing the file offset and the write operation.

For fprintf() to be used, you have to disable buffering on the file.

edited Sep 26 '11 at 08:46

Jan Hudec

73,652
13
125
172

answered Sep 26 '11 at 08:40

glglgl

89,107
13
149
217

1

**Important**: That's for `open`/`write`, *not* fopen/fwrite. – Jan Hudec Sep 26 '11 at 08:44
Right, but `write()` "lies under" `fwrite()` and `fprintf()`, i.e. is used by them. If buffering is disabled, it is even used 1:1. (But better be explicit about that, so you're right.) – glglgl Sep 26 '11 at 08:45
There's no need to disable buffering for this to work. Actually the line buffering helps to ensure that whole lines are written at a time – Per Johansson Sep 26 '11 at 08:50
1

@glglgl: Yes, `write` "lies under" `fwrite`, but nothing specifies that `fwrite` passes the data to `write` in one piece. In practice it does if buffering is off altogether or line buffering is on and the line does not exceed the buffer size. – Jan Hudec Sep 26 '11 at 09:10
even though write ensures atomicity, the order of write of two processes depends on the OS. Thus, even if process1 invokes write() before write() of process2, the order of the data written depends on OS. Therefore, to guarantee the order, using mutex is best bet. – Bill Dec 22 '12 at 19:20

score 6 · Accepted Answer · answered Sep 26 '11 at 08:36

You'll certainly have platform dependencies since Windows can't handle multiple processes appending to the same file.

Regarding synchronization problems, I think that line-buffered output /should/ save you most of the time, i.e. more than 99.99% of short log lines should be intact according to my short shell-based test, but not every time. Explicit semantics are definitely preferable, and since you won't be able to write this hack system-independently anyway, I'd recommend a syslog approach.

Thanks - that's just what I wanted to know. The Windows multiple appending is what I was slightly worried about. — Deleted, Sep 26 '11 at 09:07

score 3 · Answer 4 · answered Sep 26 '11 at 08:32

3

When your processes will be going to write something like:

"Here's process #1"
"Here's process #2"

you will probably get something like:

"Hehere's process #2re's process #1"

You will need to synchronize them.

answered Sep 26 '11 at 08:32

Andrey Atapin

7,745
3
28
34

Pete Wilson · Answer 5 · 2011-09-26T11:09:10.047

EDIT to answer your questions explicitly:

If I have multiple processes appending to the same file, will each log line appear distinctly or can they be interlaced as the processes context switch?

Yes, each log line will appear intact because according to msdn/vs2010:

"This function [that is, fwrite( )] locks the calling thread and is therefore thread-safe. For a non-locking version, see _fwrite_nolock."

The same is implied on the GNU manpage:

"— Function: size_t fwrite (const void *data, size_t size, size_t count, FILE *stream)

This function writes up to count objects of size size from the array data, to the stream stream. The return value is normally count, if the call succeeds. Any other value indicates some sort of error, such as running out of space.

— Function: size_t fwrite_unlocked (const void *data, size_t size, size_t count, FILE *stream)

The fwrite_unlocked function is equivalent to the fwrite function except that it does not implicitly lock the stream.

This function [i.e., fwrite_unlocked( )] is a GNU extension. "

Will this write block if lots of processes require access to the file, therefore causing concurrency problems?

Yes, by implication from question 1.

Thanks for the reply. The potential issue I have is concurrency so locking will cause problems with that. I've upvoted as the use case is still valid and descriptive. Thanks again! — Deleted, Sep 26 '11 at 09:57
Oh, I finally get it. By "concurrency problem" you are afraid that process2 will block while process1 is trying to write into the log file. I edited my answer to address both questions explicitly. — Pete Wilson, Sep 26 '11 at 10:30
That's the one :) - thanks again for your answer anyway. I would use this on low concurrency requirements. — Deleted, Sep 26 '11 at 10:34

score 1 · Answer 6 · answered Sep 26 '11 at 08:32

Unless you do some sort of synchronization the log lines may overlap. So to answer number two, that depends on how you implement the locking and logging code. If you just lock, write to file and unlock, that may cause problems if you have lots of processes trying to access the file at the same time.

Can multiple processes append to a file using fopen without any concurrency problems?

6 Answers6

Linked