3

I'm using the O_DIRECT flag to write to the disk directly from the user buffer. But as far as I understand, Linux doesn't guarantee that after this call, the data is written. It just writes directly from the user buffer to the physical device using DMA or anything else... Therefore, I don't understand if I can write to the user buffer after the call to 'write' function.

I'm sure that example code will help to understand my question:

char *user_buff = malloc(...); /* assume it is aligned as needed */
fd = open(..., O_DIRECT);
write(fd, ...)
memset(user_buff, 0, ...)

Is the last line (memset) legal? Is writing to the user buffer valid that is maybe used by DMA to transfer data to the device?

tshepang
  • 12,111
  • 21
  • 91
  • 136
user1868481
  • 31
  • 2

2 Answers2

2

It is legal. There's no "loopback" taking place here - consider what you should do if you used a dynamically mallocated array. Could you free it after write()? Couldn't you? - Well, the answer is that the write() function (and syscall) doesn't modify or access the user buffer after it has returned - if immediate write couldn't be performed, a copy of the data will be made. You don't have to worry about implementation details (that's the purpose of having a C standard library, after all...)

  • 1
    `write()` isn't part of the C standard library, it's a lower-level Unix system call. – Barmar Dec 01 '12 at 11:24
  • I'm confused, one answer says it's illegal other answer says it's not. @H2CO3 - if DMA is performed from the user buffer, will 'write' wait until the DMA completes before it is returned?? – user1868481 Dec 01 '12 at 11:34
  • @user1868481 I don't think there's any contradiction - Barmar's answer also says that it's guaranteed that when 'write', then your data has been written. –  Dec 01 '12 at 12:31
  • but Barmar suggested to add the O_SYNC flag in addition to the O_DIRECT. Is it necessary? I need this write to be as fast as possible. After this 'write', I fill the same user buffer with new data and then use 'write' again. I just want to understand what happens when I only use O_DIRECT and 'write' and DMA is activated. How do I know if it's safe to use the user buffer or is it being used by the DMA after 'write' returns... – user1868481 Dec 01 '12 at 13:24
  • @userXXX you know it's safe since if it wasn't, you would have had to observe implementation details, which you don't have to do. –  Dec 01 '12 at 13:30
  • OK, but I don't understand how is it possible with DMA. On one hand, the man page of open says: "File I/O is done directly to/from user space buffers". On the other hand, it says "but does not give the guarantees of the O_SYNC that data and necessary metadata are transferred". So - If I use 'write' and DMA is activated directly from user buffer, but 'write' returns before DMA completes, there is a problem... – user1868481 Dec 01 '12 at 13:41
  • @user1868481: I'm not sure if `O_DIRECT` implies `DMA` as provided by HDD controllers. I'd say it just bypasses the use of library internal buffers. There still might occur caching on hardware level or when using distributed file systems. To make sure the latter also have been flushed `O_SYNC` shall be used, so the data is **guaranteed to have been saved physically**. – alk Dec 02 '12 at 11:48
  • @user1868481 You know it's safe because it's guaranteed to be safe. Regardless of any flags you pass to open(), write() does not leave the kernel with a pointer back to your buffer in user space which it attempts to asynchronously access later on. When write() returns, the kernel will not, under any circumstances, try to access your user space buffer. You can do whatever you want with it. – nos Feb 13 '14 at 20:17
1

Use the O_SYNC flag in conjunction with O_DIRECT. Then you can be sure that the data has been written when write() returns.

Barmar
  • 741,623
  • 53
  • 500
  • 612