1

In my application I am continually writing data to file1 and flushing it to the device. In another thread, I am reading data from file1 and writing it to file2.

Every time I do the fwrite + fflush on file1, I signal to the other thread to start reading from it. The other thread reads data from file1 and dumps it into file2. Pretty simple logic. Additionally, after every few minutes, I seek back to start of file1 and start overwriting old data.

Now my problem is that once I start overwriting data in file1, the data read into file2 is sometimes the old data (i.e. data written in the previous iteration) even though writer thread has signaled that it wrote the new data (and flushed it).

I am writing to and reading from a solid state drive (128 GB SAMSUNG 470 Series, if that helps) on [C + linux + arm platfrom]. I feel that there is an issue with the processor cache. Perhaps the write goes into the cache and the read by the reader thread comes from the flash, and hence the stale data.

The catch here is that this problem occurs if the SSD is formatted with NTFS. If I format it with ext3, the problem goes away. Unfortunately, NTFS is a hard requirement. Another interesting observation is that if I have two reader threads, both get stale data at different instants.

Event after disabling the SSD write cache (with hdparm -W0 /dev/sda1), I get the same problem with NTFS. I am badly stuck up with this since more than a week.

Any idea what is happening, and why is it happening that way?

Any help will be worth its weight in gold...

EDIT Turns out that the NTFS driver does not like me overwriting a file by rewinding the file pointer. Is this a known thing?

tshepang
  • 12,111
  • 21
  • 91
  • 136
puffadder
  • 1,814
  • 3
  • 20
  • 32
  • did you try to reopen file in thread one with overwrite mode? Also, How do you read data from file1 in thread 2, did you reopen the file? – Asad Rasheed Jul 16 '11 at 16:08
  • Are you using ntfs-3g or the in-kernel ntfs driver? – bdonlan Jul 16 '11 at 18:46
  • @Asad: No, I do not reopen the file, I just do rewind(fp). I'll try your suggestion. @Bdonlan: I use tuxera embedded ntfs driver (commercial) – puffadder Jul 17 '11 at 08:45
  • I'm not familiar with the Tuxera driver, but this sounds like a bug the particular implementation of NTFS that you're using. The SSD issue is something of a hint. We'd be looking very closely at this if this was in the Microsoft NTFS driver. – jrtipton Oct 03 '11 at 22:34

1 Answers1

0

Ok, so I found the issue myself (and how rarely does that happen !!!).

I found that there was a problem with the C library buffering (fread/fwrite). So I do fflush() before every fread(). This solves my problem (I don't know what exactly went wrong with the driver but I am assuming that there is some issue with the "read" buffering of the C library I/O functions, when reading from the same location of the file second time around).

Thanks @Asad Rasheed and @jrtipton for your inputs :)

puffadder
  • 1,814
  • 3
  • 20
  • 32