2

I've encountered a race condition with LVM and a disk driver I'm working on. It looks like things like vgcreate and lvcreate do their IO in O_DIRECT mode. I discovered this when running those commands with -vvv.

Clearing start of logical volume "test"
    /dev/Finance-PG-vg/test: Added to device cache
    Opened /dev/Finance-PG-vg/test RW O_DIRECT
    Wiping /dev/Finance-PG-vg/test at sector 0 length 8 sectors
    /dev/Finance-PG-vg/test: block size is 4096 bytes
    Closed /dev/Finance-PG-vg/test

Specifically, I suspect that our reads are hitting the cache, and not getting the latest disk contents.

If something is written with O_DIRECT, my understanding is that this bypasses the cache. Therefore any reads to that sector are going to recieve the old data from cache, at least until the cache is invalidated. So if I want to read whatever O_DIRECT just wrote within a few seconds, I should be dropping the cache first?

Correct?

Ahhmyface
  • 61
  • 12
  • I've tried dropping the cache after every lvm command, no luck. I still get periodically get zeros when I read. – Ahhmyface Sep 05 '15 at 23:50

1 Answers1

1

There are several confusions here:

  • The tools you mention mostly likely use O_DIRECT to make sure the new LVM configuration is persistent. The LVM metadata is actually stored in a specific location on all the physical disks/partitions you provide.

  • Writing to LVM devices does not use by default O_DIRECT (although you can pass this flag when you open a file).

  • Bypassing the cache with O_DIRECT does not mean that you get stale data. Lets assume that you open a file, write to it, close it, then open it again with O_DIRECT, and read the file. The read is guaranteed to return the latest changes to the file. There is no stale data ever returned. There is no need to drop caches when using O_DIRECT.

Radu
  • 1,098
  • 1
  • 11
  • 22