2

I recently lost a drive in my RAID array (and got an email from the system warning me of this, which was awfully nice) and after some drive shuffling and swapping in a new drive I'm all safe and secure. But along the way, I found this thread, which got me to thinking about how you could actually test for disk errors and other Bad Things without them actually occurring. When I ran the suggested tar command:

tar c /my/raid/device/mount/point > /dev/null

it completed in a few seconds, which is clearly not long enough for the system to have actually read all the files (well over a TiB) - so I guess my first question is why this might not have worked. If I do something like this:

find . -type f | xargs md5sum

That command runs just fine, and it takes a long time to complete... but it also loads up the CPU doing all the summing. This might or might not be faster or easier than "tar" - I'm more curious as to why the tar command didn't work as I'd expected.

Anyway - second question, and more generally: is there a way to do something along these lines to do fault injection testing:

  1. find (or create) a file that I don't care about...
  2. determine a block on the disk is used to store this particular file...
  3. fake the software/OS into thinking this block is "bad" (I assume by marking it somehow, this is where my knowledge runs out)
  4. run my test scripts and/or error checking routines
  5. confirm that the array both reports the error and does whatever other corrective action is necessary...
  6. mark that block/sector as "good" again so the system/OS uses it as normal.

This seems like something that would be doable, but I don't have enough detailed knowledge of the linux tools that would allow me to mark a block as bad at the device level without it actually BEING a bad block...

thoughts on this? Or, if there's a much more elegant way to solve this I'm happy to hear that as well...

Community
  • 1
ljwobker
  • 149
  • 6

3 Answers3

5

There is a lot of useful fault injection infrastructure in Linux. One of these might be helpful to your testing efforts.

Page 7 of this presentation show an example of faking a block device issue with dmsetup.

https://mbroz.fedorapeople.org/talks/DeviceMapperBasics/dm.pdf

md(4) itself has a mode called FAULTY that you can use to simulate read/write errors.

Faulty

The FAULTY md module is provided for testing purposes. A faulty array has exactly one component device and is normally assembled without a superblock, so the md array created provides direct access to all of the data in the component device.

The FAULTY module may be requested to simulate faults to allow testing of other md levels or of filesystems. Faults can be chosen to trigger on read requests or write requests, and can be transient (a subsequent read/write at the address will probably succeed) or persistent (subsequent read/write of the same address will fail). Further, read faults can be "fixable" meaning that they persist until a write request at the same address.

Fault types can be requested with a period. In this case, the fault will recur repeatedly after the given number of requests of the relevant type. For example if persistent read faults have a period of 100, then every 100th read request would generate a fault, and the faulty sector would be recorded so that subsequent reads on that sector would also fail.

There is a limit to the number of faulty sectors that are remembered. Faults generated after this limit is exhausted are treated as transient.

The list of faulty sectors can be flushed, and the active list of failure modes can be cleared.

The options that control it are listed in mdadm(8) under -p, --layout=.

When setting the failure mode for level faulty, the options are: write-transient, wt, read-transient, rt, write-persistent, wp, read-persistent, rp, write-all, read-fixable, rf, clear, flush, none.

Each failure mode can be followed by a number, which is used as a period between fault generation. Without a number, the fault is generated once on the first relevant request. With a number, the fault will be generated after that many requests, and will continue to be generated every time the period elapses.

Multiple failure modes can be current simultaneously by using the --grow option to set subsequent failure modes.

"clear" or "none" will remove any pending or periodic failure modes, and "flush" will clear any persistent faults.

There are examples from the linux-raid mailing list archives under error injection that should help get you started with the md fault injection options too. =)

Community
  • 1
Kassandry
  • 689
  • 1
  • 8
  • 15
  • Thank you so much! Md-faulty docs could be so much better...I went through a lot of pain before figuring out that i had to use `--build` instead of `--assemble`, thanks to that mailing list example. – Mario Vitale Jul 08 '18 at 17:05
1

To your first question:

tar c /my/raid/device/mount/point > /dev/null

Is going to depend on how your distro's tar behaves in the absence of the "f" parameter. I tried this on a Debian (wheezy) system and it behaved as you might have expected - the archive was written to stdout. However on a FreeBSD system. it returns an error:

tar: Failed to open '/dev/sa0'

A more universal approach would be to explicitly specify stdout as the archive:

tar cf - /my/raid/device/mount/point > /dev/null

Edit: Doh! Or just forget about redirecting:

tar cf /dev/null /my/raid/device/mount/point

In addition to Kassandry's excellent answer, I would also recommend use of SMART if your physical drives support it for predictive failure.

Brandon Xavier
  • 2,022
  • 13
  • 15
0

Tar has an "optimization" where it does (almost) nothing if the output is /dev/null

You can try this to trick it to do the work anyway:

tar c /my/raid/device/mount/point | cat > /dev/null

David Balažic
  • 458
  • 1
  • 7
  • 19