Hardware RAID1 configuration - Possibly failed sectors on one physical disk. Does the OS automatically read from the other disk?

Question

I'm not a professional sysadmin but since I couldn't find answers to my specific case after researching for a while I was hoping I could get some help here. Our server uses P222 - a HP Smart controller array in RAID1 configuration. I believe some sectors on one of the physical hard drives have failed. I used the hpacucli tool and the output looked like:-

 $ hpacucli ctrl all show config 

 Smart Array P222 in Slot 1                (sn: PDSXH0ARH5I0SW)

 array A (SATA, Unused Space: 0  MB)


  logicaldrive 1 (2.7 TB, RAID 1, Ready for Rebuild)

  physicaldrive 2I:1:1 (port 2I:box 1:bay 1, SATA, 3 TB, OK)
  physicaldrive 2I:1:2 (port 2I:box 1:bay 2, SATA, 3 TB, Predictive Failure)

I ran the same tool again a few times to check the status and at one point I noticed that "Predictive Failure" was replaced by "Rebuilding 1%" , increased later to 2%. I don't think I did anything to initiate the rebuilding. Anyway, I let it run and checked the status after a while at which point it was back to 'Predictive Failure'.

On running smartctl long and short tests - the selftest logs reported :-

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%     14368         334201968
# 2  Short offline       Completed: read failure       90%     14367         625082211

We are running a MySQL instance on this server and it keeps failing to start complaining of a read error which indicated that it may be due to a failing hard disk/a bad sector hence the tools used above. I had a couple of questions :-

1. I don't know for sure but it looks like one of the hard disks is failing partly. In that case, shouldn't the OS (Ubuntu 12.04) just read the data from the mirrored hard disk ? (which would mean MySQL should continue running)
2. I was following the steps in http://sg.danny.cz/scsi/badblockhowto.html . The LBA 334201968 (LBA of the long test read failure) corresponds to the data file of MySQL. But I did not want to overwrite any part of this file as I'm not sure if MySQL will permanently see the file as corrupted. What would be my best option to 'fix' the corrupted parts of the disk ?

Happy to report any additional details that may be required for diagnosing/fixing this

EDIT 1: As requested the MySQL error logs like so:-

150824 10:27:00 InnoDB: Completed initialization of buffer pool
150824 10:27:00 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
150824 10:27:00  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
150824 10:27:00  InnoDB: Waiting for the background threads to start
150824 10:27:01 InnoDB: 5.5.35 started; log sequence number 2723867081864
150824 10:27:01 [Note] Server hostname (bind-address): <ip and port here>;
150824 10:27:01 [Note]   - <ip here> resolves to <ip here>;
150824 10:27:01 [Note] Server socket created on IP: <ip here>.
InnoDB: Error: tried to read 16384 bytes at offset 70 1898921984.
InnoDB: Was only able to read -1.
150824 10:27:01  InnoDB: Operating system error number 5 in a file operation.
InnoDB: Error number 5 means 'Input/output error'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/operating-system-error-codes.html
InnoDB: File operation call: 'read'.
InnoDB: Cannot continue operation.

EDIT 2: based on comment https://serverfault.com/a/716471/306555, I opened a ticket for replacing the disk and got it replaced and RAID rebuilt. The output from hpacucli looks like:-

  physicaldrive 2I:1:1 (port 2I:box 1:bay 1, SATA, 3 TB, OK)
  physicaldrive 2I:1:2 (port 2I:box 1:bay 2, SATA, 3 TB, OK)

So the Predictive Failure is gone. However, MySQL kept giving me read errors so I ran smartctl long and short test again. While the short test passed, the long one failed with a read error:-

Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%     14393         625116232
# 2  Short offline       Completed without error       00%     14392         -

I also checked syslog and noticed that every time MySQL tries to start, there's this error in the syslog

Aug 25 14:23:41 kernel: [ 1603.911185] sd 6:0:0:1: [sda] Unhandled sense code
Aug 25 14:23:41 kernel: [ 1603.911186] sd 6:0:0:1: [sda]  Result: hostbyte=invalid driverbyte=DRIVER_SENSE
Aug 25 14:23:41 kernel: [ 1603.911188] sd 6:0:0:1: [sda]  Sense Key : Medium Error [current] 
Aug 25 14:23:41 kernel: [ 1603.911190] sd 6:0:0:1: [sda]  Add. Sense: Unrecovered read error
Aug 25 14:23:41 kernel: [ 1603.911192] sd 6:0:0:1: [sda] CDB: Read(10): 28 00 46 a2 d5 a0 00 00 08 00

What would that indicate ? (looks like its a bad sector on the disk?) If that's the case, is there a way to fix that ?

Answer 1

Okay. This is a long question, but will get a short answer:

If you see "Predictive Failure" or "Failed", replace the disk.

Both of those conditions are valid for the purpose of establishing a support ticket and/or warranty parts replacement from HP.

"Predictive Failure" incorporates S.M.A.R.T. data as well as other heuristics to determine drive health. But the specifics shouldn't really matter. Plan to replace the drive.

The fact that you're seeing an impact at the application level is another sign that the correct course of action is to replace the disk. It's easy to do... even though it's a SATA drive, it's an HP part, so it has a warranty (likely 1 year, but it's tied to your server's serial number).

Call HP...

Answer 2

1. Yes, if one drive in RAID is failed, RAID controller mark it as failed and will be read with other healthy hdd.
2. Predictive failure means, that disk still work, but controlled notice you, that soon it will fails. If you receive read error, in your test, than you should replace your drive by another. Just buy spare drive in your local store/vendor support, install it and RAID controller will rebuild array to healthy status.

Answer 3

Are you using HP hard drives? Or normal consumer drives? Do the drives have Time Limited Error Recovery?

If not, the drive could lockup the controller while it tries to read a bad sector. It will take a long time for the drive to give up, so the read fails. The raid controller doesn't get a chance to try the other drive because it is stuck waiting for the first drive to decide it has failed.

This behaviour also causes drives to fall out of the raid temporarily which would explain the rebuilding.

This should only apply if you are using non-HP drives - the supported drives are programmed to quickly give up and let the raid controller handle the error.