I made a raid10 with 8x8TB drives the file system on it is XFS. I never had any issue with it. The raid is configured as near2 and internal bitmap. The drives are connected to a LSI-9211-8i in HBA mode.
Last week, I finally bought another set of 8x8TB drive, because the raid was full. Here is how I did to add that to my existing RAID :
- Tested the drive using smartctl (no problem).
- Created a new GPT partition tab and a XFS partition on each drive.
- Used the command
sudo mdadm /dev/md0 --add /dev/...to add the drives. - Grew the array by doing
sudo mdadm --grow /dev/md0 --size=max. It took first 2 days to reshape and then a day to resync. It went succesfully without any error or interruption. After that the raid was clean and active. - Resized the file system with
sudo xfs_grow /dev/md0. It completed without any issue. - Mounted the raid array and tested it. It was working fine. I resstarted the server to be sure, the array was still clean.
So then what happened?! I used the array to download new stuff using Radarr, Sonarr, Deluge (all in docker). I had to restart the server (I do that once a week to free ressources) and then something happened.
The raid array was now inactive (check below for mdadm detail). The event count was wrong on 4 out of 16 drives. After multiples attempt I manage to bring it back by doing sudo mdadm --assemble /dev/md0 /dev/sd[...]. I tested the XFS filesystem with sudo xfs_repair -n /dev/md0, pfiou... Everything seemed OK.
By curiosity, I tried to restart again today, I made sure the docker container was closed. I checked the array is was clean, so I wait 5 minutes and I did sudo reboot. Same thing happened again!! This time 6 drives where out of sync for the event count... So again I did the same as above and it seems fine, but now I am very worried since I currently have nothing to back that up (I know it's bad).
Do you guys may have any idea of what is hapening? Thank you in advance for all of your help! :)
sudo mdadm --examine... before I re-assembled again :
/dev/sda1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260976 sectors, after=1024 sectors
State : clean
Device UUID : 17c3e75a:d0360875:69201f88:cababd94
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:04 2020
Bad Block Log : 512 entries available at offset 128 sectors
Checksum : b8d092ad - correct
Events : 63706
Layout : near=2
Chunk Size : 512K
Device Role : Active device 12
Array State : AAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260976 sectors, after=1024 sectors
State : clean
Device UUID : c0f6cf6d:3c5687e3:8a745314:9fcd0ea6
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:16 2020
Bad Block Log : 512 entries available at offset 128 sectors
Checksum : d79a4cca - correct
Events : 63709
Layout : near=2
Chunk Size : 512K
Device Role : Active device 11
Array State : .AA.A.AA.AAA.AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260976 sectors, after=1024 sectors
State : clean
Device UUID : 1c3c8784:6b6bcbef:4ce604b6:5734fed2
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:04 2020
Bad Block Log : 512 entries available at offset 128 sectors
Checksum : c9597fc2 - correct
Events : 63706
Layout : near=2
Chunk Size : 512K
Device Role : Active device 8
Array State : AAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260976 sectors, after=1024 sectors
State : clean
Device UUID : f650167a:562c413d:22d8ba5f:f6f4e486
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:16 2020
Bad Block Log : 512 entries available at offset 128 sectors
Checksum : 69d80807 - correct
Events : 63709
Layout : near=2
Chunk Size : 512K
Device Role : Active device 13
Array State : .AA.A.AA.AAA.AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260976 sectors, after=1024 sectors
State : clean
Device UUID : 8397c880:8807eb34:75425522:4f1c2f37
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:16 2020
Bad Block Log : 512 entries available at offset 128 sectors
Checksum : db18bb75 - correct
Events : 63709
Layout : near=2
Chunk Size : 512K
Device Role : Active device 9
Array State : .AA.A.AA.AAA.AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260976 sectors, after=1024 sectors
State : clean
Device UUID : cb43d48b:862f335a:90bddcd7:9060894b
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:16 2020
Bad Block Log : 512 entries available at offset 128 sectors
Checksum : d54e4f12 - correct
Events : 63709
Layout : near=2
Chunk Size : 512K
Device Role : Active device 15
Array State : .AA.A.AA.AAA.AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260976 sectors, after=1024 sectors
State : clean
Device UUID : 602c75b6:00d68a1f:c04c404f:514bffed
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:16 2020
Bad Block Log : 512 entries available at offset 128 sectors
Checksum : df205817 - correct
Events : 63709
Layout : near=2
Chunk Size : 512K
Device Role : Active device 10
Array State : .AA.A.AA.AAA.AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdh1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260976 sectors, after=1024 sectors
State : clean
Device UUID : b8db9412:ae32feaa:4d330c95:5cdcd294
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:16 2020
Bad Block Log : 512 entries available at offset 128 sectors
Checksum : b352dbb1 - correct
Events : 63709
Layout : near=2
Chunk Size : 512K
Device Role : Active device 14
Array State : .AA.A.AA.AAA.AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdi1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260992 sectors, after=1024 sectors
State : clean
Device UUID : 4af973aa:e4ae2676:67017e23:343e8114
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:04 2020
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 249da51a - correct
Events : 63706
Layout : near=2
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdj1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260992 sectors, after=1024 sectors
State : clean
Device UUID : 35255445:73245125:1e05a845:0b213eca
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:04 2020
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 468f2d21 - correct
Events : 63706
Layout : near=2
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdl1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260992 sectors, after=1024 sectors
State : clean
Device UUID : f1af8db7:4e256b3b:bdc65518:b26bb193
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:16 2020
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 6ae0c513 - correct
Events : 63709
Layout : near=2
Chunk Size : 512K
Device Role : Active device 4
Array State : .AA.A.AA.AAA.AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdm1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260992 sectors, after=1024 sectors
State : clean
Device UUID : e2b66811:4f48ebc6:245f75c2:c64db0c6
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:16 2020
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 2d5a6983 - correct
Events : 63709
Layout : near=2
Chunk Size : 512K
Device Role : Active device 6
Array State : .AA.A.AA.AAA.AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdn1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260992 sectors, after=1024 sectors
State : clean
Device UUID : dd3f61c8:1ca69a95:d58a4ac2:c46365d0
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:04 2020
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : bcaf91e8 - correct
Events : 63706
Layout : near=2
Chunk Size : 512K
Device Role : Active device 5
Array State : AAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdo1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260992 sectors, after=1024 sectors
State : clean
Device UUID : 7b867d58:c5b0551a:753ebe4f:9502faa5
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:04 2020
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 348f359d - correct
Events : 63706
Layout : near=2
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdp1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260992 sectors, after=1024 sectors
State : clean
Device UUID : 72a44102:13e98e3a:844d9e0b:e58637cc
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:16 2020
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : e0871f55 - correct
Events : 63709
Layout : near=2
Chunk Size : 512K
Device Role : Active device 7
Array State : .AA.A.AA.AAA.AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdq1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260992 sectors, after=1024 sectors
State : clean
Device UUID : 6b0152f0:ca1be94f:1af1ee36:e735cf70
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:16 2020
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : b3da0199 - correct
Events : 63709
Layout : near=2
Chunk Size : 512K
Device Role : Active device 2
Array State : .AA.A.AA.AAA.AAA ('A' == active, '.' == missing, 'R' == replacing)
sudo mdadm -D /dev/md0 before I re-assembled again :
/dev/md0:
Version : 1.2
Raid Level : raid0
Total Devices : 16
Persistence : Superblock is persistent
State : inactive
Working Devices : 16
Name : Server:0
UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Events : 63709
Number Major Minor RaidDevice
- 65 1 - /dev/sdq1
- 8 1 - /dev/sda1
- 8 241 - /dev/sdp1
- 8 225 - /dev/sdo1
- 8 209 - /dev/sdn1
- 8 193 - /dev/sdm1
- 8 177 - /dev/sdl1
- 8 145 - /dev/sdj1
- 8 129 - /dev/sdi1
- 8 113 - /dev/sdh1
- 8 97 - /dev/sdg1
- 8 81 - /dev/sdf1
- 8 65 - /dev/sde1
- 8 49 - /dev/sdd1
- 8 33 - /dev/sdc1
- 8 17 - /dev/sdb1
cat /proc/mdstat before I re-assembled again :
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : inactive sdc1[15](S) sdi1[1](S) sdo1[3](S) sdm1[6](S) sdl1[4](S) sdq1[2](S) sde1[14](S) sdh1[9](S) sdd1[10](S) sdj1[0](S) sdb1[12](S) sdg1[13](S) sdp1[7](S) sdf1[8](S) sdn1[5](S) sda1[11](S)
125022314496 blocks super 1.2
unused devices: <none>
sudo mdadm -D /dev/md0 after I re-assembled again (yes I launched a mdadm check in case) :
/dev/md0:
Version : 1.2
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 7813894144 (7451.91 GiB 8001.43 GB)
Raid Devices : 16
Total Devices : 16
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Wed Mar 25 04:28:53 2020
State : clean, checking
Active Devices : 16
Working Devices : 16
Failed Devices : 0
Spare Devices : 0
Layout : near=2
Chunk Size : 512K
Consistency Policy : bitmap
Check Status : 0% complete
Name : Server:0
UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Events : 63711
Number Major Minor RaidDevice State
0 8 145 0 active sync set-A /dev/sdj1
1 8 129 1 active sync set-B /dev/sdi1
2 65 1 2 active sync set-A /dev/sdq1
3 8 225 3 active sync set-B /dev/sdo1
4 8 177 4 active sync set-A /dev/sdl1
5 8 209 5 active sync set-B /dev/sdn1
6 8 193 6 active sync set-A /dev/sdm1
7 8 241 7 active sync set-B /dev/sdp1
15 8 33 8 active sync set-A /dev/sdc1
14 8 65 9 active sync set-B /dev/sde1
13 8 97 10 active sync set-A /dev/sdg1
12 8 17 11 active sync set-B /dev/sdb1
11 8 1 12 active sync set-A /dev/sda1
10 8 49 13 active sync set-B /dev/sdd1
9 8 113 14 active sync set-A /dev/sdh1
