3
  • I have a server with six 4TB drives running Ubuntu 14.04 LTS.
  • One of the drives is used for the system/OS
  • I wanted to use the other 5 drives to get as close to 20TB of storage as possible
  • It's a short term data analysis project, so RAID 0 is fine
  • I used fdisk to create a single ~4TB partition on each of the 5 drives

After running mdadm to create a RAID 0 array, the result was just a 10TB device on /dev/md0. Half the size that I was expecting!

Here is the command that I used to create the array:

mdadm --create --verbose /dev/md0 --level=stripe --raid-devices=5 /dev/sd[bcdef]1

Below is an fdisk -l output. Any ideas on why I can't get more space out of the drives?

Disk /dev/sda: 4000.8 GB, 4000787030016 bytes
255 heads, 63 sectors/track, 486401 cylinders, total 7814037168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1  4294967295  2147483647+  ee  GPT

Disk /dev/sdb: 4000.8 GB, 4000787030016 bytes
90 heads, 3 sectors/track, 28940878 cylinders, total 7814037168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xda4f04e2

Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048  4294967294  2147482623+  83  Linux

Disk /dev/sdc: 4000.8 GB, 4000787030016 bytes
25 heads, 3 sectors/track, 104187162 cylinders, total 7814037168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xc9e57616

Device Boot      Start         End      Blocks   Id  System
/dev/sdc1            2048  4294967294  2147482623+  83  Linux

Disk /dev/sdd: 4000.8 GB, 4000787030016 bytes
25 heads, 3 sectors/track, 104187162 cylinders, total 7814037168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x53bed1ab

Device Boot      Start         End      Blocks   Id  System
/dev/sdd1            2048  4294967294  2147482623+  83  Linux

Disk /dev/sde: 4000.8 GB, 4000787030016 bytes
25 heads, 3 sectors/track, 104187162 cylinders, total 7814037168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xbc5c1069

Device Boot      Start         End      Blocks   Id  System
/dev/sde1            2048  4294967294  2147482623+  83  Linux

Disk /dev/sdf: 4000.8 GB, 4000787030016 bytes
25 heads, 3 sectors/track, 104187162 cylinders, total 7814037168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x4801480c

Device Boot      Start         End      Blocks   Id  System
/dev/sdf1            2048  4294967294  2147482623+  83  Linux

Disk /dev/md0: 10995.1 GB, 10995108413440 bytes
2 heads, 4 sectors/track, -1610614656 cylinders, total 21474821120 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 524288 bytes / 2621440 bytes
Disk identifier: 0x00000000

Disk /dev/md0 doesn't contain a valid partition table
Rob
  • 33
  • 4
  • 2
    Am I right in thinking that each of the `/dev/sdX1` partitions is only about half of each actual disc (about 4.3Gsectors, out of 7.8Gsectors)? That would certainly explain why you're missing so much space. – MadHatter Jul 28 '14 at 19:55
  • @MadHatter The partitions were created to use each drive's full resources. The default first and last sectors were used in fdisk and these were new unpartitioned drives. – Rob Jul 28 '14 at 19:59
  • I'm not knocking that. But the evidence is strong that the partitions don't go anywhere near the top of each drive; in fact, they interestingly seem to top out at 2^41 bytes, which is about 55% of your total drive. The evidence is right in front of you: compare the end sector of each partition with the total number of sectors on each drive. – MadHatter Jul 28 '14 at 20:04
  • 2
    `fdisk` should never be used on drives larger than 2TB. These require GPT parititoning to use their full space. Use `gdisk` or `parted` instead. – Michael Hampton Jul 28 '14 at 20:11

2 Answers2

7

fdisk will not create a partition bigger than 2TB (2^41 bytes). That seems to be pretty well documented; one example of many can be read here. 5*2TB=10TB, which is what you're getting.

You will need to use GNUparted to put a GPT partition table on each drive; then you'll be able to make a /dev/sdX1 partition that stretches all the way to the top of the drive.

Or, as jski suggests, you can use the raw devices instead of the partitions. I'd go for the partition approach, though, not least because you can always trim the partition to slightly below top-of-disc in case you ever need to replace a disc with a slightly smaller one (yes, I know that's a pointless consideration in RAID-0, but I feel it's a good habit to get into).

MadHatter
  • 79,770
  • 20
  • 184
  • 232
4

Drop the 1 from the end of your command:

mdadm --create --verbose /dev/md0 --level=stripe --raid-devices=5 /dev/sd[bcdef]

Should work fine. I'm assuming you want to use all the space available on all 5 of the RAID disks.

jski
  • 911
  • 1
  • 7
  • 20
  • How bazaar, that actually worked - despite the official docs saying to use partitions rather than the drives directly. – Rob Jul 28 '14 at 20:11
  • 2
    "work fine". Yeah, until you accidentally partition the drive with a tool which doesn't recognize MD-on-raw-device. Apart from that, MadHatters answer actually describes what is going on instead of just saying "do this instead because it works [for whatever reason]". – Simon Lindgren Jul 29 '14 at 09:35