0

So, I clone lots of Windows-based NTFS drives from images each day and I'm halfway through a project that could potentially help me and others to speed up this process dramatically.

I currently have a set up of Ubuntu 17.10 with mdadm and 16 SATA ports. The idea is to connect drives to each of the SATA ports and do a simultaneous clone to each at once.

If I use dd on one drive at a time, I can get a result of around ~135MB/s (max ~180MB/s) but as soon as I start cloning other drives at the same time, the speed reduces considerably.

Same thing happens with dcfldd which has the added bonus of being able to specify multiple of= outputs.

Now... initially I had the idea to utilise a RAID1 array containing all of the drives I wish to clone at once. However, there seems to be some inconsistencies with the architecture of RAID and the usability of the drives after cloning.

Here's what I've tried so far: I can create an array of multiple drives using mdadm and then proceed to run sudo dd if=/home/clone/some_image.img of=/dev/md0 bs=4096 This process seems to run absolutely fine and with brilliant speed results (~110Mb/s for every drive at once). After the process has finished, the drives do not appear to contain the data we just wrote to them. My next step would be to zero the superblock on the drives in the hopes of reverting them back to usable drives, this does not seem to work either.

I have even tried creating a new array using an older superblock version (from 1.2 to 0.9) but this yields the same results as before.

It is as if the drives will only ever be accessible via mdadm regardless of the data stored on them, which makes this idea fail completely, even though the initial results were so promising.

So I pose these questions to you: Is there any way of writing an image to a RAID1 array and then turning back into single disks? Are there any other technologies that will allow me to have the speed benefits of cloning to RAID1 but without using RAID or the necessity of third-party tools in order to mount the drives?

Thank you so very much for reading this.

Matt Dunne
  • 1
  • 2
    I would think that rather than cloning drives most people that need to install many identically configured systems use PXE boot for a scripted network installation, which saves the hassle of removing and replacing physical drives and deals better with variations in hardware configuration. – HBruijn Feb 20 '18 at 10:45
  • @HBruijn I appreciate the alternative suggestion. The other department is already using Ghost PXE over a 100/base network, but some members have stated that they would prefer managing the HDDs directly as it saves having to boot up 16 machines to a KVM sharing one VDU to broadcast an image over a slow network to ultimately have one or more fail and have to painstakingly switch through each machine just to find the one that has failed and restart the process. I know dd to be very stable and in this case, was hoping that with the 6gbps SATA interfacing instead of 100MB/s would be quicker >.<; – Matt Dunne Feb 20 '18 at 12:31
  • You can use clone/snapshot procedure on zfs filesystem or LVM. It's more faster than `dd` command. By example look for `FreeNAS`. – Mikhail Khirgiy Feb 20 '18 at 16:03
  • Manually clone D1 to D2, then Clone D1 to D3 AND simultaneously D2 to D4, then D1 to D5 AND D2 to D6 AND D3 to D7, etc... – Damon Feb 20 '18 at 16:20

2 Answers2

1

No, this can't work. If you have a Linux software RAID (or a hardware RAID really), you will have an outer management structure for the RAID that is required and that makes your disk unusable for your use case.

Sven
  • 98,649
  • 14
  • 180
  • 226
0

When you're cloning a single image to multiple disk you need to make sure your start all dd instances simultaneously. This way, all reads (but the first) come from cache and the source disk won't thrash.

If you're cloning different images your source disk needs to get faster and have a low access latency. A simple RAID 1 or 0 probably won't cut it. An SSD will most likely do this or a larger RAID with many disks.

Zac67
  • 10,320
  • 2
  • 12
  • 32