RAID10 array lost after reboot | Amazon Linux

Question

I carefully followed this procedure to setup RAID10 on Amazon Linux:

https://mongodb-documentation.readthedocs.org/en/latest/ecosystem/tutorial/install-mongodb-on-amazon-ec2.html

I also upgraded MongoDB to 3.2 and followed this procedure to deactivate THP: https://docs.mongodb.org/manual/tutorial/transparent-huge-pages/

Then I updated packages for security and rebooted the instance... When I got back in, the RAID array was gone!

I tried to set it up again (I lost my data and had to do it all over again), and after reboot, gone again...

I checked that the array is correctly specified in mdadm.conf, as pointed out in other posts. I couldn't find any working solution in other posts. Hopefully the details I have provided here will give you some new ideas...!

This is the procedure followed:

1) Create instance: Amazon Linux HVM SSD 64 bits.
At the instance launch setup you can choose the volumes. There is one Root volume by default (/dev/xvda). Add 4 extra EBS volumes as /dev/sdf, /dev/sdg, /dev/sdh and /dev/sdi. Uncheck Delete on Termination for all volumes and check Encrypted on the 4 extra volumes.

2) Implement RAID10

sudo mdadm --verbose --create /dev/md0 --level=10 --chunk=256 --raid-devices=4 /dev/sdf /dev/sdg /dev/sdh /dev/sdi
echo 'DEVICE /dev/sdf /dev/sdg /dev/sdh /dev/sdi' | sudo tee -a /etc/mdadm.conf
sudo mdadm --detail --scan | sudo tee -a /etc/mdadm.conf

Set read-ahead for better performances

sudo blockdev --setra 128 /dev/md0
sudo blockdev --setra 128 /dev/sdf
sudo blockdev --setra 128 /dev/sdg
sudo blockdev --setra 128 /dev/sdh
sudo blockdev --setra 128 /dev/sdi

Create logical volumes for data, log and journal

sudo dd if=/dev/zero of=/dev/md0 bs=512 count=1
sudo pvcreate /dev/md0
sudo vgcreate vg0 /dev/md0
sudo lvcreate -l 54%vg -n data vg0
sudo lvcreate -l 23%vg -n log vg0
sudo lvcreate -l 23%vg -n journal vg0

Create filesystems for each logical volume

sudo mke2fs -t ext4 -F /dev/vg0/data
sudo mke2fs -t ext4 -F /dev/vg0/log
sudo mke2fs -t ext4 -F /dev/vg0/journal
sudo mkdir /data
sudo mkdir /log
sudo mkdir /journal
echo '/dev/vg0/data /data ext4 defaults,auto,noatime,noexec 0 0' | sudo tee -a /etc/fstab
echo '/dev/vg0/log /log ext4 defaults,auto,noatime,noexec 0 0' | sudo tee -a /etc/fstab
echo '/dev/vg0/journal /journal ext4 defaults,auto,noatime,noexec 0 0' | sudo tee -a /etc/fstab

Mount storage devices

sudo mount /data
sudo mount /log
sudo mount /journal

Create symbolic link for db

sudo ln -s /journal /data/journal

3) Install MongoDB

echo "[10gen]
name=10gen Repository
baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64
gpgcheck=0" | sudo tee -a /etc/yum.repos.d/10gen.repo
sudo yum -y install mongo-10gen-server

Change owner

sudo chown mongod:mongod /data
sudo chown mongod:mongod /log
sudo chown mongod:mongod /journal

LATER, I did this:

4) Update packages

sudo yum update

5) Upgrade MongoDB: start by uninstalling old version

sudo yum remove mongo-10gen-server
sudo rm /etc/yum.repos.d/10gen.repo

Install new version

sudo rpm --import https://mongodb.org/static/pgp/server-3.2.asc

echo "[mongodb-org-3.2]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/amazon/2013.03/mongodb-org/3.2/x86_64/
gpgcheck=1
enabled=1" | sudo tee -a /etc/yum.repos.d/mongodb-org-3.2.repo

sudo yum install -y mongodb-org

Change owner

sudo chown mongod:mongod /data
sudo chown mongod:mongod /log
sudo chown mongod:mongod /journal

6) Disable THP (new database was displaying a warning)

sudo nano /etc/init.d/disable-transparent-hugepages

Copy the following:

#!/bin/sh
### BEGIN INIT INFO
# Provides:          disable-transparent-hugepages
# Required-Start:    $local_fs
# Required-Stop:
# X-Start-Before:    mongod mongodb-mms-automation-agent
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Disable Linux transparent huge pages
# Description:       Disable Linux transparent huge pages, to improve
#                    database performance.
### END INIT INFO

case $1 in
  start)
    if [ -d /sys/kernel/mm/transparent_hugepage ]; then
      thp_path=/sys/kernel/mm/transparent_hugepage
    elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then
      thp_path=/sys/kernel/mm/redhat_transparent_hugepage
    else
      return 0
    fi

    echo 'never' > ${thp_path}/enabled
    echo 'never' > ${thp_path}/defrag

    unset thp_path
    ;;
esac

Finally, apply and restart

sudo chmod 755 /etc/init.d/disable-transparent-hugepages

sudo chkconfig --add disable-transparent-hugepages

sudo shutdown -r now

Yes, post everything. It's OK if your post is long. More info is better than less :) — Michael Hampton, Apr 12 '16 at 01:55
@MichaelHampton Done! Thanks for the piece of advice :) I hope that's enough information now — APE, Apr 12 '16 at 02:13
Yes, at the instance launch setup you can choose the volumes. There is one Root by default (/dev/xvda). I added 4 EBS volumes as /dev/sdX — APE, Apr 12 '16 at 02:17
I added some more info: I should have mentioned the volumes are encrypted by Amazon — APE, Apr 12 '16 at 02:26

score 0 · Accepted Answer · answered Oct 07 '16 at 07:28

It seems mdadm can be executed too early in the boot process (elements not available).

Here's how I retrieved the array

Reassemble the array:

sudo mdadm --assemble --scan --verbose

Activate Virtual Groups:

sudo vgchange -ay

Mount:

sudo mount /data
sudo mount /log
sudo mount /journal

RAID10 array lost after reboot | Amazon Linux

1 Answers1