1

I am running some sequence analysis via EC2. I expect my output files to be over 2 Tb. Before I run my command I want to make sure I have enough room.I changed my instance type to one for data processing d2.4xlarge.

My question: If I am running my command, and the output file exceeds the storage capacity of xvda, will it automatically switch to xvdf??

hopefully this info is helpful:

lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0     7:0    0   18M  1 loop /snap/amazon-ssm-agent/
loop1     7:1    0 93.9M  1 loop /snap/core/9066
loop2     7:2    0   91M  1 loop /snap/core/6350
loop3     7:3    0   18M  1 loop /snap/amazon-ssm-agent/
xvda    202:0    0  4.9T  0 disk 
└─xvda1 202:1    0    2T  0 part /
xvdf    202:80   0  1.8T  0 disk 
xvdg    202:96   0  1.8T  0 disk 
xvdh    202:112  0  1.8T  0 disk 
xvdi    202:128  0  1.8T  0 disk 
xvdj    202:144  0  1.8T  0 disk 
xvdk    202:160  0  1.8T  0 disk 
xvdl    202:176  0  1.8T  0 disk 
xvdm    202:192  0  1.8T  0 disk

You will see that xvda1 does not match xvda, and that is because with EBS the max storage is 2tb ( I need to change the volume size on that)

df
Filesystem      1K-blocks     Used  Available Use% Mounted on
udev             62832116        0   62832116   0% /dev
tmpfs            12570936      824   12570112   1% /run
/dev/xvda1     2081729452 45739112 2035973956   3% /
tmpfs            62854676        0   62854676   0% /dev/shm
tmpfs                5120        0       5120   0% /run/lock
tmpfs            62854676        0   62854676   0% /sys/fs/cgroup
/dev/loop0          18432    18432          0 100% /snap/amazon-ssm-agent/
/dev/loop1          96256    96256          0 100% /snap/core/
/dev/loop3          18432    18432          0 100% /snap/amazon-ssm-agent/
/dev/loop2          93184    93184          0 100% /snap/core/
tmpfs            12570932        0   12570932   0% /run/user/

thank you!!!!!

morinajc
  • 15
  • 5

1 Answers1

2

"If I am running my command, and the output file exceeds the storage capacity of xvda, will it automatically switch to xvdf"

No. You would need to mount xvdf (it doesn't appear that you have done this yet) which will give it a path in your file system, and then you will need to configure your application to switch to that mounted location when the first location runs out of space.

"that is because with EBS the max storage is 2tb"

The official features list for AWS EBS here, states that the max volume size is 16TB.

If you need a single volume in your file system to be more than 16TB, you could look at combining EBS volumes in a RAID 0 array, as documented here.

Mark B
  • 183,023
  • 24
  • 297
  • 295
  • Hi Mark, thanks for your help! I saw that it was 16 TB as well, and that should be plenty. However, when I go to growpart the max it will go to is 2 TB. Then I found this which says that the max is actually 2TB https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/volume_constraints.html – morinajc May 28 '20 at 13:26
  • Oh I just saw your comment on the RAID array, that looks very useful – morinajc May 28 '20 at 13:30
  • @morinajc that link you posted simply says that 2TB is the limit for the MBR partition. That shouldn't be an issue for extra EBS volumes you attach to your EC2 instance. – Mark B May 28 '20 at 14:45
  • Thank you for pointing that out!! If my partition (xvda 1) fills with data, will it automatically start saving to xvda? – morinajc May 28 '20 at 15:13
  • There is no feature to "automatic" change to another partition when one fills up that is built into any operating system that I am aware of. – Mark B May 28 '20 at 15:30