are EBS volumes wiped after use?

Question

I tried to recover some data from an ebs volume on which I accidentally ran wipefs on.

I used PhotoRec (http://www.cgsecurity.org/wiki/PhotoRec)... and it got my files back but also a ton of other files that did not belong to me.

It got images, text files, code etc... They all were valid data, from not from my account.

That lead me to ask... when I delete an EBS volume, I guess my data is in the clear usable by someone else?

Tim · Answer 1 · 2017-01-26T19:48:25.737

14

From the AWS documentation

The physical block storage used by deleted EBS volumes is overwritten with zeroes before it is allocated to another account.

From an AWS rep on their forums.

I can confirm that when any customer volume is terminated (be it EBS or an instance storage volume) it is completely wiped before being made available for use by other customers.

If this is genuine and you really have someone else's data you need to get in touch with AWS. Extraordinary claims require extraordinary evidence.

TLDR; I did two sets of tests and was unable to reproduce the results that @stevelandiss produced.

Update - test one

I tried this out myself. Here's what I did and my results.

TLDR; could not reproduce.

0) I allocated an m3.medium spot instance, with gp2 and io1 (provisioned IOPS) volumes, 10GB each. I used the standard Ubuntu 16.04 AMI (ami-b7a114d7). Note that I could not mount as /dev/xvdb as the OP suggested, AWS forced me to use longer names like /dev/xvdba which makes me slightly suspicious.

1) I installed photorec/testdisk

apt-get install testdisk

2) I used lsblk to look at the volumes available

lsblk
NAME    MAJ:MIN   RM SIZE RO TYPE MOUNTPOINT
xvda    202:0      0   8G  0 disk
└─xvda1 202:1      0   8G  0 part /
xvdba   202:13312  0  10G  0 disk
xvdbb   202:13568  0  10G  0 disk
xvdca   202:19968  0   4G  0 disk

I tried to mount the disks just to check, but of course they have no file system so it failed

mount /dev/xvdba /gp2/ mount: wrong fs type, bad option, bad superblock on /dev/xvdba, missing codepage or helper program, or other error

In some cases useful info is found in syslog - try dmesg | tail or so.

3) I made file systems on each device

mkfs -t ext4 /dev/xvdba
mke2fs 1.42.13 (17-May-2015)
Creating filesystem with 2621440 4k blocks and 655360 inodes
Filesystem UUID: e32b2ed1-a0f8-49df-895d-c56b9802a009
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

root@ip-11-0-2-184:/home/ubuntu# mkfs -t ext4 /dev/xvdbb
mke2fs 1.42.13 (17-May-2015)
Creating filesystem with 2621440 4k blocks and 655360 inodes
Filesystem UUID: 4f1f7c75-bbce-4887-aac7-02e197a36c89
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

4) I mounted the disks

mount /dev/xvdba /gp2/
mount /dev/xvdbb /pio/

5) I ran photorec on each volume

photorec /dev/xvdba

GP2

IO1 provisioned IOPS

As you can see no files were found. If @stevelandiss can point out what he did differently I can try again to reproduce. For example he didn't mention any mounting, and he used a different device name. I'll try again without mounting a few minutes, but I want to save this update so I don't lose it.

Update - test two

This time I did much the same, but I didn't create a file system or mount the disk. This is closer to what to what @stevelandiss did. This made no difference, no files were recovered.

GP2

IO1 provisioned IOPS

edited Jan 26 '17 at 19:48

answered Jan 26 '17 at 01:48

Tim

31,888
7
52
78

I'm not trolling... This is easy enough to reproduce. Just allocate a new EBS volume and run photorec. The volume in both of my cases were provisioned iops volumes. – steve landiss Jan 26 '17 at 02:10
they were newly created volume that you have not used before? – Marvin Hoffmann Jan 26 '17 at 02:29
@MarvinHoffmann That is correct. If you have an AWS account, please try it out. I have now since posting this done it many times in different availability zones. – steve landiss Jan 26 '17 at 02:33
2

Steps to repro (1) Launch a Linux instance (I used Ubuntu)... (2) allocate any PIOPS EBS volume to the instance... Call it `/dev/xvdb` (3) Install photorec... `sudo apt-get install testdisk` (4) run `photorec /dev/xvdb`. – steve landiss Jan 26 '17 at 02:36
where is the step where you create a new volume? allocate=attach is not creation. Volumes are wiped at creation, not at attaching. – Marvin Hoffmann Jan 26 '17 at 03:47
@MarvinHoffmann... go to your AWS account... create a NEW EBS volume of provisioned IOPS. Then attach that to an EC2 instance. Even use hexdump on the raw volume, it is not zeroes. I can't explain this any more simpler than these steps. – steve landiss Jan 26 '17 at 03:56
4

@stevelandiss I just tried creating a new volume and hexdump-ing it, all I got was zeros. – Jan 26 '17 at 04:15
@jrdn Was it a provisioned iops volume? Also what AZ ... i will try with those settings. In the mean time if you can try photorec on that vol, that would be great. – steve landiss Jan 26 '17 at 04:17
edit these are GP2 volumes – steve landiss Jan 26 '17 at 05:19
4

If no-one else tries it I'll give it a shot in the next couple of days. I'd appreciate if you could edit your question to give clear unambiguous steps to reproduce. – Tim Jan 26 '17 at 08:04
1

I tried IOP and GP2 on eu-west-1. Nothing found. – Quentin Hayot Jan 26 '17 at 14:19
1

@stevelandiss are you using encrypted volumes? Also please confirm that these volumes are new, blank, NOT from snapshots, and please mention which regions allow you to duplicate this behavior, and please mention whether you can duplicate it on other volume types. – Michael - sqlbot Jan 26 '17 at 17:36
@stevelandiss I tried twice, using slightly different methods, outlined above. I was unable to reproduce. You need to produce really detailed steps to reproduce and screenshots if you want to be believed. – Tim Jan 26 '17 at 19:49
3

no mean to offend but, i'd bet 100 euros, that this is a layer-8 problem. – lauc.exon.nod Jan 26 '17 at 20:19
2

@lauc.exon.nod it could also be a PEBKAC issue – Tim Jan 26 '17 at 20:24

score 5 · Answer 2 · answered Jan 26 '17 at 01:45

5

from the wipefs man pages:

wipefs does not erase the filesystem itself nor any other data from the device

answered Jan 26 '17 at 01:45

Marvin Hoffmann

151
2
4

1

Understood... still this means that when ai get an EBS volume, it's possible to see someone else's data... – steve landiss Jan 26 '17 at 02:08
9

@stevelandiss You need to contact AWS support with your findings. – jscott Jan 26 '17 at 02:47
6

Ouch, put down "wipefs" as the number one improperly named command ever. :-) – Brian Knoblauch Jan 26 '17 at 13:28

lauc.exon.nod · Answer 3 · 2017-01-26T08:58:24.100

we need more information about the volume. How did you create it? Are you 100% sure that no one else created it but you?

AWS does not share how they designed the technology, so i am guessing as a NetApp certified storage guy. EBS Volumes are abstraction layers, built on RAID groups. I doubt it will just one single disk. So everytime you provision a volume, you will(would) be getting chuncks from different physical devices. That makes it very unlikely for you to get complete files.

Give us more information how you provisioned the volume. But i am guessing you are making a mistake at some point. Or else this would be a huge security violation on AWS about such a basic feature.

here is the test i made, i create a new volume, a new instance. attached the volume to the instance and then ran photoRec test. i found 0 files as expected.

PhotoRec 7.0, Data Recovery Utility, April 2015
Christophe GRENIER <grenier@cgsecurity.org>
http://www.cgsecurity.org

Disk /dev/xvdf - 1073 MB / 1024 MiB (RO)
 Partition                  Start        End    Size in sectors
P Unknown                  0   0  1   130 138  8    2097152


0 files saved in /home/ec2-user/testdisk-7.0/recup_dir directory.
Recovery completed.

do you have any other IAM users in your account? maybe they attached that volume to their instances and used that way.

Perhaps the volume was created from a snapshot? https://d0.awsstatic.com/whitepapers/aws-security-whitepaper.pdf says 'An EBS snapshot is a block-level view of an entire EBS volume. Note that data that is not visible through the file system on the volume, such as files that have been deleted, may be present in the EBS snapshot.' — Jason Martin, Jan 26 '17 at 16:05

score 1 · Accepted Answer · answered Jan 26 '17 at 16:27

https://d0.awsstatic.com/whitepapers/aws-security-whitepaper.pdf describes Amazon's published process for dealing w/EBS. Two quotes seem relevant:

Amazon EBS volumes are presented to you as raw unformatted block devices that have been wiped prior to being made available

but also

An EBS snapshot is a block-level view of an entire EBS volume. Note that data that is not visible through the file system on the volume, such as files that have been deleted, may be present in the EBS snapshot.

The most likely case is that you are creating your volume from a snapshot that has deleted data on it.

I tried to reproduce your scenario in us-east-1 with new PIOPS, gp2, and magnetic volumes and was not able to recover any data.

That said, you can further safeguard your EBS data by making use of KMS encrypted volumes.

are EBS volumes wiped after use?

4 Answers4