0

I am a developer, not an IT guy, but I am the one responsible for managing our servers currently. I have run into a problem... apparently our SQL backups are much bigger than I anticipated, and on Saturday night, our SQL backup ran, then completely filled our virtual disk.

I couldn't boot the VM, so I figured I would just attach the drive to another VM, remove the older backups, then add it back to the SQL VM. But, only a select number of files were available on the drive when I was browsing it from the other VM. Thus, I was unable to remove the old backups, leaving the disk still completely full. I then read a post that mentioned that I could delete a snapshot to free up some space. In retrospect, this was a dumb thing to do, but I did. It was the very first snapshot I captured (one I don't need anymore).

So, now I have a VMDK, and the following error message:

The parent virtual disk has been modified since the child was created. The content ID of the parent virtual disk does not match the corresponding parent content ID in the child.

enter image description here

I am currently working through fixing the mismatched CIDs, but I still don't know what to do about the full VMDK... can anyone explain how I can get into that virtual disk and delete a few files so that I can reconfigure the backups?

Update: View my answer below to see how I resolved this issue

If you do not have the time to work things out like I did, then HopelessN00b's solution is the way to go (assuming you have backups of everything).

Derek Foulk
  • 111
  • 1
  • 6
  • You should probably suggest that an admin be hired. This could have been a career altering mistake. – Jim B Mar 14 '16 at 10:50
  • `only a select number of files were available on the drive when I was browsing it from the other VM` - Can you give us more detail on that? You should have been able to see and access all files on the VMDK if you attached it to a VM that can read whatever file system that the virtual disk was formatted with by the guest operating system. If it's NTFS then attaching it to another Windows VM should have given you complete and full access to all files on the VMDK. – joeqwerty Mar 14 '16 at 11:23
  • Quick guess: You attached the first snapshot read-write (as opposed to the newer, "current" snapshot), thereby altering it. That should explain that you only saw some files and not all. Now VMWare won't let you apply the snapshot because the underlying parent has been changed. You Have A Big Problem Now. First thing: Get a large disk and backup everything left. Then you can continue trying getting your data back. – jornane Mar 14 '16 at 12:17
  • Thank you guys for your comments. After resolving this issue, I discovered that I was indeed able to view all the files on the "full" VMDK... I was just wrong about what was full... I must have misinterpreted the initial VMWare error (regarding the full disk). My understanding was that the virtual drive was full, when the datastore was actually full. I took about 4 snapshots during my installation process, which apparently (combined with the Thick/Eager provisioning & the hypervisor OS) filled our 240GB SSD very, very quickly. I have reconfigured our VMs to avoid this in the future! – Derek Foulk Mar 17 '16 at 23:55

3 Answers3

5

At this point, it's going to be much quicker and easier to restore this VMDK from the backup copy you hopefully have. Snapshots are delta disks, and by deleting the earliest one, you have probably rendered the snapshots following it useless.

It may or may not be possible to untangle your snapshot chain and end up with a current, working virtual disk at all, so ... well, learning experience. It's safer to push backups, logs, or anything else that might run a disk out of space onto a non-system disk so that you can simply expand it, instead of trying to mount a system disk, free up space and get it bootable again.

Also, for future reference, snapshots do not take up space within the file system of the VMDK, so deleting them won't free up space within the VDMK.

HopelessN00b
  • 53,795
  • 33
  • 135
  • 209
  • How do you prevent people from ending up in situations like this? – ewwhite Mar 14 '16 at 05:33
  • 6
    @ewwhite By hiring actual, competent sysadmins/engineers, rather than saying "developers... they know computer stuff, right? Let's just make one of them work two jobs for one paycheck and give ourselves a bonus for saving money!" – HopelessN00b Mar 14 '16 at 06:12
  • @ewwhite Quite easily - don't use devops (aka developpers with admin access), hire normal sysadmins. It's more expensive but it is right way how to work. As you will not let your admins to write code, you should not let your developpers to manage servers. – Ondra Sniper Flidr Mar 14 '16 at 08:37
  • Devops is not "developers with admin access" but that's certainly how many people seem to take it. – Jim B Mar 14 '16 at 10:51
0

Had the same issue my macVM Mojave would not start after installing an app(RipX) tried the hints above but no cigar...the mac diskutil would say cannot fix disk damaged save what u can had similar issues in the past with real HDD DiskWarrior save me back then so I tried that a full fix but didn't do it tried it again later on and hallelujah it worked the VM fired up and quickly deleted that app

becker
  • 101
0

To remove files from inside a virtual disk (to free space on the disk)...

1) Remove the virtual disk from your VM:

  • Open the vSphere Client
  • Right Click the affected virtual machine
  • In the Hardware tab, select the full disk ("Hard Disk #") from the list of hardware
  • Click Remove
  • Select Remove from virtual machine (DO NOT select 'Remove from virtual machine and delete file from disk')

2) Add the virtual disk to a functioning VM:

  • Open the vSphere Client
  • Right Click the a working virtual machine
  • In the Hardware tab, click Add > Hard Disk > Use an existing... > Select the "full" virtual disk

3) Browse and delete files from within the virtual disk to free up some space

  • Open the console of the functioning VM
  • Browse to the newly attached virtual disk (should be visible in File Explorer)
  • Locate and remove unnecessary files to free some space on the disk

3b) If you do not see the drive, open Disk Management

  • Locate the "missing" drive and toggle the drive from Offline to Online

To fix the mismatched CIDs error

See this guide by VMware:

Resolving the CID mismatch error: The parent virtual disk has been modified since the child was created (1007969)

Basically, you view the vmware.log file to determine what CID it's looking for, then manually set that CID on the virtual disk that is failing to attach. This guide looks intimidating, but this process is pretty simple. I used Putty to SSH into the host, then used vim to correct the CID in the .vmdk file.

To prevent this from happening in the future, I will be using larger drives on my host, and not storing any VMs on the same drive as the hypervisor...

Thank you for your help. If anyone like me runs into this issue, please don't panic like I did. Read the errors and and corresponding KB articles in their entirety, and make sure you retain backups of everything you change (aka don't delete a snapshot without taking a backup of the VM's directory first).

Update: Read Understanding VM snapshots in ESXi / ESX (1015180) | VMware KB to understand how snapshots work and you will save yourself a lot of heartache if you need to delete snapshots...

Derek Foulk
  • 111
  • 1
  • 6