1

A user deleted a directory containing an important file over a Samba share. Share was from a Nexenta box running ZFS, raid-Z3.

Last backup was 12 hours ago - is there a way to recover the 1 day's work?

Edit:
Asked about topic on Nexenta forums, similar responses.

xddsg
  • 3,392
  • 2
  • 28
  • 33
  • 1
    Is your ZFS configured to do any snapshotting? – Shane Madden Jan 19 '12 at 17:02
  • Snapshots only exist for system upgrades - the previous one was a month ago when the file did not exist. – xddsg Jan 19 '12 at 17:09
  • 2
    Then you will be of luck, I fear. Snapshotting is easy, why don't you use this, at least for intra-day purposes between regular backups? – Sven Jan 19 '12 at 17:14

3 Answers3

8

If you're not running snapshots then restoring from backup is your only option.

I would advise you to look into snapshotting, as it's extremely useful on fileservers. Users are dumb, and they overwrite/delete files way more often than you can run a backup.

Edit: As mentioned by ErikA - providing snapshots on a file server also gives users a easy way of finding the old copies on their own. In Windows you'll just have to right click the folder/file and select "Previous versions".

pauska
  • 19,620
  • 5
  • 57
  • 75
  • 3
    Additionally, the snaps can be exposed very easily to the users, enabling them to do restore things themselves. – EEAA Jan 19 '12 at 17:18
  • 1
    No way to recover based on open file handles or some zdb block-fu? – xddsg Jan 19 '12 at 17:39
  • @ErikA: ZFS snapshots can be exposed to end users over Samba? That is awesome! – StrangeWill Jan 19 '12 at 18:33
  • 2
    @StrangeWill - Yes, as well as via all their other NAS file sharing protocols. I've used it mostly via NFS. – EEAA Jan 19 '12 at 19:43
  • I've heard of people editing the disk to have ZFS use a previous state, but I don't know of a utility to do that. This would allow you to get to the files if they hadn't been over written. In addition, I've heard of some data carving tools that would be used on ZFS, mainly for forensics. – Scott McClenning Jan 26 '12 at 05:24
  • @TomPaine - zdb-fu would have only been an available option had it been attempted fairly quickly after the file was deleted. It is both possible to potentially rip the file out of an older uberblock, or it is possible to roll back the entire ZFS pool to an earlier point in time.. but in both cases, only for the last 128 or so txg's -- which depending on version and load could be gone in just over 10 minutes or an hour. This is an extremely risky procedure that should only be undertaken by someone with experience doing it. – Nex7 Feb 25 '12 at 06:22
  • If you ever find yourself realizing you just deleted something SUPER IMPORTANT, immediately export the pool and call an expert (you need to export the pool; just stopping your access is almost undoubtedly insufficient, as would be pulling the network cable from the storage device -- export the pool to be safe, and call for help). – Nex7 Feb 25 '12 at 06:24
  • @Nex7 is it possible to use a point in time snapshot if he isn't using snapshots? – pauska Feb 25 '12 at 12:34
  • @pauska Yes, it is. Sort of. Again, ZFS is keeping a rolling queue of the last 128 updates to the global pool, if you will. So effectively you have 128 point-in-time snapshots representing the last few minutes to hours (depending on configuration). You can both traverse these (if you're good, know what you're looking for, and have a bit of luck on your side), or you can 'roll back' to one of these (losing everything between it and present time, unfortunately). This is very advanced ZFS fu -- it is NOT meant to be used as a scapegoat for not using real snapshots or poor implementation/planning. – Nex7 Feb 27 '12 at 21:02
  • @Nex7 Accepted the loss and carried on but I'm interested in file-systems. Can you link to any resources where I can learn how to do what you suggested? – xddsg Mar 23 '12 at 11:09
  • 1
    @TomPaine: Here's an example (albeit from 2008) of how one might used zdb/mdb to find the older uberblock and recover the orphaned file. Probably not too practical for single file undelete, but I had an unmountable raidz pool which was mostly recoverable after winding back past a dozen corrupt uberblocks. Given infinite time I don't think I could've ever done it, but with the help of a ZFS uber-guru like George Wilson such things are possible. http://mbruning.blogspot.com/2008/08/recovering-removed-file-on-zfs-disk.html – notpeter Dec 06 '12 at 17:21
  • @notpeter if you converted this to an answer I'd accept it. – xddsg Jan 21 '13 at 19:25
3

Here's an example (albeit from 2008) of how one might use zdb/mdb to find the older uberblock and recover the orphaned file. Probably not too practical for single file undelete, but I had an unmountable raidz pool which was mostly recoverable after winding back past a dozen corrupt uberblocks. Given infinite time I don't think I could've ever done it, but with the help of a ZFS uber-guru like George Wilson such things are possible.

http://mbruning.blogspot.com/2008/08/recovering-removed-file-on-zfs-disk.html

notpeter
  • 3,515
  • 2
  • 26
  • 44
1

Right now your only option is restoring from backup as has previously been stated. I'd highly recommend using snapshots and integrating those into your backup workflow.

First of all you will only be backing up changes and new files once you've done a full backup and go with snapshots from there and it may also increase your flexibility depending on how you do backups now. Considering you're on a 12 hour backup schedule you could do hourly ZFS Snapshots in between and keep those for a few days. You could probably even integrate the snapshots themselves into your backup mechanism using send/receive. Its a great feature for incremental backups.

juwi
  • 573
  • 1
  • 5
  • 14