8

I host a lot of websites and our system makes it easy to duplicate items in these sites which is handy, but leads to lots of duplicated (and potentially quite large) files. I was wondering if these is any mechanism in linux (specifically Ubuntu) where the filesystem will only store the file once but link to it from all its locations.

I'd need this to be transparent, and also handle the case that if a user changes one of the files, it doesn't alter the contents of the main file but creates a new file for just this particular instance of the file.

The point of the exercise is to reduce wasted space used by duplicated files.

Brendon Muir
  • 243
  • 1
  • 6

2 Answers2

6

I'd need this to be transparent

ZFS-on-Linux × feature called "on-line deduplication".

UPD.: I've re-read your question once again now it looks like Aufs can be of help for you. It's very popular solution for hosting environments. And actually I can mention Btrfs by myself now as well — the pattern is you have some template sub-volume which you snapshot every time you need another instance. It's COW, so only changed file blocks would need more space. But keep in mind, Btrfs is, ergh… well, not too stable anyways. I'd use it in production only if data on it are absolutely okay to be gone.

poige
  • 9,448
  • 2
  • 25
  • 52
  • Seems this is the better solution for online deduplication. I've decided to skip out on this in favour of bigger hard drives. Space is cheaper than the ram required to make this work. – Brendon Muir Jan 22 '14 at 02:19
  • 1
    Well, yeah, ZFS is rather RAM-greedy thing. But can be controlled, so actually as always it is task-dependent. Ppl run on FreeBSD, some claim they're happy with it on Linux as well, and it think it's quite reasonable to run it on ≥ 16 GiB – poige Jan 22 '14 at 05:51
  • Hehe, I have about 6GB and Rails want to chew a lot of that up! :) Thank you for revising your question. Hopefully it will help anyone else that comes across this requirement and doesn't know where to even start. – Brendon Muir Jan 22 '14 at 22:05
1

There is a linux user space/fuse filesystem that will do this dedup.

http://sourceforge.net/p/lessfs/wiki/Home/

Linux Journal has a good article on it in it's August 2011 issue. There are also various filesystem specific options with btrfs and zfs.

Nathan Neulinger
  • 607
  • 1
  • 6
  • 17
  • «There are also various filesystem specific options with btrfs» — ORLY? – poige Jan 20 '14 at 08:57
  • https://btrfs.wiki.kernel.org/index.php/Deduplication and http://sdteffen.blogspot.com/2013/10/bedup-dedup-data-deduplication-using.html would be good places to start reading about btrfs's dedup. – Nathan Neulinger Jan 21 '14 at 13:29
  • And does any of them mention you can have on-line dedup with Btrfs? I don't think so, so, what's the better place to start then? ) – poige Jan 21 '14 at 14:17
  • 1
    Sure looks to me like the bedup stuff runs with filesystem mounted. Granted, you have to run a tool to go find the duplicates and tell btrfs about them, but the functionality is there. Not enough definition of "transparent" in the original request to know what is strictly required. – Nathan Neulinger Jan 22 '14 at 14:36
  • Thanks @nneul. I guess I was hoping it would deduplicate on write but running something every so often wouldn't be a problem. I guess it's like how they do scrubbing (unless that's changed). – Brendon Muir Jan 22 '14 at 22:08