Server Fault Stack Exchange
Server Fault Stack Exchange

Questions tagged [deduplication]

98 questions
3
votes
1 answer

Production-ready disk deduplication for KVM (Linux)?

Multiple Windows VMs, all of them with identical software installed (they're clones in fact) that I have to run for administrative+licensing reasons, give me a headache storage-wise. KVM on Linux runs great but ext4 does not provide any features re…
LetMeSOThat4U
  • 1,371
  • 2
  • 17
  • 35
3
votes
2 answers

Deduplication of tar archives

Now I'm trying to configure ZFS with deduplication on my linux backup server and have a question. For example I have 1k files in backup1.tar and 1k same files and 1 new in backup2.tar. This files will be deduplicated or not? As I can see it doesn't…
cpt.Buggy
  • 279
  • 2
  • 10
3
votes
2 answers

Accessing windows server deduplicated files from other operating systems

If you enable data deduplication on the Windows Server 2012, it will change certain files (according to the policy) and introduce reparse points. If the drive containing deduplicated volumes is detached from Windows Server 2012 and attached to…
Iravanchi
  • 330
  • 4
  • 11
3
votes
2 answers

Mixing Volume Shadow Copy and Data Deduplication in Windows Server

I'm using Windows Server 2012 and I'm creating a scheduled backup for multiple Hyper-V guests (all of them are Windows Server 2012) to a dynamic VHDX on a removable USB drive (so that I can alternate two drives and have two sets of backups). I…
Iravanchi
  • 330
  • 4
  • 11
3
votes
1 answer

Extracting and saving attachments from maildir, deduplicated

I would like to know if there exists a solution for archiving email, stripping out the attachments and saving them separately, deduplicated? The ideal would be a maildir-setup, with a script running over the messages, extracting the attachments,…
grojo
  • 429
  • 1
  • 7
  • 18
3
votes
1 answer

Cheap/fast deduplication with hardlinks?

I've got shared hosting with a few thousand Wordpress installs and I've wanted for ages to have a nice way of removing all the duplicate files in a sensible and secure way. I'm looking for better disk cache hit ratios and simpler backups. I'm just…
Nick
  • 287
  • 1
  • 10
3
votes
1 answer

Setup Raid 10 for backup store

We have HP DL380p Gen8 with Smart Array P420i: 10HDDx6TB, 2 RAID10 6 and 4 HDD, for backup store via Arcserve UDP, deduplication backup configured for 16 KB blocks, we are not satisfied with the speed of recovery, want a better option, how do we…
mgkmgk
  • 31
  • 1
2
votes
0 answers

Windows 2016 deduplication and application compatibility

I'm re-evaluating Windows 2016/2019 deduplication engine, which is way better (faster/more capable) than what shipped in previous Windows versions. I understand how it works (sparsifying files via holes + reparse point + compression), the…
shodanshok
  • 47,711
  • 7
  • 111
  • 180
2
votes
1 answer

puppet code design: how to collect arguments into a string (avoiding possible duplicate declaration errros)

I'm having trouble with puppet (version 5.5) code design. I've written a component module to deal with ucarp. It make use of the module eyp-systemd to register an ucarp service within systemd. Now I would like to consume the ucarp module from two…
C.Scharfenberg
  • 63
  • 7
2
votes
1 answer

How to read fully deduplicated files over SMB on macOS?

I have a volume in Windows Server 2016 that is a target of the data deduplication role. It works very well ~60% savings rate, with many files appearing as zero bytes on the disk. The volume is shared over SMB and mounts fine on Mac, Windows and…
Louis Waweru
  • 755
  • 1
  • 9
  • 29
2
votes
1 answer

How to find duplicate files against a reference directory structure in Linux

There are a couple of duplicate file finders for Linux listed e.g. here. I have already tried fdupes and fslint. However, from what I have seen, these will find all duplicates of the selected directory-structures/search paths and thus also…
packoman
  • 175
  • 1
  • 2
  • 8
2
votes
0 answers

2012 R2 DPM: massive increase in data transferred during syncronisation of a Deduplicated protected volume

I have been using 2012 R2 DPM to protect a 46 TB 2012 R2 deduplicated volume for the past 4 months without too much issue. The volume is a low access archive server. Current usage on the volume is 23 TB (if not deduplicated, this would be more like…
Steve
  • 133
  • 2
  • 6
2
votes
2 answers

Windows Server 2012R2 NTFS data deduplication and SVN clients

There seems to be an issue with using SVN on a NTFS deduped volume. To save space on my SSD for some huge development checkouts (> 5 GB each) i enabled data deduplication on the volume. This seems to cause SVN all kinds of grief and madness. The…
schlenk
  • 183
  • 5
2
votes
1 answer

ZFS for offline backups

I'm thinking of using ZFS to backups files to single drives that are going to be kept mostly offline. Does anyone know if the FreeBSD ZFS implementation still uses memory if a dedupped pool is exported?
Cyclone
  • 557
  • 5
  • 15
2
votes
3 answers

Fastest way to copy large file directory over slow internet

I need to copy a directory from one of our servers several hundred miles away. Both sites where I am and where the files are have business cable. The cable is 5 Mbps(Down) / 768 Kbps(Up). This poses a small problem considering I need to copy about…
AtomicPorkchop
  • 1,975
  • 8
  • 34
  • 55
1 2
3
4 5 6 7