0

At the moment, we are using git to backup our databases. I know there lots of reason to not do so, but this is not the topic of this post. We need to do it and we will do it (we have good reason for it). However, the problem is git doesn't handle large files really well and we can't push anything anymore because git run out of memory. I was wondering if any of git alternatives (hg, darcs ?) are better for this purpose.

mb14
  • 22,276
  • 7
  • 60
  • 102
  • 1
    Why not use [`rdiff-backup`](http://www.nongnu.org/rdiff-backup/) or [`attic`](https://attic-backup.org/)? They do what you'd use a VC system for: 1) store deltas/deduplicate; 2) provide history, and they allow what Git doesn't (at least not easily): pruning the old history. – kostix Jun 24 '16 at 12:10
  • What I need is mainly to be able to check at differences between to commit/backup. – mb14 Jun 24 '16 at 17:54
  • SVN handles binaries and large files really well, but there is no obliterate command (you have to dump/filter/load the repo again). I'd strongly suggest duplicity or ```borg``` backup (more up to date then attic). – tgharold Jun 28 '16 at 18:37

1 Answers1

0

You have conflicting requirements, you said that:

  1. you have to use use to use git
  2. git doesn't handle well large files

The second argument is true, every single change will reallocate the entire file.

You need a backup tool that handles efficiently diffs, what is against your first axiom.

Since you can't change #2, you have to loosen up #1 (or change git internals :) )

I would look for a backup solution specific to your database system. If none is up to your needs, go for a backup tool that uses rdiff algorithm (it hashes every 200 kB of the files, so small changes in large files only reallocate the blocks that changed).

Jonas Fagundes
  • 1,519
  • 1
  • 11
  • 18
  • I don't have to use git but I have to use a vcs. I doubt git is the best vcs to handle large file, therefore it must be another vcs better than git to do that . Or maybe a git plugin. Backup is not an option unless the backup tool has some vcs capabilities: I need to be able to inspect diff (in text mode) between different commit/snapshots. – mb14 Jun 25 '16 at 08:43
  • At the moment, git works perfectly well I n my local machine. It's only the GC which doesn't work. What I have to do is copy everything on another machine with more RAM and then push it to the main repository. – mb14 Jun 25 '16 at 08:45