Alternatives to Git for Minecraft server backup/version control

Question

Currently my Minecraft server, residing on a CentOS server, uses Git as a means of version control and 'catastrophe-management'. This works very well except for two issues:

It's big. Because the server has a central repository, master branch (on which the server actually runs) and a test-server branch, each containing every committed change ever made fills the SSD up no-end (using approximately 70GB from the last 1.5 months of usage)
It's slow. After having so much data stored in the objects directory, commits, pushes and pulls are slow as it tries to compress/uncompress and parse all this data.

I'm looking for either a solution to make Git more effective for this application, or a replacement. Here are some of the reasons I chose to use Git:

Incremental backups - I don't have to save the whole 8GB uncompressed/2GB compressed server each time I want to backup!
Cherry-pick restoration - I need to be able to restore certain parts of the server easily (such as a specific plugin configuration without restoring people's changes to the main worlds)
Ability to clone the project to home computers for offsite backup and testing
Ability to make a branch for a testing server to try unstable features before rolling them out

When we used to use a precise tarballing bash script to backup the server, we usually removed backups that were more than 2 weeks old. With incremental backups, this period should be one month or greater.

If you're unfamiliar with Minecraft's structure, it goes a bit like this:

.
|-- plugins
    |-- SomePlugin
        |-- config.yml
    |-- SomePlugin.jar
|-- world
    |-- region
        |-- (binary files of chunks, a 2000x2000 world is often 1GB in size)
    |-- mcmmo_data (third party plugin)
        |-- x coordinate
            |-- y coordinate
                |-- small flatfile
    |-- level.dat
|-- stuff.txt
|-- properties.yml
|-- server.jar

Any ideas anyone?

4 core AMD Bulldozer (3.6-3.8GHz); 8GB RAM; 128GB SSD; CentOS 6.3 x86_64 — Chris Watts, Jan 11 '13 at 14:16
Git will store the lifetime history, do you really want that? If not, a backup program would be more suited (like Amanda, for instance). — fge, Jan 11 '13 at 14:19
Ever considered a (damdamdaaaam...) database? Also it sounds like you are saving much binary data. Thats usually not what vcs are designed for. — KingCrunch, Jan 11 '13 at 14:19
As with the above comments, having JAR files in source version control doesn't make any sense. Consider using an Artifactory/Package Repository system for hosting the jar files. Although this might come as a lot of effort, Maven + Nexus are the default good choice. Also take a look at Gradle and Buildr for Maven replacement and check out their repository requirements. — hovanessyan, Jan 11 '13 at 14:26
@hovanessyan We don't necessarily want the latest revisions of plugins - but rather versions that work on our server. Besides, many plugins don't support Maven. — Chris Watts, Jan 11 '13 at 14:32
I use `rdiff-backup` for something similar (it's not Minecraft, but the scenario for what I'm doing is very similar to what you are doing). That has the advantage that you can tell it to remove all backups older than some interval, as well as using delta storage where it can to reduce space usage. — twalberg, Jan 11 '13 at 16:18

score 2 · Answer 1 · answered Jan 11 '13 at 17:03

One option you might consider, in order to store the binary/large files, is git-annex, which is designed for managing large objects inside of git. It would let you check in those huge files without cluttering the central git database itself. But, it would require some rethinking about how you mess with those files and let them change over time. It does have a nice push/pull/backup component that would likely work well, but you'll run into other "new ways of thinking" that you'll need to deal with. Try it on a test system first, of course.

Alternatives to Git for Minecraft server backup/version control

1 Answers1