2

Is there a way to push huge amounts of data to github? I have a massive collection of files I am trying to backup, but when I try to push them, git will timeout or stop working in the middle of it.

Current solution: Commit and push small portions individually (At most 1 to 2 Gigabytes at a time)

Additional Information:

  • It is a private repository being accessed using ssh.
  • I am using git gui and git bash
  • Attempting to backup 30 GB of data

What I want to do: Commit all changes, and push all data overnight. (Approx: 6hrs)

Jaedon Lippert
  • 51
  • 4
  • 1
    This doesn't sound like a job for Git. You could use GitLFS, but maybe version control was not the purpose to begin with? Git is not `rsync`, for instance. – matt May 20 '22 at 15:34
  • 1
    I agree with @matt - fundamentally, Git is not a *backup* system, it's a *version control* system. People sometimes (ab)use it as a backup system but it's not meant to do that and does a poor job of doing that. (Also, while I'm not up on current pricing, I'd bet that using Amazon S3 with a real backup system will be *far* cheaper, especially if you opt for "slow access" storage.) – torek May 20 '22 at 22:20
  • It's worth emphasizing on "version control" part of it… if your files change a lot, the Git history is going to get _really_ fat, and cloning your repo would take far longer than simply downloading the files from a more traditional solution. You definitely don't want to store your kind of backups like this. – Seth Falco May 21 '22 at 01:04

1 Answers1

0

Git is not well suited for such a huge amount of data, and most of said data would not benefit from delta/compression of any kind (because of their binary nature).

A proper tool like rclone is a possible right tool for the job, since it can:

  • Backup (and encrypt) files to cloud storage
  • Restore (and decrypt) files from cloud storage
  • Mirror cloud data to other cloud services or locally
  • Migrate data to the cloud, or between cloud storage vendors
  • Mount multiple, encrypted, cached or diverse cloud storage as a disk

Instead of using a Git repository, you would use a Cloud backup service.

VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250