3

Im using reposurgon which reads svn dump's, created simply by:

svnadmin dump /svnroot/my-repo > svn_dump

However I need to update this file because we are still committing to the repository and re-creating the entire file each time is quite time consuming (its almost 60gig).

So my question is:

Is there a way to append onto this file each time to avoid fully re-creating the dump?

Ideally this would be possible without manually having to record the ranges dumped each time

Update

I manged to do this using incremental dumps, but the first 4 lines of each file after the first need to be removed.

svnadmin dump -r0 /svnroot/my-repo --incremental                   > svn_dump
svnadmin dump -r1 /svnroot/my-repo --incremental | sed -n '5,$ p' >> svn_dump
svnadmin dump -r2 /svnroot/my-repo --incremental | sed -n '5,$ p' >> svn_dump
svnadmin dump -r3 /svnroot/my-repo --incremental | sed -n '5,$ p' >> svn_dump
svnadmin dump -r4 /svnroot/my-repo --incremental | sed -n '5,$ p' >> svn_dump

# is the same as:
svnadmin dump -r0:4 /svnroot/my-repo                               > svn_dump

However this seems fairly add-hoc, and for it to be useful Id need to write a more comprehensive script.

bahrep
  • 29,961
  • 12
  • 103
  • 150
ideasman42
  • 42,413
  • 44
  • 197
  • 320

2 Answers2

1

Apache Subversion 1.8 added support for incremental svnadmin hotcopy backups. Prior to version 1.8, svnadmin hotcopy was able to create only complete backup, refusing to copy over existing hotcopied repository.

The major benefit of using svnadmin hotcopy instead of svnadmin dump / load is it's performance which is limited to your disk I/O speed only.

bahrep
  • 29,961
  • 12
  • 103
  • 150
  • `svnadmin hotcopy` makes a copy of the repository, not an svn dump (the question explicitly states that a an svn dump is needed, since this is what reposurgon reads) – ideasman42 Nov 04 '13 at 11:57
0

You dumpfile is really big. Are you storing a lot of binary files in your repository? If so you may want to consider storing binaries in another location.

Anyway, there is the possibility of an incremental dump. From the svn manual:

Two useful options modify the dump file generator's behavior. The first is the --incremental option, which simply causes that first revision in the dump stream to contain only the files and directories modified in that revision, instead of being presented as the addition of a new tree, and in exactly the same way that every other revision in the dump file is presented. This is useful for generating a relatively small dump file to be loaded into another repository that already has the files and directories that exist in the original repository.

The second useful option is --deltas. This option causes svnadmin dump to, instead of emitting full-text representations of file contents and property lists, emit only deltas of those items against their previous versions. This reduces (in some cases, drastically) the size of the dump file that svnadmin dump creates. There are, however, disadvantages to using this option—deltified dump files are more CPU-intensive to create, cannot be operated on by svndumpfilter, and tend not to compress as well as their nondeltified counterparts when using third-party tools such as gzip and bzip2.

You can have a script that uses this functionality like:

$ svnadmin dump /var/svn/repos -r 21 --incremental > incr.dump
* Dumped revision 21.

You can also use the other useful thing - the deltas! For example you can have a weekly full backup and daily - incremental and deltas backups.

However if you are storing big amounts of binary data be aware that deltas will be very ineffective. The way that svn works internally is to blame here - it is just not meant to store binary files.

The general answer is that you cannot directly do this - i.e. you cannot append to a dump file. That's it.

Borislav Sabev
  • 4,776
  • 1
  • 24
  • 30
  • Yes, I have read this section of the svn manual already, but this answer doesn't explain how you would update an existing svn dump. Perhaps dumping every revision incrementally then appending together. but this means the entire dump is stored twice - once as small files, and again as one large file. Maybe its obvious - but for example, how would you apply a delta to an existing file? - What are the exact steps to update the dump? Also, the size of the repository is beside the point (projects use svn for binary files - its a valid use case - games, movies etc). – ideasman42 Nov 04 '13 at 01:11
  • Well the answer is that you cannot directly do this - i.e. you cannot append to a dump file. That's it. – Borislav Sabev Nov 04 '13 at 06:18
  • it can be done, see the update in my original post, however `svnadmin` doesn't support it out of the box (it seems). It shouldn't be too hard to inspect the dump, find the last written revision (or failing that - simply record it somewhere), then append all new revisions to the dump (probably <100 LOC script). – ideasman42 Nov 04 '13 at 08:18