5

this is my first question here and in fact one moved from StackOverflow as I think this site will be more in sync with the topic.

I've set up mirroring of my repository and it works nicely but I've had an issue recently.

The target repository was left with an unreleased lock somehow - from what I read this may be caused by an abort to the svnsync operation and I suspect it may be because in my post-commit hook I'm executing svnsync in blocking mode rather then pushing it into the background via &.

I do this so that the user can be sure that if the commit finished then it's in all repositories now but maybe it introduces a risk of them hitting cancel and stopping the commit hook?

I can't find clear guidelines or suggestions on which is better or what the best practice is or even if hitting cancel can cause an abort of the post commit hook and the sync executed from it - in most places I see people using & to kick of the sync in the background - does this prevent the lock corruption in case a user presses cancel to his commit while the sync is in progress? How do you make sure both repositories are really in sync or report issues? Do you need a separate notification mechanism for that?

Update:

From the two above options I decided to chose the third ;)

I'm calling svnsync in the background but at the same time I make the hook wait for it to finish:

svnsync ... &
wait $!

I think this nicely joins the best of both worlds but time will show how effective it'll be - please let me know what you think about the whole issue and what suggestions you might have to share about it.

Update2: It did not fix the problem - apparently when the hook is killed it also kills it's child processes :(

Would anyone have any tips on how to tackle this? I'd really like my users to be able to know and trust that when they see the commit complete it means all sites are in sync.

Update3: Time goes by and problems show up - for ~2 years my approach worked pretty well - I fired off the sync non blocking and didn't wait for it - and last week our mirror blew up probably due to 2 different commits performed by the users in the very same second (both succeeding on the master :) ) - the result was that svnsync failed to copy revision properties correctly but also created some strange revisions (it basically copied one commit and created a separate revision from it and attributed it to the svnsync user). And then complained about "someone" making changes directly to the mirror repository. With no way of removing commits in SVN I had to go from 0 to HEAD-2 which for a 162GB strong repository took a good while. Now - after a couple days of restoring actions - I modified my synchronization a bit as suggested below.

  1. In the commit hook I just touch a trigger file.
  2. Every minute in a cron job I check existence of this file - if it exists I get a "synchronization lock" and perform the commit.

This guarantees that the system is reasonable responsive and that svnsync is never called more then once - I'll update this with info once time provides more data.

RnR
  • 153
  • 6

1 Answers1

4

I would consider moving this from a post-commit hook to an incrond task, which would use inotify to monitor the repository for changes and then run the svnsync task.

As a more modular setup, it would provide me with more flexibility with a unified configuration - i.e. I wouldn't have to keep updating various users' post-commit hooks.

Docunext
  • 301
  • 1
  • 3
  • This will not work - I need the users to be sure that once they finished the commit all the mirrors are in sync - working in a cron job works in a backup scenario but it's not the case with what I have (a read caching mirror - the cache has to be up to date at all times) – RnR Sep 14 '10 at 11:11
  • Incrond isn't a cron job, its an live modification watcher built into the kernel, so it might even be faster than the post-commit hook. :-) – Docunext Sep 14 '10 at 12:48
  • Its not about speed but atomicnes of the operation. When its blocking your commits end when all sites are in sync and thats what I need. Can you do it this way? – RnR Sep 14 '10 at 23:06
  • I appreciate what you are aiming for! Its a worthy goal, but in my experience svnsync just isn't robust enough - my guess is that it halts the sync to prevent corruption of the replicated repositories. I can't tell you how many times I've run into that unreleased lock problem, so I gave up on it. Could you elaborate on why you are trying to do this, and what you are trying to avoid with the atomicity? – Docunext Oct 09 '10 at 17:32
  • Given atomicity which never worked + a week of working in backup mode without the mirror while the mirror is dump-loaded from revision 0 to HEAD-2 I had to choose cron (wouldn't need to if it wouldn't corrupt the repository completely - but man who would expect this... grrr ). Thanks :) – RnR Jan 30 '12 at 13:22
  • 1
    If commercial solutions are an option I know Subversion MultiSite does not have any of these issues. All commits are executed in the exact same order and much less bandwidth is used. http://www.wandisco.com/subversion/multisite – vinnyjames Jan 30 '12 at 22:02
  • Thanks for the info - but I must say my current solution seems to hit the sweet spot for now :) – RnR Feb 01 '12 at 07:52