How can I keep some modifications from propagating in mercurial?

Question

I am developing a web database that is already in use for about a dozen separate installations, most of which I also manage. Each installation has a fair bit of local configuration and customization. Having just switched to mercurial from svn, I would like to take advantage of its distributed nature to keep track of local modifications. I have set up each installed server as its own repo (and configured apache not to serve the .hg directories).

My difficulty is that the development tree also contains local configuration, and I want to avoid placing every bit of it in an unversioned config file. So, how do I set things up to avoid propagating local configuration to the master repo and to the installed copies?

Example: I have a long config.ini file that should be versioned and distributed. The "clean" version contains placeholders for the database connection parameters, and I don't want the development server's passwords to end up in the repositories for the installed copies. But now and then I'll make changes (e.g., new defaults) that I do need to propagate. There are several files in a similar situation.

The best I could work out so far involves installing mq and turning the local modifications into a patch (two patches, actually, with logically separate changesets). Every time I want to commit a regular changeset to the local repo, I need to pop all patches, commit the modifications, and re-apply the patches. When I'm ready to push to the master repo, I must again pop the patches, push, and re-apply them. This is all convoluted and error-prone.

The only other alternative I can see is to forget about push and only propagate changesets as patches, which seems like an even worse solution. Can someone suggest a better set-up? I can't imagine that this is such an unusual configuration, but I haven't found anything about it.

Edit: After following up on the suggestions here, I'm coming to the conclusion that named branches plus rebase provide a simple and workable solution. I've added a description in the form of my own answer. Please take a look.

I asked the same question on programmers. http://programmers.stackexchange.com/questions/117998/how-to-manage-ide-project-files-on-a-forked-repository — sylvanaar, Mar 04 '12 at 11:25
@sylvanaar, thanks for the pointer. But it's not the same question, since my local changes sit inside versioned files. Also, I believe git has support for selective pushing that mercurial lacks. I'm looking for a mercurial solution. — alexis, Mar 04 '12 at 11:31
I know it sounds different, but it really is the same use case. I wanted to version my IDE configuration files without having them pushed upstream. The answer was basically, structure your directory layout to keep the files separate. BTW, i do what you are talking about with MQ and it is truly a hassle. — sylvanaar, Mar 04 '12 at 11:33
Here is another idea: http://mercurial.selenic.com/wiki/ExcludeExtension — sylvanaar, Mar 04 '12 at 11:41
I can already keep some files out of version control, that's what I want to avoid. What I was doing under svn was to maintain templates in a directory, which are copied to unversioned files and modified. This means that changes to the templates (e.g., new defaults) never make it to the installed copies. A DVCS should be able to do more. — alexis, Mar 04 '12 at 11:41
Thanks for ExcludeExtension. It would allow me to automate what I was (also) doing under svn: Never committing certain files once the repository is set up. Can there really be no better way to do this? — alexis, Mar 04 '12 at 11:51

Wim Coenen · Answer 1 · 2012-03-04T13:59:55.460

2

From your comments, it looks like you are already familiar with the best practice for dealing with this: version a configuration template, and keep the actual configuration unversioned.

But since you aren't happy with that solution, here is another one you can try:

Mercurial 2.1 introduced the concept of Phases. The phase is changeset metadata marking it as "secret", "draft" or "public". Normally this metadata is used and manipulated automatically by mercurial and its extensions without the user needing to be aware of it.

However, if you made a changeset 1234 which you never want to push to other repositories, you can enforce this by manually marking it as secret like this:

hg phase --force --secret -r 1234

If you then try to push to another repository, it will be ignored with this warning:

pushing to http://example.com/some/other/repository
searching for changes
no changes found (ignored 1 secret changesets)

This solution allows you to

version the local configuration changes
prevent those changes from being pushed accidentally
merge your local changes with other changes which you pull in

The big downside is of course that you cannot push changes which you made on top of this secret changeset (because that would push the secret changeset along). You'll have to rebase any such changes before you can push them.

edited Mar 04 '12 at 13:59

answered Mar 04 '12 at 13:41

Wim Coenen

66,094
13
157
251

Thanks, I've got hg 2.0.2 now but this looks promising, and it is simpler than using patches. But I see that having a secret revision will force me to branch, commit and rebase for every single commit from then on. Could you suggest the right graft/transplant/rebase procedure? There seem to be so many options. – alexis Mar 04 '12 at 14:33
@alexis: I feel `graft` would be the best choice since that's a core command. But to be honest, I personally prefer the solution with versioned templates and unversioned config files, so I have no experience with all the ugly details of the approach I describe here. – Wim Coenen Mar 04 '12 at 14:56
Thanks! If that's considered Best Practice I'll think about it more carefully. I've been using templates for years and they've led me develop a layer of installation scripts, which don't work OOTB on Windows (they use make and unix tools). – alexis Mar 04 '12 at 15:00
Occam Razor is opposed to unnatural solutions – Lazy Badger Mar 04 '12 at 15:58

score 1 · Answer 2 · answered Mar 04 '12 at 16:49

If the problem with a versioned template and an unversioned local copy is that changes to the template don't make it into the local copies, how about modifying your app to use an unversioned localconfig.ini and fallback to a versioned config.ini for missing parameters. This way new default parameters can be added to config.ini and be propagated into your app.

alexis · Accepted Answer · 2013-03-19T13:13:46.863

Having followed up on the suggestions here, I came to the conclusion that named branches plus rebase provide a simple and reliable solution. I've been using the following method for some time now and it works very well. Basically, the history around the local changes is separated into named branches which can be easily rearranged with rebase.

I use a branch local for configuration information. When all my repos support Phases, I'll mark the local branch secret; but the method works without it. local depends on default, but default does not depend on local so it can be pushed independently (with hg push -r default). Here's how it works:

Suppose the main line of development is in the default branch. (You could have more branches; this is for concreteness). There is a master (stable) repo that does not contain passwords etc.:
```
---o--o--o   (default)
```
In each deployed (non-development) clone, I create a branch local and commit all local state to it.
```
...o--o--o   (default)
          \
           L--L    (local)
```
Updates from upstream will always be in default. Whenever I pull updates, I merge them into local (n is a sequence of new updates):
```
...o--o--o--n--n    (default)
          \     \
           L--L--N     (local)
```
The local branch tracks the evolution of default, and I can still return to old configurations if something goes wrong.
On the development server, I start with the same set-up: a local branch with config settings as above. This will never be pushed. But at the tip of local I create a third branch, dev. This is where new development happens.
```
...o--o   (default)
       \
        L--L    (local)
            \
             d--d--d     (dev)
```
When I am ready to publish some features to the main repository, I first rebase the entire dev branch onto the tip of default:
```
hg rebase --source "min(branch('dev'))" --dest default --detach
```
The previous tree becomes:
```
...o--o--d--d--d   (default)
       \
        L--L    (local)
```
The rebased changesets now belong to branch default. (With feature branches, add --keepbranches to the rebase command to retain the branch name). The new features no longer have any ancestors in local, and I can publish them with push -r default without dragging along the local revisions. (Never merge from local into default; only the other way around). If you forget to say -r default when pushing, no problem: Your push gets rejected since it would add a new head.
On the development server, I merge the rebased revs into local as if I'd just pulled them:
```
...o--o--d--d--d   (default)
       \        \
        L--L-----N    (local)
```
I can now create a new dev branch on top of local, and continue development.

This has the benefits that I can develop on a version-controlled, configured setup; that I don't need to mess with patches; that previous configuration stages remain in the history (if my webserver stops working after an update, I can update back to a configured version); and that I only rebase once, when I'm ready to publish changes. The rebasing and subsequent merge might lead to conflicts if a revision conflicts with local configuration changes; but if that's going to happen, it's better if they occur when merge facilities can help resolve them.

Are you still following this procedure? I'm quite conflicted on how to manage configuration files across environment branches—template don't feel right as one then has a number of unversioned configuration files; patches or the ExcludeExtension feel like a total cludge, and rebasing feels like a lot of work. A good answer is tougher than I expected. — Matt, Mar 18 '13 at 23:16
Yes I am. I'm managing almost a dozen clones of my project, each with local customizations as well as its connection settings, and I'm very happy with this set-up. Instead of step 3 (and 6), I used to rebase the local branch so it's always branching off the tip of default. That gives cleaner-looking history, but it made it impossible to simply revert to an older _configured_ state when something goes wrong. So, the clones' repos contain the history of the configured versions as well as that of the distributed software. — alexis, Mar 19 '13 at 13:11

Lazy Badger · Answer 4 · 2012-03-04T16:19:40.057

0

1 Mercurial have (follow-up to comments) selective (string-based) commit - see Record Extension

2 Local changes inside versioned public files can be easy received with MQ Extension (I do it for site-configs all time). Your headache with MQ

Every time I want to commit a regular changeset to the local repo, I need to pop all patches, commit the modifications, and re-apply the patches. When I'm ready to push to the master repo, I must again pop the patches, push, and re-apply them.

is a result of not polished workflow and (some) misinterpretation. If you want commit without MQ-patches - don't do it by hand. Add alias for commit, which qop -all + commit and use this new command only. And when you push, you may don't worry about MQ-state - you push changesets from repo, not WC state. Local repo can also be protected without alias by pre-commit hook checking content.

3 You can try LocalBranches extension, where your local changes stored inside local branches (and merge branches on changes) - I found this way more troublesome, compared to MQ

edited Mar 04 '12 at 16:19

answered Mar 04 '12 at 15:57

Lazy Badger

94,711
9
78
110

Right, the RecordExtension enables smaller-than-file granularity for commits. But I was referring to flagging revsets as "share" or "don't share" (which is what hg phases do, as @Wim points out). Could you please elaborate on how you use mq to hold local changes? I've tried but the only way I found seems very clunky. – alexis Mar 04 '12 at 16:27
@alexis - well, I use *almost like you*. 1 - I use THG 2 - I have vanilla file in repo 3 - Two **indepenent patches** in MQ - one per site 4 - work all time with both patches popped 5 - on deploy (by hand for now): push --move first, copy patched file from WC, pop, push --move second, copy file, pop. It's good for 2 sites, on increased amount I'll think (maybe) about parametrized alias for qpush --move + copy – Lazy Badger Mar 04 '12 at 16:59
"Every time I want to commit a regular changeset to the local repo, I need to pop all patches" -- I'm sorry but if you wanna commit something then your working dir is dirty -- pop on a dirty working dir is not allowed (nasty "use refresh" warning appears). What did I miss? – springy76 Mar 28 '12 at 11:45
@springy76 - Why you asked **me**?! I **quoted OP** and clearly (I hope) show it in my answer – Lazy Badger Mar 28 '12 at 12:00

How can I keep some modifications from propagating in mercurial?

4 Answers4

Linked