How to separate development-only files from the rest with Git?

Question

I'm working on a Lua library that involves code generation and validation among other things - a lot of development code of project is written in another language.

I'd like to have one branch for library and another for rest code. Library shouldn't know it depends on something - for user's convenience first of all. Users should be able to find rest code if they read docs though - it's not private.

Branches should be synchronized: to avoid bugs like "wrote some code in project, but it doesn't affect library" - forgot to update, to make convenient CI, to make clean git history.

Current project is two folders at the root of master branch. I'd like to separate them without breaking synchronization. Could I achieve this without using separate repos? Or using separate repos fits here and it's convenient? Reference to documentation or some project with similar systems may be enough.

Two branches of same repo seems unsuitable here - it's necessary to work simultaneously on both parts. I don't want to use separate git repos - then I need to use separate git histories, they should be identical and it's just seems inconvenient to manage multiply repos - but I don't see other option. Also, I don't know, how to properly synchronize them, google provides results "how to force repos to have identical content" - that's not that I want.

Git with separated development and releases is about how to develop in private, then push to public - I need both in parallel. The rest seem irrelevant to me.

The main thing I don't like in approaches I know is that most changes will require several commits - one for each branch/repo. Duplication (e.g. exactly the same messages) always leads to mistakes. Can it be avoided?

I've read question rules, it seems, my question is open and it's forbidden. Also, it doesn't look like helpful to other (question, answers are helpful for people like me, who didn't know about some git possibilities). Should I delete this?

Answer 1

I acknowledge that I don't know all latest trends, but intuitively, I would have a branch called integration or something like that and have two branches off of that - one for library development and the other for rest code so they can both push their changes to the integration branch. If you have multiple developers, they can branch off of their respective specialty - library or rest.

Answer 2

You can carry separated histories with the main history, all in one repository. Easiest is to not use the git submodule helper at all, it's a grab bag of tools for wrangling separate histories in separate repositories, but bypassing it pops some warnings and the boilerplate to silence them is also easy, so I've included it here.

Say your support code is in support. Quickest way to split the support code into an incorporated submodule is, from a clean checkout:

git branch support $(git commit-tree -m 'Split support code to separate history' @:support)
git rm -r support
git worktree add support
git submodule add support
git config -f .gitmodules submodule.support.url ./
git add .gitmodules
git commit -m 'Split support code to separate history'

A fresh clone of the split history will have an empty support directory, to populate it you

git submodule init
git worktree add support :support

Easiest way to switch between commits with the older and newer structure is to make separate work trees for them, either with worktree add or a local clone.

p.s. if you have enough history in your support code that you want to just convert the entire repository to the new structure, see here, which I wrote before the git worktree command was built.

Answer 3

Take a look at git subtree. It does not meet all your needs, but I think it's an option. git subtree seems deprecated because its manual cannot be found in the online docs. The command git subtree is still available and can work as described in its manual which we can find by git subtree --help.

Let's suppose that your current repository has two folders lib/ and src/ and it already has an existing branch main.

We can split main into two separate branches, lib_split for lib/ and src_split for src/.

git switch main
git branch lib_split $(git subtree -P lib split main)
git branch src_split $(git subtree -P src split main)

Now we have three different and unrelated branches. On main, the directory structure is like

.
├── lib
│   └── a.so
└── src
    └── a.lua

On lib_split, it's like

.
└── a.so

And on src_slit,

.
└── a.lua

Additionally, we can introduce another branch which you may use. Let's call it main_merge and create it from main.

git checkout --orphan main_merge main
git rm -rf lib src
git commit -m'init main_merge' --allow-empty

This new branch is used to combine lib_split and src_split.

git switch main_merge
# First add the subtrees
git subtree -P lib add lib_split
git subtree -P src add src_split

# If later lib_split and src_split get updated and have new commits,
# we can merge them to main_merge
git subtree -P lib merge lib_split
git subtree -P src merge src_split

If main gets updated, we can also update lib_split and src_split.

git switch main
revision=$(git subtree -P lib split main)
git switch lib_split
git merge $revision

git switch main
revision=$(git subtree -P src split main)
git switch src_split
git merge $revision

# The following commands would raise an error.
# When using "git subtree split main",
# make sure the current branch is right main.
git switch src_split
git merge $(git subtree -P src split main)
# As "src" does not exist on src_split,
# git subtree split raises an error

You can maintain the 4 branches, or maybe the first 3 branches, or maybe more branches, in the same repository or in different repositories. It's up to your workflow.

Answer 4

After reading the answers, I've come to the conclusion that I need separate place (branch) for library, which will be automatically updated to be up to date with the dev branch - as was pointed, I want a single commit to update to update separate histories.

So I'm going to use some script for that (in CI, probably). It'll produce separate branch for users (this need is why this question arose) and I won't need to bother with synchronization - our current workflow won't be changed at all (with CI nobody will need to use scripts locally).

I'd rather avoid copies, but subtree/submodule (and there aren't other suggestions) won't fix my problem, if I managed to understand correctly. They assumes, that final result - library, which depends on dev files, will contain dev files, even if it's not directly. Worktree can help to avoid copying by using one branch for dev-only files and another for lib-only, but it's more complicated to work with (and unexpected possibility to solve problem without affecting workflow looks as a very important benefit). Thanks for answers, they helped me to formulate my problem.