Can git filter repo create a monorepo from many repos interweaving commits by date?

Question

Using git-filter-repo is it possible to combine N repositories into a mono-repository re-writing the commits so that the commits are interwoven, or "zippered" up by date? Currently, I'm testing this with only 2 repos with each repo having their own subdirectory. After the operation, the commits for each repo are on "top" of each other rather than interwoven. What I really want is to be able to have a completely linear history by authored data without the added merge commits.


rm -rf ___x
mkdir ___x
cd ___x

echo "creating the monorepo"
git init
touch "README.md"
git add .
git commit -am "Hello World!"

declare -A data
data=( 
    ["foo"]="https://github.com/bcanzanella/foo.git"
    ["bar"]="https://github.com/bcanzanella/bar.git"
)

for d in "${!data[@]}"; 
do  {
    REPO_NAME=$d
    REPO_REMOTE=${data[$d]}

    # since we can use a foo/bar as the repo identifier, replace the / with a -
    REPO_DIR_TMP="$(mktemp -d -t "${REPO_NAME/\//-}.XXXX")"

    echo "REPO REMOTE: $REPO_REMOTE"
    echo "REPO NAME: $REPO_NAME"
    echo "REPO TMP DIR: $REPO_DIR_TMP"
    echo ""

    echo "Cloning..."
    git clone "$REPO_REMOTE" "$REPO_DIR_TMP"

    echo "filtering into ..."
    cd $REPO_DIR_TMP && git-filter-repo --to-subdirectory-filter "$REPO_NAME"
    # cat .git/filter-repo/commit-map

    ## merge the rewritten repo
    git remote add "$REPO_NAME" "$REPO_DIR_TMP"

    echo "fetching..."
    git fetch "$REPO_NAME"

    echo "merging..."
    git merge --allow-unrelated-histories "$REPO_NAME/master" --no-edit

    ## delete the rewritten repo
    echo "Removing temp dir $REPO_DIR_TMP..."
    rm -rf "$REPO_DIR_TMP"

    echo "Removing remote $REPO_NAME..."
    # git remote rm "$REPO_NAME"

    echo "$REPO_NAME done!"
} 
done

This - https://stackoverflow.com/questions/40976798/how-to-merge-several-git-repos-into-one-and-interleave-histories/41190478#41190478 - seems like a possible solution, but I did not try it myself. — tmr232, Jun 08 '20 at 20:15
Reordering revisions as they are being applied to branches can _easily_ lead to conflicts. — eftshift0, Jun 08 '20 at 20:26

LeGEC · Accepted Answer · 2020-06-15T15:39:55.477

To emphasize on eftshift0's comment : rebasing and rewriting history can lead to commits being ordered in seemingly absurd chronoogical order.

If you know for a fact that all commits are well ordered (e.g : the commit date of a parent commit is always "older" than the commit date of its child commit), you may be able to generate the correct list of commits to feed in a git rebase -i script.

[edit] after thinking about it, this may be enough for your use case :

Look at the history of your repo using --date-order :

git log --graph --oneline --date-order

If the sequence of commits matches what you expect, you can use git log to generate a rebase -i sequence script :

# --reverse   : 'rebase -i' asks for entries starting from the oldest
# --no-merges : do not mention the "merge" commits
# sed -e 's/^/pick /' : use any way you see fit to prefix each line with 'pick '
#        (another valid way is to copy paste the list of commits in an editor,
#         and add 'pick ' to each line ...)
git log --reverse --no-merges --oneline --date-order |\
  sed -e 's/^/pick /' > /tmp/rebase-apply.txt

Then rebase the complete history of your repo :

git rebase -i --root

In the editor, copy/paste the script you created with your first command, save & close.

Hopefully, you will get a non conflicting unified history.

interesting solution, i didn't think of programmatically creating an ordered set of commits for a rebase. — brianc, Jun 16 '20 at 15:23

Can git filter repo create a monorepo from many repos interweaving commits by date?

1 Answers1