Remove a file from git history using git-filter-repo on a fresh clone

Question

I'm following this answer to remove a single file containing credentials from git history. I have git 2.35.1 and filter-repo 22826b5a68b6. The command I need is apparently:

git-filter-repo --path auth.json --invert-paths

If I try to apply this to my working repo, I get this error:

Aborting: Refusing to destructively overwrite repo history since
this does not look like a fresh clone.
(expected freshly packed repo)

So I check out a fresh copy with git clone, and the command runs successfully:

Parsed 861 commits
New history written in 0.69 seconds; now repacking/cleaning...
Repacking your repo and cleaning out old unneeded objects
HEAD is now at 7212384 Update app.css
Enumerating objects: 8203, done.
Counting objects: 100% (8203/8203), done.
Delta compression using up to 24 threads
Compressing objects: 100% (2310/2310), done.
Writing objects: 100% (8203/8203), done.
Total 8203 (delta 5630), reused 8196 (delta 5623), pack-reused 0
Completely finished after 2.85 seconds.

And I can see that the file has been removed. But when I go to push:

git push --force
fatal: No configured push destination.

for some reason it's lost the remote it cloned from, so I add it back in manually:

git remote add origin git@git.example.com:abc/xyz.git

This fails with:

fatal: The current branch master has no upstream branch.

so I add that with

git push --set-upstream origin master

but this fails too:

To git.example.com:abc/xyz.git
 ! [rejected]        master -> master (fetch first)
error: failed to push some refs to 'git.example.com:abc/xyz.git'
hint: Updates were rejected because the remote contains work that you do
hint: not have locally. This is usually caused by another repository pushing
hint: to the same ref. You may want to first integrate the remote changes
hint: (e.g., 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

but I know that nothing has been pushed to this repo since checking it out. Repeating the process has the same result. If I do a git pull to update it, it then fails again with the this does not look like a fresh clone error, right back where I started.

I went round and around this a few times, eventually getting past all the errors, only to find that it has all made no difference to my repo – the file is still there.

So my question is what are the exact steps I should do to make this filtering process work on a freshly cloned repo?

Answer 1

You were so close...

The reason you need to git push --force in the previous step is because you are going to blow away commits on the remote and replace them with your new ones. Since your remote is gone, skip the first force push command, and then you simply need to add force to your final push command:

git push --set-upstream origin master --force

Side Note: I almost always prefer using --force-with-lease over --force, as it's slightly safer in that it will error if someone added new commits to the remote branch between the time when you last fetched (or in this case, cloned) and pushed, that you haven't seen yet. It might be rude to just blow them away. When using --force-with-lease if you get the error, just do git fetch, look at the new commits and decide if you're OK with deleting them. If you are, then use --force-with-lease again and it will work (unless new commits appeared again in the last minute since you fetched).

In this particular case where you are re-adding your remote, you must fetch first or else the --force-with-lease will not work, and if it were me I would probably consider doing this if there was a possibility of new commits appearing on the remote between the time you cloned and when you are about to force push your rewritten repo. In that case I would change your final command to these steps:

git fetch
# inspect origin/master to see if new commits appeared after your clone
git push --set-upstream origin master --force-with-lease

Or, perhaps in your case, as soon as you decide you're going to rewrite a branch, temporarily lock the branch (or remove permissions to it), and unlock it after your force push. Then you know for sure no one is adding commits until you're done.

Answer 2

TTT's answer helped, in particular the comment about filter-repo doing a git init – it was the order of operations that was the problem. I did this so many times before it worked, I turned it into a script to make it clear exactly what's needed and in what order:

#!/usr/bin/env bash
set -xv
git clone git@git.example.com:abc/xyz.git project
cd project
git filter-repo --path auth.json --invert-paths
git remote add origin git@git.example.com:abc/xyz.git
git push --set-upstream origin main --force

After doing this, I ran into lots of issues updating existing clones, but generally they were solved by accepting all changes from the remote.

Answer 3

To fix the issue, you need to preserve your .git/config file before running git-filter-repo, and then revert it back to the original state, e.g.:

git clone <...> .
cp ./.git/config ../
git filter-repo --invert-paths --path <...>
mv -f ../config ./.git
git push --force --all