Git LFS - fatal error different files missing on 2 different machines

Question

I was migrating my existing repository to git lfs and it seems like somehow I managed to create a perfect storm.

There are currently two machines: A and B.

Machine A is missing the contents of the file X. I get an error when calling git lfs fetch --all origin. The machine B is missing the contents for file Y.

I try calling git lfs push --all origin on both. It doesn't work on machine A because it doesn't have file X. It doesn't work on machine B, because it doesn't have file Y.

How can I possibly resolve this situation. I have no idea how this happened or even could possibly happen.

---

Output of git lfs push on one of the machines:

Git LFS: (0 of 982 files, 1300 skipped) 0 B / 1.73 GB, 1.09 GB skipped         
d8d0edd8e03f523ab08de27e72a17272ddd24170764e0a9f836c8ba95cf73006 does not 
exist in .git/lfs/objects. Tried 
Assets/weapons/models/at_mine/textures/1k/at_mine__normal_1k.bmp, which 
matches d8d0edd8e03f523ab08de27e72a17272ddd24170764e0a9f836c8ba95cf73006.

Output of git lfs fetch on one of the machines:

git lfs fetch --all
Scanning for all objects ever referenced...
* 2319 objects found
Fetching objects...
Git LFS: (207 of 207 files, 722 skipped) 524.01 MB / 524.51 MB, 870.82 MB 
skipped
[d8d0edd8e03f523ab08de27e72a17272ddd24170764e0a9f836c8ba95cf73006] Object 
does not exist on the server: [404] Object does not exist on the server
[df1dbb12f35f5392c157beebbc3613f70993c691a68f4cc65033ade528de3418] Object 
does not exist on the server: [404] Object does not exist on the server
[edaf4f6721fb14ce4d38d3adddc86aaaf6fbf0c7db11da451022a4f74af97f30] Object does not exist on the server: [404] Object does not exist on the server
... and so on and so on

I was performing the migration to LFS using bfg repo cleaner.

** EDIT **

The steps I performed:

1. Git clone of the entire repo in its old form (from before the conversion) to Machine A.
2. I run BFG on Machine A that essentially rewrote the history of the master branch (no other branches kept remotely).
3. I run git push origin master --force as you are supposed to replace the old version of master with a new version of master.
4. I run git fetch on Machine B to get all the files.
5. There were some commits on Machine B, in master that didn't get sent to the server yet - because they would cause the repo to exceed the quota. Therefore I run git rebase --onto origin/master HEAD~3 HEAD to move the unsent new commits to the new version of master.
6. I run git push origin master on Machine B.

This is where I am at now.

Answer 1

So this probably isn't related to the state of the LFS objects, but could cause some trouble with the clean-up attempt. The rebase command you show for Machine B leaves you in a somewhat awkward state. If you'd originally had

A --- B --- C --- D <--(master)

and then you fetched the results of the rewrite, giving

A --- B --- C --- D <--(master)

A' --- B' <--(origin/master)

the rebase command you specified, because it listed HEAD as the ref to rebase from, would put you in a detached head state with

A --- B --- C --- D <--(master)

A' --- B' <--(origin/master)
        \
         C' --- D' <--(HEAD)

so pushing master should fail regardless of LFS troubles. If you want to keep track of D for post-cleanup validation of some sort, you could tag it; and then with HEAD still at D' run

git branch -f master

to get

A --- B --- C --- D <--[old-master])

A' --- B' <--(origin/master)
        \
         C' --- D' <--(master)

But like I said, if each machine is missing some LFS objects, then this will not fix that. Now I'm assuming that both machines are properly configured with git-lfs. I've seen rather odd behavior if lfs install hadn't been run when it needed to have been; but unless you were doing something obscure, it's unlikely that would lead to where you are... so...

Because LFS objects are identified using a SHA-256, that two files with matching id are the same file is a ridiculously safe assumption. So one thing you can try is to find the file that is missing from Machine B under .git/lfs/objects on Machine A, and manually copy it. (Or vice versa.) If, between the two, you have a complete set of LFS objects, then the situation can be fixed in this way.

To find an object with ID d8d0edd8e03f523ab08de27e72a17272ddd24170764e0a9f836c8ba95cf73006 you would look in .git/lfs/objects/d8/d0/d8d0edd8e03f523ab08de27e72a17272ddd24170764e0a9f836c8ba95cf73006

If an object is missing on both machines, then you can try to figure out the corresponding file from the original repo and add it to one of the local LFS caches manually.

git lfs clean <original.file

Answer 2

So, apparently the whole trouble stems from the fact that I used BFG to convert my repository. DON'T DO IT. As far as I can tell it may brick your repository, the way it did brick mine. Use git-lfs-migrate.

I originally used BFG because it was easier to understand how to use it. However it has an adverse effect of not putting .gitattributes in the rewritten commits that causes all sorts of problems if you ever happen to touch or branch anything in the commits other than the latest. You also have to be extra careful to change .gitattributes manually to match the BFG command.

Basically don't use BFG for that, there is no reason to do so.

And first and foremost: MAKE A BACKUP before you attempt to rewrite the history. And make sure you do that right because forks in Bitbucket.org and some similar websites don't handle LFS correctly.