Very large TFS repo — from a subfolder

Question

I will tremendously appreciate all advice on how to proceed with the following task at hand. I have read fairly thoroughly the documentation on git-tfs clone, git-tfs quick-clone and git-tfs branch but still haven't been able to break into this problem.

Our TFS codebase is TERRIBLY LARGE (it is so because for odious reasons it contains large BLOBs and such). It is however fairly well organized and has the following structure:

 $/TeamProject/Dev (TERRIBLY LARGE)
  |
  +- $/TeamProject/Dev.EpicX (TERRIBLY LARGE)
  |
  +- $/TeamProject/Dev.EpicY (TERRIBLY LARGE)
  |
  +- $/TeamProject/Dev.EpicZ (TERRIBLY LARGE)

Each one of these branches are "legit" TFS branches (-vs- just branched "folders"). We can think of the Dev branch as the main integration branch. These branches contains our solutions and projects, as well as several other resources (as I mentioned, BLOBs, etc...)

As I said, due to the ridiculous size of these branches, most of us do not even bother fetching a branch in its entirety, but rather only the corresponding directories or solutions we are working on. For example, I am working at $/TeamProject/Dev.EpicY/Foo/Bar/SolutionDirectory in the Dev.EpicY branch. The size of the sources under SolutionDirectory is much more manageable (~200MB). This is the directory I need to create a git repo to do work on while enjoying all git goodness (local branches and so on) without affecting my colleagues who will remain using TFS.

Above all: I need to be able to forward-integrate by "merging" from the corresponding "parent" $/TeamProject/Dev/Foo/Bar/SolutionDirectory which is under constant development.

What I have tried

1st attempt: Clone with all branches

I'm not interest in past history of the project, so I am thinking to use git tfs quick-clone.

git tfs quick-clone http://tfs-server/Collection $/TeamProject/Dev.EpicY/Foo/Bar/SolutionDirectory . --branches=all

followed by:

git tfs quick-clone http://tfs-server/Collection $/TeamProject/Dev/Foo/Bar/SolutionDirectory . --branches=all

When I do this I didn't get the parent-child branching relationship between Dev and Dev.EpicY. For example:

git tfs branch

Git-tfs remote details:
default -> http://tfs-server/Collection $/TeamProject/Dev/Foo/Bar/SolutionDirectory
refs/remotes/tfs/default - 04ddfd8641096a2d02eed4c087423bc0cdeb4ed7 @ 44016

2nd attempt: Initialize the branches explicitly

After cloning. Now I even get an error:

git tfs branch --init --all

error: The use of the option '--branches=all' to init all the branches is only possible 
when 'git tfs clone' was done from the trunk!!! '$/TeamProject/Dev/Foo/Bar/SolutionDirectory'
is not a TFS branch!

3rd attempt: Go to TFS and convert the sub-folders to branches

It seems that for git tfs the distinction between regular folders vs branches in TFS is significant, so I went to TFS and converted SolutionDirectory to a branch, along with it's hierarchy:

 $/TeamProject/Dev/Foo/Bar/SolutionDirectory
  |
  +- $/TeamProject/Dev.EpicX/Foo/Bar/SolutionDirectory
  |
  +- $/TeamProject/Dev.EpicY/Foo/Bar/SolutionDirectory
  |
  +- $/TeamProject/Dev.EpicZ/Foo/Bar/SolutionDirectory

Now when I ran git tfs branch --init --all there was some progress, but it still failed with a weird error:

git tfs branch --init --all

Tfs branches found:
- $/TeamProject/Dev.EpicY/Foo/Bar/SolutionDirectory
=> Working on TFS branch : $/TeamProject/Dev.EpicY/Foo/Bar/SolutionDirectory
Branches to Initialize successively :
-$/TeamProject/Dev.EpicY/Foo/Bar/SolutionDirectory (43506)
The name of the local branch will be : Dev.EpicY/Foo/Bar/SolutionDirectory
error: an error occurs when initializing the branch. Branch is ignored and continuing...
=> Working on TFS branch : $/TeamProject/Dev/Foo/Bar/SolutionDirectory
warning: Some Tfs branches could not have been initialized:
- $/TeamProject/Dev.EpicY/Foo/Bar/SolutionDirectory

Please report this case to the git-tfs developers! (report here : https://github.com/git-tfs/git-tfs/issues/461 )
warning: Some Tfs branches could not have been initialized or entirely fetched due to errors:
- $/TeamProject/Dev.EpicY/Foo/Bar/SolutionDirectory
   =>error:error: Couldn't fetch parent branch

Fernando Espinosa · Accepted Answer · 2016-06-07T16:06:49.603

The documentation is somewhat terse but I finally succeeded in achieving the precise objective of creating a git repo with only the necessary history and branched codebase from TFS subdirectories.

It turns out that the recommendation to clone a trunk does not mean you have to clone the entire history of the trunk and its branches, not that you are forced to stay in the top-level TFS folder as branch.

Indeed, in order for git-tfs to pick up a branch from TFS, it must be a TFS branch (not just a folder that has been branched). But you can temporarily convert the desired SolutionDirectory folders into TFS branches! This is the key.

This is the correct order of actions to take:

Identify the TFS changeset when $/TeamProject/Dev was branched out to $/TeamProject/Dev.EpicY for development of ~/Foo/Bar/SolutionDirectory
- Let's say this changeset is: 43541
Ensure both $/TeamProject/Dev/Foo/Bar/SolutionDirectory and $/TeamProject/Dev.EpicY/Foo/Bar/SolutionDirectory are TFS branches

Assuming that you want to initialize the git repository in the current directory run the following command:

PS> git tfs clone --changeset=43541 --branches=all `
    http://tfs-server/Collection $/TeamProject/Dev/Foo/Bar/SolutionDirectory .

The output displayed will show how the TFS branch is picked up along thew way and the corresponding commits are fetched

Initialized empty Git repository in E:/git-tfs/SolutionDirectory/.git/
Fetching from TFS remote 'default'...
1 objects created...
C43541 = 731d29764d88e424b9d6dfb9a34c107aa4cca9c3
C43608 = aebc94b96079e73e88ea74ed859eec65440c3b03
C43609 = 64c03137f555345ec7f24fefc992162e6e082a98
...
C44016 = e1b5c55efa528733ecaa3afba31b05cf1a310cb4
Tfs branches found:
- $/TeamProject/Dev.EpicY/Foo/Bar/SolutionDirectory
=> Working on TFS branch : $/TeamProject/Dev.EpicY/Foo/Bar/SolutionDirectory
Branches to Initialize successively :
-$/TeamProject/Dev.EpicY/Foo/Bar/SolutionDirectory (43506)
The name of the local branch will be : Dev.EpicY/Foo/Bar/SolutionDirectory
C43541 = 73589d08398415549fbad191b06c55a272c7ca37
C43666 = c4bbe9b09138331041cb958cbc8b89f16f7a2902
C43670 = e7c1e4c787c6c4a745baed46da0eb5d0e3f2fc79
...
C44019 = b2401575dec8347e1debc701073f523aa09e668c
=> Working on TFS branch : $/TeamProject/Dev/Foo/Bar/SolutionDirectory

Be happy :)

score 1 · Answer 2 · answered May 06 '16 at 10:30

The clone command creates a new git repository, initialized from a TFS source tree and fetch all the changesets, which contains --branches=VALUE synopsis.

The quick-clone command creates a new git repository, initialized from the last changeset (or a specific changeset in history) in a TFS source tree, ignoring the full history, which doesn't contain --branches=VALUE synopsis.

And, to use feature Clone all the branches, all your source code folders corresponding to branches should be converted into branches.

The error message has indicated the error "error: The use of the option '--branches=all' to init all the branches is only possible when 'git tfs clone' was done from the trunk!!!", in my opinion, you need to use Clone only the trunk (with dependency branches), check: https://github.com/git-tfs/git-tfs/blob/master/doc/commands/clone.md

You are right, the _branched folders_ in TFS need to be converted to _TFS branches_ (silly distinction) in order to be picked up by the process. But it turns out I can always reset the branchness of a TFS folder at will and take advantage of this to clone ONLY the desired amount of code and history, as I describe below. Thanks anyways for your answer. — Fernando Espinosa, Jun 07 '16 at 17:26

Very large TFS repo — from a subfolder

What I have tried

1st attempt: Clone with all branches

2nd attempt: Initialize the branches explicitly

3rd attempt: Go to TFS and convert the sub-folders to branches

2 Answers2