Check out single subdirectory from minimal repository using JGit

Question

I'm using JGit 6.5.x with Java 17. I have a remote repository that is huge (gigabytes), but I only need temporary access to a single subdirectory (e.g. foo/bar/) for processing. The single subdirectory is really small (hundreds of kilobytes). Cloning a shallow, bare repository is relatively small as well:

try (final Git git = Git.cloneRepository()
    .setURI(REMOTE_REPOSITORY_URI.toASCIIString())
    .setDirectory(LOCAL_RESPOSITORY_PATH.toFile())
    .setBare(true)
    .setDepth(1)
    .call()) {
  System.out.println("cloned shallow, bare repository");
}

Is there a way to clone a shallow, bare repository like that (or any other minimal version of the repository), and then check out just the single subdirectory foo/bar to some other directory temporarily so that I can process those files using the normal Java file system API?

Note that I just now succeeded in the the clone above and haven't started looking into how I might check out just a single subdirectory from this bare repository.

Interestingly, `git help clone` has *--sparse Initialize the sparse-checkout file so the working directory starts with only the files in the root of the repository. The sparse-checkout file can be modified to grow the working directory as needed.* Now I'm guessing that might not be in your API (don't see it in docs) since that might depend on FS support and therefore is not X-platform. I'm wondering if there's a way to force that if your FS can support it. For me, it halves the clone size — g00se, May 31 '23 at 18:05

Satyajit Bhatt · Answer 1 · 2023-05-31T18:36:34.240

Try below solution :

Note : Before apply any git changes make sure you have backup for necessary files.

Use the git object to create a TreeWalk that will allow you to traverse the repository's tree and find the subdirectory you're interested in. Specify the starting path as the root of the repository:

try (Git git = Git.open(LOCAL_REPOSITORY_PATH.toFile())) {
    Repository repository = git.getRepository();

    // Get the tree for the repository's HEAD commit
    RevWalk revWalk = new RevWalk(repository);
    RevCommit commit = revWalk.parseCommit(repository.resolve(Constants.HEAD));
    RevTree tree = commit.getTree();

    // Create a TreeWalk starting from the root of the repository
    TreeWalk treeWalk = new TreeWalk(repository);
    treeWalk.addTree(tree);
    treeWalk.setRecursive(true);
    
    // Specify the path of the subdirectory you want to check out
    treeWalk.setFilter(PathFilter.create("foo/bar"));

    if (!treeWalk.next()) {
        throw new IllegalStateException("Subdirectory not found");
    }

    // Get the ObjectId of the subdirectory's tree
    ObjectId subdirectoryTreeId = treeWalk.getObjectId(0);
    treeWalk.close();
    
    // Create a new Git object with the shallow, bare repository
    Git subGit = new Git(repository);

    // Checkout the subdirectory's tree to a temporary directory
    Path temporaryDirectory = Files.createTempDirectory("subdirectory");
    subGit.checkout().setStartPoint(subdirectoryTreeId.getName()).setAllPaths(true).setForce(true).setTargetPath(temporaryDirectory.toFile()).call();

    // Now you can use the Java file system API to process the files in the temporary directory
    
    // Clean up the temporary directory when you're done
    FileUtils.deleteDirectory(temporaryDirectory.toFile());
}

In the code above, we use a TreeWalk to traverse the repository's tree and find the subdirectory you specified (foo/bar). We then get the ObjectId of the subdirectory's tree and create a new Git object with the repository. Finally, we use checkout() to check out the subdirectory's tree to a temporary directory, and you can use the Java file system API to process the files in that directory. Don't forget to clean up the temporary directory when you're done.

Note that the code assumes you have the necessary JGit and Java IO imports in place.

This approach looks promising; thanks for taking the time to write it up. It may come in useful in the future. I haven't tried it because I found an even simpler, more direct approach. Technically your solution more directly answers my original question, because I asked how to check out a single directory from a bare repository, and the approach I went with doesn't use a bare repository. Still my core question related to any approach to clone a minimal repository, so I tweaked the question to reflect this. In any case, thank you for the response. — Garret Wilson, Jun 01 '23 at 15:13
@Satyajit_Bhatt the method `setForce()` has been deprecated. Did you mean `setForced()` or `setForceRefUpdate()`? — Garret Wilson, Jun 08 '23 at 18:07
@Satyajit_Bhatt I'm not finding the `setTargetPath()` method for `CheckoutCommand`. Perhaps it has been renamed or changed in the latest version? — Garret Wilson, Jun 08 '23 at 18:13

Garret Wilson · Accepted Answer · 2023-06-01T15:19:02.743

Inspired by another answer I was able get a single-depth clone and check out only a single path without needing to do a bare clone, while using similar minimal file system space. The benefit to this approach is that only a single top-level directory is needed; the bare repository approach on the other hand requires a manual traversal and saving to a separate drop-level directory.

The key is to use setNoCheckout(true) (in addition to setDepth(1)), and then after cloning manually perform a separate checkout specifying the requested path. Note that you must specify setStartPoint("HEAD") or specify a hash starting point, as there will be no branch because there is not yet a checkout.

try (final Git git = Git.cloneRepository()
    .setURI(REMOTE_REPOSITORY_URI.toASCIIString())
    .setDirectory(LOCAL_RESPOSITORY_PATH.toFile())
    .setNoCheckout(true)
    .setDepth(1)
    .call()) {

  gitRepository.checkout()
    .setStartPoint("HEAD")
    .addPath("foo/bar")
    .call();

}

This seems to work very nicely! I would imagine it uses something similar to Satyajit Bhatt's answer under the hood.

Check out single subdirectory from minimal repository using JGit

2 Answers2

Linked