Problem with remote repository fetching a custom buildroot package

Question

My buildroot environment is downloaded from a git server named gitserver1:

$ git remote -v
origin git@gitserver1:/platforms/buildroot.git (fetch)

This is an example of a buildroot package included in package/company/mypackage/mypackage.mk

GITHOST = git@gitserver2
GIT_PATH = /packages/mypackage.git
GIT_BRANCH = branch

MYPACKAGE_SITE = $(GITHOST):$(GIT_PATH)
MYPACKAGE_SITE_METHOD = git
MYPACKAGE_VERSION = $(GIT_BRANCH)

MYPACKAGE_PKG_NAME = mypackage
MYPACKAGE_PKG_DIR = $(TOPDIR)/package/company/$(MYPACKAGE_PKG_NAME)
MYPACKAGE_BUILD_DIR = $(BUILD_DIR)/$(MYPACKAGE_PKG_NAME)-$(GIT_BRANCH)

define MYPACKAGE_BUILD_CMDS
    $(MAKE) CC="$(TARGET_CC)" CFLAGS="$(TARGET_CFLAGS)" -C $(@D)
endef

define MYPACKAGE_INSTALL_TARGET_CMDS
    $(INSTALL) -D -m 755 $(@D)/binary $(TARGET_DIR)/usr/bin/
endef

$(MYPACKAGE_PKG_NAME)-clean:
    if test -d $(MYPACKAGE_BUILD_DIR); then make clean -C $(MYPACKAGE_BUILD_DIR); fi

$(MYPACKAGE_PKG_NAME)-dirclean: $(MYPACKAGE_PKG_NAME)-clean
    rm -Rf $(DL_DIR)/$(MYPACKAGE_PKG_NAME)-$(GIT_BRANCH).tar.gz

$(MYPACKAGE_PKG_NAME)-rebuild: $(MYPACKAGE_PKG_NAME)-clean

ifeq ($(BR2_PACKAGE_MYPACKAGE),y)
$(eval $(generic-package))
endif

The source code of this buildroot package is fetched from a different git server named gitserver2. This remote will be used for the example:

$ git remote -v
origin git@gitserver2:/packages/mypackage.git (fetch)

To build this buildroot package I type:

$ make mypackage

The internal buildroot process:

• downloads a mypackage-branch.tar.gz tarball inside the BR2_DL_DIR.
• creates a local directory output/build/mypackage-branch where the tarball is uncompressed.

My purpose is getting the checksum ID of the commit used for the build of mypackage (for internal reporting). Using a branch, the only way of getting that checksum ID is having the .git directory available.

But when I check the remote repository inside the output/build directory, I get the remote pointing to gitserver1 (the buildroot .git repo), and not the expected (gitserver2):

$ git remote -v
origin  git@gitserver1:/platforms/buildroot.git (fetch)

After some responses to this post, I understand the downloaded tarball is uncompressed inside output/build/mypackage-branch, and it doens't contain the .git control information. So when I type git remote -v I get the main repository (gitserver1) and not the repository used to fetch the source code (gitserver2).

As a workaround, I defined this inside the mypackage.mk:

define MYPACKAGE_EXTRACT_CMDS
        @$(call MESSAGE, "Extracting $(MYPACKAGE_PKG_NAME) branch $(GIT_BRANCH) from $(GITHOST)")
        rm -f $(MYPACKAGE_BUILD_DIR)/.stamp_downloaded
        git clone -b $(GIT_BRANCH) $(GITHOST):$(GIT_PATH) $(MYPACKAGE_BUILD_DIR)
        touch $(MYPACKAGE_BUILD_DIR)/.stamp_downloaded
endef

Now, once the package is built, I can see the expected remote server (gitserver2) inside the output/build/mypackage-branch directory:

$ git remote -v
origin  git@gitserver2:/packages/mypackage.git (fetch)

Here, all the repo history is available, even the last commit ID for the branch used to fetch source code. And I use that information to get the checksum_ID for the compiled branch.

However, this is an awful fix. The package is fetched twice. And I understand this is not the way to go.

Can you suggest a better way to get the expected result? Thanks!

Answer 1

Buildroot downloads the source code to the BR2_DL_DIR directory. You can find the BR2_DL_DIR directory by grepping for it in the buildroot BASE_DIR :

grep BR2_DL_DIR .config

Buildroot will clone a repo to the download directory (BR2_DL_DIR). Buildroot will reset the remote to the buildroot git url. This is why you see the origin in your source package pointing to gitserver1.

If you want to update your source code while you are developing it, then there are two ways to do it.

• Develop on your local filesystem and change the site to a local uri.
• Develop in your git repo and push the changes to gitserver2.

In either method, you will need to remove the output/build directory and then make the package again :

rm -rf output/build/mypackage-branch
make mypackage

In the case of the local uri, clone mypackage locally and set it like so :

MYPACKAGE_SITE = path/to/mypackage
MYPACKAGE_SITE_METHOD = local

Once you have debugged your code, you can git add/commit and push. Then change the SITE back to the git server. All should build as expected for the rest of the team.

Answer 2

I don't know about "buildroot".

From what I understand : you are using a framework to build your package.

The build framework comes from gitserver1, the code to build comes from gitserver2, and the result of that build is placed in the output/build/mypackage-branch directory.

If this is the case : there is no reason for that build directory to be placed under the repo cloned from gitserver2.

---

You run git remote -v from output/build/mypackage-branch/, but you didn't check where the root repository is :

• you can run ls .git/ to confirm that output/build/mypackage-branch/ is not a clone of any repo,
• run git rev-parse --show-toplevel to see what repo you are viewing (my guess is : your root repository)

If I understand your script correctly, the repo from gitserevr2 is probably cloned in package/company/$(MYPACKAGE_PKG_NAME) instead.

---

If somehow you need to compare or push the build artifacts back to gitserver2, please take time update your question and explain your intention.

---

[update, following your edits]

One way to get the sha1 of a branch without cloning the complete repo is :

git ls-remote origin refs/heads/$GIT_BRANCH

The catch in your situation is : since downloading the tarball and getting the sha1 would be distinct actions, you would need some way to make sure that the sha1 matches the tarball -- e.g : that you don't get mixed if someone else happens to run git push to the remote while a build is running.

@Matt offers a way to do that in his answer : use a local clone of gitserver2 as the "remote repo" for your buildroot script.

Say you make a mirror clone of gitserver2:/packages/mypackage.git to path/to/mypackage :

git clone --mirror gitserver2:/packages/mypackage.git path/to/mypackage

Turn your build steps to :

a. update the local mirror : git --git-dir=path/to/mypackage fetch
b. run your buildroot script

Since no one else would act on your local clone, its state would stay consistent throughout the build.