1

I have a recipe that looks basically like this :

SUMMARY = "SomeLibrary"

LICENSE = "Apache-2.0"
LIC_FILES_CHKSUM = "file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57"

SRC_URI += "git://gitlab.com/some_library/some-library.git;protocol=https;nobranch=1"
SRCREV = "${PV}"

S = "${WORKDIR}/git"

inherit autotools pkgconfig

It builds successfully with bitbake some-library, and I can see there is a git2/gitlab.com.some_library.some-library.git/ directory and a git2/gitlab.com.some_library.some-library.git.done file in my downloads folder (the one DL_DIR point to).

My understanding is that if I then immediately run bitbake -c cleansstate some-library && bitbake some-library, given that there is no change in the recipe, bitbake shouldn't need to download anything (it already has everything it needs). In practice, if I turn off my network connection or add BB_NO_NETWORK="1" to my local.conf, I get the following error :

Initialising tasks: 100% |################################################################| Time: 0:00:01
Sstate summary: Wanted 12 Found 4 Missed 8 Current 251 (33% match, 96% complete)
NOTE: Executing Tasks
ERROR: some-library-v2.3.0-r0 do_fetch: Bitbake Fetcher Error: NetworkAccess('https://gitlab.com/some_library/some-library.git', 'git -c core.fsyncobjectfiles=0 ls-remote "https://gitlab.com/some_library/some-library.git" ')
ERROR: Logfile of failure stored in: /home/myusername/work/builddir/tmp/work/aarch64-poky-linux/some-library/v2.3.0-r0/temp/log.do_fetch.116252
ERROR: Task (/home/myusername/work/builddir/../../layers/meta-mymeta/recipes-core/some-library/some-library_v2.3.0.bb:do_fetch) failed with exit code '1'
NOTE: Tasks Summary: Attempted 806 tasks of which 804 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
  /home/myusername/work/builddir/../../layers/meta-mymeta/recipes-core/some-library/some-library_v2.3.0.bb:do_fetch
Summary: There was 1 ERROR message shown, returning a non-zero exit code.

Why is that ? How do other recipes avoid this pitfall ? (when I build my image, this recipe seems to be the only one trying to fetch things from the network, which suggests to me that I'm doing something wrong here)

EDIT : What really puzzles me is that bitbakes seems to behave differently with recipes other than my own. For example, the recipe for can-utils located at meta-openembedded/meta-oe/recipes-extended/socketcan/can-utils_git.bb looks like this:

SUMMARY = "Linux CAN network development utilities"
LICENSE = "GPLv2 & BSD-3-Clause"
LIC_FILES_CHKSUM = "file://include/linux/can.h;endline=44;md5=a9e1169c6c9a114a61329e99f86fdd31"

DEPENDS = "libsocketcan"

SRC_URI = "git://github.com/linux-can/${BPN}.git;protocol=https;branch=master"

SRCREV = "da65fdfe0d1986625ee00af0b56ae17ec132e700"

PV = "2020.02.04"

S = "${WORKDIR}/git"

inherit autotools pkgconfig

which is very similar, but when I set BB_NO_NETWORK="1" in my local.conf and run bitbake -c cleansstate can-utils && bitbake can-utils I get Tasks Summary: Attempted 842 tasks of which 822 didn't need to be rerun and all succeeded.

Eternal
  • 2,648
  • 2
  • 15
  • 21
  • did you make sure that there is a file with .done extension in download repository? Like package.tar.xz.done – void_brain Feb 10 '23 at 08:50
  • @void_brain I did and there is. But that's a good question, I'll add that information to my question – Eternal Feb 10 '23 at 09:11
  • can you try this command instead : bitbake-layers clean-cache – void_brain Feb 10 '23 at 09:40
  • @void_brain With `bitbake-layers clean-cache && bitbake lely-core` I get `Attempted 822 tasks of which 822 didn't need to be rerun and all succeeded`. Just to be clear, what I'm trying to do beyond the minimal example is to be able to delete my build folder including the `SSTATE_DIR`, keeping only the `DL_DIR`, and rebuild from scratch without needing to download anything – Eternal Feb 10 '23 at 09:51
  • it should be succesful, i advise you to keep your SSTATE_DIR however, it will build faster. Even if you want to rebuild from scratch you can keep and old SSTATE_DIR. – void_brain Feb 10 '23 at 09:58
  • Bitbake can try to fetch the sources even if BB_NO_NETWORK is set to "1" because it can be configured to check for the presence of the source in the cache before starting the download. If the source is not found in the cache, Bitbake generates a download error. Bitbake-layers clean-cache will completely clear the download cache and prevent Bitbake from trying to fetch online sources. – void_brain Feb 10 '23 at 10:00
  • @void_brain I wonder if I messed up earlier, because when I try to run `bitbake-layers clean-cache` again it says there is no such command (btw I updated my question because I found another recipe that looks the same but behaves differently) – Eternal Feb 10 '23 at 10:29
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/251781/discussion-between-eternal-and-void-brain). – Eternal Feb 10 '23 at 10:30

2 Answers2

0

This works for me: After configuring the project, add the following lines to build/conf/site.conf file:

# Build offline
SOURCE_MIRROR_URL ?= "file:///path/to/oe-downloads"
INHERIT += " own-mirrors"
BB_GENERATE_MIRROR_TARBALLS = "1"
BB_NO_NETWORK = "1"

After that, it might be necessary to build project once when online. After every re-configuration (different build options) the site.conf is overwritten, so I created a script to add these lines after re-configuration.

ReAl
  • 1,231
  • 1
  • 8
  • 19
  • https://docs.yoctoproject.org/3.1.22/ref-manual/faq.html (I'm working with dunfell) states that `When the build system searches for source code, it first tries the local download directory. If that location fails, Poky tries PREMIRRORS, the upstream source, and then MIRRORS in that order.`. Doesn't that mean that I shouldn't need mirrors after the first fetch, since the sources are already in the local download directory ? – Eternal Feb 10 '23 at 10:09
  • @Eternal it worked for me for Thud and now works for Kirkstone. Sorry, I do not know why without those lines it tries to access some remote gits despite oe-downloads directory exists and has all that archives and `.done` files. – ReAl Feb 11 '23 at 17:43
0

I believe I found the issue.

If I replace ${PV} (which was equal to v2.3.0 here) by the hash associated to that tag, then the issue stops happening.

If I interpret this correctly, it means that bitbake is able to tell if SRCREV is a hash or a tag, and that if it is a tag then do_fetch will always run git ls-remote to make sure that the tag has not been moved.

Eternal
  • 2,648
  • 2
  • 15
  • 21
  • Anything that isn't a sha _has_ to be looked up in the remote repository to turn it into a hash, because the tag could have moved. You can use srcrev caching but then you've got a problem where if the tag moves, you won't rebuild. Basically, don't use tags, use SHAs. – Ross Burton Apr 20 '23 at 15:26