2

I'm trying to print a list of direct dependencies in a Go project that has a vendor directory with:

go list -mod=readonly -m -f '{{ if not .Indirect }}{{ .Path }}{{ end }}' all

As I understand, I have to use -mod=readonly because go list refuses to report based on the partial information in vendor. This works on my laptop, because I have access to the private repo github.com/monzo/argo-rollouts, but in CI it fails with:

go list -m: github.com/argoproj/argo-rollouts@v0.0.0-20220309162305-84c86ea52e8f (replaced by github.com/monzo/argo-rollouts@v0.0.0-20220309162305-84c86ea52e8f): version "v0.0.0-20220309162305-84c86ea52e8f" invalid: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /home/circleci/go/pkg/mod/cache/vcs/052bf06be1713582ba9f1144e9982b362ff355fec71675863c108f9cf5a00bb4: exit status 128:
    ERROR: Repository not found.
    fatal: Could not read from remote repository.
    
    Please make sure you have the correct access rights
    and the repository exists.

Why does go list -mod=readonly need access to the private repo? I haven't provided -u.

Also, why can't it get this information locally when in vendor mode?

Edit: I've found a workaround that passes eligible modules to go list -m in vendor-mode, but it involves parsing the unstable vendor/modules.txt file intended for machines only.

% cat modules-that-provide-packages.awk
BEGIN {
        mod = ""
}

# Remember the module
/^# / {
        mod = $2
}

# Print the module if it provides a package
!/^# / {
        print mod
}


% awk -f modules-that-provide-packages.awk vendor/modules.txt /
    | sort -u \
    | xargs go list -m -f '{{ if not .Indirect }}{{ .Path }}{{ end }}'
tompreston
  • 630
  • 7
  • 24
  • (Removed the `govendor` tag: it's for the venerable (and now deprecated) tool which predates Go modules.) – kostix Jun 30 '22 at 12:58
  • you have access on local because of the `GOPRIVATE` variable you might configured. On CI, however, you have to configure the same – Neenad Jun 30 '22 at 13:08
  • Hey thanks Neenad, but I can still get output locally using `GOPRIVATE="" go list -mod=readonly -m all | grep argo-rollouts`. As I understand, GOPRIVATE only turns off the proxy and checksum DB for private repos. It does not disable them. – tompreston Jun 30 '22 at 13:48
  • 1
    I think it works locally because my GOMODCACHE is already populated, but it isn't in CI. – tompreston Jun 30 '22 at 13:52
  • In vendor-mode, `go list -m all` returns `go list -m: can't compute 'all' using the vendor directory (Use -mod=mod or -mod=readonly to bypass.)` – tompreston Jun 30 '22 at 14:10
  • Thanks blackgreen, when I specify `-mod=vendor` I get the same error. My understanding is that when `go list` detects the vendor directory, it behaves as if `-mod=vendor` has been specified. – tompreston Jun 30 '22 at 14:22
  • have you tried `-mod=mod` mode? – Neenad Jun 30 '22 at 16:06
  • @tompreston when you said you tried this `GOPRIVATE="" go list -mod=readonly -m all | grep argo-rollouts` did you try this inside a docker container? because I think your local git config is configured to login to private repo and you are not able to replicate it on local. – Neenad Jun 30 '22 at 16:08
  • Hey, Neenand I have tried `-mod=mod`. I think I understand the problem better now. `go list` needs to get the module info from somewhere, and that's either the GOMODCACHE or the vendor directory. In CI, GOMODCACHE is empty so it tries to populate it from the internet - which requires access private repos. We can get the info from the vendor directory if we provide a specific module, but if we specify `all` then `go list` is programmed to give up. – tompreston Jun 30 '22 at 16:11
  • @Neenand, `GOPRIVATE` is already "" in a container. – tompreston Jun 30 '22 at 16:13

2 Answers2

1

I think the answer is: go list -m doesn't require access to private repos, but populating an empty GOMODCACHE does.

go list -m gets the module info from either:

  • GOMODCACHE, but this must be already populated. It is empty in CI, so Go will try to download from the private repos. Hence the error.
  • The vendor directory, but you must provide a specific module. go list -m all gives up in vendor-mode.
tompreston
  • 630
  • 7
  • 24
0
  • make sure you set GOPRIVATE to your private GitHub account link eg github.com/abc* in your CI script
  • Add git config to use the GitHub account token to log in to the private repo. On your ci, you would have to do something like this in the CI script
 git config --global url."https://${{ inputs.token }}:x-oauth-basic@github.com/<useraccount>".insteadOf "https://github.com/<useraccount>"
  • Now when you run the command it will be able to access the private repo or before running the list command make sure to do go mod vendor to get all the dependencies on CI first
Neenad
  • 861
  • 5
  • 19
  • Hey thanks for the suggestions, I get the same error with `GOPRIVATE=github.com/monzo go list -m -mod=readonly all`. We're considering adding access to private repos, but why does `go list -mod=readonly -m all` need access in the first place? – tompreston Jun 30 '22 at 13:29