Fetch 'go get' project from local bare git repo

Question

I'm trying to simulate remote repository for Go dependency as local git repo. I'm aware of go mod edit -replace etc., but for my use case I need Go command to fetch dependency from git repo locally.

To simulate it with minimal reproducible example, I created bare git repo:

# in /tmp/git/repo1
git init --bare

Then cloned the repo, created go project, and pushed it back:

# in /tmp/projects/
git clone file:///tmp/git/repo1
cd repo1
go mod init git.localhost/repo1
git add go.mod && git commit -m "init: go mod init localhost" && git push origin

Just in case, checked bare repo updated successfully:

# in /tmp/git/repo1
git cat-file -p $(cat refs/heads/master) | grep init
# init: go mod init localhost

Then I created "client" go project, updated git config and tried to fetch it using go get:

# in /tmp/go-proj
go mod init example.com
git config --global url."file:///tmp/git/".insteadOf "https://git.localhost/"
go get git.localhost/repo1

But got this error:

go: unrecognized import path "git.localhost/repo1": https fetch: Get "https://git.localhost/repo1?go-get=1": dial tcp: lookup git.localhost on 127.0.0.1:53: no such host

Also, I tried to update GOPRIVATE env and set git config allow=always for file protocol:

export GOPRIVATE="git.localhost/*"
git config --global protocol.file.allow always

And checked git can clone this repo by using:

git clone https://git.localhost/repo1
# successfully cloned from bare repo

Is it possible to get dependency from local git bare repo using go get?

Update: Added GOPROXY env, so now my go env has:

GOPRIVATE="git.localhost/*"
GOPROXY="direct"

and git config --global has:

url.file:///tmp/git/.insteadof=https://git.localhost/
protocol.file.allow=always

Update2:

Verbose output for go get command:

GOFLAGS="-v -x" go get git.localhost/repo1
# get https://git.localhost/?go-get=1
# get https://git.localhost/repo1?go-get=1
# get https://git.localhost/?go-get=1: Get "https://git.localhost/?go-get=1": dial tcp: lookup git.localhost on 127.0.0.1:53: no such host
# get https://git.localhost/repo1?go-get=1: Get "https://git.localhost/repo1?go-get=1": dial tcp: lookup git.localhost on 127.0.0.1:53: no such host
go: unrecognized import path "git.localhost/repo1": https fetch: Get "https://git.localhost/repo1?go-get=1": dial tcp: lookup git.localhost on 127.0.0.1:53: no such host

VonC · Answer 1 · 2023-04-24T08:30:42.317

1

Just in case this is the issue, test your go get command with first:

export GOPROXY=direct

That should allow go get to fetch the dependency from the local Git repository without any issues.
That should be set in addition to your other settings, like your export GOPRIVATE="git.localhost/*" and your Git insteadOf directive.

The OP adds in the comments:

Looks like go get is trying to fetch some meta data via http from dependency host (git.localhost in my case), then create package git repo in go/pkg/mod/cache/vcs/{hash} and then fetch dependency code using git, where git applies isnteadOf replace.

So I'm going to try create simple http server to respond with meta information with VCS, Prefix and RepoRoot url response to http://git.localhost/?go-get=1 URL

This is indeed what is illustrated in "How can I make go get work with a repo on a local server".

edited Apr 24 '23 at 08:30

answered Apr 21 '23 at 23:44

VonC

1,262,500
529
4,410
5,250

Thanks, I tried `GOPROXY` but has the same issue (updated question) – Kirill Apr 24 '23 at 06:39
@Kirill Do you have any `HTTP(S)_PROXY` environment variable in place? – VonC Apr 24 '23 at 06:44
No, I don't have proxies, the only proxy env for `env | grep -i proxy` is `GOPROXY=direct` – Kirill Apr 24 '23 at 06:48
@Kirill Is there any clue with a `GOFLAGS="-v" go get git.localhost/repo1`? – VonC Apr 24 '23 at 07:06
I tried with `-v -x` and it looks it `go get` command tried to connect via TCP to localhost (see log in updated question). – Kirill Apr 24 '23 at 07:44
Looks like `go get` is trying to fetch some meta data via http from dependency host (`git.localhost` in my case), then create package git repo in `go/pkg/mod/cache/vcs/{hash}` and then fetch dependency code using git, where git applies `isnteadOf` replace. So I'm going to try create simple http server to respond with meta information with `VCS`, `Prefix` and `RepoRoot` url response to `http://git.localhost/?go-get=1` URL. – Kirill Apr 24 '23 at 08:20
@Kirill I have included your comment in the answer for more visibility, and added a link to another answer which could illustrate what you want to do. – VonC Apr 24 '23 at 08:31

Kirill · Accepted Answer · 2023-05-02T08:59:52.273

To solve this problem, I started primitive HTTP web server to return HTML page for go get requests with <meta> go-import tag. It specifies the location of git repository as https://git.localhost/<repo-name> and then go get uses git to fetch this repo and insteadOf git config replaces https://git.localhost with file:/// location. Also, GOINSECURE="git.localhost/* environment is required to request go-get requests using http.

This is my server handler (with submodule and semver support):

package main

import (
    "fmt"
    "log"
    "net/http"
    "regexp"
    "strings"
)

// repo names: /repo /repo/module /repo/ /repo/submodule /repo/v2
var reRepo = regexp.MustCompile(`^/([a-zA-Z0-9_-]+)/?(?:.+)?$`)

func main() {
    http.ListenAndServe(":80", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        repoParts := reRepo.FindStringSubmatch(r.URL.Path)
        if len(repoParts) == 0 {
            http.NotFound(w, r)
            return
        }
        name := repoParts[1]
        log.Printf("Resolve %s => %s", r.URL.Path, name)

        var sb strings.Builder
        sb.WriteString("<html><head>")
        sb.WriteString(fmt.Sprintf(`<meta name="go-import" content="git.localhost/%s git https://git.localhost/%s" />`, name, name))
        sb.WriteString("</head></html>")
        fmt.Fprint(w, sb.String())
    }))
}

Well done, more detailed than my answer. Upvoted. – VonC May 02 '23 at 06:33 — VonC, May 02 '23 at 06:33

Fetch 'go get' project from local bare git repo

2 Answers2