8

I've recently started with Go 1.11 and love the modules. Apart from runtime dependencies I need to work with go modules during the build, e.g. during go generate.

How can I install a specific build dependency (e.g. github.com/aprice/embed/cmd/embed) and run that specific tool from which folder? Is go get the right tool for doing so?

andig
  • 13,378
  • 13
  • 61
  • 98

4 Answers4

7

If you get an error

I was not seeing the dependency that I wanted added to the go.mod and I was getting this error:

internal/tools/tools.go:6:5: import "github.com/UnnoTed/fileb0x" is a program, not an importable package

(fileb0x is the thing I'm trying to add)

I'm not 100% clear on the sequence of events that fixed it, but I did all of these things:

Using a "tools" package

I made a tools directory:

mkdir -p internal/tools

I put the tools package inside of it (as mentioned above):

internal/tools/tools.go:

// +build tools

package tools

import (
    _ "github.com/UnnoTed/fileb0x"
)

Note that the tag is mostly not important. You could use foo:

// +build foo

However, you cannot use ignore. That's a special predefined tag.

// +build ignore

// NO NO NO NO NO
// `ignore` is a special keyword which (surprise) will cause
// the file to be ignore, even for dependencies

Updating go.mod

The best way is probably to run go mod tidy:

go mod tidy

However, before I did that I ran a number of commands trying to figure out which one would cause it to go into go.mod:

go install github.com/UnnoTed/fileb0x # didn't seem to do the trick
go get
go generate ./...
go build ./...
go install ./...
go mod vendor

Later I did a git reset and rm -rf ~/go/pkg/mod; mkdir ~/go/pkg/mod and found that go mod tidy did well enough on its own.

vendoring

In order to actually take advantage of the modules cache in a project you need to copy-in the source code

go mod vendor

That will grab all dependencies from go.mod

You also need to change nearly all of your go commands to use -mod=vendor in any Makefiles, Dockerfiles or other scripts.

go fmt -mod=vendor ./... # has a bug slated to be fixed in go1.15
go generate -mod=vendor ./...
go build -mod=vendor ./...

That includes go build, go get, go install, and any go run called by go generate (and even the go generate itself)

//go:generate go run -mod=vendor github.com/UnnoTed/fileb0x b0x.toml
package main

// ...
coolaj86
  • 74,004
  • 20
  • 105
  • 125
3

https://github.com/golang/go/issues/25922 proved helpful for me, especially

when using build-only dependencies with modules the main point is version selection (not installing these!)

To avoid installing you can modify your //go:generate directive to something like:

//go:generate go run golang.org/x/tools/cmd/stringer ARGS

There is also the best practices repo: https://github.com/go-modules-by-example/index/blob/master/010_tools/README.md

Community
  • 1
  • 1
andig
  • 13,378
  • 13
  • 61
  • 98
2

The convention is to add a file named "tools.go" that is guarded by a build constraint and imports all required tools:

// +build tools

package tools

import (
    _ "github.com/aprice/embed/cmd/embed"
)

https://github.com/golang/go/issues/25922#issuecomment-412992431

The tools are then installed as usual in one of

  • $GOBIN
  • $GOPATH/bin
  • $HOME/go/bin

You may also want to follow https://github.com/golang/go/issues/27653, which discusses future explicit support for tools.

Peter
  • 29,454
  • 5
  • 48
  • 60
  • Thanks for the related feature discussion – andig Sep 21 '18 at 12:29
  • 1
    Actually, embed doesn't end up in any of theses paths though I've added the tools.go? – andig Oct 04 '18 at 17:47
  • does this obsolete github.com/twitchtv/retool? or only for go 1.11? – Bruce Adams Nov 01 '18 at 09:54
  • 1
    @BruceAdams, this question and answer are about Go modules, which have been introduced in 1.11. vgo may work similar for earlier versions, but I never used that so I can't be sure. I'm also not familiar with retool, but judging from the readme the approach outlined here does render it obsolete. – Peter Nov 01 '18 at 13:15
  • Retool is a good solution for older versions of go. The modules solution is better for go 1.11 and later though. See https://stackoverflow.com/questions/53086136/fixing-versions-of-tools-used-by-go/53089261#53089261 – Bruce Adams Nov 07 '18 at 19:11
  • I tried that, but I get "tools/tools.go:6:5: import "github.com/UnnoTed/fileb0x" is a program, not an importable package". Other suggestions? – coolaj86 Jan 03 '19 at 18:52
  • 1
    @CoolAJ86, sounds like you forgot the build constraint. – Peter Jan 03 '19 at 20:53
2

tools.go is a great solution if you're building an app or service. But if you're building a library, tools.go still leaks dependencies to things consuming your library (your tools are still there as indirect dependencies, and go mod tidy will pull them in since it considers every possible target). That's not the end of the world since those modules never end up in the actual built binaries of the consumer, but it's still messy.

https://github.com/myitcv/gobin/issues/44 is probably the most promising approach to fixing this long term, but short term I've used a combination of the "internal module" approach explained there along with https://github.com/izumin5210/gex.

First, I install gex globally:

GO111MODULE=off go get github.com/izumin5210/gex/cmd/gex

Then before actually using gex I create a structure like this:

myproject/
\
  - go.mod: module github.com/ysamlan/myproject
  \
    internal/
    \
      tools/
       - go.mod: module github.com/ysamlan/myproject/tools

To install a build-only tool I just cd internal/tools and run gex --add (sometool), which puts that tool in internal/tools/bin. CI scripts and other folks that want to build my stuff locally just need to run cd internal/tools && gex --build to reliably and reproducibly populate the tool binaries, but the top-level go.mod is unchanged.

The key piece there is creating that internal/tools/go.mod file with a different module path than the one the root project uses, and then only running gex from that directory.

Yoni Samlan
  • 37,905
  • 5
  • 60
  • 62