multiple go projects and sharing a vendor directory (in go before 1.11)

Question

I have started learning go (1.7.4) and have a project which currently produces two executables. I have a directory structure as below following the standard go layout:

GOPATH=`pwd`
bin
src/
src/<project1>
src/<project1>/vendor
src/<project1>/glide.yaml
src/<project2>
src/<project2>/vendor
src/<project2>/glide.yaml
pkg/

Project 1 and project 2 share a lot of dependencies. Is there a way to share the vendor directory between project1 and project2 and still pin the versions to ensure reproducible builds?

I don't want to duplicate the glide.yaml and vendor directories for each project as it bloats the build and violates DRY.

The pkg directory is the obvious the way to do this but unlike vendor I don't have a dependency manager tool like glide to ensure a specific version is used (see also my related question).

A possibly related issue is how this project is organised. I believe in go it would be more conventional for each project sub-directory to map to a single github repository. However, for my project I want to build at least two executables. I realise you can do this by using different package names but it confuses go and glide. I wrestled with getting this to work under a single project and decided/discovered it was easier to use the standard go layout and work two levels up. For example, an advantage is that "go build" etc. in the subdirectories just works without having to name the package. I can also have my build, test and package machinery at the top level operate on all projects and keep my go environment separate from any others. The programs are not complex enough to warrant separate git repositories (even as submodules). If there is a pattern that makes this work it might render my original question moot.

See also https://stackoverflow.com/questions/53086136/fixing-versions-of-tools-used-by-go/53089261#53089261 which covers the same problem solution for installable commands written in go. Has answer for both go 1.11 with modules and earlier versions. — Bruce Adams, Nov 08 '18 at 11:31

score 1 · Accepted Answer · answered Oct 31 '18 at 17:23

1

It should be possible to have a shared vendor directory. The way I am doing it involves Go 1.11 and the new Go feature called modules. But I am pretty sure it should work with vendor and tools like glide and dep. To use dep/glide your directory structure might looks like this

- src
  - projects
      - project1
      - project2
      - vendor
      - Glide.yaml

And you can build it either from the projects folder using go build -o p1 project1/*.go or from individual project folder using go build

The same structure, but outside of GOPATH will work for Go 1.11 modules. You would have to set the GO111MODULE variable to "on" or "auto". Mind you that go modules store dependencies in some other location and download them automatically during the build process when needed.

Note: glide github page recommends switching to dep as the more official tool

Edit: Just tested it with dep. It works for me.

answered Oct 31 '18 at 17:23

Mad Wombat

14,490
14
73
109

both glide and dep are obsoleted by modules. However I'm using go 1.7.4 as provided with Debian 9. dep might work but is offically supported only for 1.9 above. – Bruce Adams Oct 31 '18 at 17:30
1

Thanks, I will try your solution and accept if it works for me. – Bruce Adams Oct 31 '18 at 17:30
Yes, modules are preferred, but I believe dep is the more "official" tool for Go < 1.11. If you are free to upgrade, you could use snap to get the latest Go. – Mad Wombat Oct 31 '18 at 17:32
That seems to work and answers another question I had. Using -o for out of source builds. The trick seems to be the /*.go if you try to be clever go doesn't like it, it wants to be clever for you. – Bruce Adams Nov 01 '18 at 10:07
You are welcome. And as I said, if the reason you are using such an old Go version only has to do with what is provided by Debian, you can use snap. To install snap on Debian 9 you can follow a guide here https://docs.snapcraft.io/installing-snap-on-debian/6742 and here is a link to Go snap package https://snapcraft.io/go – Mad Wombat Nov 01 '18 at 15:02
1

In production you don't want a constantly moving target. In general I try to use the versions of packages the OS vendor provides for stability. Debian and RedHat are both a little on the conservative side but 1.8 was only released late in 2016. In this case I'm copying colleagues who've recently introduced go. – Bruce Adams Nov 01 '18 at 15:23
Having moved to go 1.11 I now have no GOPATH and no vendor directory just some go.mod files and it is a simpler solution as I don't have to explain the use of third party tools. – Bruce Adams Nov 08 '18 at 11:29

score 1 · Answer 2 · answered Oct 31 '18 at 20:45

1

I recommend look at new vendoring system - https://github.com/golang/go/wiki/Modules

It allows you to fix versions of packages used:

module github.com/my/thing

require (
    github.com/some/dependency v1.2.3
    github.com/another/dependency/v4 v4.0.0
)

answered Oct 31 '18 at 20:45

Eugene Lisitsky

12,113
5
38
59

That is correct for go 1.11 which I have now switched to as it is a better solution than depending on glide and retool. I had to accept the other answer however as my question refers to an earlier version of go. – Bruce Adams Nov 08 '18 at 11:25
Yes, that's fair. Besides the question, I recommend to start learning from the fresh versions - they have big improvements. And you will not waste time relearning. I've had spent some time investigating modules after *classic* GOPATH. – Eugene Lisitsky Nov 09 '18 at 19:54

multiple go projects and sharing a vendor directory (in go before 1.11)

2 Answers2