2

What is the recommended way in Golang to download a submodule of a dependency? I think my question can be best described by examples.

Example #1:

I have a client and a server. My server is an API and has a bunch of other dependencies, such as databases, messages queues, consul, and etc. I would like my client to be a lightweight package where users download only few dependencies needed for the client.

You could say that the client and server can be on separate repositories. However, there may also be some common code amongst them, which if we follow this pattern, would again be another repository.

I'm thinking of some structure that looks like this:

service/
---> common/
------> redis.go
------> kafka.go
---> client/
------> client.go
---> server/
------> database/
------> swagger/
------> producer/
------> etc/

Example #2:

It's pretty common for projects to share models. If we have microservices that communicate through message brokers with a common model, we'd probably want some structure like this.

service/
---> model/
------> message.go
---> service1/
---> service2/

Comparison to other languages

I am from a Scala/Java background and have started to use Golang less than a month. Comparing with Scala, I could have handled this in two ways. Let's take Example #2

  1. Publish model as its own jar and import model.jar in each service
  2. Use multi-project setup in sbt: https://www.scala-sbt.org/1.x/docs/Multi-Project.html.

Some Things I Explored in Go

  1. https://blog.gopheracademy.com/advent-2015/vendor-folder/
  2. https://github.com/golang/go/wiki/Modules#faqs--multi-module-repositories

But so far they don't seem to solve my question

Final Question

What is the recommended way in Golang to tackle the issues I stated in the examples?

Thank you for your help!

Sarin Madarasmi
  • 526
  • 6
  • 11
  • 2
    Start by understanding that there simply are no "sub"modules or "sub"packages. Basically all modules and packages are equal. It absolutely doesn't matter whether you put all your code into one repo or several, whether you put your code into one module or several and how you split your code into packages, just do what fits your need. Start with one repo, one module which naturally models what you want to do: Modules are collections of packages versioned together and that is suitable for server/client code. – Volker Nov 03 '20 at 16:07
  • If a module is shared, make it a project and import it twice. For example a complicated calculation module. For shared structs I recommend duplication or using automation like git hook that copy the message to client when ever you change the server. "A little copying is better than a little dependency. Rob Pike - Go Proverbs" – ZAky Nov 03 '20 at 17:09

1 Answers1

3

Generally, try to stick to one repo and one module, as much as possible. Splitting your codebase to multiple repos and/or multiple modules incurs costs that will grow over time. You'll need to define clear version dependencies between your own modules, do phased upgrades carefully, etc. Especially for small-to-medium projects, one repo and one module is the best way to go.

Regarding your specific requirement in Example 1, if it's critical for the users of "client" to not pull "server" code, then client and server will have to be in separate modules. These modules can be either in the same repo or in different repos - this makes no difference to Go. The common code will have to be in a module of its own, e.g.:

github.com/user/myrepo/
  client/
    go.mod
  server/
    go.mod
  common/
    go.mod

Then, a module using your client would have, in their go.mod:

require github.com/user/myrepo/client <version>

And this will pull in client and common, but not server.

If it's OK for server to depend on client, you could potentially make common part of client, though I'm not sure this would save much.

For Example 2 the idea is similar - common code can go into its own module.

I strongly recommend you read the official blog posts on Using Go Modules.

For a simple Go project layout with modules, see this post

Eli Bendersky
  • 263,248
  • 89
  • 350
  • 412
  • Thanks for your suggestions! Regarding the example struct you have with common, client, and server, I was trying to do something like that. However, when I import just `go get github.com/user/myrepo/client` it does seem to still be pulling everything including the server package. Do you have suggestions on which repository I can look at this kind of example from? Or a blog post / tutorial that shows working with this kind of structure? – Sarin Madarasmi Nov 04 '20 at 00:55
  • @SarinMadarasmi: if the client and server are different modules and client has no dependency on server, then just `require`-ing client should not pull in server. As far as example, take a look at this repo: https://github.com/google/go-cloud/ -- it has a bunch of its packages wrapped in separate modules (`find . -name go.mod`), and if you just pull the main thing these packages will not be included, unless you require them explicitly – Eli Bendersky Nov 04 '20 at 04:08
  • I did `go get gocloud.dev/pubsub/rabbitpubsub` which is one of the submodules from https://github.com/google/go-cloud and my `go.mod` file shows that it only contains this dependency. However, the whole repository is still downloaded everything from the repository – Sarin Madarasmi Nov 07 '20 at 04:43
  • @SarinMadarasmi: yes, but do you really care? It should be very fast with the Go module proxy (`GOPROXY`). You can split to a separate repo of this is really important – Eli Bendersky Nov 07 '20 at 13:42
  • I see, because this was actually my question. Whether we can structure a golang project where client and server can stay in the same repository and still only download a client dependency if we want to. Performance reasons is one thing, but I think in general having unnecessary dependencies in your project can bring more pain, such as an indirect dependency that has conflicts with your existing dependency. – Sarin Madarasmi Nov 08 '20 at 09:00
  • @SarinMadarasmi: you're conflating "downloading" from actually depending. If your project has `A` in its `go.mod`, then it depends on `A`. If it doesn't, it doesn't. If it downloads `A` because it happens to be in the same repo as something else it depends on, it still doesn't depend on `A` - this is just an artifact of how Go tooling works. – Eli Bendersky Nov 08 '20 at 15:08
  • I have accepted your answer, thank you for the explanations. However, if you will, I hope you can help to answer this as I think it is still related to my original question. Sure, "downloading" is different from depending. Although, wouldn't this increase the size of my executable file, since those unused transitive dependencies are "downloaded"? This may sound trivial but can easily add up. Please let me know if I still misunderstand something. – Sarin Madarasmi Nov 12 '20 at 10:37
  • 1
    @SarinMadarasmi: the download happens while you `go build`. Only the deps your program uses go into the final binary, so its size is not going to be affected. I recommend you try it out, actually - it should take 15 minutes to test using any multi-module repo online (or create your own) – Eli Bendersky Nov 12 '20 at 13:16