How to distribute a Go module with C dependencies

Question

I have a Go package—call it foo—I've built around some existing C code, and I'm trying to determine how best to distribute it.

A bit of background... Here's a simplified version of my directory structure:

foo/
  |_ include/
      |_ <several other header files>
  |_ libs/
      |_ <several .so files>
  |_ foo.c
  |_ foo.h
  |_ foo.go

A stripped down head of foo.go

package foo

// #cgo CFLAGS: -I${SRCDIR}/include
// #cgo LDFLAGS: ${SRCDIR}/_foo.so -L${SRCDIR}/libs
// #include <stdlib.h>
// #include "foo.h"
import "C"

...

Where _foo.so is a dynamic lib generated as part of the build process. I am able to run and test my code fine, but I want to be able to distribute it and use it in a separate project, one that uses go mod. This is where things get a bit tricky, and my understanding of Go gets flimsy. Conceptually, I simply want to be able to distribute the foo/ directory (.so files and all) in a place where another application can find and import it, however there are a couple stumbling blocks I've run into:

I can't make this a binary package, as we're on Go 1.13 and as far as I understand that capability was dropped after 1.12.
The build process for the .so file is quite substantial, and I don't want other developers to have to spend 30 mins+ just building that object.

I have found a lot of CGO demos and blogs, but not a lot of explanations around how to distribute CGO packages. Any ideas?

It's worth noting that this is for my company, where the environment is standard, so I do not have to control for different OS's, etc.

Edit

Something I forgot to mention that I've tried is baking the library into the GOPATH of a Docker image, and just building from that base for the microservice that depends on it. However, when building the service with -mod=vendor, it's unable to find the library in the GOPATH:

RUN GOOS=linux \
    GOARCH=amd64 \
    CGO_ENABLED=1 \
    GOFLAGS=-mod=vendor \
    GO111MODULE=on \
    go build \
    -o service ./cmd/serve/main.go

./service
# build foo: cannot load foo: open /.../svc/vendor/foo: no such file or directory

Not ideal, but maybe just commit the compiled .so file to version control. The file will be then downloaded with the module. — , Jan 14 '20 at 21:25
I'd toyed with that idea, but it's really the other `.so` files it depends on in `foo/libs` that make it too large to commit. — TayTay, Jan 14 '20 at 21:28
C dependencies should be controlled however you would control C dependencies without Go in the system. That said, I think your problem with the docker image is that you're trying to `go build` a file -- the go commands should operate on packages. — JimB, Jan 14 '20 at 22:47

score 3 · Accepted Answer · answered Jan 17 '20 at 13:38

For anyone who might run into a similar question in the future, here are the steps I took to make this work.

I baked the generated artifacts into a base docker image, a la:

FROM golang:1.13.4

COPY ./foo /go/src/foo/foo
COPY ./foo.mod /go/src/foo

This gets around having to commit the .so files to source control, instead simply pushing the docker image to a registry.

Any service that requires this library is built from that base image:

FROM my-foo-base:latest

COPY ./vendor ./vendor
COPY ./go.mod ./
COPY ./go.sum ./

# Put foo in /vendor so it's discoverable, and put its libs where they
# can be discovered
RUN mv /go/src/foo ./vendor && \
    mv ./vendor/foo/foo/libs/*.so /usr/lib/ && \
    mv ./vendor/foo/foo/include/* /usr/include/

# Copy in app code
COPY ./my-app ./my-app

# Required so the app can find the _foo.so shared object
ENV LD_LIBRARY_PATH="/vendor/foo/foo:/usr/lib:${LD_LIBRARY_PATH}"

# Build into a single binary
RUN GOOS=linux \
    GOARCH=amd64 \
    CGO_ENABLED=1 \
    GOFLAGS=-mod=vendor \
    GO111MODULE=on \
    go build \
    -o service ./cmd/serve/main.go

This works, and the service is able to discover the CGO package.

How to distribute a Go module with C dependencies

Edit

1 Answers1