How to persuade Go to find C libraries

Question

Can anyone tell me what I'm doing wrong when building a go program that uses the ImageMagick library that has been installed to a non-standard location.

What I think I'm seeing is that the program builds, but a flag telling the program to look in the non-standard location for the dynamic library isn't set somewhere and so the program fails to be able to locate the library.

Steps to reproduce are:

Configure ImageMagick to install to a non-standard directory and build it:

# ./configure --prefix="/temp/imagemagick-temp" --without-magick-plus-plus --without-perl --disable-openmp --with-gvc=no
# make install

Add the pkgconfig directory to PKG_CONFIG_PATH so that pkg-config can find it.

# export PKG_CONFIG_PATH=/temp/imagemagick-temp/lib/pkgconfig

Check that pkg-config is happy and can find it:

# pkg-config --cflags --libs MagickWand MagickCore
-DMAGICKCORE_HDRI_ENABLE=0 -DMAGICKCORE_QUANTUM_DEPTH=16 -I/temp/imagemagick-temp/include/ImageMagick-6  -L/temp/imagemagick-temp/lib -lMagickWand-6.Q16 -lMagickCore-6.Q16

Build the program and capture all output:

# go build -v -x debug.go > debug_build.txt 2>&1

Attempt to run it:

# ./debug
./debug: error while loading shared libraries: libMagickWand-6.Q16.so.2: cannot open shared object file: No such file or directory

I've also run the program through strace to capture all system calls, and that capture is below. It shows that the program isn't looking in the right place for the libary file.

What do I need to do to make the program be able to find the library when it's in that custom location?

I get the same result if I set the CGO_CFLAGS and CGO_LDFLAGS manually, and then build with the 'no_pkgconfig' option set.

# export CGO_CFLAGS="-DMAGICKCORE_HDRI_ENABLE=0 -DMAGICKCORE_QUANTUM_DEPTH=16 -I/temp/imagemagick-temp/include/ImageMagick-6"
# export CGO_LDFLAGS="-L/temp/imagemagick-temp/lib -lMagickWand-6.Q16 -lMagickCore-6.Q16"
# go build -tags no_pkgconfig debug.go
# ./debug
./debug: error while loading shared libraries: libMagickWand-6.Q16.so.2: cannot open shared object file: No such file or directory

In case it matters I'm using "go version go1.6.2 linux/amd64" on Centos6.4

debug.go source file

package main

/*
#cgo !no_pkgconfig pkg-config: MagickWand MagickCore
#include <wand/MagickWand.h>
*/
import "C"
import "fmt"

type KernelInfoType int

const (
    KERNEL_UNDEFINED     KernelInfoType = C.UndefinedKernel
)

func main() {
    fmt.Println("Hello world")
}

This is the output of the build:

WORK=/tmp/go-build568335569
command-line-arguments
mkdir -p $WORK/command-line-arguments/_obj/
mkdir -p $WORK/command-line-arguments/_obj/exe/
cd /home/github/golang/goworkspace/src/debug
pkg-config --cflags MagickWand MagickCore
pkg-config --libs MagickWand MagickCore
CGO_LDFLAGS="-g" "-O2" "-L/temp/imagemagick-temp/lib" "-lMagickWand-6.Q16" "-lMagickCore-6.Q16" /home/github/golang/go/pkg/tool/linux_amd64/cgo -objdir $WORK/command-line-arguments/_obj/ -importpath command-line-arguments -- -DMAGICKCORE_HDRI_ENABLE=0 -DMAGICKCORE_QUANTUM_DEPTH=16 -I/temp/imagemagick-temp/include/ImageMagick-6 -I $WORK/command-line-arguments/_obj/ debug.go
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -DMAGICKCORE_HDRI_ENABLE=0 -DMAGICKCORE_QUANTUM_DEPTH=16 -I/temp/imagemagick-temp/include/ImageMagick-6 -I $WORK/command-line-arguments/_obj/ -g -O2 -o $WORK/command-line-arguments/_obj/_cgo_main.o -c $WORK/command-line-arguments/_obj/_cgo_main.c
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -DMAGICKCORE_HDRI_ENABLE=0 -DMAGICKCORE_QUANTUM_DEPTH=16 -I/temp/imagemagick-temp/include/ImageMagick-6 -I $WORK/command-line-arguments/_obj/ -g -O2 -o $WORK/command-line-arguments/_obj/_cgo_export.o -c $WORK/command-line-arguments/_obj/_cgo_export.c
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -DMAGICKCORE_HDRI_ENABLE=0 -DMAGICKCORE_QUANTUM_DEPTH=16 -I/temp/imagemagick-temp/include/ImageMagick-6 -I $WORK/command-line-arguments/_obj/ -g -O2 -o $WORK/command-line-arguments/_obj/debug.cgo2.o -c $WORK/command-line-arguments/_obj/debug.cgo2.c
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -o $WORK/command-line-arguments/_obj/_cgo_.o $WORK/command-line-arguments/_obj/_cgo_main.o $WORK/command-line-arguments/_obj/_cgo_export.o $WORK/command-line-arguments/_obj/debug.cgo2.o -g -O2 -L/temp/imagemagick-temp/lib -lMagickWand-6.Q16 -lMagickCore-6.Q16
/home/github/golang/go/pkg/tool/linux_amd64/cgo -objdir $WORK/command-line-arguments/_obj/ -dynpackage main -dynimport $WORK/command-line-arguments/_obj/_cgo_.o -dynout $WORK/command-line-arguments/_obj/_cgo_import.go
cd $WORK
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -no-pie -c trivial.c
cd /home/github/golang/goworkspace/src/debug
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -o $WORK/command-line-arguments/_obj/_all.o $WORK/command-line-arguments/_obj/_cgo_export.o $WORK/command-line-arguments/_obj/debug.cgo2.o -g -O2 -L/temp/imagemagick-temp/lib -Wl,-r -nostdlib -Wl,--build-id=none
/home/github/golang/go/pkg/tool/linux_amd64/compile -o $WORK/command-line-arguments.a -trimpath $WORK -p main -buildid f97632daf57e18f83c78ba78cc2d1ac8f96a4d1f -D _/home/github/golang/goworkspace/src/debug -I $WORK -pack $WORK/command-line-arguments/_obj/_cgo_gotypes.go $WORK/command-line-arguments/_obj/debug.cgo1.go $WORK/command-line-arguments/_obj/_cgo_import.go
pack r $WORK/command-line-arguments.a $WORK/command-line-arguments/_obj/_all.o # internal
cd .
/home/github/golang/go/pkg/tool/linux_amd64/link -o $WORK/command-line-arguments/_obj/exe/a.out -L $WORK -extld=gcc -buildmode=exe -buildid=f97632daf57e18f83c78ba78cc2d1ac8f96a4d1f $WORK/command-line-arguments.a
cp $WORK/command-line-arguments/_obj/exe/a.out debug

This is the output of the strace capture showing that the program isn't looking in the correct place.

execve("./debug", ["./debug"], [/* 28 vars */]) = 0
brk(0)                                  = 0xc44000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7facec440000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=30417, ...}) = 0
mmap(NULL, 30417, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7facec438000
close(3)                                = 0
open("/lib64/tls/x86_64/libMagickWand-6.Q16.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/lib64/tls/x86_64", 0x7fff50c5d9e0) = -1 ENOENT (No such file or directory)
open("/lib64/tls/libMagickWand-6.Q16.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/lib64/tls", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0
open("/lib64/x86_64/libMagickWand-6.Q16.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/lib64/x86_64", 0x7fff50c5d9e0)   = -1 ENOENT (No such file or directory)
open("/lib64/libMagickWand-6.Q16.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/lib64", {st_mode=S_IFDIR|0555, st_size=12288, ...}) = 0
open("/usr/lib64/tls/x86_64/libMagickWand-6.Q16.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/tls/x86_64", 0x7fff50c5d9e0) = -1 ENOENT (No such file or directory)
open("/usr/lib64/tls/libMagickWand-6.Q16.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/tls", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0
open("/usr/lib64/x86_64/libMagickWand-6.Q16.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/x86_64", 0x7fff50c5d9e0) = -1 ENOENT (No such file or directory)
open("/usr/lib64/libMagickWand-6.Q16.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/usr/lib64", {st_mode=S_IFDIR|0555, st_size=20480, ...}) = 0
writev(2, [{"./debug", 7}, {": ", 2}, {"error while loading shared libra"..., 36}, {": ", 2}, {"libMagickWand-6.Q16.so.2", 24}, {": ", 2}, {"cannot open shared object file", 30}, {": ", 2}, {"No such file or directory", 25}, {"\n", 1}], 10./debug: error while loading shared libraries: libMagickWand-6.Q16.so.2: cannot open shared object file: No such file or directory
) = 131
exit_group(127)                         = ?
+++ exited with 127 +++

You mean to persuade Go to go find libraries? What do you mean by find libraries? Surely Go has an implementation of find, unless it's gone? — hauron, Jul 10 '16 at 20:47
@hauron I've clarified the question. The program is meant to use a C library which would get loaded dynamically.....this is a library linking error. — Danack, Jul 10 '16 at 21:03

score 3 · Answer 1 · edited Jun 20 '20 at 09:12

The iMagick library is set up to rely on pkg-config, by default, in order to make it compatible out of the box with standard ImageMagick installs from package managers. It allows you to use a build tag to tell it not to use pkg-config and to instead rely on your own provided env to locate the includes and libs.

This is actually defined in the project README:

Build tags

If you want to specify CGO_CFLAGS/CGO_LDFLAGS manually at build time, such as for building statically or without pkg-config, you can use the "no_pkgconfig" build tag:
go build -tags no_pkgconfig gopkg.in/gographics/imagick.v2/imagick

In my own project that used iMagick, I used the custom build tag with a custom ImageMagick location.

Update

Since you have updated your question, it is apparent now that your problem is not really a Go problem, but rather a standard linux problem of a binary needing to dynamically load a library at runtime (You can see details about how shared libraries work here)

Do the following and it should fix your runtime problem:

$ export LD_LIBRARY_PATH=/temp/imagemagick-temp/lib:$LD_LIBRARY_PATH

LD_LIBRARY_PATH contains a list of paths that should be searched by the linker when a program starts. You have actually succeeded in providing the right info when building the program, as you can confirm by doing the following:

$ ldd ./debug

In that previously mentioned link you can also read about how to provide an rpath when building your binary. This is a path that gets baked into the binary, to tell the linker to search in that location for dependencies. It would allow you to avoid setting the LD_LIBRARY_PATH to your custom ImageMagick location for that particular binary.

Yes, that's what I thought I had done by setting the variable CGO_LDFLAGS to be "-L/usr/sbin/lib -lMagickWand-6.Q16 -lMagickCore-6.Q16" - my understanding is that -L instructs the built program to look in that directory for the libraries. But it doesn't seem to be having any effect. — Danack, Jul 12 '16 at 20:29
@Danack, so you definitely used the build tag? Did you build with -v and -x to see what the build is doing? — jdi, Jul 13 '16 at 00:18
I've updated the question with full build output and a standalone go file. — Danack, Jul 16 '16 at 13:35

How to persuade Go to find C libraries

1 Answers1

Update