compile multiple cuda files (that have dynamic parallelism) and MPI code

Question

I have a bunch of .cu files that use dynamic parallelism (a.cu, b.cu, c.cu.., e.cu, f.cu), and a main.c file that uses MPI to call functions from a.cu on multiple nodes. I'm trying to write a make file to compile the executable, but I keep facing the following errors:

cudafiles.o: In function `__cudaRegisterLinkedBinary_66_tmpxft_00001a84_00000000_17_cuda_device_runtime_compute_61_cpp1_ii_8b1a5d37':
link.stub:(.text+0x1fb): undefined reference to `__fatbinwrap_66_tmpxft_00001a84_00000000_17_cuda_device_runtime_compute_61_cpp1_ii_8b1a5d37'

Here is my makefile:

INCFILES=-I/usr/local/cuda-8.0/include -I/opt/mpi/mvapich2-gnu/2.2/include -I./
LIBFILES=-L/usr/local/cuda-8.0/lib64 -L/opt/mpi/mvapich2-gnu/2.2/lib
LIBS=-lcudart -lcudadevrt -lcublas_device -lmpi 
ARCH=-gencode arch=compute_60,code=sm_60
NVCC=nvcc -ccbin g++


default: all

all: clean final.o

io.o: io.cpp
        g++ -c -std=c++11  io.cpp 


final.o: io.o a.cu b.cu c.cu d.cu e.cu f.cu main.cpp
        $(NVCC) -std=c++11 $(INCFILES) $(LIBFILES) $(LIBS) -g -G -Xptxas -v -dc $(ARCH) a.cu b.cu c.cu d.cu e.cu f.cu
        $(NVCC) -std=c++11 $(ARCH) $(INCFILES) $(LIBFILES) $(LIBS) -rdc=true -dlink a.o b.o c.o d.o e.o f.o io.o -o cudafiles.o
        mpicxx -O3 $(INCFILES) $(LIBFILES) -c main.cpp -o main.o
        mpicxx $(INCFILES) $(LIBFILES) $(LIBS) cudafiles.o a.o b.o c.o d.o e.o f.o io.o main.o -o exec

clean:
        rm -rf *.o exec

1. add `-dlink` to your 2nd `(NVCC)` line. 2. change `main.c` to `main.cpp`, both the filename itself as well as all references in the makefile. 3. switch from `mpicc` to `mpicxx` Note that you don't show `g.o` getting built anywhere, and you don't show `io.o` getting used anywhere, so I'm guessing this not really the makefile you are using. — Robert Crovella, Apr 03 '18 at 20:00
Thanks Robert. I've made the edits you suggested (see edited file above). Now, I get the following errors: "undefined reference to cudaFree, cudaSetDevice") — user2330963, Apr 03 '18 at 20:35
Thanks Robert. Even after adding -lcudart to libs, I get linking errors (as shown above). Thanks for helping again! — user2330963, Apr 03 '18 at 21:17
1. define your `LIBS` variable like this: `LIBS=-lcublas_device -lcudadevrt -lmpi -lcudart` 2. In your final link step, move the `$(LIBS)` reference to the end of the line, like so: `mpicxx $(INCFILES) $(LIBFILES) cudafiles.o a.o b.o c.o d.o e.o f.o io.o main.o -o exec $(LIBS)` — Robert Crovella, Apr 03 '18 at 21:49

Robert Crovella · Accepted Answer · 2018-04-03T23:16:01.123

The original problem reported was an undefined reference to main. This was arising from this line in the Makefile:
```
$(NVCC) -std=c++11 $(ARCH) $(INCFILES) $(LIBFILES) $(LIBS) -rdc=true a.o b.o c.o d.o e.o f.o io.o -o cudafiles.o
```
As constructed, this actually instructs nvcc to perform full/final linking. However the intent of this line was to perform the device-link step only, required when compiling with -rdc=true or -dc, and when not performing the final link with nvcc. In this case, the final link was being performed by mpicc/mpicxx. To perform the device-link step only, we need to specify -dlink. Without that switch, nvcc expects to do final linking, but fails because none of the supplied objects contain a main function. The correct solution, since we have no intent to do final link at this point, is to use the -dlink switch.
I also suggested converting everything to C++ style linking, since nvcc links that way. It might be possible to sort out a C-style link with a C++-style link, but this just seems troublesome to me. Therefore I suggested converting the only .c file (main.c) to a .cpp file, and convert from mpicc to mpicxx
The next problem that arose was undefined references to e.g. cudaSetDevice() and cudaFree(). These are part of the CUDA runtime API library ("libcudart"). When performing final link with nvcc, these are linked automatically. But since final link is being performed by mpicxx (basically a wrapper on g++), it's necessary to call out the link against that library specifically with -lcudart.
Finally, the remaining problem was a link-order problem. In a nutshell, link dependencies need to be satisfied from left to right in the linker command line. Different compilers are more or less picky about this. The final reordering changes were to specify the libraries to link against in the correct order, and also to specify these libraries at the end of the link command line, so that any dependencies on these libraries, to their left in the link command line, are satisfied.

compile multiple cuda files (that have dynamic parallelism) and MPI code

1 Answers1