2

I submitted a package to CRAN (sundialr) around 6 months back which was recently archived as it fails with the parallel version of make. The exact error message from one of CRAN maintainers is as follows

This have just failed to install for me with a parallel make:

g++ -std=gnu++98 -std=gnu++98 -shared 
-L/data/blackswan/ripley/extras/lib64 -L/usrlocal/lib64 -o sundialr.so 
cvode.o RcppExports.o -L/data/blackswan/ripley/R/R-patched/lib -lRlapack 
-L/data/blackswan/ripley/R/R-patched/lib -lRblas -lgfortran -lm 
-lquadmath -L../inst/ ../inst/libsundials_all.a
g++: error: ../inst/libsundials_all.a: No such file or directory
make[1]: *** [/data/blackswan/ripley/R/R-patched/share/make/shlib.mk:6: 
sundialr.so] Error 1

The library was still being made ....

It seems the package fails to generate the static library with the parallel make. The easiest solution I could think of for this problem was to force a serial version of make using the .NOTPARALLEL phony command in Makevars and Makevars.win (https://github.com/sn248/sundialr/blob/master/src/Makevars). I have made this change and it seems to work on my machine and on testing on TravisCI and Appveyor (https://github.com/sn248/sundialr).

However, before I re-submit to CRAN, I wanted to get an opinion as to will this be enough to get rid of error with parallel make? Sorry for cross-posting on r-package-devel, it seems my email to the list yesterday did not get to the forum.

Ralf Stubner
  • 26,263
  • 3
  • 40
  • 75
Satya
  • 1,708
  • 1
  • 15
  • 39
  • 2
    I am an r-package-devel co-admin, and I did not a bounce. Subscribe, and make sure you email from the exact same address. – Dirk Eddelbuettel Jan 10 '19 at 21:46
  • Thanks Dirk. I was indeed emailing from a different address. I have re-sent my message. Hopefully, it will get posted to the forum successfully. – Satya Jan 10 '19 at 21:57
  • 1
    There is some stuff in _Writing R Extensions_ about parallel `make` and targets. You could punt and build the static library from `configure` ... – Dirk Eddelbuettel Jan 10 '19 at 22:05
  • Thanks for the comment. The guide does state -_Be careful to create all the necessary dependencies, as there is no guarantee that the dependencies of all will be run in a particular order (and some of the CRAN build machines use multiple CPUs and parallel makes)_ I do not want to sound lazy but for a large external library (from which some of the files are used here), creating that dependency tree sounds like a lot of work and really understanding what is going on in the third party code .. would forcing a serial make not work, since it builds on a single build machine.. – Satya Jan 10 '19 at 22:30
  • 2
    Your dependencies are probably not complete. For instance, you did not tell make that `sundialr.so` depends on `../inst/libsundials_all.a`. In most cases telling make what the dependencies are is not more difficult than telling it what to do: if you have a recipe that looks like `BUILD -o OUPUT INPUTS`, then the dependencies are just `OUPUT: INPUTS` (and, incidentally, you can use [make automatic variables](https://www.gnu.org/software/make/manual/make.html#Automatic-Variables) to simplify your recipe: `BUILD -o $@ I$^`). – Renaud Pacalet Jan 11 '19 at 10:33
  • 1
    I find this really odd. You have `($SHLIB): $(OBJECTS) ../inst/libsundials_all.a` in your [`Makevars`](https://github.com/sn248/sundialr/blob/dbb40e076e6f0576e52cf0a135e29c6b49799c89/src/Makevars#L69), so why does `make` try to build `sundialr.so` before the static library is finished? – Ralf Stubner Jan 11 '19 at 11:06
  • @RalfStubner - Thank you. This happens only with parallel `make` as mentioned in CRAN message, but I agree, I am not sure why `sundialr.so` is built before the static file is finished making. I think `($SHLIB): $(OBJECTS) ../inst/libsundials_all.a` does imply that the static library is a dependency for the shared object, so this target should be built after making the library even in the parallel version. Any thoughts about `.NOTPARALLEL:` phony target usage? – Satya Jan 11 '19 at 14:36
  • @RenaudPacalet - Thank you for your comment. Does `($SHLIB): $(OBJECTS) ../inst/libsundials_all.a` not specify that `libsundials_all.a` is a dependency for the `sundialr.so`? – Satya Jan 11 '19 at 14:48
  • @RalfStubner - Another quick question, hope you can help, is there any way to test the `r` package build with parallel `make` (i.e., locally or using `rhub` etc.)? Thank you. – Satya Jan 11 '19 at 14:53
  • 2
    @SN248: If there is a make variable named `S` and its value is `FOO`, `($SHLIB)` is expanded by make as `(FOOHLIB)`, that is not `sundialr.so`. And if there is no make variable named `S` or its value is the empty string, `($SHLIB)` expands as `(HLIB)`, not `sundialr.so` neither. So, no, this cannot declare the missing dependency. Maybe should it be `$(SHLIB)`, but even then, you should also have a make variable named `SHLIB` somewhere (I could not find it) and its value should be `sundialr.so`. – Renaud Pacalet Jan 11 '19 at 14:54
  • 1
    @RenaudPacalet I think the miss-placed parenthesis is the solution! `$(SHLIB)` is defined by R's build environment. – Ralf Stubner Jan 11 '19 at 15:01
  • 1
    For testing parallel make you can set the environment variable `MAKEFLAGS` to `-j 2`, c.f. https://github.com/coatless/sitmo/blob/ea425902f15baf90e92628dd6cf60795d963b833/.travis.yml#L7. – Ralf Stubner Jan 11 '19 at 15:03
  • @RalfStubner Thank you for your hint regarding `$(SHLIB)` and `MAKEFLAGS`. It is very interesting that while `$(SHLIB)` in `Makevars` works, on Windows, i.e., for `Makevars.win`, `$(SHLIB)` gives me linking errors, but `($SHLIB)` works fine and even passes `CRAN` checks. Very odd! – Satya Jan 12 '19 at 02:46
  • 1
    The linking error on Windows is odd. However, `$(SHLIB)` is documented, so this might warrant a new question. – Ralf Stubner Jan 12 '19 at 10:49

0 Answers0