R and data.table on AWS

Question

This is a very strange error I am running into trying to install a specific R library on AWS EC2 instance (Amazon Linux AMI). A link in the AWS forums here, posted by someone else, actually highlights my issue well.

The main / relevant part of the error message is:

data.table.h:6:12: fatal error: omp.h: No such file or directory
#include <omp.h>

I did some research (while I'm not sure), I think this is related to whether or not there is an OpenMP enabled compiler on the server. The data.table installation page on GitHub itself discusses this a bit here, but I'm not sure how to update or fix this on my EC2 instance.

Any help with this is appreciated.

EDIT - this is a new problem, as i was able to successfully install data.table on a previous, similar EC2 instance less than a month ago.

EDIT 2 - I got around this issue by taking a previous EC2 instance of mine with data.table already installed on it, and creating a custom AMI from it. By using this custom AMI when launching new instances, they already came with the data.table library installed. If I notice AWS resolve this issue on its own, I'll try to remember to come back and update this post!

I deleted the instance that I was struggling with, so I'm not sure what that file had in it. — Canovice, Feb 08 '18 at 06:01
I found the following instructions and these worked for us: https://stackoverflow.com/questions/48307251/aws-linux-server-install-r-package — smooth, Feb 12 '18 at 16:03
this had a similar answer: https://stackoverflow.com/questions/48307251/aws-linux-server-install-r-package — snowkeep, Mar 03 '18 at 00:47
This answer may help: https://stackoverflow.com/questions/48307251/aws-linux-server-install-r-package — snowkeep, Mar 03 '18 at 00:50

score 11 · Accepted Answer · answered Feb 10 '18 at 19:19

The problem here is that data.table doesn't play nice with the default gcc compiler (gcc72-c++.x86_64 gets installed as a dependency of R-devel.x86_64). Point R to an older version by adding

CC=gcc64

in ~/.R/Makevars. If you start from a "clean" Amazon Linux AMI this file doesn't exist and you can just type

mkdir ~/.R
echo "CC=gcc64" >> ~/.R/Makevars

score 10 · Answer 2 · answered Apr 17 '18 at 10:45

For some reason setting CC=gcc64 in the ~/.R/Makevars did not work for me. R was still using default gcc to compile.

However there is another option. You can edit the Makeconf file that R uses during compilation directly. If you're using Amazon Linux the file location is /usr/lib64/R/etc/Makeconf. Once you locate the file the trick is just the same, that is to change the CC = gcc to CC = gcc64. You also might want to make sure that gcc64 is installed by running sudo yum install gcc64.

`sudo sed -i 's/CC = gcc -m64/CC = gcc64 -m64/g' /usr/lib64/R/etc/Makeconf` — nate, Dec 20 '18 at 00:22

score 2 · Answer 3 · answered Apr 16 '18 at 18:44

I have the following workaround.

1) Connect with ssh.

2) Download package source

wget https://cran.r-project.org/src/contrib/data.table_1.10.4-3.tar.gz

3) Install package

R CMD INSTALL data.table_1.10.4-3.tar.gz

This will install the package in the local package directory, not necessarily the one of the RStudio user(s).

4) Use .libPaths() in R and RStudio to figure out where packages are stored.

5) Copy package into the desired directory using sudo cp -R SOURCE DEST.

Remark: This is a workaround not a solution because you need to install each package manually this way, i.e. handling dependencies can be cumbersome, but at least data.table works now.

R and data.table on AWS

3 Answers3

Linked