How to install libparquet-dev on Docker so I can use R's {arrow}?

Question

I am basing my docker image on https://hub.docker.com/r/rocker/tidyverse/dockerfile

So I tried to add the following line to the docker file to try and install libparquet-dev which is required to use Arrow from R.

RUN apt-get update -qq && apt-get -y --no-install-recommends install \ libparquet-dev which complains about E: Unable to locate package libparquet-dev, so I tried to follow this guide and added the follwoing lines

RUN apt update && \
        apt install -y -V apt-transport-https gnupg lsb-release wget && \
        wget -O /usr/share/keyrings/apache-arrow-keyring.gpg https://dl.bintray.com/apache/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-keyring.gpg && \
        sudo tee /etc/apt/sources.list.d/apache-arrow.list <<APT_LINE \
        deb [arch=amd64 signed-by=/usr/share/keyrings/apache-arrow-keyring.gpg] https://dl.bintray.com/apache/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/ $(lsb_release --codename --short) main \
        deb-src [signed-by=/usr/share/keyrings/apache-arrow-keyring.gpg] https://dl.bintray.com/apache/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/ $(lsb_release --codename --short) main \
        APT_LINE  && \
        apt update && \
        apt install -y -V libarrow-dev && \
        apt install -y -V libarrow-glib-dev && \ 
        apt install -y -V libarrow-flight-dev && \
        apt install -y -V libplasma-dev && \
        apt install -y -V libplasma-glib-dev && \
        apt install -y -V libgandiva-dev && \
        apt install -y -V libgandiva-glib-dev && \
        apt install -y -V libparquet-dev  && \
        apt install -y -V libparquet-glib-dev

which is now complain about

2019-11-13 03:56:56 (116 KB/s) - ‘/usr/share/keyrings/apache-arrow-keyring.gpg’ saved [44156/44156]

tee: 'signed-by=/usr/share/keyrings/apache-arrow-keyring.gpg]': No such file or directory
tee: 'https://dl.bintray.com/apache/arrow/debian/': No such file or directory
tee: '[signed-by=/usr/share/keyrings/apache-arrow-keyring.gpg]': No such file or directory
tee: 'https://dl.bintray.com/apache/arrow/debian/': No such file or directory

So how do I install libparquet-dev on Docker?

Edit Putting the above into a .sh file and then just run that instead of putting them in a RUN command seems to help but I get another error now

The following packages have unmet dependencies:
 libplasma-dev : Depends: libarrow-cuda-dev (= 0.15.1-1) but it is not going to be installed
                 Depends: libplasma15 (= 0.15.1-1) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

score 0 · Answer 1 · answered Nov 13 '19 at 05:01

Firstly, that's wrong Docker syntax so

put the instruction in an .sh file and just run that file instead.
Shorten the installation to below so you only install the libparquet-dev and not all the others

apt update
apt install -y -V apt-transport-https curl gnupg lsb-release
tee /etc/apt/sources.list.d/backports.list <<APT_LINE
deb http://deb.debian.org/debian $(lsb_release --codename --short)-backports main
APT_LINE
curl --output /usr/share/keyrings/apache-arrow-keyring.gpg https://dl.bintray.com/apache/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-keyring.gpg
tee /etc/apt/sources.list.d/apache-arrow.list <<APT_LINE
deb [arch=amd64 signed-by=/usr/share/keyrings/apache-arrow-keyring.gpg] https://dl.bintray.com/apache/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/ $(lsb_release --codename --short) main
deb-src [signed-by=/usr/share/keyrings/apache-arrow-keyring.gpg] https://dl.bintray.com/apache/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/ $(lsb_release --codename --short) main
APT_LINE
curl https://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add -
tee /etc/apt/sources.list.d/llvm.list <<APT_LINE
deb http://apt.llvm.org/$(lsb_release --codename --short)/ llvm-toolchain-$(lsb_release --codename --short)-7 main
deb-src http://apt.llvm.org/$(lsb_release --codename --short)/ llvm-toolchain-$(lsb_release --codename --short)-7 main
APT_LINE
apt update
apt install -y -V libparquet-dev

the finalised Dockerfile should look like, and noticed that installation of {arrow} should come after installation of libparquet-dev because we need to install {arrow} from source, and libparquet-dev needs to be there when installing from source.

FROM rocker/tidyverse

COPY create-parquet.sh create-parquet.sh

RUN chmod +x ./create-parquet.sh

RUN ./create-parquet.sh

RUN install2.r --error \
    --deps TRUE \
    disk.frame \
    arrow

RUN R -e "arrow::install_arrow()"

CMD ["R"]

How to install libparquet-dev on Docker so I can use R's {arrow}?

1 Answers1