0

i want to use multi stage build in docker. the first stage is to install faiss library, and the second stage is to install the main images. here is my script

FROM conda/miniconda3 as install_faiss

WORKDIR /app

RUN conda install -c conda-forge faiss-cpu

FROM python:3.7.11 as base

WORKDIR /app

COPY requirements.txt ./app/requirements.txt
COPY data_train.csv ./app/data_train.csv
COPY train.py ./app/train.py

RUN pip3 install -qr ./app/requirements.txt
COPY --from=install_faiss ./app ./app
RUN python3 ./app/train.py

but after that in may file train.py error getting like cannot import faiss, so i assume that module faiss not copy in the second image. any help ? thanks

newbie programmer
  • 11
  • 2
  • by `COPY --from=install_faiss ./app ./app`, do you mean the conda libraries are installed to workdir? – Lei Yang Mar 14 '22 at 07:41
  • python library installation is never as portable as you think. if the modules can be copied like that, no need to invent `pip` or `conda`. – Lei Yang Mar 14 '22 at 07:49
  • 1
    as long as you dont need some dev packages to build the wheel yourself, there is almost no benefit in going for multi staging. At the end, you need to have that package in your system. If pip can fetch the pre compiled wheel and put it there, then you should use pip to do that. Otherwise, you need to copy the dependencies from the first state into the second stage. – The Fool Mar 14 '22 at 07:59

1 Answers1

2

I'm not sure how you expect this to work?

The conda install in the first stage runs in the /app working directory, but it will install the faiss-cpu package in conda's site-packages directory. I don't know how the conda/miniconda3 docker image is set up, but I'm quite sure it does not simply install packages in the current working directory.

In the second image you have a python 3.7.11 installation where you installed your requirements.txt. These requirements will be installed in python 3.7.11's site-packages directory. That's also where you want to install faiss-cpu.

There's probably a way to figure out where the correct site-packages directories are in both the miniconda and python base images and then use the correct COPY --from command.

But why do you use conda to install the first dependency and pip for the others? Can't you use a single image and do:

pip3 install faiss-cpu
GeertPt
  • 16,398
  • 2
  • 37
  • 61