How to install xvfb on Scrapinghub for using Selenium?

Question

I use Python-Selenium in my spider (Scrapy), for using Selenium i should install xvfb on Scrapinghub.

when i use apt-get for installing xvfb i have this error message:

E: Could not open lock file /var/lib/dpkg/lock - open (13: Permission denied)                                                                                                                                                                
E: Unable to lock the administration directory (/var/lib/dpkg/), are you root?

Is there any other way for installing xvfb on Scrapinghub?

UPDATE 1

I read this, I tried to use docker, I am stuck at this stage

shub-image init --requirements path/to/requirements.txt

i read this

If you are getting an ImportError like this while running shub-image init: You should make sure you have the latest version of shub installed by running:

$ pip install shub --upgrade

but i have always this error

Traceback (most recent call last):
  File "/usr/local/bin/shub-image", line 7, in <module>
    from shub_image.tool import cli
  File "/usr/local/lib/python2.7/dist-packages/shub_image/tool.py", line 42, in <module>
    command_module = importlib.import_module(module_path)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/usr/local/lib/python2.7/dist-packages/shub_image/push.py", line 4, in <module>
    from shub.deploy import list_targets
ImportError: cannot import name list_targets

developer_hatch · Answer 1 · 2017-06-14T20:32:58.360

3

did you try:

sudo apt-get install xvfb

Another way is to compile manually the packages, a sort of:

apt-get source xvfb
./configure --prefix=$HOME/myapps
make
make install

And the third way, is download the .deb from the source web page https://pkgs.org/download/xvfb

after download it, you can mv it to the path of the downloaded sources:

mv xvfb_1.16.4-1_amd64.deb /var/cache/apt/archives/

then you change your directory and do:

sudo dpkg -i xvfb_1.16.4-1_amd64.deb

and that's all!

edited Jun 14 '17 at 20:32

answered Jun 09 '17 at 15:19

developer_hatch

15,898
3
42
75

On Scrapinghub we can't use sudo, because we are not root – parik Jun 09 '17 at 15:21
@parik okey, I updated the answer, hope that helps, let me know – developer_hatch Jun 09 '17 at 15:40
Thanks for your answer Damian, is it possible to put it in setup.py ? – parik Jun 09 '17 at 15:52
@parik Sure, check this https://www.pydanny.com/python-dot-py-tricks.html, glad to help :) – developer_hatch Jun 09 '17 at 15:55
thanks, i think that it's better to try Docker, because on scrpainghub i don't know what i put instead of ./configure --prefix=$HOME/myapps, i try docker and i wait for an answer from someone from Scrapinghub – parik Jun 09 '17 at 16:05

score 1 · Accepted Answer · answered Jun 14 '17 at 10:05

I resolved my problems ( use selenium in scrapinghub )

1- for xvfb in docker i use

RUN apt-get install -qy xvfb

2- for creating docker image i used this and for installing geckodriver i use this code

#
# Geckodriver Dockerfile
#

FROM blueimp/basedriver

# Add the Firefox release channel of the Debian Mozilla team:
RUN echo 'deb http://mozilla.debian.net/ jessie-backports firefox-release' >> \
  /etc/apt/sources.list \
  && curl -sL https://mozilla.debian.net/archive.asc | apt-key add -

# Install Firefox:
RUN export DEBIAN_FRONTEND=noninteractive \
  && apt-get update \
  && apt-get install --no-install-recommends --no-install-suggests -y \
    firefox \
  # Remove obsolete files:
  && apt-get clean \
  && rm -rf \
    /tmp/* \
    /usr/share/doc/* \
    /var/cache/* \
    /var/lib/apt/lists/* \
    /var/tmp/*

# Install geckodriver:
RUN export BASE_URL=https://github.com/mozilla/geckodriver/releases/download \
  && export VERSION=$(curl -sL \
    https://api.github.com/repos/mozilla/geckodriver/releases/latest | \
    grep tag_name | cut -d '"' -f 4) \
  && curl -sL \
  $BASE_URL/$VERSION/geckodriver-$VERSION-linux64.tar.gz | tar -xz \
  && mv geckodriver /usr/local/bin/geckodriver

USER webdriver

CMD ["geckodriver", "--host", "0.0.0.0"]

from here

Hi parik! I made another update to the last answer, so there are two option to solve the problem, I'm glad you could find a way before, sorry I couldn't answer sooner :/ — developer_hatch, Jun 14 '17 at 20:33
@DamianLattenero thanks man, your answer helped me a lot, i didn't accept it because it doesn't covered all the steps of problems, thanks again — parik, Jun 15 '17 at 07:55

How to install xvfb on Scrapinghub for using Selenium?

2 Answers2