I'm trying to create a web crawler using crawlee and apify With NodeJS. This crawler works when I run on the localhost, but when I run on docker container, I receive a timeout error waiting for locator. The docker container was built correctly but when the web crawler starts running I get the timeout error.
I'm using Playwright and Firefox browser. I need to know, what are the possible reasons why this is happening?
The error:
INFO PlaywrightCrawler: Collecting results from league 2
WARN PlaywrightCrawler: Reclaiming failed request back to the list or queue. page.waitForSelector: Timeout 30000ms exceeded.
=========================== logs ===========================
waiting for locator('.vrl-MeetingsHeaderButton') to be visible
============================================================
at navigateToLeague (file:///home/myuser/dist/handlers/league.js:3:16)
The Dockerfile:
# Specify the base Docker image. You can read more about
# the available images at https://crawlee.dev/docs/guides/docker-images
# You can also use any other image from Docker Hub.
FROM apify/actor-node-playwright-firefox:16 AS builder
# Copy just package.json and package-lock.json
# to speed up the build using Docker layer cache.
COPY --chown=myuser package*.json ./
# Install all dependencies. Don't audit to speed up the installation.
RUN npm install --include=dev --audit=false
# Next, copy the source files using the user set
# in the base image.
COPY --chown=myuser . ./
# Install all dependencies and build the project.
# Don't audit to speed up the installation.
RUN npm run build
# Create final image
FROM apify/actor-node-playwright-firefox:16
# Copy just package.json and package-lock.json
# to speed up the build using Docker layer cache.
COPY --chown=myuser package*.json ./
# Install NPM packages, skip optional and development dependencies to
# keep the image small. Avoid logging too much and print the dependency
# tree for debugging
RUN npm --quiet set progress=false \
&& npm install --omit=dev --omit=optional \
&& echo "Installed NPM packages:" \
&& (npm list --omit=dev --all || true) \
&& echo "Node.js version:" \
&& node --version \
&& echo "NPM version:" \
&& npm --version \
&& rm -r ~/.npm
# Copy built JS files from builder image
COPY --from=builder --chown=myuser /home/myuser/dist ./dist
# Next, copy the remaining files and directories with the source code.
# Since we do this after NPM install, quick build will be really fast
# for most source file changes.
COPY --chown=myuser . ./
# Run the image. If you know you won't need headful browsers,
# you can remove the XVFB start script for a micro perf gain.
CMD ./start_xvfb_and_run_cmd.sh && npm run start:prod --silent
I was hoping that the dockerfile templates provided by apify would make my crawler work like it works locally
