0

I have Django and Scrapyd both running in separate containers, Django works just fine on my host's localhost:8001 but when I try to access Scrapyd via localhost:8000 I get a "connection reset" error from Firefox. I'm trying to connect both contaiers to each other, a docker network exists already. This worked fine on my host before I moved to Docker. How can I fix this?

docker compose

docker-compose: version: "3.9"

services:   web:
    build: .\web
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - .\web:/usr/src/remindme_web
    ports: 
      - 8001:8000

  scrapy:
    build: .\scraper
    command: scrapyd
    volumes:
      - .\scraper\:/usr/src/remindme_scraper/
    ports: 
      - 8000:8080

Scraper docker file

FROM python:3.9

ENV PYTHONUNBUFFERED=1

WORKDIR /usr/src/remindme_scraper

COPY requirements.txt .

RUN pip install -r requirements.txt

WORKDIR /usr/src/remindme_scraper/remind_me_scraper

COPY . .

RUN ["apt-get", "update"]
RUN ["apt-get", "install", "-y", "vim-tiny"]

Scrapyd log

2021-10-07T21:21:03+0000 [-] Loading /usr/local/lib/python3.9/site-packages/scrapyd/txapp.py...

2021-10-07T21:21:03+0000 [-] Scrapyd web console available at http://127.0.0.1:8080/

2021-10-07T21:21:03+0000 [-] Loaded.

2021-10-07T21:21:03+0000 [twisted.scripts._twistd_unix.UnixAppLogger#info] twistd 21.7.0 (/usr/local/bin/python 3.9.7) starting up.

2021-10-07T21:21:03+0000 [twisted.scripts._twistd_unix.UnixAppLogger#info] reactor class: twisted.internet.epollreactor.EPollReactor.

2021-10-07T21:21:03+0000 [-] Site starting on 8080

2021-10-07T21:21:03+0000 [twisted.web.server.Site#info] Starting factory <twisted.web.server.Site object at 0x7f0882999b80>

2021-10-07T21:21:03+0000 [Launcher] Scrapyd 1.2.1 started: max_proc=48, runner='scrapyd.runner'

Line that connects to API in Django app

scrapyd = ScrapydAPI('http://scrapy')

Scrapyd conf

[scrapyd]
eggs_dir    = eggs
logs_dir    = logs
items_dir   =
jobs_to_keep = 5
dbs_dir     = dbs
max_proc    = 0
max_proc_per_cpu = 4
finished_to_keep = 100
poll_interval = 5.0
bind_address = 0.0.0.0
http_port   = 8080
debug       = off
runner      = scrapyd.runner
application = scrapyd.app.application
launcher    = scrapyd.launcher.Launcher
webroot     = scrapyd.website.Root

[services]
schedule.json     = scrapyd.webservice.Schedule
cancel.json       = scrapyd.webservice.Cancel
addversion.json   = scrapyd.webservice.AddVersion
listprojects.json = scrapyd.webservice.ListProjects
listversions.json = scrapyd.webservice.ListVersions
listspiders.json  = scrapyd.webservice.ListSpiders
delproject.json   = scrapyd.webservice.DeleteProject
delversion.json   = scrapyd.webservice.DeleteVersion
listjobs.json     = scrapyd.webservice.ListJobs
daemonstatus.json = scrapyd.webservice.DaemonStatus
Denzel Hooke
  • 65
  • 6
  • When you say you're trying to connect the containers together, you mean you want django to make a request to scrapy? And in django, it's configured to make that request to localhost:8000? – Rickkwa Oct 07 '21 at 23:41
  • Yeah I want to make a request to the Scrapyd api but no, in my code I'm, attempting to make a request like https:// scraper since scraper is my host name for the scraper service. – Denzel Hooke Oct 08 '21 at 02:21
  • Can you show me the line of python that starts up the scraper api? And to be clear, currently you're just trying to get localhost:8000 to work on your browser rather than the other container? – Rickkwa Oct 08 '21 at 02:39
  • Yes correct, though I would rather get it to work within the Django container. I've added the line to my answer that attempts to call the API. The line that starts the scraper API is located in the command section of the scraper service in the docker compose, "scrapyd". – Denzel Hooke Oct 08 '21 at 03:04
  • Ya just seen your answer to binding it to 0.0.0.0...this is very strange. It should be working – Denzel Hooke Oct 08 '21 at 03:11

1 Answers1

0

Turns out the problem was that I accidentally commented out my url variable within the scrapy.cfg file. Don't forget to un-comment that variable out!

url = http://localhost:8080/
Denzel Hooke
  • 65
  • 6