2

I have scrapyd installed in Ubuntu.

I also have a website with SSL enabled, I need to make request to links like https://IP_HERE:6800/listjobs.json?project=default inside my website.

But it looks like Scrapyd does not work with HTTPS.

Even if I open link in browser it just keeps loading and loading.

But if I make request using http:// instead of https:// it works. But I want it to work with HTTPS.

I thought I need to edit my SSL conf file to work with port 6800. I did but still its not working.

Here is my SSL config file looks like.

<IfModule mod_ssl.c>
    <VirtualHost *:443 *:6800>
    .... and rest of confguration...
Umair Ayub
  • 19,358
  • 14
  • 72
  • 146
  • Odd, can you paste the spider output when trying to connect to the https website? – Rafael Almeida Mar 14 '17 at 08:14
  • @RafaelAlmeida not spider sir ... on 6800 port, I have deployed `Scrapyd`, not `Scrapy` ... and then visiting URL with HTTPS it shows `This site can’t be reached` – Umair Ayub Mar 14 '17 at 09:22

1 Answers1

1

By looking at the source code of scrapyd, it uses a TCPServer from Pythons socketserver module. It is not possible to enable SSL in a Python module via the Apache config file.

What you want to use is a HTTPS-to-HTTP proxy, which wraps up scrapyd's HTTP into an HTTPS protocol. You can use Apache for that, see this tutorial from Digital Ocean or this blog post.

rfelten
  • 181
  • 6