Scrapy does not have command 'crawl'

Question

I started to learn Scrapy but right away I get an error Unknown command: crawl. I do not know why im getting this, but in py Scrapy commands I do not have that command. Im using python 3.6 and pycharm as editor.

(venv) C:\Users\Pc\PycharmProjects\web skreper\venv\Scripts>scrapy crawl quotes
Scrapy 1.7.3 - no active project

Unknown command: crawl

Use "scrapy" to see available commands

(venv) C:\Users\Pc\PycharmProjects\web skreper\venv\Scripts>scrapy
Scrapy 1.7.3 - no active project

Usage:
  scrapy <command> [options] [args]

Available commands:
  bench         Run quick benchmark test
  fetch         Fetch a URL using the Scrapy downloader
  genspider     Generate new spider using pre-defined templates
  runspider     Run a self-contained spider (without creating a project)
  settings      Get settings values
  shell         Interactive scraping console
  startproject  Create new project
  version       Print Scrapy version
  view          Open URL in browser, as seen by Scrapy

This is the basic code that I wrote:

import scrapy

class My_Scraper(scrapy.Spider):

    name = 'quotes'

    #url sajtova koje scrapujem
    start_urls = [
        'https://www.nekretnine.rs/'
        ]

    def parse(self, response):

        title = response.css('title').extract()

        yield {'title text': title}

Possible duplicate of [unknown command: crawl error](https://stackoverflow.com/questions/10123104/unknown-command-crawl-error) — Rob Bricheno, Sep 25 '19 at 11:24
You should run scrapy crawl spider_name command being in a scrapy project folder, where the scrapy.cfg file resides. https://stackoverflow.com/questions/10123104/unknown-command-crawl-error — Rob Bricheno, Sep 25 '19 at 11:25

score 8 · Accepted Answer · answered Sep 25 '19 at 12:24

You need to be inside the project folder within the Scrapy folder.

You are currently trying to run the command from C:\Users\Pc\PycharmProjects\web skreper\venv\Scripts

but it should be something like C:\Users\Pc\PycharmProjects\web skreper\venv\Scripts\Scrapy\My_Scraper

score 7 · Answer 2 · answered Sep 26 '19 at 08:01

7

You must get out from the script folder and move to the project directory where scrapy.cfg file placed.

Run the scrapy crawl from there it will work.

answered Sep 26 '19 at 08:01

Arun Augustine

1,690
1
13
20

score 3 · Answer 3 · edited Aug 30 '20 at 07:01

3

I know this is an old post but I would also like to contribute an additional instance that may result in the missing of crawl command

if you accidentally delete the cfg file, crawl command will disappear too.

edited Aug 30 '20 at 07:01

Arghya Sadhu

41,002
9
78
107

answered Aug 30 '20 at 06:31

Andes Lam

192
2
8

Scrapy does not have command 'crawl'

3 Answers3