How could I launch my scrapy script with PHP?

Question

this is for example my script ,i type "scrapy crawl quotes" in the shell to execute it , so how can i launch this command in php ?

import scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = [
    'http://quotes.toscrape.com/page/1/',
    'http://quotes.toscrape.com/page/2/',
]

def parse(self, response):
    for quote in response.css('div.quote'):
        yield {
            'text': quote.css('span.text::text').extract_first(),
            'author': quote.css('small.author::text').extract_first(),
            'tags': quote.css('div.tags a.tag::text').extract(),
        }

score 1 · Answer 1 · answered Apr 27 '18 at 13:02

1

I give you what I use for my project

chdir('/Path_to_folder');
shell_exec('scrapy crawl yourSpider -a firstParam="'.$firstParam.'"' . ' -a secondParam="'.$secondParam.'"');

for more information

Chdir

shell_exec

answered Apr 27 '18 at 13:02

parik

2,313
12
39
67

thank you for the answer , i tried ur script , but how can i print the output of the crawling in php ? – HeyHude Apr 27 '18 at 13:32
you want to print the items? – parik Apr 27 '18 at 13:40
and there is the condition to execute " scrapy crawl myspider " in the main directory of the project ? – HeyHude Apr 27 '18 at 13:41
yes the items in a table for example – HeyHude Apr 27 '18 at 13:41
For your first question read the doc :) and for the second one, if you have no obligation to print your items, it's better to export your result to csv or json https://stackoverflow.com/a/29971406/4527978 – parik Apr 27 '18 at 14:08

score 0 · Accepted Answer · edited Apr 27 '18 at 14:42

0

If I understand you correctly, the correct way to run php files in command line is

php file.php

If this is not what you are asking for please clarify.

EDIT

<?php
$output = shell_exec('scrapy crawl quotes');
echo "<pre>$output</pre>";
?>

This will run the terminal command in php.

Reference

edited Apr 27 '18 at 14:42

parik

2,313
12
39
67

answered Apr 27 '18 at 12:45

Riz-waan

603
3
13

thanks for the answer ,, so how i can define the path of the python script ? – HeyHude Apr 27 '18 at 13:19
You can always just do python /path/to/file/file.py in the shell_exec – Riz-waan Apr 27 '18 at 13:25
@ZouhirSarrar If the python script is in the same location as phone script then no need to do file path. Also it will run the command as www-data user. – Riz-waan Apr 27 '18 at 13:26
@ZouhirSarrar my answer also prints the outcome onto the screen in code format as you can see. If this helped I would appreciate an upvote or check. Thanks – Riz-waan Apr 27 '18 at 23:13

score 0 · Answer 3 · answered Apr 27 '18 at 12:50

To execute an external program from PHP you want to use exec.

An example in your case would be:

<?php

exec('scrapy crawl quotes');

Be careful to consider the current working directory if the PHP is being executed by Apache or Nginx. You'll most likely need a full path to scrapy unless it's already in your $PATH.

How could I launch my scrapy script with PHP?

3 Answers3