3

This question is all over SO but I'm not entirely sure if this solution even works.

I have a scrapy script that I would like to convert to an .app file. There's a program called Platypus[] that uses a GUI to convert python projects into an application. Ultimately I want to just have something I or someone else can interact with and have the spider start running.

So i was thinking

make a script to run spider -> feed that into platypus -> create program that someone would just have to click on

The argument I normally run through terminal is

scrapy crawl [spiderName] -t csv -o - > "pathName"

After some research I found this might be the most simple way of pulling it off

import os
os.system('scrapy crawl [spiderName] -t csv -o - > "pathName"')

I have this program in the same directory as my spider, but when I run it nothing happens. [Spider does not run]

Another Question: I know I would have to adjust the script to run the spider on someone else's machine for the directory (which is fine) but would I have to install scrapy on the other person's machine as well?

Thanks for any help!

Edit: When trying to use subprocesses, same result where Spider was not ran

subprocess.Popen('scrapy crawl [spiderName] -t csv -o - > "pathName"', shell = True)

Edit #2: The way to do subprocesses is this:

process = subprocess.Popen(['scrapy', 'crawl', 'homeDepotSpider'] , shell=True)

However when I run this I don't this it is being ran in the correct directory even though the runspider program is in the same directory as my spider.

After typing 'ls' into terminal I found out that I'm in the desktop directory

enter image description here

chrisHG
  • 80
  • 1
  • 2
  • 18
  • have you looked into using `subprocess.Popen`? As `os.system` is simply a wrapper function for `Popen`, you might have more control over how things are run and have better success. For your second question, yes you'd most likely have to install scrapy on the other person's machine as well, but you can make an install script that should make things seamless. – M Z Jul 07 '20 at 18:57
  • @MZ thanks! I'll look into it! – chrisHG Jul 07 '20 at 19:21

1 Answers1

2

Figured it out, feel dumb for missing this one.

I wasn't in the correct directory (Not sure if this is something only for OSX)

os.chdir(path name of folder)
chrisHG
  • 80
  • 1
  • 2
  • 18