0

I've created a scraper with Scrapy and wxPython which works as expected, exporting a file with results to the desktop in CSV format. I'm attempting to package this into an executable with cx_Freeze using the below command prompt line:

cxfreeze ItemStatusChecker.py --target-dir dist

This seems to work fine, building the dist directory with ItemStatusChecker.exe

However, when I open ItemStatusChecker.exe, I get the below error in the command prompt and my GUI does not launch:

Traceback (most recent call last):
    File "C:\Python27\lib\site-packages\cx_Freeze\initscripts\Console.py", line 27, in <module>
      exec code in m.__dict__
    File "ItemStatusChecker.py", line 6, in <module>
    File "C:\Python27\lib\site-packages\scrapy\__init__.py", line 6, in <module>
      __version__ = pkgutil.get_data(__package__, 'VERSION').strip()
    File "C:\Python27\lib\pkgutil.py", line 591, in get_data
      return loader.get_data(resource_name_)
IOError: [Errno 2] No such file or directory: 'scrapy\\VERSION'

I've tried running it through py2exe as well. This also seems to work fine, creating the dist directory, but I get a very similar error when trying to launch the exe:

Traceback (most recent call last):
  File "ItemStatusChecker.py", line 6, in <module>
  File "scrapy\__init__.pyc", line 6, in <module>
  File "pkgutil.pyc", line 591, in get_data
IOError: [Errno 2] No such file or directory: 'scrapy\\VERSION'

I'm a python newbie, so let me know if I left out any necessary details. Thank you in advance for any advice you can offer!

user994585
  • 661
  • 3
  • 13
  • 28
  • 1
    It sounds like scrapy doesn't work well in frozen applications. Can I shamelessly plug [Pynsist](http://pynsist.readthedocs.org/en/latest/)? It avoids these problems by making an installer that makes start menu shortcuts directly to your Python code. – Thomas K Apr 29 '14 at 01:01
  • @ThomasK Thanks Thomas! I just tried working with Pynsist, but I'm having trouble and getting a 404 Client Error: Not Found. I believe I'm not using the right function in the 'entry_point', but I don't know what I should put. [This](http://jsfiddle.net/yD9RM/) is the core file of my program. I don't have a 'main' function within it. This is probably a noob mistake, but what should I be putting for 'entry_point=myapp:main'? Any advice you can offer is greatly appreciated! – user994585 Apr 29 '14 at 17:18
  • The easiest way to use Pynsist is to tweak your program to have a function that launches it. It's an easy change to make - have a look here: https://gist.github.com/takluyver/11408873 – Thomas K Apr 29 '14 at 18:59
  • @ThomasK Thanks that makes a lot of sense. I've implemented the changes, but I seem to still be getting the same error, so I think it may be something different that the 'entry_point' causing this. [This is the full error](https://gist.github.com/anonymous/156cd76c9d95a8256e09). Any more ideas? I really appreciate the help! – user994585 Apr 29 '14 at 19:59
  • When you specify the Python version in the `installer.cfg` file, it needs to be a version with precisely 3 parts, like `2.7.6`. If you leave that line out, it should default to `2.7.6` if you're using Python 2. I'll try to make the error message clearer. – Thomas K Apr 29 '14 at 21:35
  • 1
    @ThomasK Thanks, that worked! I managed to get everything packaged into an installer. When I install, everything appears to install correctly, but when I open the software from the start menu, nothing happens. Any advice on how I can troubleshoot this? Thanks! – user994585 May 05 '14 at 18:20
  • If you look in the installation directory (probably `C:\Program Files\YourApp`), there should be a `.log` file with a record of the error that your program hit. If not, try running it from the command line with `py "C:\Program Files\YourApp\yourapp.launch.py"`. – Thomas K May 05 '14 at 18:35
  • Thanks @ThomasK - That worked, I was able to fix the issue. Now I'm able to install the program and open my GUI successfully. However, the main function of my program is not working. On clicking the submit button on my GUI, my scrapy pipeline exports my scraped data to CSV and saves it to file on the desktop. However, it seems to skip over this on the installed version. [This](https://gist.github.com/anonymous/6509439bd00e49f50e06) is the scrapy pipeline I'm using to determine the path - I think it may be a path issue, but not sure since it works fine as a py script. Any advice on how to fix? – user994585 May 06 '14 at 17:35
  • I'm not familiar with scrapy, but try running the installed copy of the script from the command line (with the command I posted before) to see if there are error messages. BTW, what was the change you had to make for the last problem? It would be good to know what sort of issues people have when using Pynsist. – Thomas K May 06 '14 at 23:37
  • @ThomasK I just tried from the command line and it actually works 100% that way with no errors. It's when I run it through the start menu that it seems to skip the CSV somehow. This leads me to believe it has something to do with my Pynsist setup. Is that a fair assumption to make? The issue I had before is that I was saving the lines entered in my GUI into an existing text file before processing to my spider. It couldn't find the text file, which I assume was a path error, but not sure. I ended up removing that aspect of my code, sending straight to the spider instead, which solved the issue. – user994585 May 07 '14 at 20:13
  • Just checked and it also works by just double clicking the `myapp.launch.pyw` file directly from `Program Files/MyApp`. It seems to be the start menu shortcut specifically where the issue is occuring... any ideas? Appreciate all your help with this. – user994585 May 07 '14 at 20:17
  • The start menu shortcut sets the working directory to your home directory (`%HOMEDRIVE%\%HOMEPATH%`), whereas when you double click the file, the working directory will be the application directory. Could that make a difference? – Thomas K May 08 '14 at 15:58
  • @ThomasK I think this is the issue, though I don't know why. I did some more testing in the command line and I must have been in the application directory when running yesterday. If I run your command from the default C:\myuser\ directory(which I believe is %HOMEDRIVE%\%HOMEPATH%, right?), it does not work, but if I cd into the application directory and run it, everything works fine. Anything I can do to make the working directory be the application directory on the start menu shortcut? – user994585 May 08 '14 at 16:58
  • I just managed to get it working by changing the "Start in" directory for my start menu shortcut to `%PROGRAMFILES%\MyApp`. Is there any way I can have it do this by default when installing, so that users I distribute to don't need to change it manually? – user994585 May 08 '14 at 18:32
  • It's good practice to write your application so that it doesn't depend on where it's started. As a quick workaround, you can change the working directory in your code to the folder the script is in: `os.chdir(os.path.dirname(sys.argv[0]))`. I've opened [an issue](https://github.com/takluyver/pynsist/issues/20) to think about making this configurable in Pynsist, but I'm not convinced yet. – Thomas K May 09 '14 at 21:33
  • @ThomasK Thanks for the advice! Your workaround worked - I'd like it to work independently of this, but being my 1st python app, I'm not sure what is causing it or how to fix. I've got the app running on a coworkers computer thanks to Pynsist, but they had to install pywin32 separately as I could not get it packaged with Pynsist. I've tried adding win32 & pywin32 to the packages section of installer.cfg, but it says that those modules do not exist. I'm not sure if I'm using the wrong name or do something else wrong, but I can't get it packaged within Pynsist. Any advice on troubleshooting? – user994585 May 14 '14 at 14:16
  • Great! Pywin32 installs a [collection of modules](http://docs.activestate.com/activepython/2.7/pywin32/modules.html) - you'll need to list the ones that your application needs under `packages`. – Thomas K May 14 '14 at 17:11
  • Thanks Thomas. It appears win32api is the problem, but I havent been able to resolve. It doesnt seem to install correctly unless I run the actual pywin32 installer. When I run through the command line, I get [this error](https://gist.github.com/anonymous/e430684cb42034136969). On running `from scrapy.core.downloader.handlers import s3` in the python cmd line, I get [this error](https://gist.github.com/anonymous/f38ce84ac9fc8888521e). On adding win32api to the Pynsist packages, it seems to only add `win32api.pyd` to the `pkgs` folder. I'm guessing there's DLL that didn't transfer, but not sure? – user994585 May 15 '14 at 13:58
  • .pyd files are DLLs with a different extension, IIRC. If you just start a Python command line, it won't look in your `...\Application\pkgs\` folder for imports. What happens if you start the Python command line in the pkgs folder, and then try that import? P.S. Spotted [another bug](https://github.com/takluyver/pynsist/issues/24) from one of your gists - thanks. – Thomas K May 16 '14 at 00:44
  • Thanks Thomas - when I change directories to the `Program Files\App\pkgs` folder and run the command, I get pretty much [the same error](https://gist.github.com/anonymous/d779459f62875d22774a), except the last line says `ImportError: DLL load failed: The specified module could not be found.` instead of `ImportError: No module named win32api`. – user994585 May 16 '14 at 14:05
  • I used [Dependency Walker](http://www.dependencywalker.com/) to examine win32api.pyd. It appears to need `pywintypes27.dll`, which is also installed as part of pywin32. It might be easiest to just copy everything from the `PLATLIB` directory of the pywin32 installer into pynsist_pkgs (described in [this FAQ entry](http://pynsist.readthedocs.org/en/latest/faq.html#building-on-other-platforms)). – Thomas K May 16 '14 at 19:37

2 Answers2

0

It sounds like cxfreeze and py2exe aren't picking up the scrapy package automatically. For cx_freeze, you'll probably need to take look at the --include-modules directive:

For py2exe, you will want to create a setup.py script that has an includes list specified. Here's a tutorial that shows how to do it:

Mike Driscoll
  • 32,629
  • 8
  • 45
  • 88
  • Thanks Mike! I've created the a setup.py using the example you linked to, with `includes = ['scrapy']` but I'm still receiving the same error. Am I doing something wrong? [Full setup.py here](http://jsfiddle.net/MyFKK/) – user994585 Apr 29 '14 at 16:25
  • Looks right to me. Is scrapy installed as an egg? I know py2exe doesn't get along with eggs very well at all. You may have to go down the rabbit hole and ask the guys on the py2exe list. – Mike Driscoll Apr 29 '14 at 17:48
0

Take a look to (http://cx-freeze.readthedocs.org/en/latest/faq.html#using-data-files)

My solution using cx-freeze:

Modify the file C:\Python2.7\Lib\site-packages\scrapy__init__.py

import sys
# Scrapy version
import pkgutil
if getattr(sys, 'frozen', False):
    __version__ = 'VERSION'.decode('ascii').strip()
else:
    __version__ = pkgutil.get_data(__package__, 'VERSION').decode('ascii').strip()

and the file C:\Python2.7\Lib\site-packages\scrapy\responsetypes.py

import sys
from mimetypes import MimeTypes
from pkgutil import get_data
...
   def __init__(self):
        self.classes = {}
        self.mimetypes = MimeTypes()
        if getattr(sys, 'frozen', False):
            mimedata = 'mime.types'
        else:
            mimedata = get_data('scrapy', 'mime.types')
        self.mimetypes.readfp(StringIO(mimedata))

And in setup.py include this build exe option

"include_files": [("C:\\Python27\\Lib\\site-packages\\scrapy\\VERSION","VERSION"),
    ("C:\\Python27\\Lib\\site-packages\\scrapy\\mime.types","mime.types")]}
mcandal
  • 407
  • 4
  • 10