Running Map/Reduce python programs from Sublime Text 2

Question

I just started a tutorial series on map reduce and Hadoop. The set up instructions call for using an IDE called Canopy with MRjob. I have installed both, and everything works. But... If Canopy is just a Python IDE couldn't i use anything in its place (e.g. intellij or Sublime Text)?

when i run the same python script in Sublime Text i get the following error:

Traceback (most recent call last):
   File "../Rating-Counter.py", line 1, in <module> from mrjob.job import MRJob
ImportError: No module named mrjob.job

here is the script:

from mrjob.job import MRJob

class MRRatingCounter(MRJob):
    def mapper(self, key, line):
        (userID, movieID, rating, timestamp) = line.split('\t')
        yield rating, 1

    def reducer(self, rating, occurences):
        yield rating, sum(occurences)

if __name__ == '__main__':
    MRRatingCounter.run()

I just used pip install for the MRjob, is there somewhere i can do a custom install and put it in a place that Sublime Text can see it? i have multiple versions of python, could it be the case that pip installed MRjob for 2.7 and not 3.4?

Update: I have tried using pip3 install MRjob with this error returning:

Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/gj/h8lrdpmx7lg3bq2_9t9j5kbr0000gn/T/pip-build-le53azna/MRjob

Answer 1

Virtualenv:

virtualenv is a tool to create isolated Python environments. virtualenv creates a folder which contains all the necessary executables to use the packages that a Python project would need.

Install virtualenv via pip:

$ pip install virtualenv

Basic Usage Create a virtual environment for a project:

$ cd my_project_folder
$ virtualenv venv

virtualenv venv will create a folder in the current directory which will contain the Python executable files, and a copy of the pip library which you can use to install other packages. The name of the virtual environment (in this case, it was venv) can be anything; omitting the name will place the files in the current directory instead.

This creates a copy of Python in whichever directory you ran the command in, placing it in a folder named venv.

You can also use a Python interpreter of your choice.

$ virtualenv -p /usr/bin/python2.7 venv

This will use the Python interpreter in /usr/bin/python2.7

To begin using the virtual environment, it needs to be activated:

$ source venv/bin/activate

The name of the current virtual environment will now appear on the left of the prompt (e.g. (venv)Your-Computer:your_project UserName$) to let you know that it’s active. From now on, any package that you install using pip will be placed in the venv folder, isolated from the global Python installation.

Install packages as usual, for example:

$ pip install requests

If you are done working in the virtual environment for the moment, you can deactivate it:

$ deactivate

This puts you back to the system’s default Python interpreter with all its installed libraries.

To delete a virtual environment, just delete its folder. (In this case, it would be rm -rf venv.)

After a while, though, you might end up with a lot of virtual environments littered across your system, and its possible you’ll forget their names or where they were placed.

Referenced from:

http://docs.python-guide.org/en/latest/dev/virtualenvs/

This will do exactly what you want, and is made for different python requirements on the same machine. It works great.