0

Python newbie here.

I am looking for a way to save my python scripts online so that I can work on them from both work and at home. I would like to build some (simple) web scraping tools, using Mechanize, and perhaps Scrapy.

After doing a little bit of research, I guess that what I need is an online IDE (Am I right?). I found c9.io. The problem is that c9.io proved too slow, even for really simple tasks such as opening and reading a page with Mechanize.

Have you any ideas on what I should use for writing, saving and running web scraping scripts online?

Bear in mind that I can not save or run scripts using command line at work. I need something online.

DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
Theo
  • 87
  • 7
  • 1
    Most online IDEs and similar environments are going to be either slow or restricted for doing things like web scraping. You can always just get a *nix shell account somewhere and run your Python scripts there. Or set up a way to ssh/tunnel/whatever into your home computer from work, and just run them on your home computer. – abarnert Aug 13 '13 at 20:31
  • http://repl.it/languages/Python – scohe001 Aug 13 '13 at 20:32
  • @Josh: Considering that `repl.it` doesn't have any way to add third-party modules like `scrapy` and `mechanize`, I'm not sure how that could help. – abarnert Aug 13 '13 at 20:33
  • `lxml` and try http://github.com it's actually really great. – Torxed Aug 13 '13 at 20:33
  • @Torxed: What is lwxml? The DirectoryScanner at http://lwxml.org, or some other thing? And how does github help you run code online? – abarnert Aug 13 '13 at 20:35
  • @abarnert First off, i miss-spelled `lxml` and have a shitty internet connection so it took me a min to fix it :) Secondly, "save my python scripts online" `==` github to me. – Torxed Aug 13 '13 at 20:36
  • @Torxed: How does `lxml` help you save or run scripts online? Second, "save and run my python scripts online" != "save my python scripts online and not have any way to run them anywhere". – abarnert Aug 13 '13 at 20:46
  • @abarnert, I will look into setting up a *nix shell account. Thanks for the tip. – Theo Aug 13 '13 at 21:01

1 Answers1

2

Sounds like https://scraperwiki.com/ is exactly what you need. It even has some great scraping modules for python built in.

ScraperWiki is a free tool that allows you to write scrapers in Python (and other languages). It then provides the data as an API or download as well as offering some display tools as well. Their environment isn't bad for writing scripts in - but it is not like working on your home machine.

CodeBeard
  • 515
  • 3
  • 10