4

I am a newborn programmer still programming from the book on my Alt+Tab. One of the first programs I want to create is to help my mom in her work. I need to know if I can use Python to create it.

It needs to:

  1. Go on-line and log-in with account / pass.

  2. Do a search with specific criteria (use the site's search engine)

  3. View all the results and pick only the newest ones.

  4. Sort them out.

  5. Notify me so that the newest adds are noticed the moment they are posted on the website.

From what I see the site says : .cgi in the end.

I know python can connect, download the text from a page and sort the wanted info, but can it log in, use the search engine and pick the options I need?

I don't want to skip my learning process, but I am so serious about this project I am ready to put Python on hold and start learning some language that can do it!

I will very much appreciate your guidance! Thank you for your time!

AJ

hymloth
  • 6,869
  • 5
  • 36
  • 47
troty_master
  • 315
  • 1
  • 2
  • 11
  • 6
    Python is a general purpose language. Yes, it can do this, or basically anything else. – agf Mar 30 '12 at 14:48
  • 1
    Yes, you can do this. Read the material linked in the below answers, and try to make a start on the project. Use google to search for solutions to your problems (there are a lot of python recipes out there, and the official python.org docs have a wealth of information), and look for existing answers here that match your problem. Once you are stuck, come here and ask a concrete question with a clear, concise, and complete example including your code, and the problematic input and output. Good luck, and happy coding! – Marcin Mar 30 '12 at 14:55
  • You will probably benefit from a basic understanding of how the web works (read about HTTP, HTML vs XHTML, etc). Also, people very much like to have their good answers voted up, and to have their answer accepted if it is a complete solution to your question. – Marcin Mar 30 '12 at 15:14

7 Answers7

4

I think you should take a look at the Mechanize project. It is a stateful browser in Python, so that you can easily submit forms and things of that nature and maintain a session. You'll want to then use lxml.html or BeautifulSoup to inspect the document (although Mechanize itself contains some tools for inspecting documents).

Thomas Allen
  • 393
  • 1
  • 4
4

To add to the brilliant list of packages commented by many, you can use Selenium too.

Zain Khan
  • 3,753
  • 3
  • 31
  • 54
3

Yes, it can. Specifically you'll want to use something like mechanize to handle the web-driving part.

regularfry
  • 3,248
  • 2
  • 21
  • 27
1

Python is just as capable of doing this as any other language, however you're going to want to look into things like httplib and understand cookies and http requests. So yes it can.

This will become your best friend: http://docs.python.org/

dennmat
  • 2,618
  • 2
  • 17
  • 20
1

If I understand well, the idea of your program is to do an automated browsing session. So yes, it's possible. It's not important in what the website is programmed (cgi, php etc). All you need is to send data through post/get (like a real browser) and process the return (regexp and so on).

Good luck

Yann Sagon
  • 547
  • 3
  • 21
1

I would point out that depending upon what site you are on, there may be a more efficient way (perhaps an exposed web service) than scraping data from the page and working with mechanize/selenium to do what you want. If you are on the web, browser driver tools are the hammers, and they will get the screws in the wood, but sometimes another tool will work better.

Silas Ray
  • 25,682
  • 5
  • 48
  • 63
0

You can do this with a lot of fun with the irobotsoft visual tool. Check quick demos here: http://www.irobotsoft.com/help/, then ask questions in their forum.

seagulf
  • 380
  • 3
  • 5