3

In some website for which I have access, there are some input fields. In the sixth field I need to enter some input string from a list of 10000 strings, then a new page appears, for which I would just need to count the number of lines. Finally I would like to get a table with two columns like input string and number of resulting lines. Since I have to manually enter the info for all the different 10000 strings, I wonder therefore what is the best approach to enter a string into a generic formular field and get the resulting text. I heard about curl but I am not sure whether this is the easiest one.

P.S. Example of interactive way: I type some string o words into google search and then I get a new page with the search results. Previously I have introduced my google username and password, so the results will be probably filtered according to my profile.

Example of non-interactive way: A script somehow introduces my user information, search query and saves to some text file the search results. Imagine the same idea but for a more complicated website like this.

Open the way
  • 26,225
  • 51
  • 142
  • 196
  • 1
    The vote was cast for "too localized", which means "This question is unlikely to ever help any future visitors; it is only relevant to a small geographic area, a specific moment in time, or an extraordinarily narrow situation that is not generally applicable to the worldwide audience of the internet.". Your question is quite specific to your needs; perhaps if you rephrased it as a general question, with your specific situation as an example, it would be more appropriate. – Tom Anderson Jun 22 '11 at 09:06
  • Ok, I understand. I rephrased the question – Open the way Jun 22 '11 at 09:10
  • I have found so far that I can be done with libcurl – Open the way Jun 24 '11 at 09:03
  • I don't really understand, can you give a more specific example? – daniel.herken Jun 27 '11 at 10:00
  • I rephrased the question, hope it gets more clear now – Open the way Jun 27 '11 at 15:37

2 Answers2

2

What you want to do is to send a HTTP POST with specific data. This can be done with any proper HTTP client code, and one such is libcurl (or the pycurl binding or even using the curl command line tool). On the response from the post, you probably get a redirect and then the results, or you need to do a separate request for the results and then you're done and go back to do the next POST. Repeat until all POSTs are done.

What you may need to take into account is that you may have to deal with cookies and possibly to follow a redirect from the POST. A good approach is to record a "manual session" as done with a browser (use firebug or LiveHTTPHeaders etc) and then use that recording to help you repeat the same thing with a HTTP client.

A decent tutorial to get some starting up details on this kind of work can be found here: http://curl.haxx.se/docs/httpscripting.html

Daniel Stenberg
  • 54,736
  • 17
  • 146
  • 222
0

You could also use JMeter to run all the posts. You may use the CSV input to set the 10000 strings. Then you save the result as xml and extract the necessary data.

ghm1014
  • 675
  • 6
  • 12