0

I like to access some data from web pages that are arranged like a catalog/shop from an android app.

For a concrete example: This is the URL for Amazons listing on Mark Twains books:

http://www.amazon.com/s/ref=nb_sb_noss/180-5768314-5501168?url=search-alias%3Daps&field-keywords=mark+tain&x=0&y=0#/ref=nb_sb_noss_1?url=search-alias%3Daps&field-keywords=mark+twain&rh=i%3Aaps%2Ck%3Amark+twain

1) If I have the above URL how do I obtain e.g.

  • the number of entries and
  • for each entry the line with the title (and maybe the image)? Which probably includes how to iterate through all the follow-up pages and access each entry.

What is the best (correct + compatible + efficient) way to do this? I got the impression that jquery might be of use. But so far my knowledge of HTML and Javascript is just about basic.

2) How to query for the URL for all of Mark Twains books?

3) Any suggested readings for this and similar kind of topics?

Thanks for your time and have a good day!

Thomas

abraxas
  • 1

1 Answers1

2

You would be very well advised to not "screen scrape" other web sites. Besides being difficult to maintain (as the web site changes, etc.) - this will actually be against the terms of use / service (TOS) for many web sites.

Instead, see if the desired web sites offer a web service that you can use. These will return data in a much more consumable format, such as JSON or XML. You'll usually also get your own developer key (to track requests against), as well as other possible features that you wouldn't get if going directly against the HTML.

Amazon, in particular, certainly offers this. See https://affiliate-program.amazon.com/gp/advertising/api/detail/main.html for details. (Don't be confused by the naming of "advertising".)

ziesemer
  • 27,712
  • 8
  • 86
  • 94
  • Ok. Seems Amazon was a bad example to use. I just took it because it was widely known and probably a bit more general than one of the specific pages I had in mind. For example I like to navigate and extract the quotes from pages like http://www.quotationspage.com Other sites I have in mind are even less known and surely won't have such an API. – abraxas Jan 22 '12 at 05:42