0

I'm having some trouble trying to write a basic webscraper for a Minecraft server site. I'm pretty new to python (but fluent in C/java/.net) and can't seem to get the damn thing to work.

I'm trying to reach 

amishsmp.net/player.php?playerName=Leth

The following code works to the extent of successfully reaching and scraping from the site. But the page returned is an error saying no playerName was provided. So there must be something wrong with the param not being appended correctly?

I've tried as much as I can to fix it. Any suggestions?

import httplib, urllib

params = urllib.urlencode({'playerName':'Leth'})
headers = {"Content-type": "application/x-www-form-urlencoded","Accept": "text/plain"}
conn = httplib.HTTPConnection("amishsmp.net")
conn.request("GET", "/player.php", params, headers)
response = conn.getresponse()

print response.status, response.reason
print "####################"
data = response.read()
print data
conn.close()

EDIT: Doing this wihtout using the params does actually work...but i dont feel like thats the way it should be done:

conn.request("GET", "/player.php?playerName=Leth")
P4r0dy
  • 49
  • 7
  • 3
    `?playerName=Leth` indicates a GET request, yet you are sending a POST, are you certain that's what the server expects? – Martijn Pieters Sep 07 '12 at 20:26
  • using GET returns the same thing – P4r0dy Sep 07 '12 at 20:28
  • @P4r0dy: Viewing the page in the browser is a GET request. If you wish to scrape that information then you need to perform a GET request. – Simeon Visser Sep 07 '12 at 20:29
  • Perhaps the site then doesn't allow scraping, probably by detecting scripts based on the headers sent? – Martijn Pieters Sep 07 '12 at 20:29
  • added an edit, doing this works: "conn.request("GET", "/player.php?playerName=Leth")" which doesnt use the params seperately....i somehow feel like thats a bit of a hack? or is that pretty straightforward :P – P4r0dy Sep 07 '12 at 20:30
  • Using this: `conn.request("GET", "/player.php?{}".format(params))` might be a little cleaner. – Gerrat Sep 07 '12 at 20:37
  • @P4r0dy: Parameters are part of the URL in a GET request. So I wouldn't call it a hack. – Simeon Visser Sep 07 '12 at 20:45

1 Answers1

0

It looks like I'll jsut go with the edit i made. As it turns out, this site is terribly optimized, bringing every single user on the server into the source of a single user page (1.5mb html file)

P4r0dy
  • 49
  • 7