Python reads data from webpages

Question

I have a list of a bunch of IP addresses. I am wondering if it is possible to use python to determine the country name of the IP addresses by extracting the information from this website (http://www.whatip.com/ip-lookup). Please see the screenshot below. e.g: IPlist = ["100.43.90.10","125.7.8.9.9"]

Here is my code: I understand i could change the search url by concatenating the actual url with the suffix (=my IP address). And I want to get "United States"

Here is the screenshot of where "United States" is located:

    import urllib.request
    with urllib.request.urlopen('http://www.whatip.com/ip/100.43.90.10') as response:
        html = response.read()
        print (html)
        text = html.decode()                

        start = text.find("<td>Country:</td>")

I checked there is only one "Country" in the source code. I understand that I need to find the index of "Country", and then print out "United States" but I got stuck. Anyone plz tell me how to do it? Many thanks!!

You can do one better and not use Python at all. There are loas of Linux command line utilities to do this. — Akshat Mahajan, Mar 30 '16 at 15:25
First you have to try something, i suggest using urllib2 and beautifulsoup :) there are a lot of tutorials of data from webpage extraction in python :) — Marko Mackic, Mar 30 '16 at 15:29
You can append the IP to the URL `http://www.whatip.com/ip/125.7.8.9` and find the geolocation that way. Alternatively, you can try beautifulsoup which will make the web scraping easier. Or use an API such as the one from http://ip-api.com/ for a more programmatic approach. — Ryan Jackman, Mar 30 '16 at 15:31
@MarkoMackic thx! i have modified my questions by providing my code made so far. but I got stuck again on how to print out the stuff showing after the key word. — yingnan liu, Mar 30 '16 at 15:48
@MarkoMackic: urllib2 is for Python2 and is replaced by urllib.* in Python3. But I do agree on BeautifoulSoup :-) — Serge Ballesta, Mar 30 '16 at 16:00
Please see the [Terms of Use](http://www.whatip.com/terms-of-use) for http://whatip.com. Specifically, "*The users shall agree and consent that they cannot use agents, automated scripts and application for automatically querying this website in any manner without explicit permission.*" — Robᵩ, Mar 30 '16 at 16:22
oops.... thx a lot! i guess there is no way i could generate country names for larger amount of IP addresses. I only need it cuz of academic purpose — yingnan liu, Mar 30 '16 at 23:46

score 1 · Answer 1 · answered Mar 30 '16 at 15:39

1

You can use this site : http://whatismyipaddress.com/ip/

All you need to do is write a Python script. The Python Script will be making use of the urllib3 library. This library is used to create connections to the web, setup an array of IP Addresses and loop through them, each time appending the IP address to the above given site. Create a http request using urllib , once the response is received, you can use the .data property of response to get the response data. Once you receive the response data, use a simple regex for locating country field name, and then just grab the country name.

Just go through the urllib documentation, which is small ! and you're done !

p.s. I did a similar thing a month back !

answered Mar 30 '16 at 15:39

DebashisDeb

392
5
13

thx! I have made some python codes and got the question updated. Seems like I need to figure out a way to get "United States" printed. – yingnan liu Mar 30 '16 at 15:49
@yingnanliu what you can do is search for the entire : ` Country: US ` using a regex, where only the country name will vary and the rest is fixed. once found, just use another regex to get the starting index of the country name. Does this help ? – DebashisDeb Mar 30 '16 at 16:03
thx! thats exactly what i did so far, but i got stuck for the next step when i need to print out "United States" locating in the next line. Would u plz see the code I just updated? – yingnan liu Mar 30 '16 at 16:05

score 1 · Accepted Answer · answered Mar 30 '16 at 16:12

I would suggest using one of the many REST APIs available for IP geolocation.

This doesn't require you to install any new modules or perform any web page scraping. The request returns a json object that you can use the inbuilt module to parse and immediately create a python dictionary.

I had a quick play with nekudo and it appear to work well:

import json
from http import client

# Connect to the client
conn = client.HTTPConnection("geoip.nekudo.com")

# Make the request and extract the data
conn.request("GET","/api/172.217.3.110/full")
json_data = conn.getresponse().read().decode()

# Convert the JSON to a Python object
data = json.loads(json_data)

data is now a Python dictionary containing all the information you need

>>> data['registered_country']['names']['en']
'United States'

>>> data['location']
{'latitude': 37.4192, 'metro_code': 807, 'time_zone': 'America/Los_Angeles', 'longitude': -122.0574}

score 1 · Answer 3 · answered Mar 30 '16 at 16:19

1

I find it almost always easier to use an API than the screenscrape a web page. Here is one solution using ip-api.com:

import requests
import json

IPlist = ["100.43.90.10","125.7.8.9.9"]

request = json.dumps([{'query':ip, 'fields':'country'} for ip in IPlist])
response = requests.post('http://ip-api.com/batch', data=request).json()

print '\n'.join('{}: {}'.format(ip, data.get('country', 'Unknown'))
                for ip, data in zip(IPlist, response))

answered Mar 30 '16 at 16:19

Robᵩ

163,533
20
239
308

Please read and follow ip-api.com's terms of use (as you should any web service). Please note the immediate ban for violating traffic volume limits there, and how to rescind the ban. – Robᵩ Mar 30 '16 at 19:29

Python reads data from webpages

3 Answers3