Python flask requests load page

Question

Create script which loads page from my server (with server IP etc.) - all OK, but if I want to click on any link I landed to 404 error page, because link is - some thing like this:

...37.139.17.81:5000/html/privacy-check.php

My code:

from flask import Flask
import requests

application = Flask(__name__)

@application.route("/")
def hello():
        result = requests.get("http://ipinfo.info/index.php")
        return result.content

if __name__ == "__main__":
    application.run(host='0.0.0.0')

Here's a live example :

http://37.139.17.81:5000/

How I can parse click URL and get this link content?

Answer 1

You are basically trying to make your page act like a proxy for the remote page. In order to do this completely, you need to handle all links in the remote page.

So for example if there is a link like /something/something flask will automatically try to match it with a local url (http://yourserver.com/something/something). Given the fact that you only define a single route ("/") the app will determine that any other page does not exist and will return a 404.

To handle this correctly you can try the following:

import urlparse

@application.route("/")
@application.route("/<url:path>")
def hello(url=None):
    baseurl = "http://ipinfo.info/"
    if not url:
        result = requests.get(urlparse.urljoin(baseurl,"index.php"))
        return result.content
    else:
        result = requests.get(urlparse.urljoin(baseurl,url))
        return result.content

A word of warning: this approach might break in various cases (css and js loading for example) so you might need to check the results after the page loads.

Answer 2

Your "script" is a flask application running on a local server http://37.139.17.81:5000/.

When you click on a link from a page loaded from a different site your flask application, reasonably, thinks its a link to a page in the flask application and so goes to try to load the page on the local application.
The flask application looks for the links on the local server probably because the links on the page you loaded are relative links.

To parse the links you could use something like urlparse

from urlparse import urlparse
o = urlparse('http://www.cwi.nl:80/%7Eguido/Python.html')
o   
ParseResult(scheme='http', netloc='www.cwi.nl:80', path='/%7Eguido/Python.html',
            params='', query='', fragment='')

I have to ask why are you trying to load a php page into a flask application?

Answer 3

For href="/html/privacy-check.php" you should do

@application.route("/html/privacy-check.php")
def hello():
        result = requests.get("http://ipinfo.info/index.php")
        return result.content

Since it cannot find any url match for /html/privacy-check.php on your server, it's throwing 404 error.