Kivy UrlRequest called as a class - Methods not executed?

Question

Trying to experiment and build a class resolving around the use of UrlRequest just to check if a given URL is valid. Turns out being a bit more difficult than anticipated!

The issue is that the on_success and on_failure/error methods defined as part of the class are never called. The script throws the following output (based on the print commands):

http://www.google.com
request sent
URL doesn't work

Now my suspicion is that I´m getting the return code ("None") from the test_connection method, and not the connectionSuccess or connectionFailure. How can I make the call wait for one of the latter to give a return? Any suggestion welcome. Thanks.

from kivy.app import App
from kivy.uix.floatlayout import FloatLayout
from kivy.network.urlrequest import UrlRequest

class WebExplorer():      
    def test_connection(self, path):
        self.path = path
        print (self.path)
        req = UrlRequest(self.path,on_failure=self.connectionFailure,on_error=self.connectionFailure,on_success=self.connectionSuccess)
        print ("request sent")

    def connectionSuccess(self,*args):
        print ("connectionSuccess")
        return 0

    def connectionFailure(self,*args):
        print ("connectionFailure")
        return 1


class MainScreen(FloatLayout):
    def __init__(self, **kwargs):
        super(MainScreen, self).__init__(**kwargs)
        self.address = 'http://www.google.com'

        if WebExplorer().test_connection(self.address) == 0:
            print ("URL works")
        else:
            print ("URL doesn't work")

class App(App):
    def build(self):
        return MainScreen()


if __name__ == "__main__": 
    App().run() 

UPDATE 2016-09-27 I changed my code and I have spent hours on trying to figure out the problem. First the code:

from kivy.app import App
from kivy.uix.floatlayout import FloatLayout
from kivy.network.urlrequest import UrlRequest

class WebExplorer():

    def test_connection(self, path):
        self.path = path
        req = UrlRequest(self.path,on_failure=self.connectionFailure,on_error=self.connectionFailure,on_success=self.connectionSuccess)
        req.wait()
        return (self._return_value)

    def connectionSuccess(self, req, results):
        print ("Success")
        self._return_value = [0,results]

    def connectionFailure(self, req, results):
        print ("Failure")
        self._return_value = [1,results]


class MainScreen(FloatLayout):
    def __init__(self, **kwargs):
        super(MainScreen, self).__init__(**kwargs)
        self.URLtest = ['http://www.ikea.com/','https://www.google.com','https://www.sdfwrgaeh.com']
        for URL in self.URLtest:
            self.returnCode = WebExplorer().test_connection(URL)
            if self.returnCode[0] == 0:
                print ("Correct URL")
            else:
                print ("Wrong URL")

class App(App):
    def build(self):
        return MainScreen()

if __name__ == "__main__": 
    App().run() 

Why 3 URLs? Because one is "correct" (IKEA), one is a redirect (Google) and one is completely bogus. Turns out, the code works for the first one only. req.wait doesn't work when the result is a failure/error (btw I have zero idea what's the difference between these two).

So the question is how to make req.wait process a failure, alternatively how to exit the class with the correct error code. I considered Clock.schedule_interval to periodically check the status, ut since the event methods are not even executed when the URL is incorrect, I have nowhere to set my variales -_-

Answer 1

UrlRequest is an asynchron, which means that the actual request is processed in the background while your application proceeds. The functions on_failure etc. are callbacks, i.e. they will be called when the request is done - but that means that the return value is "lost". The return value you get is indeed from the test_connection function.

You mentioned on the mailing list that you want to make a call to a root url, do some processing on what you get back and then issue new requests to suburls depending on content and user interaction.

The following should provide a skeleton for what you are trying to do.

class WebExplorer

    def __init__(self, gui):
        self._visited_urls = [] # to prevent endless redirect loops
        self._visiting = set() # to check if we are still doing something
        self.gui = gui # used to display buttons in process_result

    def get_url(self, url):
        self._visited_urls.append(url)
        self._visiting.add(url)
        UrlRequest(url, 
            on_success=self.process_result,
            on_failure=self.failure,
            on_error=self.failure, # what is the difference between failure and error?
            on_redirect=self.redirect)

    def redirect(self, req):
        self._visiting.discard(req.url)
        self.get_url(req.req_headers["Location"]) # check if correct handling of 3xx

    def process_result(self, req):
        self._visiting.discard(req.url)
        #TODO process the body to extract urls you want to visit

        box = BoxLayout()
        for url in urls_you_want_to_visit:
            button = Button()
            button.text = url
            # I'm not sure if the following works due to some scoping oddities
            # if - no matter which button is pressed - the same url is called
            # this line is probably at fault
            button.bind(on_press=lambda: self.get_url(url)) 
            box.add_widget(button)

        gui.add_widget(box)

Of course there is still a lot to improve. It's also not tested so it can contain bugs.