2

How should I fetch multiple URLs from within a method in a Circuits framework Controller in Python 3? Here's a trivial example of what I want except with urllib3. It would be preferable to request both URLs at the beginning and when they both are back, continue execution.

# -*- coding: utf-8 -*-
__author__ = 'jscarbor'
import urllib3
from circuits.web import Server, Controller, Static

http = urllib3.PoolManager()


class Root(Controller):
    def index(self):
        self.response.headers["Content-Type"] = "text/plain"
        a = http.request('GET', 'https://www.w3.org/services/html2txt?url=http%3A%2F%2Fwww.example.com%2F').data
        b = http.request('GET', 'http://home.hiwaay.net/~jimes/checklist.txt').data

        return "%s %s" % (a, b)


(Server(8011) + Root()).run()

2 Answers2

2

You need to register the circuits.web.client.Client Component to your Controller with a different channel than the Controller (because the events name are equal in the client and the server component). Then you can fire request events into this channel and wait for the response. In your specific example you need to register a handler for the request event to hook into the response process. I currently don't have a working example but this is a base for a start point:

from circuits.web.client import Client, request as request_event
from circuits.web import Server, Controller
from circuits import handler


class Root(Controller):

    @handler('request')
    def _on_request(self):

        a = yield self.wait(request_event('GET', 'https://www.w3.org/services/html2txt?url=http%3A%2F%2Fwww.example.com%2F'), channel='url-fetching')
        b = yield self.wait((request_event('GET', 'http://home.hiwaay.net/~jimes/checklist.txt'), channel='url-fetching')

        self.response.headers["Content-Type"] = "text/plain"
        self.response.body = "%s %s" % (a.value.read(), b.value.read())


(Server(('0.0.0.0', 8011)) + Root() + Client(channel='url-fetching').run()
spaceone
  • 121
  • 3
  • An explanation of your code would be more useful than ('something like this'). – Nander Speerstra Jan 27 '17 at 10:03
  • 1
    @NanderSpeerstra better? – spaceone Jan 27 '17 at 12:04
  • 1
    Yes, that's better! – Nander Speerstra Jan 27 '17 at 12:07
  • I have refined a bit, but I'm running into two problems. Here's the new code https://gist.github.com/ke4roh/f19cdc704559b5ea482533636e345167 . If I omit line 29, it appears to fetch each URL in turn, but it still displays a 404 not found result. If I include line 29 to defer the waiting until both requests are processed, it appears that the secure and insecure requests get conflated. I see this error: ... File "/usr/lib64/python3.5/ssl.py", line 633, in do_handshake self._sslobj.do_handshake() OSError: [Errno 9] Bad file descriptor Thoughts? – Jim Scarborough Feb 14 '17 at 15:52
  • [Issue 226](https://github.com/circuits/circuits/issues/226) blocks the ideal solution where the two URLs are fetched in parallel. – Jim Scarborough Feb 21 '17 at 22:51
0

I improved your gist so that it does these things. I am not yet sure why one can't fire 2 request events at the same and wait afterwards for them.

#!/bin/env python3
from circuits.web.client import Client, request as request_event
from circuits.web import Server, Controller
from circuits import handler, Debugger


class Root(Controller):

    def index(self, event):
        request, response = event.args

        urls = [
            'https://www.w3.org/services/html2txt?url=http%3A%2F%2Fwww.example.com%2F',
            'http://home.hiwaay.net/~jimes/checklist.txt'
        ]
        events = []
        results = []
        for url in urls:
            event_ = request_event('GET', url)
            client = Client(channel=url)
            client.register(self)
            events.append((event_, client))
            self.fire(event_, url)
        #for event_, client in events:
            er = yield self.wait(event_, *event_.channels)
            client.unregister()
            results.append(er)

        response.headers["Content-Type"] = "text/plain"
        yield " ".join([r.value.read().decode("utf-8") for r in results])


(Server(8011) + Root() + Debugger()).run()
spaceone
  • 121
  • 3