How to abstract from pycurl the correct way

Question

I would like to write a pseudo module that makes one do a GET request that keeps on going (pretty much like the one to consume the Twitter Streamming API), but making unnecessary to give in all the parameters everytime someone wants to call a function to make that same GET request.

In my module.py I have

class viewResults():
    def __init__(self,username,password,keyname,consume):
        self.buffer = ""
        self.consume = consume
        self.conn = pycurl.Curl()  
        self.conn.setopt(pycurl.USERPWD, "%s:%s" % (username, password))  
        self.conn.setopt(pycurl.URL, "http://crowdprocess.no.de/"+keyname+"/results") 
        self.conn.setopt(pycurl.WRITEFUNCTION, self.on_receive)
    #   self.conn.setopt(pycurl.VERBOSE, 1)
    #   self.conn.setopt(pycurl.DEBUGFUNCTION, self.debug)
        self.conn.perform()

#   def debug(self,debug_type,debug_message):
#       print 'type: '+str(debug_type)+' message'+str(debug_message)

    def on_receive(self, data):  
        self.buffer += data  
        if data.endswith("\r\n") and self.buffer.strip():  
            content = json.loads(self.buffer)
            self.consume(content)
            self.buffer = ""

And on index.py I have

from module import viewResults

def consume(content):
    print content

viewResults('username','password','keyname',consume)

So I wanted to pass only the parameters username, password, keyname and the "consume" function that should be called when the buffer is full of valid JSON data...

What's happening is that the request is actually made, if VERBOSE is on I can see all the data arriving, but that "higher level consume" function get's nothing...

How can I achieve this ? Thanks.

score 1 · Answer 1 · answered Nov 22 '11 at 07:50

As I understand you want to archive debug data?

Create your custom debug function to store you data: custom_debug(debug_type, debug_msg)

>>> import human_curl as hurl
>>> import json
>>> r = hurl.get("http://crowdprocess.no.de/"+keyname+"/results"",
... debug=custom_debug, auth=('username', 'password'),)
>>> consume(json.loads(r.content))

score 0 · Answer 2 · answered Oct 30 '13 at 21:59

I could not see in your post code that on_receive(self, data): print something.
Add to it sys.stderr.write("%s\n" % data)

def on_receive(self, data):
        # -- print data to stderr --
        import sys
        sys.stderr.write("%s\n" % data)
        # -- end --
        self.buffer += data  
        if data.endswith("\r\n") and self.buffer.strip():  
            content = json.loads(self.buffer)
            self.consume(content)
            self.buffer = ""

How to abstract from pycurl the correct way

2 Answers2