0

I've tried to modify an example from "Building the Realtime User Expereince" by Ted Roden to send live twitter data to a client pushed by a server.

My app has a class that makes a call to the Twitter Streaming API via tweepy and filters the results by a keyword which the user enters in his browser. The aim is it for then the twitter stream to be sent back to the client as per filtered results.

The problem I keep on having is that I am unable to display a real-time data to the client, in the future I wanted to modify this so that I could send any type of data to the client in real time.

I'm very new to this and I am really struggling with this.....as I thought making a few changes to the example code would have worked. Here is my code:

The Executor module:

import tornado.httpserver
import tornado.ioloop
import tornado.options
import tornado.web
from tornado.options import define, options
import os.path, logging
from Tweeter import StreamerRt
import Queue
import threading
import simplejson as json
import uuid

queue = Queue.Queue(0)

define("port", default=8000, help='run on the given port', type=int)

class IndexHandler(tornado.web.RequestHandler):
    def get(self):
        self.render('templates/index.html')

class Tweet(object):
    waiters = []
    cache = []
    cache_size = 200
    def wait_for_messages(self, callback, cursor=None):
        cls = Tweet
        if cursor:
            index = 0
            for i in xrange(len(cls.cache)):
                index = len(cls.cache) - i - 1
                if cls.cache[index]["id"] == cursor: break
            recent = cls.cache[index + 1:]
            logging.info("%r recent tweets", len(recent))
            if recent:
                    callback(recent)
                    return
        cls.waiters.append(callback)

    def new_tweets(self, messages):
        cls = Tweet
        for callback in cls.waiters:
            try:
                callback(messages)
            except:
                logging.error("Error in waiter callback", exc_info=True)
        cls.waiters = []
        cls.cache.extend(messages)
        if len(cls.cache) > self.cache_size:
            cls.cache = cls.cache[-self.cache_size:]


class TweetsEventsHandler(tornado.web.RequestHandler, Tweet):
    @tornado.web.asynchronous
    def post(self):
        cursor = self.get_argument("cursor", None)
        self.wait_for_messages(self.async_callback(self.on_new_tweets),
                           cursor=cursor)
        self.render('templates/results.html')

    def on_new_tweets(self, tweets): 
        if not self.request.connection.stream.closed():
            self.finish(dict(tweets=tweets))


class TweetExtractor(threading.Thread):
    def run(self):

        while True:
            tweet = queue.get()
            if tweet:

                try:
                    message = {
                        "id": str(uuid.uuid4()),
                        "tweet": tweet
                        }
                    message["html"] = "<div class=\"message\" id=\"m" + message["id"] +     "\"><strong>" + ['author']['author_id'] + ": </strong>" + ['source'] + "</div>"
                    Tweet().new_tweets([message])

                except Exception, e:
                    pass

class TweetFinder(threading.Thread):
    def run(self):
        results = StreamerRt().filter(track=[self.get_argument('event')])        
        for tweets in results:
            if len(tweets)>2:
                queue.put(tweets)

local_static_path = os.path.join(os.path.dirname(__file__), "static") 

app = tornado.web.Application([
            (r"/", IndexHandler),
            (r"/results", TweetsEventsHandler),
    ],
    static_path=local_static_path)    


if __name__ == "__main__":
    print 'Listening on http://localhost:8000'
    tornado.options.parse_command_line()

    # setup the twitter threads
    threads = []
    threads.append(TweetExtractor())
    threads.append(TweetFinder())

    # for each thread, set daemon mode to true and start them up
    for t in threads:
        t.daemon = True
        t.start()

    http_server=tornado.httpserver.HTTPServer(app)
    http_server.listen(options.port) 
    tornado.ioloop.IOLoop.instance().start()

The js file is as this:

// -*- indent-tabs-mode: nil -*-
google.load("jquery", "1");

google.setOnLoadCallback(
    function() {
        $(document).ready(function() { T.init(); T.poll(); });
    }
);

var T = {
    cursor: false,
    types: ['hashtags', 'retweets', 'ats', 'links']
};

T.init = function() {
    $('input[type=checkbox]').click(T.click);
};

T.click = function(ev) {
    if(ev.target.id == 'all') {
        for(i in T.types)
            $('#' + T.types[i]).attr('checked', $('#all').attr('checked'));
        }
        else
            $('#all').attr('checked', false);
};


T.poll = function () {
    var args = {};
    if(T.cursor) args.cursor = T.cursor;
    $.ajax({
               url: "/results.html",
               type: "POST",
               dataType: "json",
               data: $.param(args),
               success: T.new_tweets
           });
};

T.new_tweets = function(response) {
    if(!T.cursor)
         $('#waiting').remove();

    T.cursor = response.tweets[response.tweets.length - 1].id;

    for(var i = 0; i < response.tweets.length; ++i) {
        if(T.should_show_tweet(response.tweets[i])) {
            var html = response.tweets[i].html;
            var regexp = /(http\:\/\/[^\ |\<]+)/g;
            if((result = regexp.exec(html))) {
                console.log(result);
                for (var n = 1; n < result.length; ++n)
                    html = html.replace(result[n], '<a target=blank href="' + result[n]       + '">' + result[n] + '</a>');

            }

            var d = $(html);
            d.css('display', 'none');
            $('#content').prepend(d);
            d.slideDown();
        }
    }

    // clean up the DOM
    var limit = 100;
    var messages = $('.message');
    for(var x = messages.length-1; x > limit; --x)
        $(messages[x]).remove();

    T.poll();
};

T.should_show_tweet = function(tweet) {
    show_tweet = false;

    $('#all-count').text(parseInt($('#all-count').text())+1);

     for(x in T.types) {
        var type = T.types[x];

        // does the tweet have the specified type?
        if(tweet.stats[type].length) {
            var count_div = $('#' + type + '-count');
            count_div.text(parseInt(count_div.text())+1);

            // does the user want to see it?
            if($("#" + type).attr('checked'))
                show_tweet = true;
        }
    }
    return show_tweet;
};


T.click = function(ev) {
    if(ev.target.id == 'all') {
        for(i in T.types)
            $('#' + T.types[i]).attr('checked', $('#all').attr('checked'));
    }
    else
        $('#all').attr('checked', false);
};

I am a complete newbie to js.

Thanks

user94628
  • 3,641
  • 17
  • 51
  • 88
  • Try to push your own data to client first. Twitter API is tricky, if you newbie. – Nikolay Fominyh Oct 30 '12 at 04:38
  • Thanks, I'm able to get make a call to the twitter api, and store to my DB. The problem I have is more in general pushing data to the client. – user94628 Oct 30 '12 at 18:19
  • I've read your js code and haven't got it. =) Take a look at `socket.io + tornadio2`. It will dramatically simplify socket connection. – Nikolay Fominyh Oct 31 '12 at 04:33
  • I appreciate you taking the time to have looked at the js code. I just used the code give in the example in the book, but I didn't really understand it. Thanks for the tip, I'll look at the socket.io + tornadio2. – user94628 Oct 31 '12 at 10:22
  • I got it. Try to rename "success: T.new_tweets" to "complete: T.new_tweets". Also check in firefox, if it is really polling data or not. – Nikolay Fominyh Oct 31 '12 at 14:07
  • I tried that still not working. Seems that my StreamerRt method is not being called and so nothing is displayed. The server brings it up as a 404 error to. Thanks – user94628 Nov 01 '12 at 12:11

0 Answers0