Real-time output from multithreaded extensions in the IPython notebook

Question

I have a c++ function that spawns a child thread to provide asynchronous output (eventually this will be used to implement a progress monitor reporting on the state of a multithreaded algorithm). I want the child thread to be able to output to any place including the IPython notebook. For this reason I provide the child thread with a callback routine that takes in a string. If I want to output to the Python stdout, I initialise this callback with a pointer to an exposed cython routine (using the api keyword).

It took me some time to correctly implement all the GIL book keeping that would allow me to execute Python code from the child thread but eventually I got the code to work in the classic Python prompt. Working means that every half second (more or less) I get updated output from the child thread on the Python prompt.

The problem is that when I execute this code from either the IPython qtconsole or IPython notebook, the output does not appear until the function returns. At this point all the output appears at once (this is obviously not acceptable for a progress monitor).

I suspect this has something to do with how the GIL is passed from the Python kernel to the IPython kernel-side client? How do I allow the kernel-side IPython client to send new output to the notebook?

Content of pymontor.hpp

#include "Python.h"
#include "compel_api.h"

void printSomeOutput(PyObject *callable);
void printSomeOutputMT(PyObject *callable);

Content of pymonitor.cpp

#include "pymonitor.hpp"
#include <boost/date_time/posix_time/posix_time.hpp>
#include <boost/thread/thread.hpp> 

struct Worker {
    PyObject *callback;
    size_t n;
    void operator()()
    {
        std::string msg = "Hello!";
        for(size_t i = 0; i < n; ++i)
        {
            boost::this_thread::sleep(boost::posix_time::milliseconds(500));
            PyGILState_STATE gstate;
            gstate = PyGILState_Ensure();
            cy_print_message(msg, callable);
            PyGILState_Release(gstate);
        }
    }
};

void printSomeOutputMT(PyObject *callable)
{
    import_compel();
    Py_INCREF(callable);
    Worker w;
    w.callable = callable;
    w.n = 5;
    PyEval_InitThreads();
    Py_BEGIN_ALLOW_THREADS
    boost::thread thr(w);
    thr.join();
    Py_END_ALLOW_THREADS
    Py_DECREF(callable);
}

Content of compel.pdx:

cdef extern from "pymonitor.hpp" namespace "compel":
    cdef void printSomeOutputMT(object callable)

Content of compel.pyx:

cdef public api cy_print_message(string message, object callback):
    callback(message)

def test_printMT(object callback):
    printSomeOutputMT(callback)

Test at the Python / IPython-notebook command line:

from compel import *
def f(m):
    import sys
    print(m)
    sys.stdout.flush()
test_printMT(f)