How to show a 'processing' or 'in progress' view while pyramid is running a process?

Question

I've got a simple pyramid app up and running, most of the views are a fairly thin wrapper around an sqlite database, with forms thrown in to edit/add some information.

A couple of times a month a new chunk of data will need to be added to this system (by csv import). The data is saved in an SQL table (the whole process right till commit takes about 4 seconds).

Every time a new chunk of data is uploaded, this triggers a recalculation of other tables in the database. The recalculation process takes a fairly long time (about 21-50 seconds for a month's worth of data).

Currently I just let the browser/client sit there waiting for the process to finish, but I do foresee the calculation process taking more and more time as the system gets more usage. From a UI perspective, this obviously looks like a hung process.

What can I do to indicate to the user that:-

1. That the long wait is normal/expected?

2. How MUCH longer they should have to wait (progress bar etc.)?

Note: I'm not asking about long-polling or websockets here, as this isn't really an interactive application and based on my basic knowledge websockets/async are overkill for my purposes.

I guess a follow-on question at this point, am I doing the wrong thing running processes in my view functions? Hardly seem to see that being done in examples/tutorials around the web. Am I supposed to be using celery or similar in this situation?

Answer 1

You're right, doing long calculations in a view function is generally frowned upon - I mean, if it's a typical website with random visitors who are able to hung a webserver thread for a minute then it's a recipe for a DoS vulnerability. But in some situations (internal website, few users, only admin has access to the "upload csv" form) you may get away with it. In fact, I used to have maintenance scripts which ran for hours :)

The trick here is to avoid browser timeouts - at the moment your client sends the data to the server and just sits there waiting for any reply, without any idea whether their request is being processed or not. Generally, at about 60 seconds the browser (or proxy, or frontend webserver) may become impatient and close the connection. Your server process will then get an error trying writing anything to the already closed connection and crash/raise an error.

To prevent this from happening the server needs to write something to the connection periodically, so the client sees that the server is alive and won't close the connection.

"Normal" Pyramid templates are buffered - i.e. the output is not sent to the client until the whole template to generated. Because of that you need to directly use response.app_iter / response.body_file and output some data there periodically.

As an example, you can duplicate the Todo List Application in One File example from Pyramid Cookbook and replace the new_view function with the following code (which itself has been borrowed from this question):

@view_config(route_name='new', request_method='GET', renderer='new.mako')
def new_view(request):
    return {}


@view_config(route_name='new', request_method='POST')
def iter_test(request):
    import time

    if request.POST.get('name'):
        request.db.execute(
            'insert into tasks (name, closed) values (?, ?)',
            [request.POST['name'], 0])
        request.db.commit()


    def test_iter():
        i = 0
        while True:
            i += 1
            if i == 5:
                yield str('<p>Done! <a href="/">Click here</a> to see the results</p>')
                raise StopIteration
            yield str('<p>working %s...</p>' % i)
            print time.time()
            time.sleep(1)

    return Response(app_iter=test_iter())

(of cource, this solution is not too fancy UI-wise, but you said you didn't want to mess with websockets and celery)

Answer 2

So is the long running process triggered by browser action? I.e., the user is uploading the CSV that gets processed and then the view is doing the processing right there? For short-ish running browser processes I've used a loading indicator via jQuery or javascript, basically popping a modal animated spinner or something while a process runs, then when it completes hiding the spinner.

But if you're getting into longer and longer processes I think you should really look at some sort of background processing that will offload it from the UI. It doesn't have to be a message based worker, but even something like the end user uploads the file and a "to be processed" entry gets set in a database. Then you could have a pyramid script scheduled periodically in the background polling the status table and running anything it finds. You can move your file processing that is in the view to a separate method, and that can be called from the command line script. Then when the processing is finished it can update the status table indicating it is finished and that feedback could be presented back to the user somewhere, and not blocking their UI the whole time.