1

I am using Mojolicious with a plugin called pgAsync in order to listen for NOTIFY events that are being emitted from a Postgres database. The code I currently have is working fine, but I am seeing that the number of database connections is only ever increasing.

Back-end

#!/usr/bin/env perl

use Mojolicious::Lite;
use Mojo::IOLoop;
use Mojolicious::Plugin::PgAsync;

app->secret('awdawdawdawd');

plugin PgAsync => {dbi => ['dbi:Pg:dbname=;host=;port=;', '', '', {AutoCommit => 0, RaiseError => 1}]};

any '/api/listen' => sub {
  my $self = shift;

  my $saved_tx = $self->tx;

  Mojo::IOLoop->stream($self->tx->connection)->timeout(300);

  $self->res->headers->add('Content-Type' => 'text/event-stream');
  $self->res->headers->add('Cache-Control' => 'no-cache');
  $self->res->headers->add('Access-Control-Allow-Origin' => '*');

  # required for IE
  $self->write(" " x 2048);
  $self->write("\nretry: 2000\n\n");

  my $drain_cb;
  $drain_cb = sub {
    my $c = shift;
    $c->render_later;
    $c->tx($saved_tx);
    $c->pg_listen('foo', sub {
      my $notify = shift;
      my $payload = $notify->{payload};
      $c->write("id:1\ndata:$payload\n\n", $drain_cb);
    });
  };

  $self->pg_listen('foo', sub {
    my $notify = shift;
    my $payload = $notify->{payload};
    $self->tx($saved_tx);
    $self->write("id:1\ndata:$payload\n\n", $drain_cb);
  });

};

app->start;

As you can see, I needed to do a bit of hackery to get this to work as I wanted it to. pg_listen seemed to be undefining the callback after every call. This was causing an error when I called pg_listen again (since it was undefined), and I had to modify line 34 of PgAsync::Db.pm to add a check for this:

$self->callback->($notify_hash, $notify_hash);

->to->

$self->callback->($notify_hash, $notify_hash) if defined $self->callback

Front-end

I am using JavaScript with an EventSource object to listen for the push notifications from the Mojolicious script:

var es = new EventSource("/api/listen");
var listener = function (event) {
  console.log(event.data);
};
es.addEventListener("open", listener);
es.addEventListener("message", listener);
es.addEventListener("error", listener);

The Problem

Looking closer, it appears that this system is continuously creating more database connections:

  1. Starting from nothing, there is 1 database connection, which is me looking at how many database connections there are with select count(*) from pg_stat_activity;

  2. I start morbo and the Mojolicious script with DEBUG_PG=1 morbo mojopush.pl. There are now 2 database connections (the 1st is me in psql, the 2nd is morbo)

  3. I open up the webpage with the EventSource object. It correctly sets the GET request with the content-type of text/event-stream and holds the connection open. There are now 3 db connections

  4. I go into psql and send NOTIFY foo, 'test', the Mojo script detects it, and the webpage displays 'test. There are still 3 db connections.

  5. I refresh the page, and there are now 4 db connections. I wait for the 300 second inactivity timeout, and yet another database connection is created resulting in 5 connections.

If anyone could help point me in the right direction, that would be extremely appreciated!

Mike
  • 4,071
  • 21
  • 36

2 Answers2

1

Not 100% sure so this is a bit of a guess based on the information provided, but I wonder if each page load is starting a new db connection which is then listening for notifications. If this is the case, I wonder if the db connection is effectively removed from the pool and so gets created on the next page load.

If this is the case, my recommendation would be to have a separate DBI database handle dedicated to listening for notifications so that these are not active in the queue. This might be done outside your page workflow.

Chris Travers
  • 25,424
  • 6
  • 65
  • 182
1

I know this is late answer, but I am with Mojolicious for not more than two weeks. The issue seems to be unresolved, and I have some observation about PgAsync I might share.

I found this post when googling around after some excercises with PgAsync. I also could observe the "connection leak", not with a listen/notify case, but with using the "pg" helper to execute a series of independend queries, on a scenario similar to the one given here: https://groups.google.com/forum/#!topic/mojolicious/titaWRImLt0 .

Particularly, the leak was happening when making more than one simultaneous requests. At the same time, I could see some of the requests of two simultaneous continously running loops of requests, were left with uncompleted query execution and timing out.

Still not understanding too much from the plugin mechanics, and generally Mojo:: event hadling, I guessed that maybe a database connection just made free might too quickly be picked for another request.

So, after some more or less blind hacking, I tried to introduce a dumb delay between a database connection finishing its request and getting back to a free pool. I did it by changing this line of code:

https://metacpan.org/source/ROMANENKO/Mojolicious-Plugin-PgAsync-0.03/lib/Mojolicious/Plugin/PgAsync/Db.pm#L49

to:

$reactor->timer(1 => sub { $self->make_free->($self) });

After that change both the connection leaking and incomplete requests disappeared. This is definitely not a fix, but a workaround. Maybe that could also work for your case.

However, I can see that there is still connection leak in the case when client disconnects (Ctrl+C) while a query is executing. Such a case leaves the database connection "orphaned" at db_pool, and never returned to db_free.

Unless the author gets it quicker, I hope to dive deeper into the code of the plugin, to find a true fix, and maybe add some enhancements. I need this plugin a lot in my current work.

vahid abdi
  • 9,636
  • 4
  • 29
  • 35