0

I've caught a problem I can't explain using pygresql for python. This is a simple SELECT query and as far as I can tell the only query that has the problem in the whole script.

Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "run_daily_script.py", line 122, in DelegateAPINodes
    vm_api_group = Execute(db_conn, fetch_api_group, (cust_id, nodes["node_id"]))
  File ".../PreparedQueries.py", line 14, in Execute
    result = db.execute(command)
  File "/usr/local/lib/python2.7/site-packages/pgdb.py", line 1030, in execute
    return self.executemany(operation, [parameters])
  File "/usr/local/lib/python2.7/site-packages/pgdb.py", line 1065, in executemany
    raise _op_error("Internal error in '%s': %s" % (sql, err))
OperationalError: Internal error in 'SELECT * FROM vm_api_group WHERE cust_id=3 AND node_id=18': insufficient data in "T" message

I ran the query via the console and it works just fine. I monitored psql logs and there were no errors from the database. It seems to be an internal operation with pygresql.

What does "T" mean and how do I fix this?

Dan
  • 1,812
  • 3
  • 13
  • 27
  • [The documentation](https://www.postgresql.org/docs/current/static/protocol-message-formats.html) says that it is a "RowDescription" message from the backend. If it works with `psql` from the same machine, I'd suspect a bug in the driver. – Laurenz Albe Jun 29 '18 at 20:25
  • I've filed a bug with their mailing list. We'll see what they say. – Dan Jun 29 '18 at 21:28
  • This looks like a problem in the network communication between the client (libpq) and the server. Maybe the libpq version on the client is not compatible with the PostgreSQL version on the server. It could also indicate a network problem. Or maybe it's a problem with multithreading, since your stack trace shows you have multiple threads. E.g. your libpq could be compiled without multithreading support, or two threads might have tried to access the same connection object. The PyGreSQL driver is well tried and tested, and I don't think it could cause such an error message. – Cito Jul 11 '18 at 18:09
  • I've tested in a single threaded replica and it showed the same error for the same query. I'll check the versions against eachother to see if there's a conflict between the driver and our server. Interestingly we were short on time so we swapped it out with another python lib psycopg2 and it worked without the error. – Dan Jul 11 '18 at 23:11
  • 1
    I'm suspicious that this happened inside Thread-3 - postgresql drivers (which build on libpq) are _not_ thread safe, you must be careful to open a separate connection per thread. – reedstrm Jul 13 '18 at 17:53
  • We tried on a single threaded test as well. Same problem. We switched to psycopg2, built with multithreading in mind, and have had zero issues. – Dan Jul 13 '18 at 17:54
  • Psycopg2 has a threadsafety of 2 (connection level), PyGreSQL has threadsafety of 1 (module level). That can explain the difference in behavior. If you can provide a reproducible test case that causes such a problem in a single threaded program, I will gladly investigate further. – Cito Jul 17 '18 at 14:27
  • Ive said it a million f*n times I have tried it single threaded as well. There's clearly a problem as other libs have worked fine. We are now using other libs. This has solved the problem for me. Goodbye. – Dan Jul 18 '18 at 14:49

0 Answers0