0

I know that there exists a package called aiopg for working with Postgresql in asyncio-friendly connections. Is there a similar version/package for Presto? I'm trying to implement concurrency for querying in Presto connections in Python.

Thanks in advance!

Manfred Moser
  • 29,539
  • 13
  • 92
  • 123
Samuel Cortez
  • 45
  • 6
  • How many concurrent queries to you expect to send to Presto from your Python app? – Piotr Findeisen Dec 29 '19 at 14:09
  • Hi @PiotrFindeisen! Thanks for taking the time to respond. I'm looking at ~5 Presto-connection memory intensive queries. I was hoping to gather these along with my Postgres-connection queries (~15 count) and run them in one `asyncio` loop (using their respective connections). The goal would be that if each query takes at least 5 mins or greater to complete, then at least at one point, early in the loop, all the queries would have been sent (to save time). But let me know if you have other suggestions. Thanks! – Samuel Cortez Dec 30 '19 at 17:07

1 Answers1

1

Presto Python API does not provide asyncio integration, so you need to treat it as any other blocking IO in asyncio world.

I am not familiar with asyncio, but run_in_executor looks like the way to do this.

Since you're not running thousands of concurrent queries (which wouldn't make sense in Presto anyway), this isn't going to hurt performance of your application.

I recommend that you also create a feature request in https://github.com/prestosql/python-client/issues/new (this is a new repo, so you can get very low, cool issue number).

Piotr Findeisen
  • 19,480
  • 2
  • 52
  • 82