0

With Pandas, we are able to create persistent connections, which allows (for example) creating temporary tables against which we can join. For example:

import pandas as pd
import sqlalchemy as sa

engine = sa.create_engine("postgresql://me@server:port/my_db")

# create a temporary table
sql = """
CREATE TEMPORARY TABLE my_table (
  name varchar(50),
  age SMALLINT,
  birth_date DATE
);
INSERT INTO my_table VALUES ('me', 38, '1980-01-01');
"""

conn.execute(sql)

# now perform a SELECT using the persistent connection
df = pd.read_sql_table("my_table", conn)

However, polars appears to use a string as a connection, and presumably creates a new connection with each query. Is there a way to use polars along with persistent connections?

NedDasty
  • 192
  • 1
  • 8

1 Answers1

0

Polars uses ConnectorX in read_sql, which is a really thin and convenient wrapper around pl.from_arrow(cx.read_sql(connection_uri, sql,...)). Polars uses Arrow as the underlying data structure, meaning that pl.from_arrow is (mostly) zero-copy and efficient. Thus any python SQL query package that returns an Arrow table would work. An example would be TurbODBC, but there may be more that return an Arrow table and maintain a persistent connection.

jvz
  • 1,183
  • 6
  • 13