0

In a single process, I could write an Polars Expr and then use it on DataFrame by with_column or select.

But in another scenario of realtime financial data, I have a rust process which has a lot of history and realtime data(DataFrame) as a server. There are multiple python process(client) need to put their own Expr/PyExpr/LogicalPlan on the same data.

Because the data is changing time by time, so I need to avoid frequently data copy from rust to python. Is there a way to send the Expr/PyExpr/LogicalPlan from python client to rust server, then calculate the result in rust with its data so the copy job would be much smaller.

Hakase
  • 211
  • 1
  • 12

1 Answers1

1

Because the data is changing time by time, so I need to avoid frequently data copy from rust to python

You can move DataFrames and Series between python and rust for free within the same process. We just swap arrow pointers.

You can use the pyo3-polars crate to make this easy.

https://github.com/pola-rs/pyo3-polars

ritchie46
  • 10,405
  • 1
  • 24
  • 43
  • Thank you. If the data source api is an independent rust process, could it swap coming arrow pointers to some other python processes and then wrapped as PyDataFrame? – Hakase Jan 11 '23 at 08:50
  • No, the process must be the same for pointer reads. Otherwise you must serialize the data and deserialize on the other side. – ritchie46 Jan 11 '23 at 09:23