Questions tagged [rust-polars]
271 questions
0
votes
0 answers
How to split the same LazyFrame without .clone()?
I have a quite big .arrow file (150Gb) I need to split into various parts based on some filters. I started using the Python Polars implementation for prototyping (JIT compiler really helped here) and now I was trying to port my working Python…
Jona Rodrigues
- 992
- 1
- 11
- 23
0
votes
0 answers
Polars: Tracking Data lineage(New,Existing,Updates,Deletes) between 2 Dataframes
I have 2 datasets (Day1 and Day2) that can have 100s of columns.
The problem statement is I have to track on Day2 dataset
What records are existing from Day 1
What records got updated.
What records are brand new.
And lastly, what records in Day1…
0
votes
1 answer
Creating a Datetime Column in Polars DataFrame from i64 Vector in Rust
I'm working with the Polars library in Rust and am struggling to find documentation related to its dtype-datetime feature, especially when compared to its Python counterpart which seems a lot richer with examples.
My goal is to create a dataframe…
deepvalue
- 81
- 3
0
votes
0 answers
Polars: Joining on a struct column of lazy dataframes and then collecting gives Panic Exception
Getting Panicked Exception while collecting on a lazy data frames join.
The Exception I get:
thread '' panicked at 'not implemented', D:\a\polars\polars\crates\polars-core\src\series\series_trait.rs:60:13 note: run with RUST_BACKTRACE=1…
0
votes
0 answers
Docker build failing when installing polars-lts-cpu
I'm getting the following error when building my Dockerfile
0.894 Downloading polars_lts_cpu-0.18.15.tar.gz (1.9 MB) …
0
votes
2 answers
Add prefix and suffix in existing column and add that as a separate new column in Polars rust
I have a DataFrame df:
A
B
1
value
3
value
now for Column A, I want to add prefix "test_" and suffix "_new" as a new column "TEST".
Output should be:
A (i64)
B
TEST
1
value
test_1_new
3
value
test_2_new
Please help me!!.…
Mahima Misra
- 1
- 1
0
votes
0 answers
Rust Polars - `drop_nulls` equivalent but for `NaN`
I'm working with a DataFrame that contains both null values and NaN, which I'm loading from a CSV file with CsvReader.
I would like to ignore any row which contains either a null or a NaN in any column. For example:
foo | bar | baz…
cameron1024
- 9,083
- 2
- 16
- 36
0
votes
0 answers
Counting the number of occurences parameterized the column
I am trying to create a groupby aggregation that takes as input a set of columns indices, lhs, and a single column rhs. What I would like to do is output two aggregations:
groupby the LHS, and count all the distinct occurrences of the columns…
user22284368
- 1
- 2
0
votes
1 answer
Polars Lazyframe Left join and ~is_in filter not working as expected
When I use lazyframe and use left join along with ~is_in filter, nulls are created instead of dropping the rows. This happens in lazyframe but not in eager
pl_df = pl.DataFrame({
"group":["A","B","C","D","E","F"],
"values":…
0
votes
0 answers
How to unnest a JSON string in a parquet file in polars into separate columns
I have a string column in parquet which I would like to unnest:
df = df.select([pl.col("parameters").apply(json.loads)]).unnest("parameters")
I assume I need to do something similar to this (which I found in this answer):
fn…
Bahadir Cambel
- 422
- 5
- 12
0
votes
0 answers
How do I overwrite a dtype that Polars has inferred when reading a .CSV file?
I am attempting to read data from a .CSV file into a Polars dataframe for analysis. I followed this prior Stack Overflow answer to get help on the layout of the CsvReader's chained methods.
fn read_csv_to_dataframe(path: &str) ->…
Stephen Tanksley
- 31
- 5
0
votes
1 answer
Rust Polars Dataframe String replacement
I am writing a Rust using Polars. I would like to know how can I manipulate one string dataframe column.
For example, I have the following dataframe:
Id
Text
1
Some foo text
2
Other text
And I would like to replace all values that has…
lucaszane
- 3
- 3
0
votes
0 answers
Rust-Python interop in Polars
Is it anyhow possible to load a dataset in memory using rust, keep the program running, then somehow interact with the dataframe still loaded in rust - with python?
Example: if i have an axum application/service where i would load a dataframe, then…
mspehar
- 527
- 1
- 6
- 19
0
votes
1 answer
The reason why to use map twice for applying polars Rust
let o = GetOutput::from_type(DataType::UInt32);
// this adds new column len, two is unchanged
let lf = lf.with_column(col("two").alias("len").apply(str_to_len, o));
fn str_to_len(str_val: Series) -> Result {
let x = str_val
…
Crispy13
- 230
- 1
- 3
- 16
0
votes
1 answer
In Rust, how to rename all columns of a Polars Dataframe?
Rust newbie here, I want to use Polars to summarize a dataset that didn't come with column names. If I just use df directly, the error was: cannot borrow df as mutable because it is also borrowed as immutable. How to fix it?
use…
PaulDong
- 711
- 7
- 19