Stack Overflow
Stack Overflow

Questions tagged [python-polars]

Polars is a DataFrame library/in-memory query engine.

The Polars core library is written in Rust and uses Arrow, the native arrow2 Rust implementation, as its foundation. It offers Python and JavaScript bindings, which serve as a wrapper for functionality implemented in the core library.

Links

1331 questions
6
votes
4 answers

Index operation on list column data in polars

I'm working with polars 0.13.46 for Python and I have a column with a list of Strings for which I need to check if a particular String occurs before another. I have created the following code example that works, but needs to break out of polars…
Thomas Lutterbeck
  • 71
  • 6
6
votes
1 answer

Polars: how to add a column with numerical?

in pandas： df['new'] = a where a is a numerical Series or just a number. while in polars we can add a char df.with_column( [ pl.all(), pl.lit('str').alias('new') ] ) but how to add a numerical Series or a number as a new column in…
lemmingxuan
  • 549
  • 1
  • 7
  • 18
6
votes
1 answer

How is Python Polars treating the index?

I want to try out polars in Python so what I want to do is concatenate several dataframes that are read from jsons. When I change the index to date and have a look at lala1.head() I see that the column date is gone, so I basically lose the index. Is…
daeda
  • 369
  • 5
  • 14
6
votes
1 answer

Extract value of Polars literal

If I have a Polars literal, how can I extract the value? import polars as pl expr = pl.lit(0.5) val = float(expr) # TypeError: float() argument must be a string or a real number, not 'Expr'
drhagen
  • 8,331
  • 8
  • 53
  • 82
6
votes
0 answers

How to exchange Polars-DataFrame between Rust and Python

I want to write a Python extension using Rust with Ctypes or Pyo3 to get better performance than native Python. But how to exchange data such as Polars DataFrame or ndarray type between Rust and Python?
Hakase
  • 211
  • 1
  • 12
6
votes
3 answers

How to filter a polars dataframe by date?

df.filter(pl.col("MyDate") >= "2020-01-01") does not work like it does in pandas. I found a workaround df.filter(pl.col("MyDate") >= pl.datetime(2020,1,1)) but this does not solve a problem if I need to use string variables.
keiv.fly
  • 3,343
  • 4
  • 26
  • 45
6
votes
1 answer

polars categorical feature and lazy api doesn't work like expected

I'm trying to join two Dataframes with the help of categorical features and the lazy API. I tried to do it the way it was decribed in the user guide(https://pola-rs.github.io/polars-book/user-guide/performance/strings.html) count =…
seb2704
  • 390
  • 1
  • 5
  • 17
5
votes
0 answers

Reduce polars memory consumption in unique()

I have a dataset that fits into RAM, but causes an out of memory error when I run certain methods, such as df.unique(). My laptop has 16GB of RAM. I am running WSL with 14GB of RAM. I am using Polars version 0.18.4. Running df.estimated_size() says…
stressed
  • 328
  • 2
  • 7
5
votes
4 answers

Detect the format of a datetime string in Python

I'm looking for a way to detect the strftime-style format of a datetime string in Python. All datetime libraries I've found have functionalities for parsing the string to create a datetime object, but I would like to detect the format or pattern…
pietz
  • 2,093
  • 1
  • 21
  • 23
5
votes
1 answer

How to implement rolling mean ignoring null values

I am trying calculate RSI indicator. For that I need rolling-mean gain and loss. I would like to calculate rolling mean ignoring null values. So mean would be calculated by sum and count on existing values. Example: window_size = 5 df =…
Joris Medeišis
  • 101
  • 4
5
votes
2 answers

convert 2 columns of polars dataframe to dictionary having its key as first column elements and second column elements as values

I am using below dataframe to convert to dictionary in specific format. However, I am getting an error TypeError: unhashable type: 'Series' import polars as pl #input (polars eager dataframe): polar_df = pl.DataFrame( "foo": ['a', 'b', 'c'], "bar":…
Rakesh Chaudhary
  • 55
  • 5
5
votes
2 answers

How to convert float to string with specific number of decimal places in Python polars?

I have a polars DataFrame with multiple numeric (float dtype) columns. I want to write some of them to a csv with a certain number of decimal places. The number of decimal places I want is column-specific. polars offers format: import polars as…
FObersteiner
  • 22,500
  • 8
  • 42
  • 72
5
votes
2 answers

How to extract date from datetime column in polars

I am trying to move from pandas to polars but I am running into the following issue. import polars as pl df = pl.DataFrame( { "integer": [1, 2, 3], "date": [ "2010-01-31T23:00:00+00:00", …
brokkoo
  • 157
  • 7
5
votes
1 answer

polars equivalent to pandas groupby shift()

Is there an equivalent way to to df.groupby().shift in polars? Use pandas.shift() within a group
Michael WS
  • 2,450
  • 4
  • 24
  • 46
5
votes
3 answers

Optimal way to get all duplicated rows in a polars dataframe

I want to filter all duplicated rows from a polars dataframe. What I've tried: df = pl.DataFrame([['1', '1', '1', '1'], ['7', '7', '2', '7'], ['3', '9', '3', '9']]) df shape: (4, 3) ┌──────────┬──────────┬──────────┐ │ column_0 ┆ column_1 ┆ column_2…
DataWiz
  • 401
  • 6
  • 14
1 2
3
…
88 89