Stack Overflow
Stack Overflow

Questions tagged [python-polars]

Polars is a DataFrame library/in-memory query engine.

The Polars core library is written in Rust and uses Arrow, the native arrow2 Rust implementation, as its foundation. It offers Python and JavaScript bindings, which serve as a wrapper for functionality implemented in the core library.

Links

1331 questions
3
votes
1 answer

Adding a column based on condition in Polars

Let's say I have a Polars dataframe like so: df = pl.DataFrame({ 'a': [0.3, 0.7, 0.5, 0.1, 0.9] }) And now I need to add a new column where 1 or 0 is assigned depending on whether a value in column 'a' is greater or less than some threshold. In…
NotAName
  • 3,821
  • 2
  • 29
  • 44
3
votes
1 answer

How to connect polars write_database with mssql+pyodbc?

Before polars>=0.16.10 I was using .to_pandas().to_sql() to send a polars dataframe to the database. Now it should be possible to use the wrapper pl.Dataframe.write_database(), however, altough my connection_uri is working for pl.read_database() and…
gilnribeiro
  • 83
  • 6
3
votes
2 answers

Polars: fill nulls with the only vaild value within each group

Each group only has one valid or not_null value in a random row. How do you fill each group with that value? import polars as pl data = { 'group': ['1', '1', '1', '2', '2', '2', '3', '3', '3'], 'col1': [1, None, None, None, 3, None, None,…
steven
  • 2,130
  • 19
  • 38
3
votes
2 answers

Correlation dataframe convertion from results from pl.corr

I have a simple dataframe as follows: import polars as pl df = pl.DataFrame( { "group": [1, 1, 1, 1, 2, 2, 2, 2], "a": [1, 2, 3, 4, 1, 2, 3, 4], "b": [5, 1, 7, 9, 2, 4, 9, 7], "c": [2, 6, 3, 9, 1, 5, 3, 6], …
lebesgue
  • 837
  • 4
  • 13
3
votes
1 answer

how to limit the display width in polars so that wide dataframes are printed in a legible way?

Consider the following example pd.set_option('display.width', 50) pl.DataFrame(data = np.random.randint(0,20, size = (10, 42)), columns = list('abcdefghijklmnopqrstuvwxyz123456789ABCDEFG')).to_pandas() You can see how nicely the…
ℕʘʘḆḽḘ
  • 18,566
  • 34
  • 128
  • 235
3
votes
2 answers

Polars: Select first (or any) non-null column value row-wise

How can we select first (or any) non-null struct from polars dataframe to new column? Input data: structs_a = [ {"a_key_1": 1, "a_key_2": "a" }, {"a_key_1": None, "a_key_2": None } ] structs_b = [ {"b_key_1": None, "b_key_2": None }, …
Krank
  • 141
  • 1
  • 8
3
votes
1 answer

Polars cumulative sum over consecutive groups

I have a DataFrame like so: | Date | Group | Value | |------------|-------|-------| | 2020-01-01 | 0 | 5 | | 2020-01-02 | 0 | 8 | | 2020-01-03 | 0 | 9 | | 2020-01-01 | 1 | 5 | | 2020-01-02 | 1 | -1 | |…
Syafiq Kamarul Azman
  • 860
  • 2
  • 8
  • 27
3
votes
2 answers

how custom sort of rows in polars

How to sort row with spesific order df = pl.DataFrame({"currency": ["EUR","EUR","EUR","USD","USD","USD"], "alphabet": ["A","B","C","A","B","C"]}) i need to descending the currency and custom sort of alphabet expected to be like…
Muhammad D Vikar
  • 91
  • 6
3
votes
3 answers

Trouble with strptime() conversion of duration time string

I have some duration type data (lap times) as pl.Utf8 that fails to convert using strptime, whereas regular datetimes work as expected. Minutes (before :) and Seconds (before .) are always padded to two digits, Milliseconds are always 3 digits. Lap…
Dorian
  • 33
  • 8
3
votes
2 answers

Create dictionary of each row in polars Dataframe

Lets assume we have below given dataframe. Now for each row I need to create dictionary and pass it to UDF for some logic processing.Is there a way to achieve this using either polars or pyspark dataframe ?
pbh
  • 186
  • 1
  • 9
3
votes
1 answer

Access newly created column in .with_columns() when using polars

I am new to polars and I am not sure whether I am using .with_columns() correctly. Here's a situation I encounter frequently: There's a dataframe and in .with_columns(), I apply some operation to a column. For example, I convert some dates from str…
Thomas
  • 1,199
  • 1
  • 14
  • 29
3
votes
2 answers

How do I write polars dataframe to external database?

I have big polars dataframe that I want to write into external database (sqlite for example) How can I do it? In pandas, you have to_sql() function, but I couldn't find any equivalent in polars
Shahar Luftig
  • 41
  • 5
3
votes
3 answers

Python Polars: How to get the row count of a DataFrame?

The CSV file I have is 70 Gb in size. I want to load the DF and count the number of rows, in lazy mode. What's the best way to do so? As far as I can tell, there is no function like shape in lazy mode according to the documentation. I found this…
roei shlezinger
  • 53
  • 1
  • 5
3
votes
1 answer

Dataframe conversion from pandas to polars -- difference in the final dimensions

I'm trying to convert a Pandas Dataframe to a Polar one. I simply used the function result_polars = pl.from_pandas(result). Conversion proceeds well, but when I check the shape of the two dataframe I get that the Polars one has half the size of the…
081N
  • 35
  • 4
3
votes
2 answers

Polars DataFrame save to sql

Is there a way to save Polars DataFrame into a database, MS SQL for example? ConnectorX library doesn’t seem to have that option.
Dennis L
  • 33
  • 3
1 2 3
…
88 89