Stack Overflow
Stack Overflow

Questions tagged [apache-arrow]

Apache Arrow™ enables execution engines to take advantage of the latest SIM D (Single input multiple data) operations included in modern processors, for native vectorized optimization of analytical data processing.

For installation details see this

595 questions
1
vote
1 answer

Read csv file separated by semicolons (";") using the {arrow} package in R

I have a semicolon-separated csv file that has millions of rows of data. I want to use arrow to read it faster, but arrow doesn't provide a function like readr::read_csv2 or read.csv2 in base. How can I read a semicolon-separated csv file using…
Bastián Olea Herrera
  • 454
  • 4
  • 14
1
vote
1 answer

Convert CSV to Apache Arrow in Rust

I need to convert a csv file to apache arrow. Here is the structure of my csv file (much more rows than that…
Jona Rodrigues
  • 992
  • 1
  • 11
  • 23
1
vote
1 answer

How to filter rows from arrow::table based on a certain condition in Apache Arrow C++?

I want to do equivalent of pandas operation df[df['certain_date'] > '2023-05-26'] . I have gone through almost all the Apache Arrow related answers on this site. I have been trying some combination of is_in compute function here -…
Abhishek Kumar
  • 729
  • 6
  • 20
1
vote
0 answers

Rust + Apache Flight: how to get around a library trait method defined on &self

The Apache Arrow Flight protocol's Rust implementation defines a FlightService trait with a static lifetime pub trait FlightService: Send + Sync + 'static { ... and a non-mutable handshake method: // Required methods fn handshake<'life0,…
jwimberley
  • 1,696
  • 11
  • 24
1
vote
1 answer

Is it possible to partition a dataset with arrow by columns and not by column values?

I have a function that creates a data.table with around 29 million rows and a user defined number of columns based on an input sample list. It reads individual sample files with an index column and joins them column-wise to a master index column to…
basesorbytes
  • 85
  • 7
1
vote
0 answers

R arrow json reader calls notimplemented function

I'm trying to use the arrow json reader in R: library(arrow) read_json_arrow("http://stats.oecd.org/sdmx-json/data/QNA/all/all?startTime=2009-Q1&endTime=2009-Q4") It gives the error: Error: NotImplemented: Call to R (readBin() on R connection) from…
pluke
  • 3,832
  • 5
  • 45
  • 68
1
vote
1 answer

How to append new data to an existing parquet file?

I have parquet files with some data in them. I want to add more data to them frequently every day. I want to do this without having to load the object to memory and then concatenate and write again. instead, directly append to the end if the table…
Keyhan
  • 11
  • 1
1
vote
0 answers

How can I append data to each partition of an Arrow.Table created from a GroupedDataFrame in Julia?

I have a GroupedDataFrame GDF1 that I want to save as an Arrow.Table with each subdataframe as a separate partition. I am currently using Arrow.append to achieve this. However, I want to be able to append data to each partition after it is created…
phntm
  • 511
  • 2
  • 11
1
vote
1 answer

How can I create Arrow Builders from a Schema in Rust?

Given an arrow schema, what would be the idiomatic way to create builders for each field so that I can populate these fields with values that match the schema so that they may later be written to a parquet file that uses this schema? For example,…
Mark
  • 2,260
  • 18
  • 27
1
vote
0 answers

Cannot setup Apache Arrow v11 on Mac

I am getting all sorts of weird errors when setting up Apache Arrow v11 on my Macbook. I'm following steps from Apache Arrow documentation. Step 1 : Clone the repo - Success git clone https://github.com/apache/arrow.git cd arrow git submodule update…
Chirath LV
  • 48
  • 4
1
vote
0 answers

error of converting pandas data frame to cudf data frame

I would like to convert a pandas data frame to a cudf data frame on linux. My code: import cudf import pandas as pd test_data = { 'session_id':[1, 2], 'val' : [1.1, 2.2] } pd_df = pd.DataFrame(test_data) …
mtnt
  • 31
  • 5
1
vote
1 answer

Is there a native S3 filesystem implementation for Apache Arrow Java?

I'm working with Apache Arrow in Java and I want to know if there is an implementation in the java library that provides a native S3 filesystem implementation like the one provided in the Python implementation of Arrow (pyarrow) which uses the…
Arjun Kashyap
  • 623
  • 2
  • 10
  • 24
1
vote
1 answer

How do I enable TLS on an Apache Arrow FlightClient in Java?

The documentation for the Java Apache Arrow (v11.0.0) FlightClient.Builder has several methods related to constructing a TLS-enabled client: clientCertificate(InputStream clientCertificate, InputStream clientKey) useTls() overrideHostname(String…
c_sagan
  • 482
  • 3
  • 15
1
vote
0 answers

How to rbind two feather into one file?

I am using arrow to combine two feather file. #data collection_from…
doraemon
  • 439
  • 3
  • 10
1
vote
0 answers

Apache Arrow from C# to Julia or Python - footer issue

I am writing a struct array in C# using the following code: var structField = new StructType( new [] { new Field("field1", new StringType(), nullable: false), new Field("field2", new…
BAR
  • 15,909
  • 27
  • 97
  • 185
1 2 3
…
39 40