1

Given an arrow schema, what would be the idiomatic way to create builders for each field so that I can populate these fields with values that match the schema so that they may later be written to a parquet file that uses this schema?

For example, given the schema:

let foo_field = Field::new("FOO", DataType::Utf8, false);
let baa_field = Field::new("BAA", DataType::UInt16, false);

let schema = Schema::new(vec![
    foo_field,
    baa_field,
]);

Then I can create builders which will match the schema using

let mut foo_builder = GenericStringBuilder::<i32>::new();
let mut baa_builder = UInt16Array::builder(batch_size);

but I would ideally like to create them from the schema so that I can be certain they will match but I can't see a way to do this.

For context, my dependencies are currently:

[dependencies]
arrow = "37.0.0"
parquet = "37.0.0"
itertools = "0.10.5"

but I am open to changing libraries and versions if needed.

Mark
  • 2,260
  • 18
  • 27

1 Answers1

1

arrow-rs do have a make_builder API to dynamically create builders from data types. however some types (for example lists and maps) are not supported at the moment.

richselian
  • 731
  • 4
  • 18