how to import parquet files from s3 to postgresql rds

Question

I'm trying to load a parquet file from s3 to AWS postgresql RDS. The data import using aws_s3.table_import_from_s3 is working fine for csv file but when I tried the similar way for parquet file, I get the below error:

ERROR: invalid byte sequence for encoding "UTF8": 0x00

In AWS docs, I see option for custom delimiter or zip file. Is it possible to import parquet data?

Did you find a way to solved it? – Benzi Jul 12 '21 at 11:52 — Benzi, Jul 12 '21 at 11:52

score 1 · Answer 1 · answered Jul 14 '21 at 07:21

1

Data import into an AWS PostgreSQL RDS supports what COPY does. Although there is a PostgreSQL binary file format it doesn't support Parquet so in order to import the data you have to convert it to a text file or the PostgreSQL binary file format first.

answered Jul 14 '21 at 07:21

voroninman

269
1
2
9

I made a library to convert Parquet to Postgres’ binary format: https://github.com/adriangb/pgpq – LoveToCode Feb 14 '23 at 22:15

how to import parquet files from s3 to postgresql rds

1 Answers1