1

I am trying to upload a very big pandas df (13230 rows x 2502 cols) into a postgres database. I am uploading the dataframe with the function df_to_sql but it gives me this error:

tables can have at most 1600 columns

Therefore I split the df into two dfs (13230 rows x 1251 cols each df) with the idea to merge them later. But when I try to upload the first df into the db I receive the following error:

row is too big: size 8168, maximum size 8160

How can I manage this? I would love to upload the df as a whole (13230 rows x 2502 cols) without merging it later.

arrabattapp man
  • 383
  • 1
  • 3
  • 11
  • 3
    I believe you are hitting the limit of the block in which the row is stored which is likely being that you have so many columns (I'm guessing the majority of the columns are an int or similar). You may need 3 tables to hold the data, but you won't be merging them into one inside of postgres as you will hit your per-row-storage-limit or per-row-column-limit. Can the data be normalized before write? I've encountered some monster tables, but 2500 columns seems like bad schema. – JNevill Jan 17 '22 at 21:17

0 Answers0