Data lost after insert to PostgreSQL Data base

Asked Nov 10 '22 at 07:50

Active Nov 10 '22 at 08:05

Viewed 101 times

I build pretty easy etl procces that get data from one DB and put it to another one.

eng1=sqlalchemy.create_engine(url1)  
eng2=sqlalchemy.create_engine(url2) 
for df in pd.read_sql(query,eng,chuncksize=1000):  
    df.to_sql(table_name,eng2, if_exists='append')

During it I catch two mysterious fails.

First is that some data is lost. It hasn't reached destination table.
Second is that all the data that's reached the destination is being expired in a few moments.

My colleague tried to build dummy ETL with another table and it works pretty nice. But the destination table couldn't be changed and it's kinda interesting what's wrong with the table or process.

edited Nov 10 '22 at 08:05

asked Nov 10 '22 at 07:50

Daniel

1

Did you forget to commit the inserting transaction? – Laurenz Albe Nov 10 '22 at 08:34
@LaurenzAlbe Sure I did! – Daniel Nov 10 '22 at 10:39
We try to upload the same data with the same process to another table and everything went good. The problem is with this mystery table – Daniel Nov 10 '22 at 10:40
Log SQL statements in PostgreSQL and see what actually happens. – Laurenz Albe Nov 10 '22 at 12:13

Data lost after insert to PostgreSQL Data base

0 Answers0