0

I am using Meltano for ETL to extract data from a source PostgreSQL database using tap-postgres (MeltanoLabs variant) and load it into a destination PostgreSQL database using target-postgres (TransferWise variant). However, I have noticed that when I delete rows from the source PostgreSQL database, the deleted rows are not removed from the destination PostgreSQL database.

I have tried using the --full-refresh option with the meltano run command, but this does not seem to delete the rows from the destination database. Is there a way to reset the data in the destination database before running the job with Meltano?

Here is the command I am using to perform the ETL:
meltano run tap-postgresql target-postgres --full-refresh

I have verified that the rows have been deleted from the source PostgreSQL database by running a SQL query, but the deleted rows still exist in the destination PostgreSQL database after running the job.

Any help or suggestions would be greatly appreciated. Thank you.

Sanushi Salgado
  • 1,195
  • 1
  • 11
  • 18
vuonglv
  • 31
  • 2

1 Answers1

1

You'll need to turn on the hard_delete option for the tap and the add_metadata_columns See https://github.com/transferwise/pipelinewise-target-postgres

Aleksandar
  • 844
  • 1
  • 8
  • 18
Sven Balnojan
  • 11
  • 1