Can I copy data table folders in QuestDb to another instance?

Question

I am running QuestDb on production server which constantly writes data to a table, 24x7. The table is daily partitioned. I want to copy data to another instance and update it there incrementally since the old days data never changes. Sometimes the copy works but sometimes the data gets corrupted and reading from the second instance fails and I have to retry coping all the table data which is huge and takes a lot of time.

Is there a way to backup / restore QuestDb while not interrupting continuous data ingestion?

Answer 1

QuestDB appends data in following sequence

1. Append to column files inside partition directory
2. Append to symbol files inside root table directory
3. Mark transaction as committed in _txn file

There is no order between 1 and 2 but 3 always happens last. To incrementally copy data to another box you should copy in opposite manner:

1. Copy _txn file first
2. Copy root symbol files
3. Copy partition directory

Do it while your slave QuestDB sever is down and then on start the table should have data up to the point when you started copying _txn file.