1

I have a sawtooth 1.1 dockerized network, and i'm trying to backup the database from the validators so i can put down every container and then recover in case of a disaster.

Trying to achieve this i proceed as followed:

  1. Stopped the all containers;

  2. Backed up all the files of one of the validators on /var/lib/sawtooh/ using

    cp --sparse=always [file] [file_backup]

  3. Removed all the containers using docker-compose down

  4. Started a fresh network with docker-compose up

  5. Stopped all containers using docker-compose stop

  6. Copied the files backed up on step 2 to the new validators using the command of that same step

  7. Restarted all network using docker-compose restart

After this i could repare that the states were correct, users on the blockchain have the same balance as before of the docker-compose down. But the blockchain doesn't process new transactions. The only error that i've found in the logs, was in the sawtooth-poet-engine i believe during the consensus as it show on the this image, ERROR_IMAGE.

So my question is, does anybody tried to do this with success or have any idea of what i'm doing wrong?

rbn.araujo
  • 11
  • 3

1 Answers1

1

I just tried the same thing and it worked for me. One possible problem is file permissions and ownership. Use the cp -p option to preserve ownership and permissions:

cp -p --sparse=always [file] [file_backup]

Also verify the ownership and permissions are correct with ls -l /var/lib/sawtooth . They should be all read/write by owner and owned by user/group sawtooth. If not, fix the ownership with something like

chown sawtooth:sawtooth /var/lib/sawtooth /var/lib/sawtooth/* chmod u+rw /var/lib/sawtooth /var/lib/sawtooth/* chmod ugo+r /var/lib/sawtooth/block-* /var/lib/sawtooth/txn_receipts-00.lmdb*

Dan Anderson
  • 2,265
  • 1
  • 9
  • 20
  • Tried what you said and now i don't have the error, but can't do transactions and i don't see any error on any log, and noticed that the permissions are read and write to root, even before. Tried the backup with cp -p and then fix the owner like you said, but i still have the same problem. There is a piece of information that i didn't said that maybe as something to do with this. To be able to use cp --sparse=always i've created 3 volumes (one for each validator) that points to the /var/lib/sawtooth and then i execute my backup and restore commands pointing to the volume dir on the host. – rbn.araujo Feb 27 '19 at 11:23
  • i've noticed on the poet-engine logs that they are stuck, i can try do transactions but even with the logs on debug nothing new happens there. In this link is the log, i've marked the moment i changed the LMDB files, pls take a look https://justpaste.it/5360d – rbn.araujo Feb 27 '19 at 11:35
  • You are supposed to stop everything when you copy the files. – Dan Anderson Feb 27 '19 at 16:46
  • Yes, and i do everything with containers stopped and then i start the containers. – rbn.araujo Feb 27 '19 at 17:28
  • My compose is similar to this https://sawtooth.hyperledger.org/docs/core/releases/1.1/app_developers_guide/sawtooth-default-poet.yaml Should i save the files on poet-share and then do a restore of them too? – rbn.araujo Feb 27 '19 at 21:55
  • I am not sure the problem is copying the files. Look at the file and directory permissions of the old and copied files and see if they are the same. Use the `cmp --verbose` command and compare the old and copied files--they should be the same. – Dan Anderson Feb 27 '19 at 23:42
  • It could be that restarting the containers are the problem. Can you stop and restart the containers OK without copying the files? – Dan Anderson Feb 27 '19 at 23:42
  • As far as what's in the containers, the contents will stay there if you don't reboot the guest host where the containers run. Otherwise you need to save /etc/sawtooth/, /var/lib/sawtooth/, /var/log/sawtooth/ and the ~/.sawtooth directory where you have the user keys (often /root/.sawtooth). – Dan Anderson Feb 27 '19 at 23:42