2

We are backing up 1TB+ of postgres database using barman. Barman is backing up full database daily. Every day we are dumping 1TB+ database. So it's consuming huge space. We have following configuration in barman:(broad - level config)

backup_method=rsync

compression=pzip

reuse_backup=copy

Due to hardware constraint we can't use reuse_backup=link, means we can't take incremental backup.

We don't know even after setting up compression=pzip, why barman is not compressing the backup.

So what we want is that, can we have a post script which will compress the database backup or any other option to compress the backup taken using barman.

Version details: Barman : ver2.12 Postgresql:13 OS: Ubuntu 20.04

Thanks in advance.

SAKA RAM DEWASI
  • 39
  • 5
  • Barman doesn’t support backup compression it only supports wal compression i.e when you have archival enabled. – Raj Verma Jul 07 '21 at 20:50
  • Looking at the [docs](http://docs.pgbarman.org/release/2.12/). It's not clear whether `pzip` is even a valid compression option. Have you tried `gzip`? Also the `network_compression` option looks interesting, but I'm not really sure how it works or whether it could be helpful for you. – bfris Jul 07 '21 at 22:02

3 Answers3

1

pgbarman does not do what you want. Maybe you can look into something else, like pgbackrest. (I have no experience using that, but the description I've read makes it sound like it works for this).

With barman, you could compress the backup after the fact. You could even do that automatically, with something like:

post_backup_script = gzip -r $BARMAN_BACKUP_DIR/

But, that will interfere with the reuse=copy configuration. Also, you would have to decompress it before using for restoration purposes (maybe with pre_recovery_script). And even then, you do still need the entire backup completely on disk before you can initiate compression on it.

jjanes
  • 37,812
  • 5
  • 27
  • 34
1

First things first:

The tool, Barman indeed is a really good tool but for your used case I seldom believe its coming out as a productive one.

Second, you need to redo some of your backup strategy work. Looking at the backups your are consuming, and I do not know what your retention is(i'm assuming it won't be high looking at the size of the backups) here are my 2 cents:

  1. Compressing backups may save time and space but it adds an overhead and time when your want to restore(again this doesn't apply to smaller DB's).

  2. Taking daily backups of a DB which is TB's(might grow as well), is not a good option when you compare it with having incremental and logs on top of it.

  3. Have a base backup every week, differential daily and then logs on top of it. You can always fine tune this along with the retention.

  4. I'm not sure if you environment supports snapshots but this is another way out and really speeds up the restore and backups for you, its snapshots(at machine level, storage level etc.) In essence take the snapshot, tune your pg_starts and pg_stop along with the snapshot timings and then put archival logs(this is if someone archives their logs)

The above would help in speeding up the process and also give you back space.

Now coming back to the tool. Pgbackrest is an excellent option to perform all of the above (except 4.). I see it working for your used case better than barman if you do not want to redesign the backup strategies(which I would recommend doing irrespective of which tool you use)

I would recommend against taking a backup and then zipping it, just to save space, this is taking more time even after the backup is done and restorability takes a hit. Futuristically this approach will not scale well too.

Raj Verma
  • 1,050
  • 1
  • 7
  • 19
0

PG barman is doing only wal compression. If you need to do the full backup compression, PGBackrest is the best option for sure.

Wimal Sudusingh
  • 1