bacula virtual full backup

Question

I've searched a lot without finding a clear documentation. Following the instructions provided here https://www.bacula.org/9.4.x-manuals/en/main/Migration_Copy.html#SECTION003050000000000000000 it seem that to run a virtual full backup, bacula need at least 2 storage daemons... I have tried to set 2 pools with with the same storage daemon and the job stuck with this message:storage01-tarquini-sd JobId 127: JobId=127, Job RPROXY.2020-01-05_16.11.39_54 waiting to reserve a device. I have bacula dir and sd on the same machine that write on a local mounted NFS. Is it possible to run 2 sd on the same machine? Is it the right way to do it? Thank you!

score 1 · Answer 1 · answered Jan 06 '20 at 14:35

Found more infos here: https://www.bacula.org/9.4.x-manuals/en/main/Migration_Copy.html#SECTION003001000000000000000 Virtual full is treated as migration/copy so is a sd to sd operation. Those types of operations need 2 storage daemons. I set a new SD in the bacula-dir.conf that point to the same local storage daemon but with another name. In this way i can have 2 storage devices one for the incremental pool and one for the next pool needed by the virtual full backup. Tested and working, however any suggestion is appreciated.

score 0 · Answer 2 · answered Mar 22 '23 at 16:21

In the interest of future people struggling to make Virtual full backups, here's an overview of how I've managed to configure the virtual full backup setup for a remote client on a remote storage daemon. That is, with three hosts (which are assumed to be reachable over DNS or via hosts file);

bacula.acme.com  --> The director
stor02.acme.com  --> The storage host
backup.myclient.com --> The computer we want backed up.

Overall; there needs to be full 6-way communication between the 3 hosts for the program to work. All communication is assumed to be over the internet here so encyption is used for data transfer.

It's taken quite a bit of fiddling and experimentation to get right; the documentation on it is quite fragmented: if you're trying to do something other than the default setup, I find bacula fights you every step of the way.

The goal here is to have an incremental backup done after each workday, and to use the weekend to consolidate the incremental backups into a virtualFull. This minimizes data transfer and storage use.

Note that the configuration supports multiple clients at the same time, unlike the default which will do only one backup at a time. This is useful if say the bacula-sd host is say a powerful server in a well-connected area like a datacenter, but the clients have much slower connections and disk speeds.

If a longer retention period is required, change the Backups to keep directive to a higher number than 1, set it to 1 + 5 * 'number of weeks to keep'. Also change the number of tapes used for incremental backups from 3 to at least 2 + 'number of weeks to keep'. This can be found under maximum Volumes in the pool config. I configured one extra tape so as to minimize the possibility of running out of tapes if users do extra manual backups.

It's mostly useful for backing up backups done via other software remotely: there's already multiple versions of files kept.

There are two folder structures. One for the director, one for the sd. The director structure looks like;

/etc/bacula
    /etc/bacula/certs
    /etc/bacula/filedefs
    /etc/bacula/jobdefs
    /etc/bacula/pooldefs
    /etc/bacula/storagedefs
    /etc/bacula/schedules

And the sd directory structure looks like:

/etc/bacula
    /etc/bacula/devices

Where each file in each of these directories is included in bacula-dir.conf with code like so:

@|"find /etc/bacula/clientdefs -name '*.conf' -type f -exec echo @{} \;"
@|"find /etc/bacula/jobdefs -name '*.conf' -type f -exec echo @{} \;"
@|"find /etc/bacula/storagedefs -name '*.conf' -type f -exec echo @{} \;"
@|"find /etc/bacula/pooldefs -name '*.conf' -type f -exec echo @{} \;"
@|"find /etc/bacula/filedefs -name '*.conf' -type f -exec echo @{} \;"
@|"find /etc/bacula/schedules -name '*.conf' -type f -exec echo @{} \;"

It is useful to split the configuration into multiple smaller files. Templating etc. can be used to replicate config changes for many clients rapidly. The configuration is more readable than in one monolithic thing.

Then, for each computer to backup using virtualFulls, create files in each of these directories.

In certs, create a public key and self-signed certificate using openSSL.

In clientdefs, create the client definition, named backup.myclient.com-client.conf. This will look like:

Client {
  Name = "backup.myclient.com-fd"
  Address = backup.myclient.com
  FDPort = 9102
  Catalog = MyCatalog
  Password = "bowoa33bOAIEgklqiowWEEIQmd910GAOIeyglU5N5vr"
  File Retention = 30 days
  Job Retention = 6 months
  AutoPrune = yes
  TLS Enable = yes
  TLS Require = yes
  TLS Certificate = /etc/bacula/certs/backup.myclient.com.crt
  TLS Key = /etc/bacula/certs/backup.myclient.com.key
  TLS CA Certificate File = /etc/bacula/certs/myca.crt
}

Note; the password here has to be configured under director in the file daemon's bacula-fd.conf. Similarly, the keys and certificates have to be copied over and set there as well. The same applies later to the storage definitions.

In the jobs folder there's a file named backup.myclient.com-job.conf. This contains the following:

Job {
  Name = "backup.myclient.com-job"
  Client = "backup.myclient.com-fd"
  FileSet = backup.myclient.com-FileSet
  Pool = backup.myclient.com-Inc-Pool-stor02
  Priority = 10
  # Keeps track of deleted files.
  Accurate = yes
  Full Backup Pool = backup.myclient.com-Full-Pool-stor02
  VirtualFull  Backup Pool = backup.myclient.com-Inc-Pool-stor02
  Differential Backup Pool = backup.myclient.com-Diff-Pool-stor02
  Incremental Backup Pool = backup.myclient.com-Inc-Pool-stor02
  # This consolidates backups so you have 1 full and 1-8 incremental at all times.
  Backups To Keep = 1
  # If the job is blocked (lost connection / not enough volumes?) continue with other jobs.
  Max Wait Time = 1h
  # Every saturday, the Full backup is copied and the new incremental added.
  # This deletes the old copy.
  Delete Consolidated Jobs = yes
  Enabled = Yes #enable automatic execution
  Type = Backup
  Schedule = monthly_virtualfull_03
  Messages= Standard
  Storage = ds_stor02_backup.myclient.com
}

Next, the pooldefs directory contains the storage pool configs for each client, so in the example the file is named backup.myclient.com-pool.conf

Pool {
  Name = backup.myclient.com-Full-Pool-stor02
  Pool Type = Backup
  Recycle = yes           # automatically recycle Volumes
  AutoPrune = yes         # Prune expired volumes
  Volume Retention = 31 days
  Maximum Volume Jobs = 2
  Label Format = backup.myclient.com-Full-
  Maximum Volumes = 2
  Storage = ds_stor02_vf_backup.myclient.com
}
Pool {
  Name = backup.myclient.com-Diff-Pool-stor02
  Pool Type = Backup
  Recycle = yes
  AutoPrune = yes
  Volume Retention = 40 days
  Maximum Volume Jobs = 1
  Label Format = backup.myclient.com-Diff-
  Maximum Volumes = 8
  Next Pool = backup.myclient.com-Diff-Pool-stor02
  Storage = ds_stor02_backup.myclient.com
}

Pool {
  Name = backup.myclient.com-Inc-Pool-stor02
  Pool Type = Backup
  Recycle = yes           # automatically recycle Volumes
  AutoPrune = yes         # Prune expired volumes
  # One day more than the longest month.
  Volume Retention = 10 days
  Maximum Volume Jobs = 5
  Label Format = backup.myclient.com-Inc-
  Maximum Volumes = 3
  Storage = ds_stor02_backup.myclient.com
  }

For the schedules, there's one file needed, virtualfull_03.conf;

Schedule {
    Name = "monthly_virtualfull_03"
    Run = VirtualFull sat at 10:00
    Run = Incremental mon-sat at 00:30
}

When picking times at night: be careful not to pick any time between 01:00 and 03:00 if you are in any part of the world that still observes DST (which is most of it), or DST transitions can and will make your backups not run (or try to run twice). Alternatively: configure your director to use UTC (and don't care that backups will run an hour earlier part of the year).

Of course, if clients need to run at different dates and times more schedules can be added to make this happen.

Next, the storagedefs. By creating separate storages, we can have the individual behaviour we want; For virtualFull, you need to create two storagedefs, both pointing to the same storage daemon.

Storage {
    Name = ds_stor02_backup.myclient.com
    Address = stor02.acme.com   # N.B. Use a fully qualified name here
    SDPort = 9103
    Password = 98C_DcJmmAjWqjqleIIo19xIiGHGHqzi
    Device = stor02_backup.myclient.com
    Media Type = mt_stor02_backup.myclient.com
    Maximum Concurrent Jobs = 50

    TLS Enable = yes
    TLS Require = yes
    TLS Certificate = /etc/bacula/certs/stor02.acme.com.crt
    TLS Key = /etc/bacula/certs/stor02.acme.com.key
    TLS CA Certificate File = /etc/bacula/certs/myca.crt
}
Storage {
    Name = ds_stor02_vf_backup.myclient.com
    Address = stor02.acme.com   # N.B. Use a fully qualified name here
    SDPort = 9103
    Password = 98C_DcJmmAjWqjqleIIo19xIiGHGHqzi
    Device = stor02_vf_backup.myclient.com
    Media Type = mt_stor02_vf_backup.myclient.com
    Maximum Concurrent Jobs = 50
    Autochanger = yes

    TLS Enable = yes
    TLS Require = yes
    TLS Certificate = /etc/bacula/certs/stor02.acme.com.crt
    TLS Key = /etc/bacula/certs/stor02.acme.com.key
    TLS CA Certificate File = /etc/bacula/certs/myca.crt
}

Finally, you'll need at least three devices for virtualFull. One tape drive to read the full backup tape the current full backup. One tape drive to write to the other full backup tape the consolidated result, and the third tape drive to read the incremental backup. (Note: If you're using actual tape drives, and not files as in the example, adding a second autochanger for the incremental reads with one drive for each tape you're using would be optimal for fully automated backups).

These devices are configured on the storage daemon, not on the director, in side a devices folder which is included in bacula-sd.conf (via same method as above).

Here, we also define an "autochanger" between air-quotes. The bacula-code for read/writing multiple jobs from the same drive using random access media is buggy. It won't do it, and will either endlessly stall, complain about mediatypes, or one of a host of other problems. Making a fake tape-drive setup will work, and flawlessly at that. This looks like this (backup.myclient.com-device.conf):

Device {
    Name = stor02_backup.myclient.com
    Media Type = mt_stor02_backup.myclient.com
    Archive Device = /zfs1/myclient/backups/backup.myclient.com/
    LabelMedia = yes;
    Random Access = yes;
    AutomaticMount = yes;
    RemovableMedia = no;
    AlwaysOpen = no;
    Maximum Concurrent Jobs = 2
}

Autochanger {
   Name = stor02_vf_backup.myclient.com
   Device = stor02_vf_backup.myclient.com_1, stor02_vf2_backup.myclient.com_2
   Changer Device = /dev/null
   Changer Command = ""
}


Device {
    Name = stor02_vf_backup.myclient.com_1
    Media Type = mt_stor02_vf_backup.myclient.com
    Archive Device = /zfs1/myclient/backups/backup.myclient.com_vf/
    LabelMedia = yes;
    Random Access = yes;
    AutomaticMount = yes;
    RemovableMedia = no;
    AlwaysOpen = no;
    Maximum Concurrent Jobs = 2
}


Device {
    Name = stor02_vf2_backup.myclient.com_2
    Media Type = mt_stor02_vf_backup.myclient.com
    Archive Device = /zfs1/myclient/backups/backup.myclient.com_vf/
    LabelMedia = yes;
    Random Access = yes;
    AutomaticMount = yes;
    RemovableMedia = no;
    AlwaysOpen = no;
    Maximum Concurrent Jobs = 2
}

bacula virtual full backup

2 Answers2