This quick guide presents various techniques and strategies to backup VMware ESXi hosts using vSphere with Bacula Enterprise Edition.
The plugin allows complete virtual machines back up and uses the VMware’s Changed Block Tracking (CBT) technology o ensure only blocks that have changed since the initial Full, and/or the last Incremental or Differential Backup are copied to the backup stream, providing more efficient backups and reduced network load.
The plugin provides, among others, the following features:
- VADP-based online backups
- Automatic Job creation routine for each VM, with filters by name, tags etc. (BWeb vSphere Integration).
- Full, Differential and Incremental image-level backups of virtual machines
- Instant recovery of a VM or single file granular (with Bacula Enterprise Single-Item Plugin)
- Complete automatic virtual machine image restore
- Restores vmdk files to an alternate directory
- TCP/IP and SAN (FC/ISCSI) VMware Datastore access support
VM Image Backup with the Bacula Enterprise Edition vSphere Plugin
The Bacula VMware Vsphere Plugin allows client-less backup of the complete Virtual Machines. This means it is not really necessary to install a Bacula Client in the VMs Operating Systems. However, it is also possible to use Bacula Client and application-specific plugins (e.g. database ones) in order to provide more restore options (e.g. restoring a single database).
With the image backup level strategy, the Bacula Enterprise Edition vSphere plugin will save the VMs disks at the raw level, in the VMware/vSphere context. Bacula’s vSphere plugin will contact your host VMware ESXi server to read and save the contents of your virtual machine disks using NBD (Network Block Device) or SAN access.
When directly accessing a vmdk image stored on your Datastore, Bacula doesn’t need to walk through the Client filesystem to open/read/close/stat files, so it consumes fewer resources on your ESXi infrastructure than a backup on each guest machine would.
When using block differential techniques such as those used by the vSphere plugin, you need to ensure that all incremental backups are available for restore, and their retention periods are long enough to recover all data. If one of your incremental jobs is missing at the restore time, Bacula will not be able to create a consistent image. Using the Differential level permits reducing the number of Jobs that are required for restore, and thus reducing the risk that something may be lost.
The granular VMs files restore is provided by another plugin – the Bacula Enterprise Single-Item Restore Plugin, which is the object of another specific quick guide/whitepaper.
Installation
Packages are available for RedHat Enterprise and Ubuntu LTS and should installed together with a Bacula Client (bacula-fd), if not installed. Contact us to get them. Example:
# rpm -ivh bacula-enterprise-libs-8.8.3-1.el7.x86_64.rpm # rpm -ivh bacula-enterprise-client-8.8.3-1.el7.x86_64.rpm # rpm -ivh bacula-enterprise-vsphere-8.8.3-1.el7.x86_64.rpm # rpm -ivh bacula-enterprise-vsphere-vixdisk-6.0.2-3566099.1.el7.x86_64.rpm
The Bacula Client and Plugin can be installed on any machine with the network or SAN access to the vCenter or ESXi nodes, in order to perform VM backup. However, if you decide to install the Single Item restore plugin (optional), this machine MUST BE the Bacula Storage Daemon one in order to perform the granular file restore.
Release firewall required ports. The Plugin connects to vSphere over 443 and 902/TCP.
The vSphere network access should be configured in /opt/bacula/etc/vsphere_global.conf.
% cat /opt/bacula/etc/vsphere_global.conf [vsphere] username = root password = vspherepassword server = 192.168.0.1 url = https://192.168.0.1/sdk thumbprint = 34:F5:0F:10:82:59:EF:2D:DB:96:CC:5B:C4:66:33:83:DC:91:AF:09
The vCenter thumbprint can be fetched with the following command:
/opt/bacula/bin/vsphere-ctl thumbprint 192.168.0.1
Note that you can specify multiple vSphere servers in the vsphere_global.conf file, as in the following example:
% cat /opt/bacula/etc/vsphere_global.conf [vsphere] username = root password = vspherepassword server = 192.168.0.1 url = https://192.168.0.1/sdk default_datastore = datastore1 default_restore_host = esx1 thumbprint = 34:F5:0F:10:82:59:EF:2D:DB:96:CC:5B:C4:66:33:83:DC:91:AF:09 [vsphere2] username = root password = vspherepassword server = 192.168.0.2 url = https://192.168.0.2/sdk default_datastore = abigdatastore thumbprint = 59:F5:0F:10:82:59:EF:2D:DB:96:CC:5B:C4:66:33:83:DC:91:AF:C4
To test the vSphere plugin, you can use the following command as your Bacula FileDaemon user (usually root):
# /opt/bacula/bin/vsphere-ctl update 1: 3 squeeze2 2: 4 squeeze.esx
The vsphere-ctl update command should print a list of all virtual machines that are defined in your ESXi server. If not, please check that your credentials in vsphere-global.conf are properly set.
The list command permits display information that is detected on ESX hosts and datastores.
# /opt/bacula/bin/vsphere-ctl list
Display host list available and their datastores:
esxi.lan
datastore1
datastore2
...
As shown in table 1, a [global] section can also be set in vsphere_global.conf with options that apply to all vSphere connections. Other directives are also listed and described.
| Section | Option | Required | Default | Info |
|---|---|---|---|---|
| global | keep_generation | No | 100 | Maximum number of backups between two Full |
| global | profile_all_vm | No | vsphere_all_vm.profile | Internal filename used to store virtual machine profile information |
| global | root_directory | No | /opt/bacula/working/vsphere | Working directory of the vSphere Plugin |
| global | vddk_path | No | /opt/bacula/bin/vddk | |
| vsphere | username | Yes | Username allowed to connect to vSphere | |
| vsphere | password | Yes | Username password allowed o connect to vSphere | |
| vsphere | server | Yes | vSphere ESXi server used for Backup | |
| vsphere | url | Yes | vSphere ESXi server URL used for SOAP call | |
| vsphere | default_datastore | No | datastore1 | Default datastore for restore |
| vsphere | default_restore_host | No | Default ESX server used for restore if multiple ones are available in the vCenter |
Table 1. vSphere plugin vsphere_global.conf options
Backup Configuration
Once Bacula Client is attached to the director, it is time to create a new FileSet. This can be done via text or using bweb GUI, as exhibited in Figure 1.
Figure 1. bweb vSphere FileSet Edit Plugin screen
The only really required Plugin option is the vsphere one, that will perform the backup of all found VMs.
FileSet {
Name = vSphere
Include {
Options { Signature=MD5 }
Plugin = "vsphere:"
}
}
In order to perform the backup of a single machine, just specify its name in the host= directive:
FileSet {
Name = vSphere
Include {
Options { Signature=MD5 }
Plugin = "vsphere: host=vm_name"
}
}
Multiple “Plugin = vsphere: etc.” lines can be specified, and the host_include or host_exclude directives can be used to provide Java RegExp patterns that match VM names.
Some of the vSphere plugin current options are listed and described in Table 2.
| Option | Required | Default | Info | Example |
|---|---|---|---|---|
| host | No | Guest hostname | host=srv1 | |
| host_include | No | Guest name regexp pattern to include | host_include=srv3 | |
| host_exclude | No | Guest name regexp pattern to exclude | host_exclude=srv[12] | |
| disk_exclude | No | VSphere disk number to exclude | disk_exclude=0,2,4 | |
| keep_cbt | No | Don’t try to activate CBT | keep_cbt | |
| server | No | vsphere | Specify a vsphere server | server=vsrv2 |
| debug | No | Enable debug | debug |
Table 2. vSphere plugin options
Note: many other options can be found on the official Bacula Systems portal Bsysdoc.
Finally create a new backup Job to backup a FileSet with one, a few or several machines. It is NECESSARY to enable the Accurate Bacula Job option.
It is possible to test the new backup Job and FileSet with the Bacula estimate command:
*estimate listing job=vmware1 Using Catalog "MyCatalog" Connecting to Client 127.0.0.1-fd at 127.0.0.1:8102 -rwx------ 0 root root 0 2012-01-10 18:22:16 /@vsphere/3-squeeze2 -rw-r----- 1 root root 1923 2012-01-10 18:22:19 /@vsphere/3/vmbkp_generation.profile -rw-r----- 1 root root 4693 2012-01-10 18:22:19 /@vsphere/3/3.ovf -rw------- 1 root root 2147483648 2012-01-10 18:22:20 /@vsphere/3/0.bvmdk -rw------- 1 root root 104857600 2012-01-10 18:22:21 /@vsphere/3/1.bvmdk -rwx------ 1 root root 0 2012-01-10 18:22:26 /@vsphere/79-squeeze.esx ... 104857600 2012-01-10 18:22:30 2000 OK estimate files=10 bytes=2,462,069,574
Run the backup Job, It is possible to verify VMs snapshot messages in Bacula log.
Situational Configurations and Troubleshooting
Incremental/Differential Backup Backup Problems
To check if a virtual disk has CBT enabled, open the vSphere Client, select a powered-off virtual machine without snapshots.
- Right-click the virtual machine and click Edit Settings.
- Click the Options tab.
- Click General under the Advanced section and then click Configuration Parameters. The Configuration Parameters dialog opens.
- Click Add Row.
- Add the ctkEnabled parameter and then set its value to true.
- Click Add Row, add scsi0:0.ctkEnabled,and set its value to true.
Note: scsi0:0 in scsi0:0.ctkEnabled indicates the SCSI device assigned to the hard disk that is added to the virtual machine. Every hard disk added to the virtual machine is given an SCSI device that appears similar to scsi0:0, scsi0:1, or scsi1:1.
During the first Full backup, the vSphere Plugin will try to enable CBT automatically if the virtual machine is powered off. To disable this feature, use the keep_cbt option in the Plugin command string.
Plugin = "vsphere: keep_cbt host=guest1"
No snapshots must exist on a virtual machine at the time of enabling CBT on it. For details, refer to http://kb.vmware.com/kb/1033816.
Using SAN Access Backup (LANFree)
To use the SAN transport method, your backup server where the vsphere plugin is installed should have access to all LUNs exported to your ESX server.
Once your disks are visible as /dev/sda, /dev/sdb, … The vSphere plugin will open each disk to get the UUID and compare it with what the ESX server is providing.
When mapping a LUN to the Bacula Server, you need to scan the new disks:
for i in /sys/class/scsi_host/host*/scan ; do echo - - - > $i ; done
When using iSCSI, the lsscsi command will display them as:
% lsscsi [5:0:0:0] disk IET VIRTUAL-DISK 0 /dev/sdb
You can verify that SAN transport is used during backup by using the debug option in the Plugin command line and verifying the vddk trace file located in:
% grep san: /opt/bacula/working/vsphere/<moref>/<seq>/0.log DISKLIB-LIB : Opened "san://3-snapshot-18[datastore2] test/test_2.vmdk...
You can check the transport method used in the Bacula backup log.
Debug Traces
According to Table 3, the following trace files provide information, depending on the Bacula vSphere plugin error.
| File | Component | Note |
|---|---|---|
| /opt/bacula/*/vsphere-ctl.log | vsphere-ctl | This file is produced by the Java vsphere-ctl program that sends commands to the ESXi/vCenter server. |
| /opt/bacula/working/vsphere/*/*/*.log | vddk | This file is produced by the C++ vddk program that read/write to the VMDK. |
| /opt/bacula/working/*.trace | bacula-fd | This file is produced Bacula File Daemon when activating the debug. |
Table 3. vSphere Plugin Traces
Excluding Specific VM Disks from Backup
To exclude a specific disk, you can activate the independent mode in the vSphere console, or using the disk_exclude option in the VM backup FileSet.
Restore
Restoring to a New Virtual Machine
Using bweb (for example), web restores, choose a vSphere terminated backup job to restore. On file selection, mark one or more VMs files directories that you want to restore.
If you run the restore using the where=/ Bacula restore option, the vSphere Plugin will try to restore your disks on a new virtual machine created during the restore with the same attributes (disks, controller, CPU type, . . . ).
If a machine with the same name exists, another one will be created with a modified name, although this behavior can be changed in future plugin versions to replace the VM if the Bacula Replace=always restore option is set.
The ESX host and the datastore that will be used to restore your guest will be detected automatically, however, you can change it during the restore Advanced Options or configure default destinations in vsphere_global.conf (see default_restore_host and default_datastore).
Note that you need to have at least one VM configured on your ESX server to restore a VM from Bacula automatically. We plan to remove this limitation in a future version.
Bacula will not recreate all network settings during the restore, you will need to check the network configuration.
Restoring Machines to a Local Disk
By using where=/path/to/dir in the restore option, the Plugin will automatically restore selected files to this location. Then, it is possible to import machines to vCenter manually or, if desired, you may mount the image locally using VMWare vmware-mount tool or qemu-nbd and perform file level restores. These operations are automated and bweb graphically performed with Bacula Single Item Restore Plugin.
It is also possible to copy the raw image to any devices or to mount it and restore files directly.
modprobe nbd max_part=16 qemu-nbd -c /dev/nbd0 /tmp/0.bvmdk partprobe /dev/nbd0 fdisk -l /dev/nbd0 mount /dev/nbd0p1 /mnt/image # example ls /mnt/image
Troubleshooting
If you experience restoration problems such as importing OVF, such as:
# Bacula Job log: ... 2018-10-01 22:02:24 pocbacula-fd JobId 148: Error: No default OVF defined, aborting... 2018-10-01 22:02:24 pocbacula-fd JobId 148: Error: Unable to restore the host. null 2018-10-01 22:02:24 pocbacula-fd JobId 148: Warning: Unable to create VM, logs are in /opt/bacula/working/vsphere-ctl-0.log 2018-10-01 22:02:24 pocbacula-fd JobId 148: Error: Import of the OVF structure probably failed... ... # and cat /opt/bacula/working/vsphere-ctl-0.log .. 2018-10-02 12:19:28 WARN GlobalManager:446 - W: Line 1: Unable to parse 'backing.exclusive' for attribute 'key' on element 'Config'. 2018-10-02 12:19:28 WARN GlobalManager:485 - The operation is not supported on the object. 2018-10-02 12:19:28 WARN GlobalManager:487 - Could not obtain nfc lease. ..
Make sure your VSphere user can create machines and move them to the ESXi and the required datastores through the VCenter console.
Note: backing up VSphere templates is not allowed by the VMware API.
The VCenter log (/var/log/vmware/vpxd/vpxd.log) can provide useful information for problems in backing up VMs. Similarly, using the debug keyword in the FileSet Plugin line (vsphere: debug …) will create a debug log on the machine running the Plugin at:
/opt/bacula/working/vsphere/vm-xxx/xxx/0.log
Replace xxx with VM ID and incremental sequence of active debug backups. It is possible to increase debug level and even tuning VSphere backup by creating a /opt/bacula/etc/vixdisklib.conf file. Example:
vixDiskLib.nfc.LogLevel=7 vixDiskLib.san.LogLevel=7 vixDiskLib.transport.LogLevel=7 tmpDirectory=/tmp logDirectory=/tmp
Vixdisklib options are available at: https://pubs.vmware.com/vsphere-51/index.jsp?topic=%2Fcom.vmware.vddk.pg.doc%2FvddkFunctions.6.11.html
Reference
VMware Backup Using Bacula Enterprise Edition – https://www.baculasystems.com
Disponível em: 
Português (Portuguese (Brazil))
English
Español (Spanish)
