VMware VSphere Enterprise Bacula Plugin Quick Guide

This quick guide presents various techniques and strategies to backup VMware ESXi hosts using vSphere with Bacula Enterprise Edition.

The plugin allows complete virtual machines back up and uses the VMware’s Changed Block Tracking (CBT) technology o ensure only blocks that have changed since the initial Full, and/or the last Incremental or Differential Backup are copied to the backup stream, providing more efficient backups and reduced network load.

The plugin provides, among others, the following features:

  • VADP-based online backups
  • Automatic Job creation routine for each VM, with filters by name, tags etc. (BWeb vSphere Integration).
  • Full, Differential and Incremental image-level backups of virtual machines
  • Instant recovery of a VM or single file granular (with Bacula Enterprise Single-Item Plugin)
  • Complete automatic virtual machine image restore
  • Restores vmdk files to an alternate directory
  • TCP/IP and SAN (FC/ISCSI) VMware Datastore access support

VM Image Backup with the Bacula Enterprise Edition vSphere Plugin

The Bacula VMware Vsphere Plugin allows client-less backup of the complete Virtual Machines. This means it is not really necessary to install a Bacula Client in the VMs Operating Systems. However, it is also possible to use Bacula Client and application-specific plugins (e.g. database ones) in order to provide more restore options (e.g. restoring a single database).

With the image backup level strategy, the Bacula Enterprise Edition vSphere plugin will save the VMs disks at the raw level, in the VMware/vSphere context. Bacula’s vSphere plugin will contact your host VMware ESXi server to read and save the contents of your virtual machine disks using NBD (Network Block Device) or SAN access.

When directly accessing a vmdk image stored on your Datastore, Bacula doesn’t need to walk through the Client filesystem to open/read/close/stat files, so it consumes fewer resources on your ESXi infrastructure than a backup on each guest machine would.

When using block differential techniques such as those used by the vSphere plugin, you need to ensure that all incremental backups are available for restore, and their retention periods are long enough to recover all data. If one of your incremental jobs is missing at the restore time, Bacula will not be able to create a consistent image. Using the Differential level permits reducing the number of Jobs that are required for restore, and thus reducing the risk that something may be lost.

The granular VMs files restore is provided by another plugin – the Bacula Enterprise Single-Item Restore Plugin, which is the object of another specific quick guide/whitepaper.

Installation

Packages are available for RedHat Enterprise and Ubuntu LTS and should installed together with a Bacula Client (bacula-fd), if not installed. Contact us to get them. Example:

# rpm -ivh bacula-enterprise-libs-8.8.3-1.el7.x86_64.rpm 
# rpm -ivh bacula-enterprise-client-8.8.3-1.el7.x86_64.rpm 
# rpm -ivh bacula-enterprise-vsphere-8.8.3-1.el7.x86_64.rpm  
# rpm -ivh bacula-enterprise-vsphere-vixdisk-6.0.2-3566099.1.el7.x86_64.rpm

The Bacula Client and Plugin can be installed on any machine with the network or SAN access to the vCenter or ESXi nodes, in order to perform VM backup. However, if you decide to install the Single Item restore plugin (optional), this machine MUST BE the Bacula Storage Daemon one in order to perform the granular file restore.

Release firewall required ports. The Plugin connects to vSphere over 443 and 902/TCP.

The vSphere network access should be configured in /opt/bacula/etc/vsphere_global.conf.

% cat /opt/bacula/etc/vsphere_global.conf
[vsphere]
username = root
password = vspherepassword
server = 192.168.0.1
url = https://192.168.0.1/sdk
thumbprint = 34:F5:0F:10:82:59:EF:2D:DB:96:CC:5B:C4:66:33:83:DC:91:AF:09

The vCenter thumbprint can be fetched with the following command:

/opt/bacula/bin/vsphere-ctl thumbprint 192.168.0.1

Note that you can specify multiple vSphere servers in the vsphere_global.conf file, as in the following example:

% cat /opt/bacula/etc/vsphere_global.conf
[vsphere]
username = root
password = vspherepassword
server = 192.168.0.1
url = https://192.168.0.1/sdk
default_datastore = datastore1
default_restore_host = esx1
thumbprint = 34:F5:0F:10:82:59:EF:2D:DB:96:CC:5B:C4:66:33:83:DC:91:AF:09
[vsphere2]
username = root
password = vspherepassword
server = 192.168.0.2
url = https://192.168.0.2/sdk
default_datastore = abigdatastore
thumbprint = 59:F5:0F:10:82:59:EF:2D:DB:96:CC:5B:C4:66:33:83:DC:91:AF:C4

To test the vSphere plugin, you can use the following command as your Bacula FileDaemon user (usually root):

# /opt/bacula/bin/vsphere-ctl update
  1: 3 squeeze2
  2: 4 squeeze.esx

The vsphere-ctl update command should print a list of all virtual machines that are defined in your ESXi server. If not, please check that your credentials in vsphere-global.conf are properly set.

The list command permits display information that is detected on ESX hosts and datastores.

# /opt/bacula/bin/vsphere-ctl list
Display host list available and their datastores:
  esxi.lan
    datastore1
    datastore2
...

As shown in table 1, a [global] section can also be set in vsphere_global.conf with options that apply to all vSphere connections. Other directives are also listed and described.

Section Option Required Default Info
global keep_generation No 100 Maximum number of backups between two Full
global profile_all_vm No vsphere_all_vm.profile Internal filename used to store virtual machine profile information
global root_directory No /opt/bacula/working/vsphere Working directory of the vSphere Plugin
global vddk_path No /opt/bacula/bin/vddk
vsphere username Yes Username allowed to connect to vSphere
vsphere password Yes Username password allowed o connect to vSphere
vsphere server Yes vSphere ESXi server used for Backup
vsphere url Yes vSphere ESXi server URL used for SOAP call
vsphere default_datastore No datastore1 Default datastore for restore
vsphere default_restore_host No Default ESX server used for restore if multiple ones are available in the vCenter

Table 1. vSphere plugin vsphere_global.conf options

Backup Configuration

Once Bacula Client is attached to the director, it is time to create a new FileSet. This can be done via text or using bweb GUI, as exhibited in Figure 1.

VMware VSphere Enterprise Bacula Plugin Quick Guide 1

Figure 1. bweb vSphere FileSet Edit Plugin screen

The only really required Plugin option is the vsphere one, that will perform the backup of all found VMs.

FileSet {
  Name = vSphere
  Include {
   Options { Signature=MD5 }
   Plugin = "vsphere:"
  }
}

In order to perform the backup of a single machine, just specify its name in the host= directive:

FileSet {
  Name = vSphere
  Include {
   Options { Signature=MD5 }
   Plugin = "vsphere: host=vm_name"
  }
}

Multiple “Plugin = vsphere: etc.” lines can be specified, and the host_include or host_exclude directives can be used to provide Java RegExp patterns that match VM names.

Some of the vSphere plugin current options are listed and described in Table 2.

Option Required Default Info Example
host No Guest hostname host=srv1
host_include No Guest name regexp pattern to include host_include=srv3
host_exclude No Guest name regexp pattern to exclude host_exclude=srv[12]
disk_exclude No VSphere disk number to exclude disk_exclude=0,2,4
keep_cbt No Don’t try to activate CBT keep_cbt
server No vsphere Specify a vsphere server server=vsrv2
debug No Enable debug debug

Table 2. vSphere plugin options

Note: many other options can be found on the official Bacula Systems portal Bsysdoc.

Finally create a new backup Job to backup a FileSet with one, a few or several machines. It is NECESSARY to enable the Accurate Bacula Job option.

It is possible to test the new backup Job and FileSet with the Bacula estimate command:

*estimate listing job=vmware1
Using Catalog "MyCatalog"
Connecting to Client 127.0.0.1-fd at 127.0.0.1:8102
-rwx------ 0 root root           0 2012-01-10 18:22:16  /@vsphere/3-squeeze2
-rw-r----- 1 root root        1923 2012-01-10 18:22:19  /@vsphere/3/vmbkp_generation.profile
-rw-r----- 1 root root        4693 2012-01-10 18:22:19  /@vsphere/3/3.ovf
-rw------- 1 root root  2147483648 2012-01-10 18:22:20  /@vsphere/3/0.bvmdk
-rw------- 1 root root   104857600 2012-01-10 18:22:21  /@vsphere/3/1.bvmdk
-rwx------ 1 root root           0 2012-01-10 18:22:26  /@vsphere/79-squeeze.esx
...
104857600 2012-01-10 18:22:30
2000 OK estimate files=10 bytes=2,462,069,574

Run the backup Job, It is possible to verify VMs snapshot messages in Bacula log.

Situational Configurations and Troubleshooting

Incremental/Differential Backup Backup Problems

To check if a virtual disk has CBT enabled, open the vSphere Client, select a powered-off virtual machine without snapshots.

  • Right-click the virtual machine and click Edit Settings.
  • Click the Options tab.
  • Click General under the Advanced section and then click Configuration Parameters. The Configuration Parameters dialog opens.
  • Click Add Row.
  • Add the ctkEnabled parameter and then set its value to true.
  • Click Add Row, add scsi0:0.ctkEnabled,and set its value to true.

Note: scsi0:0 in scsi0:0.ctkEnabled indicates the SCSI device assigned to the hard disk that is added to the virtual machine. Every hard disk added to the virtual machine is given an SCSI device that appears similar to scsi0:0, scsi0:1, or scsi1:1.

During the first Full backup, the vSphere Plugin will try to enable CBT automatically if the virtual machine is powered off. To disable this feature, use the keep_cbt option in the Plugin command string.

Plugin = "vsphere: keep_cbt host=guest1"

No snapshots must exist on a virtual machine at the time of enabling CBT on it. For details, refer to http://kb.vmware.com/kb/1033816.

Using SAN Access Backup (LANFree)

To use the SAN transport method, your backup server where the vsphere plugin is installed should have access to all LUNs exported to your ESX server.

Once your disks are visible as /dev/sda, /dev/sdb, … The vSphere plugin will open each disk to get the UUID and compare it with what the ESX server is providing.

When mapping a LUN to the Bacula Server, you need to scan the new disks:

for i in /sys/class/scsi_host/host*/scan ; do echo - - - > $i ; done

When using iSCSI, the lsscsi command will display them as:

% lsscsi 
[5:0:0:0]     disk     IET    VIRTUAL-DISK    0    /dev/sdb

You can verify that SAN transport is used during backup by using the debug option in the Plugin command line and verifying the vddk trace file located in:

% grep san: /opt/bacula/working/vsphere/<moref>/<seq>/0.log
DISKLIB-LIB   : Opened "san://3-snapshot-18[datastore2] test/test_2.vmdk...

You can check the transport method used in the Bacula backup log.

Debug Traces

According to Table 3, the following trace files provide information, depending on the Bacula vSphere plugin error.

File Component Note
/opt/bacula/*/vsphere-ctl.log vsphere-ctl This file is produced by the Java vsphere-ctl program that sends commands to the ESXi/vCenter
server.
/opt/bacula/working/vsphere/*/*/*.log vddk This file is produced by the C++ vddk program that read/write to the VMDK.
/opt/bacula/working/*.trace bacula-fd This file is produced Bacula File Daemon when activating the debug.

Table 3. vSphere Plugin Traces

Excluding Specific VM Disks from Backup

To exclude a specific disk, you can activate the independent mode in the vSphere console, or using the disk_exclude option in the VM backup FileSet.

Restore

Restoring to a New Virtual Machine

Using bweb (for example), web restores, choose a vSphere terminated backup job to restore. On file selection, mark one or more VMs files directories that you want to restore.

If you run the restore using the where=/ Bacula restore option, the vSphere Plugin will try to restore your disks on a new virtual machine created during the restore with the same attributes (disks, controller, CPU type, . . . ).

If a machine with the same name exists, another one will be created with a modified name, although this behavior can be changed in future plugin versions to replace the VM if the Bacula Replace=always restore option is set.

The ESX host and the datastore that will be used to restore your guest will be detected automatically, however, you can change it during the restore Advanced Options or configure default destinations in vsphere_global.conf (see default_restore_host and default_datastore).

Note that you need to have at least one VM configured on your ESX server to restore a VM from Bacula automatically. We plan to remove this limitation in a future version.

Bacula will not recreate all network settings during the restore, you will need to check the network configuration.

Restoring Machines to a Local Disk

By using where=/path/to/dir in the restore option, the Plugin will automatically restore selected files to this location. Then, it is possible to import machines to vCenter manually or, if desired, you may mount the image locally using VMWare vmware-mount tool or qemu-nbd and perform file level restores. These operations are automated and bweb graphically performed with Bacula Single Item Restore Plugin.

It is also possible to copy the raw image to any devices or to mount it and restore files directly.

modprobe nbd max_part=16
qemu-nbd -c /dev/nbd0 /tmp/0.bvmdk
partprobe /dev/nbd0
fdisk -l /dev/nbd0
mount /dev/nbd0p1 /mnt/image # example
ls /mnt/image

Troubleshooting

If you experience restoration problems such as importing OVF, such as:

# Bacula Job log:
...
2018-10-01 22:02:24 pocbacula-fd JobId 148: Error: No default OVF defined, aborting...
2018-10-01 22:02:24 pocbacula-fd JobId 148: Error: Unable to restore the host. null
2018-10-01 22:02:24 pocbacula-fd JobId 148: Warning: Unable to create VM, logs are in /opt/bacula/working/vsphere-ctl-0.log
2018-10-01 22:02:24 pocbacula-fd JobId 148: Error: Import of the OVF structure probably failed...
...

# and

cat /opt/bacula/working/vsphere-ctl-0.log
..
2018-10-02 12:19:28 WARN  GlobalManager:446 - W: Line 1: Unable to parse 'backing.exclusive' for attribute 'key' on element 'Config'.
2018-10-02 12:19:28 WARN  GlobalManager:485 - The operation is not supported on the object.
2018-10-02 12:19:28 WARN  GlobalManager:487 - Could not obtain nfc lease.
..

Make sure your VSphere user can create machines and move them to the ESXi and the required datastores through the VCenter console.

Note: backing up VSphere templates is not allowed by the VMware API.

The VCenter log (/var/log/vmware/vpxd/vpxd.log) can provide useful information for problems in backing up VMs. Similarly, using the debug keyword in the FileSet Plugin line (vsphere: debug …) will create a debug log on the machine running the Plugin at:

/opt/bacula/working/vsphere/vm-xxx/xxx/0.log

Replace xxx with VM ID and incremental sequence of active debug backups. It is possible to increase debug level and even tuning VSphere backup by creating a /opt/bacula/etc/vixdisklib.conf file. Example:

vixDiskLib.nfc.LogLevel=7
vixDiskLib.san.LogLevel=7
vixDiskLib.transport.LogLevel=7
tmpDirectory=/tmp
logDirectory=/tmp

Vixdisklib options are available at: https://pubs.vmware.com/vsphere-51/index.jsp?topic=%2Fcom.vmware.vddk.pg.doc%2FvddkFunctions.6.11.html

Reference

VMware Backup Using Bacula Enterprise Edition – https://www.baculasystems.com

 

Disponível em: pt-brPortuguês (Portuguese (Brazil))enEnglishesEspañol (Spanish)

Leave a Reply