Bacula Enterprise NDMP Plugin - Quick Guide - Bacula Latin America & Brazil

Overview

This Quick Guide is intended to provide information about the considerations and processes required to implement a backup strategy using the NDMP protocol with Enterprise Bacula backup software. It introduces several techniques and strategies for backing up NAS filers using the NDMP or NFS/ CIFS protocols with Bacula Enterprise Edition, including granular single file restoration and NAS snapshot capabilities [1].

There are a number of NAS Lines approved for use with the NDMP Plugin, among others:

EMC Celerra
Hitachi BlueArc
NetApp
Huawei OceanStor
EMC Isilon
EMC Unity

The current version of the NDMP plugin also supports DataONTAP version 7.3, 8.0, 8.1, 8.2 and several other NetApp vendors.

No NDMP Plugin Backup

According to Figure 1, there are several protocols that can be used to back up NAS equipment. NDMP is generally better, because using NFS or CIFS to present files for backup is very slow, especially when dealing with millions of small files. Directory scanning and network latency tend to dramatically decrease backup throughput. Using the NFS or CIFS protocols to perform a complete restoration on a large NAS can be very time-consuming, especially in an urgent disaster recovery situation.

Figure 1. Backup diagram without NDMP plugin

Backing up with the Bacula NDMP Plugin

NDMP has the main advantage of speed over NFS / CIFS protocols for backing up NAS volumes. The most important thing to understand about NDMP is that it is a control protocol, not a backup format. The backup format of the NDMP is arbitrary (implementation-specific) and individually defined by each storage platform vendor. Thus, in general, an NDMP backup made to a vendor’s NAS can not be restored to a different vendor’s NAS can.

The Bacula NDMP plug-in triggers snapshots on storage devices and generates backups within Bacula’s standard backup data stream, allowing it to be multiplexed with other backup streams and all advanced features. Within the saved data stream that contains this NDMP stream, the actual NDMP data remains opaque (vendor-specific, sometimes proprietary format). Attributes such as ACLs are included in the backup stream and handled correctly by the NAS vendor for backup and restore.

It is possible to perform a single file restore using the single item restore technique (use_hist option). Restoring a single file with the NDMP plug-in is also possible by restoring the NDMP data stream on a local machine (the same as Backup with NDMP has the NDMP plug-in) and opening the resulting file. This requires that you know the data format of the vendor and have a program that can read it. As it involves restoring the entire backup and extracting the desired files, the latter is not very efficient.

The plug-in supports the NetApp SMTAPE backup option.

Granular File Level Recovery

On NDMP systems that support DAR, the use_hist plugin option can be used to enable the “Single Item Restore” feature. With the use_hist option, all filenames included in a dump are stored in the Bacula catalog, and a restore session will have the ability to choose individual files or directories to be restored. Only selected files are restored to the target NDMP system. The correct dump file is selected automatically during the file selection process.

Plugin Installation

The Bacula Enterprise Edition NDMP plug-in consists of three files that must be installed with a File Daemon running on the same server that is running a Bacula Storage daemon that will store your data. This avoids sending backup data over the LAN from the file daemon to the storage daemon. You can install the NDMP plug-in on any other host, but doing so is less efficient because it will cause data to travel over two network links – once from the NAS to the File Daemon and another time from the File Daemon to Storage Daemon.

Binary packages are available for a limited number of platforms supported by Bacula Enterprise Edition. Please contact us to access them.

rpm -ivh bacula-enterprise-ndmp_8.10.x.rpm
# or
dpkg -i bacula-enterprise-ndmp_8.10.x.deb

These packages will ensure that your version of Bacula Enterprise Edition is compatible with the NDMP plug-in and will install the bndmp and ndmp-fd programs.

/opt/bacula/plugins/ndmp-fd.so
/opt/bacula/bin/bndmp
/opt/bacula/bin/idx_dump.pl

Configuration

Your File Daemon must have direct network access to the NAS, you can test the communication using telnet.

# telnet nasbox 10000
Connected to nasbox.
Escape character is '^]'.

Note that NDMP, when doing the actual backup, connects back to the Bacula server. This may require that firewalls, including firewalld and iptables, be configured accordingly. Using the data_port_range option, you can control which TCP ports the NAS will try to use, allowing for a more restricted firewall configuration. When the network connection is not possible, the error NDMP4_CONNECT_ERR can be issued.

Depending on your configuration, you can set the authentication method to md5 or text.

The possible plug-in configurations are listed as follows.

Option	Required	Default	Info	Example
host	Yes		Hostname NAS	host=192.168.0.50
user	Yes		Username	user=root
pass	Yes		Password	pass=password
port	No	10000	Porta Conexão	port=10000
auth	No	md5	md5, text, none	auth=text
profile	No		nome perfil	profile=root
volume	Yes		Volume para fazer backup	volume=/vol/vol0
volume_format	No		Formato de volume específico	volume_vormat=/vol/
file	No	/	Diretório para fazer backup	file=/home
type	No	dump	dump, tar, SMTAPE, config	type=dump
data_port_range	No		Intervalo de portas backup	data_port_range=2000-3000
abort_on_error	No		Abortar job havendo erro	abort_on_error
debug	No		Habilitar depuração	debug
use_hist	No		Armazenar informação HIST no catálogo	use_hist
hist_retention	No		Podar informação HIST após determinado tempo	hist_retention=30days

Custom Volume Format

The NDMP plug-in must be aware of the structure organization to detect whether the administrator wants to restore a new volume (where = / dev / vol_tmp) or within a subdirectory of the target volume (where = / tmp).

The NDMP Plugin will automatically detect the following structure:

/vol/ (used in NetApp)
/root_vdm/ (used in some EMCs)

If your volumes are using a different naming scheme, you must use the volume_format option.

/dev/volume_home -> volume_format=/dev/
/rootvolume/volume_tmp -> volume_format=/rootvolume/
/VG/volume_var -> volume_format=/VG/
FileSet {
  Name = NDMPFS
  ...
    Include {
      Plugin = "ndmp:host=nasbox user=root pass=root file=/dev/vol1 volume_format=/dev/"
    }
}

For example, at NetApp, the volume format is:

/vol/vol1
/vol/vol2
...

The common part for distinguishing volumes and directories is /vol/. On Solaris and ZFS, volumes are stored in /dev/.

/dev/Volume01
/dev/Volume02
/dev/Volume03

If your volumes are organized as in this example:

/FS1/
/FS2/
/FS3/

Volume_format must be configured with /FS – note that there is no final slash here!

As shown in Figure 2, if you have Bweb’s EBacula GUI, you can graphically configure the NDMP FileSet plug-in, and later a Job.

Figure 2. BWeb NDMP Plug-in configuration

FileSet Examples

Back up everything in /vol/vol1 and/vol/vol2:

FileSet {
Name = NDMP
  Include {
    Plugin = "ndmp:host=nasbox user=root pass=root volume=/vol/vol1"
    Plugin = "ndmp:host=nasbox user=root pass=root volume=/vol/vol2"
  }
}

Backup of /home on volume /vol/vol2:

FileSet {
  Name = NDMP_home
    Include {
      Plugin="ndmp:host=nas user=root pass=root volume=/vol/vol2 file=/home"
    }
}

Backup of volume /vol/vol2 using SMTAPE:

FileSet {
  Name = NDMP_home
    Include {
      Plugin="ndmp:host=nas user=root pass=root volume=/vol/vol2 type=SMTAPE"
    }
}

Backup of a specific snapshot to /vol/vol2

FileSet {
  Name = NDMP_snapshot
    Include {
      Plugin="ndmp:host=nas user=root pass=root volume=/vol/vol2/.snapshot/snap1"
  }
}

When using NDMP as opposed to volumes mounted by NFS/CIFS, the NDMP NAS software is responsible for backing up all information pertaining to files, including ACLs.

Note: the NDMP plug-in does not support FileSet’s “sparse” option.

Using ndmp.conf to store credentials

If you do not want to expose the NAS password on the Plug-in’s command line, you can use an ndmp.conf file stored in the File Daemon, which will contain the credential information.
In the following example, the “root” profile will refer to the NAS host “nasbox”, the connection will use MD5 as the authentication method (“/ m”), the username will be “root” and the password will be “password”.

# cat /opt/bacula/etc/ndmp.conf
[--root]
-D nasbox/m,root,password

The format of the -D parameter is as follows:

HOST[:PORT][/FLAGS][,USERNAME,PASSWORD]

Where:

HOST: It is the host name or IP address of the NAS
: PORT: optional port number. If not provided, the port number is 10000.
USERNAME: a username that will be recognized by the NAS. Whether this is a general user name or a special NAS account depends on the implementation.
PASSWORD: The password corresponding to the username. The password field must not contain special characters, such as: “,”,!, $, / Or #
/ FLAGS: Optional flags to indicate the desired NDMP version or authentication method.
The standard version is negotiated to be as high as possible. The default authentication method is text (NDMP_AUTH_TEXT).
2: Use NDMP version 2.
3: Use NDMP version 3.
4: Use NDMP version 4.
n: does not use authentication (NDMP_AUTH_NONE).
t: Use text authentication (NDMP_AUTH_TEXT). The username and password are sent over the network as clear text (no encryption).
m: Use MD5 challenge/response authentication (NDMP_AUTH_MD5).
The remote NAS is asked for a challenge. The password is used as a shared secret and is never sent over the network.

In the following example, the profile “11” will refer to the NAS host “10.1.1.11”, the connection will use MD5 as an authentication method (“/m”), the username will be “root” and the password will be “My password”. A second profile “12” is also shown in this sample file ndmp.conf.

[root@lxbackup ~]# cat /opt/bacula/etc/ndmp.conf
[--11]
-D 10.1.1.11/m,root,mypassword
[--12]
-D 10.1.1.12/m,root,mypassword2

In the following FileSet example, two backups of NDMP plug-ins are configured – one for each of the two NAS profiles configured in the ndmp.conf example above.

Fileset {
  Name = NDMP_FS
    Include {
      Plugin = "ndmp: host=10.1.1.12 profile=12 volume=/vol/DMZ_backup type=smtape"
      Plugin = "ndmp: host=10.1.1.11 profile=11 volume=/vol/LAN_backup type=smtape"
    }
}

Schedule Considerations

The native dump format has a serious limitation of eight for the number of incremental jobs you can perform between two full or differential backups. Doing more than eight Incrementals will not allow you to restore your system accurately.

All files will be restored correctly, but subsequent restores from incremental backups will not recreate the file system as it was during the final incremental backup.

NAS Volume Configuration and Testing

When specifying type = config as an argument of the NDMP plugin, Bacula will back up an overview of the configuration of all NAS volumes to the /@ndmp/.config file.

To view the configuration, simply select the file using the restore command, perform the restore and Bacula will display the NAS volume configuration in the job log, as shown in the example.

The main NetApp configuration file is located by default in /vol/vol0/etc. So, if you want to restore the NAS from scratch, you must include the /vol/vol0/etc directory in your backup FileSet.

The type = config option can also be used to check the network configuration and connection information between the Bacula Enterprise Edition NDMP plug-in and the NAS. If this backup job is not successful, check your configuration or password.

Plugin = "ndmp:host=nasbox user=root pass=pass type=config"

A sample of the NAS configuration output is displayed as follows.

...
JobId 3: QR File system /vol/vol2
JobId 3: QR physdev
JobId 3: QR unsupported 0x0
JobId 3: QR type WAFL
JobId 3: QR status online
JobId 3: QR space 805306 total, 118784 used, 8052948 avail
JobId 3: QR inodes 288238 total, 100 used
JobId 3: QR empty default env
JobId 3: QR
JobId 3: QR File system /vol/vol1
JobId 3: QR physdev
JobId 3: QR unsupported 0x0
JobId 3: QR type WAFL
JobId 3: QR status online
JobId 3: QR space 8589938 total, 6594560 used, 858334 avail
JobId 3: QR inodes 307450 total, 1703 used
JobId 3: QR empty default env
JobId 3: QR
...

Restore Using NDMP

To restore backup files with the NDMP protocol, you must select all files present in the /@ndmp/ virtual directory. You can change the where parameter to restore your files to an alternate location. However, the RegexWhere option is not available with this method.

The where parameter can contain a directory within the original volume or a directory within an alternate volume. The volume_format option allows you to configure how the NDMP Plug-in will distinguish between volumes and directories.

In the following example, if volume_format is /vol/, the NDMP stream will be restored to the /tmp directory on the “newvolume” volume.

* restore where=/vol/newvolume/tmp

In the following example, the NDMP stream will be restored to the original volume in the /tmp directory.

* restore where=/tmp

Restoring files without the NDMP host

In some cases, you can extract files generated by the NDMP host, first by restoring the NAS backup on your NDMP plug-in host machine and then using restore or tar. For this purpose, you can use the where = option during restoration with the following format:

* restore where=">/tmp"

The NDMP files would be extracted to the local /tmp/directory. (Note that the quotes in the example above should only be used when using the command line argument, they are not needed when editing options via the restore menu).

Restoring with NetApp SMTAPE

You can back up and restore entire volumes with the SMTAPE option using the path to the root of the volume (for example, / vol / foo). It is like backing up and restoring a hard disk partition using dd or partimage. Bacula does not need to know anything about the underlying structure of SMTAPE data, because when the volume is brought back online after the restoration, the archiver simply recognizes it as a WAFL file system and happily walks by.

To restore a NetApp volume with the SMTAPE option, you need to take the volume offline with the vol command on the NetApp host.

netapp> vol offline /vol/vol1
Volume "vol1" is now offline.

Then you need to specify where=/vol/vol1 in the restore command.

Attention, with this method, any data in the volume would be overwritten by the data saved previously.

Backup Considerations Using NFS or CIFS as an Alternative

If you are unable to use the EBacula NDMP plug-in, some alternative considerations for NFS and CIFS mount backups are designed as follows.

Using NFS ACLs

In order to back up the file’s Access Control Lists (ACLs), you must enable ACL support in the mount command as follows:

# mount -t nfs -o soft,intr,proto=tcp,acl nas:/vol/vol1 /nas/vol/vol1

You also need to add the ACL Support option to your FileSet:

FileSet {
  Name = FS_NAS_VOL1
    Options {
      ACL Support = yes
    ...
    }
    File = /nas/vol/vol1
}

Using CIFS ACLs

If you mount a CIFS volume on Linux, you cannot back up ACLs. To back up ACLs on CIFS volumes, you must mount the volume on a Windows server on which the Bacula File Daemon is installed.

References

[1] Backup with NDMP Using Bacula Enterprise Edition Whitepaper. http://baculasystems.com

Disponível em: Português (Portuguese (Brazil))EnglishEspañol (Spanish)