Enterprise Bacula S3 and CEPH Cloud Storage Driver Quick Guide

This quick guide is intended to provide information on the processes required to implement stored backups of S3, CEPH and other cloud service providers using the Bacula Community or Enterprise Cloud Driver.

Amazon Web Services (AWS), Wasabi, Huawei Cloud and BackBlaze are some of the tested object storage providers, but most S3 or CEPH technologies are compatible with Bacula Cloud Driver. Oracle Cloud, Azure, Google Cloud, Swift (etc.) have specific Bacula Enterprise exclusive Drivers.

The Driver provides one more type of Bacula storage device, allowing the Bacula administrator to manage local disks, tapes and now multiple backups stored on cloud devices using any desired architecture or distribution.

Bacula is all about software. The object storage service should be hired from your favorite service provider.

Cloud Transmission and Cache

The Bacula Cloud Driver transmission to the cloud is only limited by, backup clients disk read, local and to the cloud network capabilities.

In order to avoid Internet connection bottlenecks, the Driver supports advanced local cache and data splitting techniques. The cache may be kept for faster local restores or automatically discarded (truncated), according to the Bacula Administrator will and available local disk storage.

These are the new bacula-sd.conf Cloud resource Directives:

TruncateCache=AfterUpload # Each cache Volume part will be removed just after it is uploaded. Note, if this option is specified, all restores will require a download from the Cloud.
# TruncateCache=AtEndOfJob # At the end of the Job, every part that has been successfully
uploaded to the Cloud will be removed (truncated). Note, if this option
is specified, all restores will require a download from the Cloud.
# TruncateCache=No # In this case, the administrator must specify or manually truncate or by a console command script admin (e.g. cloud truncate).

Upload=EachPart # Uploads a part (volume subdivision) when it reaches its define maximum size, even if during a Job execution. 
# Upload=AfterJob # Upload the volume parts only when the backup job is terminated. 
# Upload=No # Do not upload automatically. The administrator must perform the upload manually or by a command console do script admin (e.g. cloud upload allpools allfrompool storage=<storage_name>).

It is also possible to limit transmission bandwidth.

Installation

The Driver is installed on the same machine that hosts the Bacula Storage Daemon. To install both, add a new repository to your RedHat/CentOS distribution, replacing the @customer-string@ to your exclusive customer code, and @version@ to the desired Bacula Enterprise version:

cat /etc/yum.repos.d/bacula.repo

[Bacula]
name=Bacula Enterprise Edition
baseurl=https://www.baculasystems.com/dl/@customer-string@/rpms/bin/@version@/rhel7-64/
enabled=1
protect=0
gpgcheck=0

[BEECloudPlugin]
name=BEE Cloud Plugin
baseurl=https://www.baculasystems.com/dl/@customer-string@/rpms/cloud/@version@/rhel7-64/
enabled=1
protect=0
gpgcheck=0

The Cloud Driver is also available for Debian. E.g.:

cat /etc/apt/sources.list.d/bacula.list

#Bacula Enterprise
deb https://www.baculasystems.com/dl/@customer-string@/debs/bin/@version@/jessie-64/ jessie main
deb https://www.baculasystems.com/dl/@customer-string@/debs/cloud/@version@/jessie-64/ jessie cloud

Use the package manager to update and install the Driver. E.g.:

yum install bacula-enterprise-cloud-storage

# or

apt-get update && apt-get install bacula-enterprise-cloud-storage

Configuration

As shown in Figure 1, create a new Cloud connection configuration. It is possible to use bweb to perform that.

Enterprise Bacula S3 and CEPH Cloud Storage Driver Quick Guide 1

Figure 1. New Cloud Connection Storage Daemon Configuration

Alternative text AWS and CEPH Cloud Service Providers configurations examples are listed as follows:

Cloud {
  Name = AWS
  Driver = "S3"
  HostName = "s3.amazonaws.com"
  BucketName = "your_bucket"
  AccessKey = "xyz"
  SecretKey = "abc"
  Protocol = HTTPS
  UriStyle = VirtualHost
  Truncate Cache = AfterUpload
  Upload = EachPart
  Region = "us-east-1" 
  MaximumConcurrentUploads=20
  MaximumConcurrentDownloads=20
  # MaximumUploadBandwidth = 5MB/s # Optional
}
Cloud {
  Name = CEPH_S3
  Driver = "S3"
  HostName = ceph.mydomain.lan
  BucketName = "CEPHBucket"
  AccessKey = "xxxXXXxxxx"
  SecretKey = "xxheeg7iTe0Gaexee7aedie4aWohfuewohxx0"
  Protocol = HTTPS
  Upload = EachPart
  Truncate Cache = AfterUpload
  UriStyle = Path  # Must be set for CEPH
  MaximumConcurrentUploads=20
  MaximumConcurrentDownloads=20
}

After the Cloud configuration, as viewed in Figure 2, add a Bacula Storage Daemon Device that will write to the configured bucket. It should have the Cloud DeviceType, and use the same Cloud configuration previously created. ArchiveDevice is the local disk cache path, typically with a large available space. MediaType should be a unique name in relation to all other existent Bacula Devices, and MaximumPartSize is the size of split data that Bacula will transfer to the bucket.

Enterprise Bacula S3 and CEPH Cloud Storage Driver Quick Guide 2

Figure 2. New Cloud Storage Daemon Device

Here is an text configuration example of a single Storage Daemon Device (note that the Cloud plugin is also compatible with the Autochanger, which are Bacula Device groups):

Device {
  Name = MyCloudAWS
  Device Type = Cloud
  Cloud = AWS
  Archive Device = /mnt/backups # local volume cache
  Maximum Part Size = 500 MB # important: it is the object upload granularity
  Media Type = Cloud01
  LabelMedia = yes
  Random Access = Yes;
  AutomaticMount = yes
  RemovableMedia = no
  AlwaysOpen = no
}

Finally, as exhibited in Figure 3, it is time to attach the new Storage Daemon Device to your Bacula Director. Create a new Director Storage. The Address and Password are usually the same already used by the same Storage Daemon devices. The DeviceName and MediaType must match the values defined in the recently created Storage Daemon Device configuration.

Enterprise Bacula S3 and CEPH Cloud Storage Driver Quick Guide 3

Figure 3. Bacula Director New Storage Addition

It is advisable to create one or more new Pools to host the Cloud backups, using the newly created Storage. A new CacheRetention Pool directive time value (x days, months etc.) be specified, in order to expire cached volumes.

Modify Bacula Jobs and Schedules to use the new Cloud Pools, at will.

Apply changes and run a test backup job.

Test and Troubleshooting

The S3 CLI can be used to test AWS and other Cloud Storage connections. It is possible to install it using pip:

yum -y install python-pip
pip install awscli

The following command might be used to set up the bucket connection:

aws configure

Bucket creation and a list can be performed as follows (e.g.):

aws s3api --endpoint-url='https://s3.amazonaws.com' 
create-bucket --bucket bsyssdsync
aws s3 ls

More commands, details and general information about the Bacula Enterprise Cloud Driver can be found in the referenced whitepaper.

Reference

Bacula Cloud Backup – Bacula Enterprise Edition. http://baculasystems.com

 

Disponível em: pt-brPortuguês (Portuguese (Brazil))enEnglishesEspañol (Spanish)

Leave a Reply