*Updated with rclone-changer fork compatible with Python 3.5 and higher to avoid failing to create volumes with the label barcodes.

This tutorial aims to deploy a working Bacula Storage Device to the cloud, using rclone  (https://rclone.org/) and the rclone-changer script (https://github.com/wanderleihuttel/rclone-changer). The last one presents cloud stored virtual tapes as a sort of a VTL to Bacula. Amazon S3 was used for this and we have notice of working Oracle Cloud and Google Drive deploys. It should work with all the rclone supported advertised cloud storages: Amazon Drive, Amazon S3, Backblaze B2, Box, Ceph, Dreamhost, Dropbox, FTP Google Cloud Storage, Google Drive, HTTP, Hubic, Memset Memstore, Microsoft Azure Blob Storage, Microsoft OneDrive, Minio, OVH, Openstack Swift, Oracle Cloud Storage, QingStor, Rackspace Cloud Files, SFTP, Wasabi, Yandex Disk and the local filesystem.

Rclone features:

  • MD5/SHA1 hashes checked at all times for file integrity
  • Timestamps preserved on files
  • Partial syncs supported on a whole file basis
  • Copy mode to just copy new/changed files
  • Sync (one way) mode to make a directory identical
  • Check mode to check for file hash equality
  • Can sync to and from network, eg two different cloud accounts
  • Optional encryption (Crypt)
  • Optional FUSE mount (rclone mount)

Rclone Installation and Configuration

As root, just run the following commands:

yum -y install unzip wget
wget -qO- https://downloads.rclone.org/rclone-current-linux-amd64.zip -O /tmp/tmp.zip; unzip /tmp/tmp.zip -d /usr/src; rm -f /tmp/tmp.zip
cd /usr/src/rclone-v*
cp rclone /usr/bin/
chown root:root /usr/bin/rclone
chmod 755 /usr/bin/rclone
mkdir -p /usr/local/share/man/man1
cp rclone.1 /usr/local/share/man/man1/
mandb

Now it is time to access your cloud storage and fetch programmatic access credentials.

Since using Amazon S3, I must access the IAM identification service, create and user, grant rights to the S3 service and generate a Access Key ID and a Secret key. Copy them to a notepad since we are needing this to set up rclone. These steps are shown in the following screenshots:

Bacula Storage in Any Cloud with Rclone and Rclone-changer 1 Bacula Storage in Any Cloud with Rclone and Rclone-changer 2 Bacula Storage in Any Cloud with Rclone and Rclone-changer 3

In order to generate rclone configuration, use the following command as root:

rclone config

Follow the detailed instructions of rclone documentation in order to set up your favorite cloud storage: https://rclone.org/docs/

After providing a name for this access and answering all other further questions, rclone config will save the configuration at: /root/.config/rclone/rclone.conf

In my example, aws was the name of my remote configuration and this was the output:

[aws]
env_auth = true
access_key_id = AKIAIFTEYM3JK5VUXGBA
secret_access_key = hjqF6wG/tNBqgpBbwBmQ3EQpok5U33W2ewSXD
region = sa-east-1 
endpoint = 
location_constraint = 
acl = private
server_side_encryption = 
storage_class =

It is very important to access rclone access to the cloud. Run some test commands as follows:

# Fetch bucket list:
rclone lsd aws:
root@hfaria-asus-i5:/# rclone lsd aws:
          -1 2014-10-01 20:27:41        -1 baculaheitor

# If doesn't exist, create a new bucket:
rclone mkdir aws:new_bucket_name

# List bucket contents:
rclone ls aws:baculaheitor

  6311 logs/2014-10-02-01-17-25-7F0DBBBFD43A20B7
     2721 logs/2014-10-02-01-17-31-BFC439050C2B0AA9
      666 logs/2014-10-02-02-17-36-D47447EC8F6759EE
     2079 logs/2014-10-02-02-17-47-24DD21E3B85C5CAF
    40108 logs/2014-10-02-03-17-10-4E9B7A829CEC7842

# Test diectory copy to the bucket:
rclone copy /etc aws:baculaheitor

# Test directory copy from the bucket:
rclone copy aws:baculaheitor /tmp

Finally, copy the work configuration version to the path that rclone-changer expects, as follows:

cp ~/.config/rclone/rclone.conf /opt/bacula/etc/rclone.conf
chown bacula /opt/bacula/etc/rclone.conf

It is time to deploy rclone-changer.

Rclone-changer installation and configuration

Install the python and python-yaml dependencies:

apt install -y python python-yaml
# or 
yum install -y python python-yaml

Download rclone and create needed folders:

wget https://github.com/wanderleihuttel/rclone-changer/archive/master.zip -O /tmp/tmp.zip; unzip /tmp/tmp.zip -d /usr/src; rm -f /tmp/tmp.zip
cp /usr/src/rclone-changer-master/rclone-changer /usr/sbin/
chmod +rx /usr/sbin/rclone-changer
chown bacula /opt/bacula/working

Then, create the folder and file that will host the virtual tape recording. Also create a directory for Bacula local disk spooling, since advisable for faster backups and safer copy to the cloud, as follows:

mkdir /mnt/vtapes && touch /mnt/vtapes/tape && chown -R bacula /mnt/vtapes/
mkdir /mnt/bacula-spool && chown bacula /mnt/bacula-spool

Add a similar device to the Bacula Storage Daemon Configuration file (bacula-sd.conf):

Autochanger {
  Name = "rclone_aws"
  Device = aws
  Changer Device = 'aws:baculaheitor' # remote rclone configuration name and bucket. You can also specify a cloud directory
  Changer Command = "/usr/sbin/rclone-changer %c %o %S %a" # don't change this
}

Device {
  Name = aws
  Media Type = aws
  Maximum Changer Wait = 18000 # this should be increase according to the size of the volumes 
  Archive Device = /mnt/vtapes/tape # must match the file created on the prior step
  Autochanger = yes
  LabelMedia = yes;                  
  Random Access = Yes
  AutomaticMount = no             
  RemovableMedia = no
  AlwaysOpen = no
  Spool Directory = /mnt/bacula-spool
  Maximum Spool Size = 524288000
}

Now, tie the new Storage Daemon device to your Director and also create a Pool for it:

Autochanger {
  Name = Cloud_aws
  Address = sd_address                
  SDPort = 9103
  Password = "EKPNGPoIe7wXO/oe6xwm9cXzHDgmiuS0qF3hyAhpEg+i"
  Device = rclone_aws
  Media Type = aws
  Autochanger = Cloud_aws            
  Maximum Concurrent Jobs = 10       
}

Pool {
  Name = Offsite
  Pool Type = Backup
  Recycle = yes                       
  AutoPrune = yes                    
  Storage = Cloud_aws
  Maximum Volume Bytes = 1073741824 # recommended
  AutoPrune = yes
  Volume Retention = 4 weeks
}

It is important to have developer considerations in mind if changing Maximum Volume Bytes:

It is recommended that you use many smallish volumes rather than large ones. Try to stay at a size that your average file fits into without getting too tiny. The reasoning here is in a recovery situation if you’re working on random files you want to minimize the amount of unrelated data you need to copy localy in order to recover. If you have a 10G virtual tape and only need 500M of it, you still need to wait for the full 10G to download before you can beging recovering data. Hence, if your typical file size is averaging 700M or so, 1G volumes are probably a good idea. [https://github.com/travisgroth/rclone-changer]

Edit and verify that the python version and the rclone-changer directories are correct and the Linux bacula user is allowed to write to them. You can also change the number of slots your virtual robot will have.

vi /usr/sbin/rclone-changer

#!/usr/bin/env python
...
    #=====================================================
    # Default params config
    #=====================================================
    rclone     = '/usr/bin/rclone'
    config     = '/opt/bacula/etc/rclone.conf'
    logFile    = '/opt/bacula/working/rclone.log'
    logChanger = '/opt/bacula/working/rclone-changer.log'
    lockFile   = '/opt/bacula/working/.rclonelock'
    stateFile  = '/opt/bacula/working/rclone-changer.state'
    slots      = 100
    #=====================================================

Then, create virtual tape Bacula labels. Use bconsole for that:

label barcodes storage=Cloud_aws pool=Offsite

If have trouble creating tables, check all the prior commands and permission grants. In order to reset the VTL and try again delete the vtape and the rclone-changer state file:

rm -f /mnt/vtapes/tape*
rm -f /opt/bacula/working/rclone-changer.state
touch /mnt/vtapes/tape && chown -R bacula /mnt/vtapes/

All done! Run a test backup job to the Offsite pool and create Schedule routines to the cloud, according to your needs. Virtual tapes will be changed  and uploaded to the cloud automagically, as in the following screenshots:

Bacula Storage in Any Cloud with Rclone and Rclone-changer 4 Bacula Storage in Any Cloud with Rclone and Rclone-changer 5

Rclone-changer Troubleshooting

These are the places where rclone and rclone-changer saves their log messages:

cat /opt/bacula/working/rclone.log
cat /opt/bacula/working/rclone-changer.log

 

Disponível em: pt-brPortuguês (Portuguese (Brazil)) enEnglish esEspañol (Spanish)

Close Menu