*Updated with rclone-changer fork compatible with Python 3.5 and higher to avoid failing to create volumes with the label barcodes.
This tutorial aims to deploy a working Bacula Storage Device to the cloud, using rclone (https://rclone.org/) and the rclone-changer script (https://github.com/wanderleihuttel/rclone-changer). The last one presents cloud stored virtual tapes as a sort of a VTL to Bacula. Amazon S3 was used for this and we have notice of working Oracle Cloud and Google Drive deploys. It should work with all the rclone supported advertised cloud storages: Amazon Drive, Amazon S3, Backblaze B2, Box, Ceph, Dreamhost, Dropbox, FTP Google Cloud Storage, Google Drive, HTTP, Hubic, Memset Memstore, Microsoft Azure Blob Storage, Microsoft OneDrive, Minio, OVH, Openstack Swift, Oracle Cloud Storage, QingStor, Rackspace Cloud Files, SFTP, Wasabi, Yandex Disk and the local filesystem.
Rclone features:
- MD5/SHA1 hashes checked at all times for file integrity
- Timestamps preserved on files
- Partial syncs supported on a whole file basis
- Copy mode to just copy new/changed files
- Sync (one way) mode to make a directory identical
- Check mode to check for file hash equality
- Can sync to and from network, eg two different cloud accounts
- Optional encryption (Crypt)
- Optional FUSE mount (rclone mount)
Rclone Installation and Configuration
As root, just run the following commands:
yum -y install unzip wget wget -qO- https://downloads.rclone.org/rclone-current-linux-amd64.zip -O /tmp/tmp.zip; unzip /tmp/tmp.zip -d /usr/src; rm -f /tmp/tmp.zip cd /usr/src/rclone-v* cp rclone /usr/bin/ chown root:root /usr/bin/rclone chmod 755 /usr/bin/rclone mkdir -p /usr/local/share/man/man1 cp rclone.1 /usr/local/share/man/man1/ mandb
Now it is time to access your cloud storage and fetch programmatic access credentials.
Since using Amazon S3, I must access the IAM identification service, create and user, grant rights to the S3 service and generate a Access Key ID and a Secret key. Copy them to a notepad since we are needing this to set up rclone. These steps are shown in the following screenshots:
In order to generate rclone configuration, use the following command as root:
rclone config
Follow the detailed instructions of rclone documentation in order to set up your favorite cloud storage: https://rclone.org/docs/
After providing a name for this access and answering all other further questions, rclone config will save the configuration at: /root/.config/rclone/rclone.conf
In my example, aws was the name of my remote configuration and this was the output:
[aws] env_auth = true access_key_id = AKIAIFTEYM3JK5VUXGBA secret_access_key = hjqF6wG/tNBqgpBbwBmQ3EQpok5U33W2ewSXD region = sa-east-1 endpoint = location_constraint = acl = private server_side_encryption = storage_class =
It is very important to access rclone access to the cloud. Run some test commands as follows:
# Fetch bucket list:
rclone lsd aws:
root@hfaria-asus-i5:/# rclone lsd aws:
-1 2014-10-01 20:27:41 -1 baculaheitor
# If doesn't exist, create a new bucket:
rclone mkdir aws:new_bucket_name
# List bucket contents:
rclone ls aws:baculaheitor
6311 logs/2014-10-02-01-17-25-7F0DBBBFD43A20B7
2721 logs/2014-10-02-01-17-31-BFC439050C2B0AA9
666 logs/2014-10-02-02-17-36-D47447EC8F6759EE
2079 logs/2014-10-02-02-17-47-24DD21E3B85C5CAF
40108 logs/2014-10-02-03-17-10-4E9B7A829CEC7842
# Test diectory copy to the bucket:
rclone copy /etc aws:baculaheitor
# Test directory copy from the bucket:
rclone copy aws:baculaheitor /tmpFinally, copy the work configuration version to the path that rclone-changer expects, as follows:
cp ~/.config/rclone/rclone.conf /opt/bacula/etc/rclone.conf chown bacula /opt/bacula/etc/rclone.conf
It is time to deploy rclone-changer.
Rclone-changer installation and configuration
Install the python and python-yaml dependencies:
apt install -y python python-yaml # or yum install -y python python-yaml
Download rclone and create needed folders:
wget https://github.com/wanderleihuttel/rclone-changer/archive/master.zip -O /tmp/tmp.zip; unzip /tmp/tmp.zip -d /usr/src; rm -f /tmp/tmp.zip cp /usr/src/rclone-changer-master/rclone-changer /usr/sbin/ chmod +rx /usr/sbin/rclone-changer chown bacula /opt/bacula/working
Then, create the folder and file that will host the virtual tape recording. Also create a directory for Bacula local disk spooling, since advisable for faster backups and safer copy to the cloud, as follows:
mkdir /mnt/vtapes && touch /mnt/vtapes/tape && chown -R bacula /mnt/vtapes/ mkdir /mnt/bacula-spool && chown bacula /mnt/bacula-spool
Add a similar device to the Bacula Storage Daemon Configuration file (bacula-sd.conf):
Autochanger {
Name = "rclone_aws"
Device = aws
Changer Device = 'aws:baculaheitor' # remote rclone configuration name and bucket. You can also specify a cloud directory
Changer Command = "/usr/sbin/rclone-changer %c %o %S %a" # don't change this
}
Device {
Name = aws
Media Type = aws
Maximum Changer Wait = 18000 # this should be increase according to the size of the volumes
Archive Device = /mnt/vtapes/tape # must match the file created on the prior step
Autochanger = yes
LabelMedia = yes;
Random Access = Yes
AutomaticMount = no
RemovableMedia = no
AlwaysOpen = no
Spool Directory = /mnt/bacula-spool
Maximum Spool Size = 524288000
}Now, tie the new Storage Daemon device to your Director and also create a Pool for it:
Autochanger {
Name = Cloud_aws
Address = sd_address
SDPort = 9103
Password = "EKPNGPoIe7wXO/oe6xwm9cXzHDgmiuS0qF3hyAhpEg+i"
Device = rclone_aws
Media Type = aws
Autochanger = Cloud_aws
Maximum Concurrent Jobs = 10
}
Pool {
Name = Offsite
Pool Type = Backup
Recycle = yes
AutoPrune = yes
Storage = Cloud_aws
Maximum Volume Bytes = 1073741824 # recommended
AutoPrune = yes
Volume Retention = 4 weeks
}It is important to have developer considerations in mind if changing Maximum Volume Bytes:
It is recommended that you use many smallish volumes rather than large ones. Try to stay at a size that your average file fits into without getting too tiny. The reasoning here is in a recovery situation if you’re working on random files you want to minimize the amount of unrelated data you need to copy localy in order to recover. If you have a 10G virtual tape and only need 500M of it, you still need to wait for the full 10G to download before you can beging recovering data. Hence, if your typical file size is averaging 700M or so, 1G volumes are probably a good idea. [https://github.com/travisgroth/rclone-changer]
Edit and verify that the python version and the rclone-changer directories are correct and the Linux bacula user is allowed to write to them. You can also change the number of slots your virtual robot will have.
vi /usr/sbin/rclone-changer
#!/usr/bin/env python
...
#=====================================================
# Default params config
#=====================================================
rclone = '/usr/bin/rclone'
config = '/opt/bacula/etc/rclone.conf'
logFile = '/opt/bacula/working/rclone.log'
logChanger = '/opt/bacula/working/rclone-changer.log'
lockFile = '/opt/bacula/working/.rclonelock'
stateFile = '/opt/bacula/working/rclone-changer.state'
slots = 100
#=====================================================Then, create virtual tape Bacula labels. Use bconsole for that:
label barcodes storage=Cloud_aws pool=Offsite
If have trouble creating tables, check all the prior commands and permission grants. In order to reset the VTL and try again delete the vtape and the rclone-changer state file:
rm -f /mnt/vtapes/tape* rm -f /opt/bacula/working/rclone-changer.state touch /mnt/vtapes/tape && chown -R bacula /mnt/vtapes/
All done! Run a test backup job to the Offsite pool and create Schedule routines to the cloud, according to your needs. Virtual tapes will be changed and uploaded to the cloud automagically, as in the following screenshots:
Rclone-changer Troubleshooting
These are the places where rclone and rclone-changer saves their log messages:
cat /opt/bacula/working/rclone.log cat /opt/bacula/working/rclone-changer.log
Disponível em: 
Português (Portuguese (Brazil)) 
English 
Español (Spanish)
