SUMMARY

Notes, advanced configuration, and troubleshooting for Cloudhook.

ISSUE

Describe additional Cloudhook information that may be useful. 

RESOLUTION

CloudHook Notes

Bucket/Container Names

Note that cloud bucket names are only supported with lower case letters, numbers, dash '-' and period '.'.   Each of the cloud providers has different syntax rules for bucket names and their access methods, and these are the common characters allowed for all cloud providers.  From the CloudHook menu, the bucket/container name is validated against these characters before saving it. 

Note that Rackspace containers for CloudHook must be preceeded by the hostname ‘auth.api.rackspacecloud.com’.  For example:  auth.api.rackspacecloud.com/mycontainer.  This hostname is automatically added when the Unitrends UI creates a Rackspace cloud storage instance.

Rackspace Container Types

CloudHook currently supports Rackspace ‘Private’ containers, but not Rackspace Public/CDN containers.

Amazon Container Types

CloudHook supports AWS S3 containers and new with release 10.3.x also supports S3 IA (Infrequent Access) containers, but does not support Glacier containers. CloudHook supports AWS Signature V2 and Signature V4. To enable V4 signatures, see Enabling AWS V4 Authentication.

Google Cloud Container Types

CloudHook supports Google Cloud standard and DRA containers. 

Other "S3 compatible" containers from 3rd parties:

Though unsupported by Unitrends directly, and titled entirely under "use at your own risk", it may be possible for end users to create CloudHook connections for cold copy use from a Unitrends appliance to other providers that advertise full S3QL compatibility via standard S3 API.  Customers are advised 1) this process should be heavily tested including not just simple upload and recovery, but recovery of "aged" data as well, 2) a full understanding Unitrends Support staff will refuse to assist with, diagnose, or escalate any issues with cloud storage systems other than Amazon's own S3 or S3 IA, or Google's Nearline and that data may be at risk of loss or inability to recover or be subject to fees from 3rd party.  This stated, customers may use comand line tools to try and test connectivity with other providers at their own risk using this command syntax: 

cloudadm create <storagename> s3c <hostname:port/bucketname> cloud <APIkey> <SecretKey>

Username/Password

Cloud provider terms used for CloudHook username and password (API access):

Folders

When using folders for Amazon and Google, they appear in the cloud provider menu as a filename prefix only, unless you specify the folder with a trailing ‘/’ (e.g.  mybucket/myfolder/).  Then the data block files will appear in the cloud provider web menu underneath the folder.

Firewall / Proxy Issues

Sometimes the customer network will have a firewall or proxy that prevents read/write access to certain ports.  The ports that CloudHook requires are ports 80 and 443, and if those ports are restricted, it can cause the underlying handshake functions to hang, waiting on a response from the port.   The solution is to open ports 80 and 443 for the CloudHook system. 

DNS is required

Resolving the IP address of cloud storage requires DNS, so make sure a DNS server is configured.

Connection Speed to the Cloud Storage

You can verify that your connection speed to the external network is sufficient to handle archiving to the cloud by running /usr/bp/bin/speedtest.sh.  This will run a standard speed test from speedtest.net.   Some sample results are Download: 74.31 Mbits/s, Upload: 78.23 Mbits/s.

Limits on archive throughput to Cloud Storage

Depending on your connection speed, the number of CPUs, and your available memory, you can determine how much data can be archived per day.   With the sample speed results above, 16 CPU cores, and 48 GB of memory, a test CloudHook archive job throughput was 8.2 MB/s (66 Mbps), so this would yield up to 696 GB over 24 hours.  A test case with 8 cores and 16GB memory showed throughput of 3.4 MB/s (27 Mbps), yielding throughput up to 287 GB over 24 hours.  Your speed and memory size will likely be different, and results would vary depending on other activity on the system.    Conduct your own test in your environment to understand your throughput.  You will have to restrict your CloudHook archiving if your archive data size is beyond what your connection speed and throughput supports.

Rotate to a new Cloud Storage bucket periodically

Since accumulating lots of cloud filesystem data over time slows down cloud archives, adds overhead and some volatility to the remote cloud filesystem, it is recommended that you open a new bucket for use at least once every year, or more often if the archive size is large.  The previous buckets are still there for history. 

CloudHook data encryption

The s3ql software does use SSL encryption for data in transit by default, unless the --no-ssl option is specified, and CloudHook never specifies --no-ssl.   See https://bitbucket.org/nikratio/s3ql/overview and http://www.rath.org/s3ql-docs/backends.html for details.  

RequestTimeTooSkewed Error with AWS S3

This happens when your local system time is more than a few seconds different than the AWS server. To compare the local and remote times:

Get the UTC/GMT time from Amazon

# http://s3.amazonaws.com -v

Get the UTC on your local machine

# date –u

To resolve this, configure ntp in the Setup Wizard or /etc/ntp.conf, including these servers:

server 0.amazon.pool.ntp.org iburst
server 1.amazon.pool.ntp.org iburst
server 2.amazon.pool.ntp.org iburst
server 3.amazon.pool.ntp.org iburst

If not using the Setup Wizard, you also need to run

service ntpd restart

HTTPError: 403 Forbidden error with Amazon S3

This happens when the AWS key for this bucket either has not been created or does not have full rights to S3 storage.  From the AWS web console, create the AWS key if necessary, and set the AWS policies to give that key full rights to S3 storage.  Consult Amazon support for further questions. 

BusyError: database is locked error in cloudadm.log

This error message has two possible causes:

1. There is another process using the s3ql database with a lock pending, which is true in this case. Check for this with:

   ps –ef |grep mount.s3ql

   If so, resolve this by applying cloudadm v1.41 or later from EAPP-1116.  After that, future mounts of the cloud storage will succeed.
2. This can happen when the CloudCacheDir (default=/backups/cloudcache) resides on an NFS share. If this system is a UEB with /_Stateless mounted on NFS, then the latency of the cloudcache can cause BusyError.  If so, resolve this by changing /usr/bp/bpinit/master.ini so that the [CloudHook] CloudCacheDir=/var/cloudcache, or a similar directory on a local filesystem.  This can also be done from the RRC UI Settings/System/General(Advanced) menu.

"FullError: database or disk is full" in cloudadm.log 

This error message indicates that the size of the cloud filesystem database has exceeded the CloudCacheSize allotted.  You can increase this size from 512000 to 1024000 in /usr/bp/bpinit/master.ini  with this command. 
    /usr/bp/bin/bputil -p CloudHook CloudCacheSize 1024000 /usr/bp/bpinit/master.ini 
Note that the cloud bucket size is probably too large if this occurs and should be rotated to a fresh cloud bucket.

Bucket is already mounted by <asset>

CloudCache directory is not fast enough

Bucket is in an AWS Signature V4-only region

More Cloud Storage Provider Information

The Unitrends CloudHook feature utilizes S3QL software in Linux to transparently access various Cloud storage providers.  See these references for more information about configuring the Cloud provider storage backend.

• S3QL Storage Backends:   http://www.rath.org/s3ql-docs/backends.html
• Amazon S3 Documentation: http://docs.aws.amazon.com/AmazonS3/latest/gsg/GetStartedWithS3.html
• Google Cloud Storage docs: https://cloud.google.com/storage/docs/overview
• Rackspace Cloud Files docs: http://www.rackspace.com/knowledge_center/getting-started/cloud-files

Advanced CloudHook Settings

You can access these settings from the ‘Settings / System, Updates, and Licensing / General Configuration (Advanced)’ menu.    They are stored in the /usr/bp/bpinit/master.ini file. 

CloudCacheSize   512000

Maximum size of cloud cache in KB. Default setting is 512,000 KB, which should be optimal for most cases. You might want to increase the cloud cache size for archive sets larger than 500 GB.

EnforceMax  1

Enforces the user-specified purging threshold. Setting is enabled (1) by default. When this setting is enabled, archive jobs fail if they require storage space beyond the threshold. If this setting is disabled (0), the appliance still attempts to archive to the storage.

AlwaysPurgeGB   0

Always purge. Setting is disabled (N=0) by default. Enabling it and specifying a number (N) causes all cloud archive jobs to purge if there is less than N GB of free space below the purging threshold.  This has the same effect as if the purge checkbox had been checked for those jobs.

RemoveFiles 0

Remove files also when removing an archive set manually from the Archive Status menu. Setting is disabled (0) by default. If you enable this setting (1), archived data is deleted from the archive media when you manually remove an archive set from the Unitrends database. If this setting is disabled, only archive reference information is removed when you remove an archive set and the archived data is not impacted.

CloudCacheDir   /backups/cloudcache

Specifies the cloud cache directory. If you change this directory, be sure the new location has enough space for the cloud cache.

NOTES

Additional Information