Veeam has rolled out new backup features that integrate with Amazon S3 to help users save on storage costs for their backups. This tech is some serious secret sauce.
For users of AWS, backup vendor Veeam offers native integration with Amazon Simple Storage Service (S3).
As many AWS users are aware, S3 offers high availability and extremely high durability at reasonable cost, without requiring users to manage file servers.
One of the limitations of S3, however, is that it provides only object storage. If users require block storage for running an operating system or a database, they need to provision volumes on Amazon Elastic Block Storage (EBS).
Object storage stores individual files such as images, documents, and audio and video files. One of the limitations of S3 as object storage is that it lacks the ability to "de-duplicate" that IT professionals have become accustomed to in on-prem storage solutions.
As such, if versioning is turned on, for instance, S3 will store a full copy of every version of a file, e.g. a 20MB file with ten versions will take up 200MB of storage in S3.
Veeam has developed features to allow it to take advantage of S3 object-based storage for backups and at the same time avoid unnecessary duplication of data.
Veeam will create backup that includes the backup data and metadata. When the backup tool lifecycles older backups into S3 in accordance with the user's configuration rules, it pushes to S3. When this data hits S3, it's no longer a block – instead, Veeam shards the data into hundreds or even thousands of individual objects within a single S3 bucket.
The beauty of Veeam's solution is that it uses the metadata to avoid unnecessary duplication for all the backup versions that are stored in S3. Each time a new backup version is shipped to S3, Veeam references the metadata and will only create copies of objects that do not already exist in that current state in S3. It completely avoids recopying the same objects.
It’s worth noting that although this feature prevents duplication, it works differently from what we call “deduplication” in traditional on-prem storage solutions.
As a working name for this feature, a Veaam representative suggested “spaceless full backups.”
Veeam's native integration with S3 and its "spaceless full backups" feature are a true boon to Veeam customers who use AWS.
The massive potential of storage cost reduction, combined with the insanely high durability of S3, allows customers to have confidence in the security and integrity of their backups and save on costs and/or retain longer periods of their ever-growing quantities of data.