TechOverflow Logo
TechOverflow

How to fix EC2 [error] Veeam Cannot find a compatible file system for storing snapshot data

Problem:

You are trying to run a Veeam backup on your EC2 machine (e.g. to a remote SMB or NFS service), but the backup fails immediately after the backup job is being started. The log looks like this:

21:52:29   Job BackupJob1 started at 2020-12-23 21:52:29 GMT
21:52:30   Preparing to backup
21:52:40   [error] Failed to create volume snapshot
21:52:41   [error] Failed to take volume snapshot
21:52:41   [error] Failed to perform backup
21:52:41   [error] Cannot find a compatible file system for storing snapshot data
21:52:41   [error] Processing finished with errors at 2020-12-23 21:52:41 GMT

The detailed log in /var/log/veeam/veeamsvc.log shows these errors:

[23.12.2020 21:52:41.069] <140589764957952> lpbcore|       Searching for the best candidate to store snapshot data.
[23.12.2020 21:52:41.069] <140589764957952> lpbcore|         Logical block size limit 512 bytes
[23.12.2020 21:52:41.071] <140589764957952> lpbcore|       Searching for the best candidate to store snapshot data. Failed.
[23.12.2020 21:52:41.071] <140589764957952> lpbcore| ERR |No suitable devices for snapshot data storage were found.
[23.12.2020 21:52:41.071] <140589764957952> lpbcore| >>  |An exception was thrown from thread [140589764957952].
[23.12.2020 21:52:41.071] <140589764957952> lpbcore|     Detecting snapshot storage parameters. Failed.
[23.12.2020 21:52:41.071] <140589764957952> lpbcore|   Creating snapshot storage. Storage type: stretch file Failed.
[23.12.2020 21:52:41.071] <140589764957952> lpbcore| Creating machine snapshot using VeeamSnap kernel module Failed.
[23.12.2020 21:52:41.071] <140589764957952> lpbcore| ERR |Snapshot creation operation has failed.
[23.12.2020 21:52:41.071] <140589764957952> lpbcore| >>  |Cannot find a compatible file system for storing snapshot data
[23.12.2020 21:52:41.071] <140589764957952> lpbcore| >>  |--tr:Failed to create machine snapshot
[23.12.2020 21:52:41.071] <140589764957952> lpbcore| >>  |An exception was thrown from thread [140589764957952].
[23.12.2020 21:52:41.071] <140589764957952>        | Thread finished. Role: 'snapshot operation'.
[23.12.2020 21:52:41.102] <140590156179200> lpbcore| ERR |Cannot find a compatible file system for storing snapshot data
[23.12.2020 21:52:41.102] <140590156179200> lpbcore| >>  |--tr:Failed to create machine snapshot
[23.12.2020 21:52:41.102] <140590156179200> lpbcore| >>  |--tr:Failed to finish snapshot creation process.

Solution

Veeam currently fails to detect the EC2 EBS boot device /dev/xvda as a proper device to store snapshot data on.

  1. You need to create a separate EBS block volume and attach it to the VM. The recommended size is computed as follows: *15% * total size of the disks that will be backed up.*In case you’re backing up files or directories, use 0.25 * the maximum anticipated size of the files or directories to be backed up.
  2. Attach that new EBS block device to the VM (e.g. as /dev/xvdb - use lsblk to find the correct drive !)
  3. Create a new partition table and a new partition on the EBS device using something like
sudo fdisk /dev/xvdb

then enter these commands into fdisk: g to create a new GUID partition table, then n to create a new partition. You can use the default parameters everywhere. Then, run w to write the changes to disk and exit fdisk using Ctrl+D) 4. Create a new filesystem on the new partition e.g. using

sudo mkfs.ext4 /dev/xvdb1
  1. Mount the partition somewhere (e.g. on /mnt/) using a command like mount /dev/xvda1 /mnt. In order to auto-mount on boot, use a line like
/dev/xvdb1 /mnt ext4 defaults,auto 0 0

in /etc/fstab. lsblk should now tell you that the partition is mounted, e.g.

xvdb    202:16   0    2G  0 disk
└─xvdb1 202:17   0    2G  0 part /mnt
  1. Re-run veeam. The backup should now work properly.

I don’t know why exactly this issue occurs but EC2’s /dev/xvda doesn’t seem to be a normal block device from Veeam’s viewpoint.

Note that veeam computes the minimum space for a snapshot store for entire machine backups as 10% * free space + 5% * used space. I don’t know if these factors are constant or determined dynamically, hence my recommendation of 15% * total space is much more conservative.