Veeam Backup fails with log message: CBT Data is Invalid

Having your backup jobs fail is really a frustrating experience. 99,5% of the time, i’m experiencing successful backups, either over TSM, or Veeam Backup.

However, there were more than a handful of times that the CBT (Change Block Tracking) stopped working, generally causing backup jobs to fail.

The message listed in the Action log is the following:

I’ve tried to narrow the reasons this happens, to mostly being Datastore capacity outage or the speed of snapshots being removed during the backup process.

Luckily, there is a solution and it lurks in the VMWare KB2139574 (Resetting Changed Block Tracking for VMware vSphere virtual machines) .

You can generally follow one of the two recommended actions according to VMWare:

  • Power off the VM and reset CBT
  • Run the provided script to┬áreset CBT on all VMs, which uses a snapshot method to save having to power the VMs off.

Personally, although i have tried both actions, i prefer the second approach, as most of the virtual Infrastructures contain VMs that are not easy to power off, or have different maintenance windows.

In any case, after reattempting to execute the Veeam backup job, the error is no longer there, and Change Block Tracking is back in use.

Update standalone ESXi host with esxcli

There are multiple ways available to patch a VMware Infrastructure. Two of the very common ones are through Update Manager or esxcli.

Update Manager enables centralized, automated patch and version management for VMware vSphere and offers support for VMware ESXi hosts, and virtual machines. Unfortunately it is available only on vCenter deployments and you cannot use it on standalone hosts.

When you need to update a standalone host you can follow the procedure described below.

In my case I wanted to patch an ESXi host from 6.7 Build 8941472 to 6.7 Build 9484548.

First you download the patch definitions from VMware site. In my case I needed to download the below bundle package:

Then, the package should be uploaded to the local or any available datastore so that host can access it. Also ssh should be enabled so that one can connect on the host with root account.

file upload can be performed from the available esx host GUI

change directory and find the location of the file

Lastly execute the below command, and reboot the host

The version available after the reboot, should be the new one

Map Windows OS disks with VMware hard disks

It is a bit tricky to map Windows OS disks with VMware disks especially when two or more disks have the same size, because vmware does not provide the disk name, but only scsi identifiers. As a result if I had to extend a particular disk, I must find the right one.

From VMware hard disk Advanced Settings, you can find a SCSI code for a particular disk 1:2 that vmware names hard disk 2. Note that hard disk 2 could ne different from Hard disk 2 that the Windows OS would name. You could determine that from the capacity of the Disk if they were different, but it could be also different even when the capacity (GB) were the same.

 

So how could you do the mapping?

First of all you should run the following powershell command. It would give you scsi identifiers for all Windows OS disks.

 

Windows OS start numbering controllers from a random positive number and adds one for each subsequent controllers. As a result if the numbering starts from 3, then 3 scsiport would match to 0 scsiport in VMware (because numbering starts from 0). In the particular example if you would see 4:2 from the powershell command, it would be the disk that you can see in the picture above.