Cisco switch flooded with MAC Addresses VMware Distributed switch

Recently a client had a problem with some VM’s on its infrastructure. In more detail it was detected that some packets were missing/not transmitting correctly to some virtual servers.

The investigation of the vendor (Cisco) showed that the problem was due to a hard limitation of MAC addresses on the switch. It seemed that a limitation of 70.000 MAC addresses created this problem. But how so many addresses have been created on VMware infrastructure?

In the infrastructure two new chassis with Flex nodes have been added. Those chassis contain around 20 servers each that were included in the virtual Distributed switch. In the vDS existed around 210 VLAN’s. Everytime a new chassis was active on the switch around 3000 new MAC addresses created. This number is approximately the number of VLANs x host number = 20 x 210 = 4200

The problem occurred due to Distributed switch health check mechanism. As VMware states:

vSphere Distributed Switch health check helps you identify and troubleshoot configuration problems with the vSphere Distributed Switch (VDS), and mismatched configurations between the VDS and your environment’s physical network. By default, health check is turned off. You can enable health check to identify and resolve network problems you might be experiencing. Depending on the options that you select, vSphere Distributed Switch health check can generate a significant number of MAC addresses for testing teaming policy, MTU size, and VLAN configuration. These MAC addresses result in extra network traffic, which can affect network performance.


Use health check to troubleshoot network problems, and then disable it after you identify and resolve the problem. After you disable vSphere Distributed Switch health check, the generated MAC addresses age out of your physical network environment according to your network policy. For more information, see Knowledge Base article KB 2034795.

As a result only enable virtual distributed switch health check mechanism for troubleshooting and disable on production environments as it may cause problems on your VM’s.

Extend Disk with diskpart.exe – Windows server 2003

Below is the procedure on how to extend a disk space on Windows server 2003 through diskpart.exe. Tool is located under C:\Windows\system32\diskpart.exe

The extra disk space is provided through VMware hypervisor.

after the extend from VMware disk space is available on Disk Management.
commands to extend disk space

Veeam Backup fails with log message: CBT Data is Invalid

Having your backup jobs fail is really a frustrating experience. 99,5% of the time, i’m experiencing successful backups, either over TSM, or Veeam Backup.

However, there were more than a handful of times that the CBT (Change Block Tracking) stopped working, generally causing backup jobs to fail.

The message listed in the Action log is the following:

CBT data is invalid, failing over to legacy incremental backup. No action is required, next job run should start using CBT again.
If CBT data remains invalid follow KB1113 to perform CBT reset. Usual cause is power loss.

I’ve tried to narrow the reasons this happens, to mostly being Datastore capacity outage or the speed of snapshots being removed during the backup process.

Luckily, there is a solution and it lurks in the VMWare KB2139574 (Resetting Changed Block Tracking for VMware vSphere virtual machines) .

You can generally follow one of the two recommended actions according to VMWare:

  • Power off the VM and reset CBT
  • Run the provided script to┬áreset CBT on all VMs, which uses a snapshot method to save having to power the VMs off.

Personally, although i have tried both actions, i prefer the second approach, as most of the virtual Infrastructures contain VMs that are not easy to power off, or have different maintenance windows.

In any case, after reattempting to execute the Veeam backup job, the error is no longer there, and Change Block Tracking is back in use.