You can follow the below procedure to extend an RDM disk with virtual compatibility mode on VMware. As described in the article attached on the bottom, you can accomplish that either by migrating the RDM or by shutting down the VM and performing the procedure below. The article implements the shut down way.
The first step is to extend the RDM disk from storage. Find the UUID descriptor and provide some additional space.
Then you need to locate the VMware disk that you want to extend and record the SCSI controller and SCSI ID because it is needed on a later stage.
Then you should delete the disk from virtual machine. You can also select (delete files from datastore) as only the RDM pointer will be deleted and not the actual disk data.
After that you should rescan storage devices and volumes on the ESXi host and locate if the device has identified the new capacity.
Then edit virtual machine settings and select add new device that will be of type RDM Disk
The next dialog shows the new disk with the extended capacity.
Click OK and be careful in order to add the same SCSI ID and controller as previously.
Perform a rescan inside the Operating system and the capacity will be available for use
