If you try to setup passwordless authentication for a Linux machine on your ESXi host and follow the same procedure that you would follow for a simple Linux box, the result will be a failure.
Normally you should do
to create your pub, private keys and then
to copy your keys on the esxi host. However this will not work and an additional step is required.
You should copy by your own the public key of your Linux machine to the ESXi host.
First of all enable ssh on the host and connect to the host.
The go to esxi and copy your public key under /etc/ssh/keys-root on authorized_keys. Keep in mind that permissions of this file should not be changed, otherwise it will not work.
After those actions you will be able to login passwordless on your esxi.