After applying IP filters on VMWare esxi's firewall settings for Web Access / Web Client rules, the allowed IPs work fine for some time like 5 hours.
Only the IP addresses I allow can access the webclient.
However, after some time I no longer can accesss the web client. It's like the port is closed.
What works is that I access the esxi shell using the KVM my hosting company provides, and disabling and then re-enabling the firewall which will grant me access for few more hours.
This started when I upgraded my esxi servers to 7.0+, before that I didn't have this issue. I still have an esxi 6.5 server and I don't have this problem there.