We have some standalone ESXi hosts in our company. Recently we encountered a werid issue where the mac address of the ESXi host flapping.
Issue description:
I found that the IP of some esxi hosts will occasionally be unable to ping, but the vm network on the host will not be interrupted. At this time, if you ping the IP of the ESXi host on a server in the same VLAN, the ESXi host network will be restored.
At first, I thought it was due to the EEE function of the NIC. So I upgraded ESXi version(ESXi 6.5 -6.7-7.0), disabled the igbn driver and enabled the ign driver, disabled EEE function but these did not work.
I finally found that when the issue occurred, ESXi host MAC would flap to G2/1 port. G2/1 port is connected to Sangfor Access Control Device and AC device is connected to the firewall. mac flapping log:
Jun 30 12:38:13.923 GMT: %C4K_EBM-4-HOSTFLAPPING: Host 08:94:EF:30:BB:D0 in vlan 99 is moving from port Gi2/1 to port Gi2/16
Jun 30 12:38:23.491 GMT: %C4K_EBM-4-HOSTFLAPPING: Host 40:F2:E9:96:EF:72 in vlan 99 is moving from port Gi2/1 to port Po10
Jun 30 12:45:17.847 GMT: %C4K_EBM-4-HOSTFLAPPING: Host 40:F2:E9:96:E9:7A in vlan 99 is moving from port Gi2/1 to port Gi2/33
Jun 30 12:45:20.231 GMT: %C4K_EBM-4-HOSTFLAPPING: Host 00:50:56:6F:76:8C in vlan 99 is moving from port Gi2/1 to port Gi2/29
Jun 30 12:45:20.287 GMT: %C4K_EBM-4-HOSTFLAPPING: Host 00:50:56:66:89:DB in vlan 99 is moving from port Gi2/1 to port Gi2/29
Jun 30 12:45:20.327 GMT: %C4K_EBM-4-HOSTFLAPPING: Host 00:50:56:6A:88:90 in vlan 99 is moving from port Gi2/1 to port Gi2/29
Jun 30 13:12:15.275 GMT: %C4K_EBM-4-HOSTFLAPPING: Host 08:94:EF:30:C3:40 in vlan 99 is moving from port Gi2/1 to port Gi2/16
Jun 30 13:16:26.499 GMT: %C4K_EBM-4-HOSTFLAPPING: Host BC:97:E1:86:0B:AC in vlan 99 is moving from port Gi2/1 to port Gi2/16
Our senario:
- 2 cs4506 switches are configured with portchannel
- every ESXi host is connected to the cisco cs4506
- ESXi host port is configured as access
- no portchannel or LACP between the switch and ESXi server
- ESXi host in VLAN99
Here are some troubleshooting steps I have taken:
1.The CPU usage of the switch is not high and this issue only occurs on ESXi hosts.I don't think this caused by network loop.
JD1KY2F-CS4506-01#show process
CPU utilization for five seconds: 22%/1%; one minute: 20%; five minutes: 20%
2.Someone on Sangfor’s forum said it might be caused by STP, so I checked the STP update time of the switch, but it did not match the time when flap occurred in the log.
JD1KY2F-CS4506-01#show spanning-tree detail | in ieee|from|occur|is exec
VLAN0099 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 19461 last change occurred 4d12h ago
from GigabitEthernet2/5
3.According to this kb STP may cause temporary loss of network connectivity when a failover or failback event occurs (1003804).I set portfast but not work.
4.There is no duplication ESXi host MAC in switch mac address table.
Currently, I am configuring port security on the ESXi host port to avoid MAC flapping:
interface GigabitEthernet2/11
switchport access vlan 99
switchport mode access
switchport port-security maximum 50
switchport port-security
switchport port-security violation restrict
But this is only a temporary solution. I would like to configure a portchannel between the ESXi host and the switch, so I need to resolve this issue. Do you guys have some suggestions? THX.
