I use a pair of Vyatta VM’s to route between VLAN’s at home. One is running on ESXi, the other on VMware Server 2 (yeah, I really want to move that to ESXi, haven’t had time to move all of the services I’m running on the host onto VM’s). They are redundant, I use VRRP on the user and server facing VLAN’s and OSPF on the VLAN facing my Cisco 1720 that’s connected to the Internet. The VM running on ESXi is prioritized in OSPF and VRRP to make lifer simpler and troubleshooting easier.

Drawing2

I recently noticed how slow the performance was when accessing FTP and HTTP for one particular host. From outside, a FTP session to this particular host would average maybe 10 KB/s. One moderate size image hosted via HTTP on that host basically would not load because the connection was going so slow (1 KB / s in this case). Off and on, I noticed slow performance connecting to other hosts. For a while, I focused my troubleshooting on the host exhibiting most of the performance issues. Oddly enough though, when I failed everything over to the other Vyatta VM running on VMware server, the performance was much better. I was able to receive 40 - 50 KB / s with that FTP session. Connecting the host directly to the VLAN between my Cisco router and the Vyatta VM’s also resulted in much improved performance. Nothing obvious jumped out from tcpdump’s or logs on the Vyatta VM’s.

I started looking at what inconsistencies there were between the two VM’s, and between the interfaces on each VM. These VM’s originally were made from Vyatta’s OVF template. I must have added one additional NIC to each of them. Two of the NIC’s on the ESXi Vyatta instances were VMXNET, and the third was E1000. Eventually, after troubleshooting this issue for quite a while, I found this particular topic on the Vyatta forums. When I changed the two VMXNET interfaces to VMXNET3, all of the performance issues on the ESXi Vyatta instance went away. (The Vyatta instance running on VMware server still has the NIC’s on VMXNET and doesn’t appear to suffer performance-wise from it). From other posts in that topic, it sounds like this may be related to a issue with the open-vm-tools included in past versions of Vyatta, that hadn’t been fixed in the OVF yet.

Another post on the Vyatta forum linked to a VMware KB article. The KB article describes performance issues on Linux guests that forward traffic if “Large Receive Offload” is enabled. I’m not sure if this is related to the issues I was having or not. Currently, LRO is enabled on my ESXi Vyatta instance, and performance still seems OK. So, I’m guessing that must be a separate issue from the one I was having.