2

I had a Gigabit network, which I upgraded recently to 40Gbps. To do so, I replace my TPLINK switch with a Cisco Nexus 9396PX switch, which has a module with 12x 40G ports. The 40G network interconnects DELL servers with ESXi7 installed.

The problem is that I am not able to achieve the throughput speed of a 40Gbps network (~1250 MB/s x 4).

  • When I run iperf between two virtual machines, I get a bandwidth ranging from 6 Gbits/sec to 10 Gbits/sec. I never been able to reach the 40 Gbits bandwidth.
  • When I copy files with scp I am getting a transfer speed of about 110-120 MB/s which is very far from the theoretical bandwidth of 1GB/s x 4 ~ 4G/s for a 40G network.

Recently I discovered that encryption reduce transfer speed so, I set up FTP server. Transfer speed doubled with FTP. I got a transfer rate about 240-280 MB/s. I also updated MTU to 9000 on both hosts and VMs, but transfer rates are low.

Can anybody, please, help me to identify and solve the issue? I want 40G Network bandwidth as configured in the Switch to be reflected in Virtual machines, and the transfer speed to reflect as close as possible transfer speed of 40G network (~1250 MB/s x 4).

This is the current architecture details:

  • Cisco Nexus 9396PX Switch with 40G module
  • 4 Dell PowerEdge R930 Servers (VMware ESXi7 hosts)
  • NVME disks on each server
  • 40 Gbps Mellanox NIC on ESXi7 hosts
  • QSFP+ DAC cables to connect servers to switch

UPDATES

I just made 3 more tests. And I can not say there are improvements

FIRST TEST

  1. In Switch, I checked MTU on the server interface and noticed it was not set to Maximum. I tried to set it to 9000 but was denied. It can only be set either to 1500 or 9216. I chosed the highest value.
  2. Still in the Switch I fixed speed to 40G/s and disabled speed auto negociation
  3. I disabled speed auto negociation on ESXi hosts from VCenter.

Then I run the tests again. These changes did not affect the scp copy speed transfer speed between VMs (Ubuntu 22.04). But I noticed a light improvement when downloading from FTP server (it is installed on one of the two ubuntu VMs).

SECOND TEST

I run the following tests on VMs with Windows 10 installed.

  • I added two Windows VM to the host and configured them with MTU set to 9000 too.
  • I created a shared folder for copy betwnn the two Windows VMs.

When I copy files from shared folder to local, the speed is around 600MB/s. But when I copy from local to shared folder, the copy starts at around 350MB/s and the speed decreases when it reaches ~30%.

THIRD TEST

For the last test I installed Filezilla on one the the windows VMs and dowloaded files from/to FTP server (on Ubuntu VM).

  • Dowload speed is around 450MB/s when the download starts and at about ~50% it decreases to ~200MB/s
  • Upload speed is around 200MB/s
Improve this question
3
  • Not my area of expertise, but this answer was posted yesterday and could be helpful. Especially the second paragraph.
    – Gerald Schneider
    Mar 30 at 12:17
  • 1
    @yaoviametepe - Yeah, I agree with Gerald, specifically the second paragraph of my answer to that other user - 40Gbps isn't what most people think it is.
    – Chopper3
    Mar 30 at 13:23
  • @GeraldSchneider thank you for your answers. This is a valuable information you gave. May be I should focus on how to achieve 1G/s of dowload speed with the one line of 10G. Because I can not even achieve this speed.
    – yaoviametepe
    Mar 30 at 15:14

1 Answer 1

Reset to default
2

40 gig is basically a bonded 4*10 gig connection. To benefit from the aggregated bandwidth you need multiple TCP sessions running at the same time. You’ll never reach combined estimated bandwidth with just one process and a single TCP session. Some good description could be read here:

https://www.starwindsoftware.com/blog/lacp-vs-mpio-on-windows-platform-which-one-is-better-in-terms-of-redundancy-and-speed-in-this-case-2

Improve this answer
3
  • Thank you for the answer. It helped me confirm that I am on 10Gbps link, not 40G. I do not know if it could be related to the virtual settings or not. The command iperf3 -c 192.168.1.80 -P 8 -t 60 output summary is: ...[ 19] 0.00-60.02 sec 7.15 GBytes 1.02 Gbits/sec receiver [SUM] 0.00-60.00 sec 57.0 GBytes 8.16 Gbits/sec 140 sender [SUM] 0.00-60.02 sec 56.9 GBytes 8.15 Gbits/sec receiver
    – yaoviametepe
    Mar 30 at 19:09
  • A 40G interface is not the same as four bonded 10G interfaces. On the former you can get a single flow up to 40G (with some tuning) and on the latter only up to 10G.
    – Zac67
    Mar 30 at 20:26
  • @Zac67 Can you please tell me what values needs to be tuned to achieve 40 Gbits/s throughput in my VMs?
    – yaoviametepe
    Mar 31 at 0:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .