0

We have a VMWare ESXi 4.1 server hosting a number of Linux and Windows guests. Recently a new Linux guest was added to this server and seemed to be performing well. Tomcat and some other applications on this server were then installed which seem to have caused the server to run really slowly without any obvious resource issues.

Slow performance include:

  • The time taken to bring up the password prompt over ssh takes a few seconds when it was previously instantaneous.
  • The time taken to unzip a zip file which was previously a few seconds now takes around 30 seconds
  • The time taken to compile vmware tools has increased by similar factors

Both the VMWare console and monitoring commands don't report any issues with high CPU or memory usage but something is obviously slowing the server down somehow.

Does anyone have any ideas what may be causing this issue and how it can be resolved?

Thanks,

Tom

Edit

As per your questions I’ve looked at some of the performance indicators on both the VM host and VM guest indicated.

Firstly I tried reserving the full amount of memory (3gb) for this VM – no other machines on this server have any memory reservation.

The swap in rate and swap out rate for the VM host and guest are now both zero.

Balloon memory on the guest is zero and on the host is 3.5gb (total memory on the host is 12gb)

The swap rate for the guest is also zero.

Swap used by the host is 200mb on average.

Compression and decompression rates for the host and guest are zero.

Command aborts for the host are zero.

Read latency is very low – maximum 10ms average 0.8ms.

Write latency is higher – a few spikes to 170ms but mostly around 25ms – is this bad?

Queue command latency is zero .

Physical disk read latency averages 5ms but often 10ms

Physical disk write latency averages 15ms but is often 20ms

I hope this helps - let me know if you need any more information.

Improve this question
6
  • How's the disk I/O usage look?
    – Shane Madden
    Sep 21, 2011 at 14:55
  • CPU ready? disk queue lengths? IOPS? IO wait time? there's got to be some form of clue
    – Chopper3
    Sep 21, 2011 at 14:58
  • Looks like swapping at host level or inside VM.
    – Mircea Vutcovici
    Sep 21, 2011 at 15:00
  • Thanks for the additional information - I've been through part of the VMWare performance troubleshooting guide and didn't see any figures way out of the ordinary. I hope I've posed all the figures you need above but let me know if there's any missing - thanks
    – Loftx
    Sep 22, 2011 at 16:31
  • Can you please provide detail on the version and distribution of Linux you're using? RedHat? CentOS? Version 4? 5? 6?
    – ewwhite
    Sep 22, 2011 at 18:28

4 Answers 4

Reset to default
2

Java tends to confuse the VMWare memory management systems, as it adds yet another layer of memory management.

There is a KB article, linked below. But the key one is to size the reserved VM memory (not just the VM memory size) to be >= to the memory of Java (tomcat).

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1008480

Improve this answer
2
  • Hi there, thanks for this information - I've set the reserved VM memory to the full amount of memory for this system (3gb) but it doesn't seem to have made any difference after rebooting the VM. It's also not just tomcat which is performing slowly - just copying files etc. is slow too.
    – Loftx
    Sep 22, 2011 at 16:33
  • It would be everything that is slow, it just happens to be that Tomcat is using a huge chunk of memory.
    – Jeff Warnica
    Sep 22, 2011 at 17:59
2

The disk latency numbers you have posted are not really low, they are pretty high. 10ms of disk latency or more is going to make interactive use very sluggish. Take a look at your storage system and see if you are oversubscribed for IOPS.

http://www.techrepublic.com/blog/datacenter/calculate-iops-in-a-storage-array/2182

Improve this answer
2

What you have here is double virtualization. You have a guest operating system, Java, running inside of another Guest (LINUX) on a virtualization host. All of your calls from Java must be double translated before they hit actual hardware resources. This is a recipe for slower than expected performance when compared to native in most situations.

It is typical for guest operating systems like Java and Database solutions to have their own natural coalescing layers, to be able to consolidate multiple instances on a single piece of hardware, thereby achieving the same cost savings as would normally be the case for turning and entire host operating system into a guest under a virtualization solution. The core benefit of using the native consolidation mechanism is that you avoid adding another arbitration layer on the way to hitting the actual hardware resource using by the guest operating system (Java). Sometimes the hypervisor will be your friend on allowing access to resources, many times not.

Have you considered cutting out one arbitration layer for access to hardware and looking at a more native VM tomcat instance such as VMWARE's tc server, http://www.vmware.com/products/vfabric-tcserver/ or even a Java OS instance like JNODE or SANOS?

Improve this answer
2

This happened to us too on a Debian Lenny (5.0) Server, mainly the SSH login time problem, after installing Tomcat. After seeing lots of comments on DNS resolution we found that what was slowing our system was the Avahi-Daemon and not Tomcat.

Doing an 'apt-get purge avahi-daemon' solved the ssh login delay.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .