The virtualization debate is over and virtualization won. It’s time to maximize your datacenter using virtualization. No longer will physical servers sit at 5 to 10 percent utilization. Those physical servers can become virtual hosts so that you can run many more servers on top as virtual machines. The CPU and RAM utilization can be maximized, thanks to a virtualization hypervisor, so that you can reduce costs and improve efficiency. To do that, you’ll be pushing the resources of your physical servers to their max.
It’s a great plan for financial return, company agility, and administrator efficiency but what about the performance of those virtual machines and their critical company applications? How do you maintain performance of those applications when, at the same time, you are trying to pack as many VMs on a host as possible?
The answer is to follow our top 10 VMware vSphere performance tips. We’ll start with number 10 at the bottom, and work our way up.
Many sysadmins incorrectly think of VMware Tools as just being for mouse sync when opening a VM console. While this is an important (and convenient) benefit, VMware Tools is much more than that. As discussed in this VMware publication on VMware Tools, not only does VMware Tools contain the mouse driver but it also has many other drivers. And it’s those other drivers that can actually increase performance on the VM in which the the tools are installed. Even better, those drivers can increase performance on other virtual machines when you have a resource shortage.
The driver installed by VMware Tools that is most important is the MemCtl, or memory control driver. This driver is responsible for memory ballooning for the VM. So, if there is a memory shortage on the ESXi host, the ballooning driver can be used to free up memory from your VMs that have too much. If you turn that around, and your VM is the one that is short on memory, and if other VMs have the ballooning driver installed with VMware Tools, then memory can be freed from those VMs to help your VM.
In the graphic below you can see which of my VMs have VMware Tools installed and which VMs don’t.
Check this link for more information on how to install VMware Tools on just about any operating system (including Linux).
The hardware version of a virtual machine reflects the virtual machine’s supported virtual hardware features. These features correspond to the physical hardware available on the ESXi host on which you create the virtual machine. Virtual hardware features include BIOS and EFI, available virtual PCI slots, maximum number of CPUs, maximum memory configuration, and other characteristics typical to hardware.
As part of any upgrade process, you should upgrade your virtual machine hardware (along with VMware Tools, VMFS versions, vCenter, vSphere Client, ESXi, and more). The vCenter VM inventory can tell you what version of VM hardware you are using on all your VMs:
More information on Virtual Machine Hardware Versions can be found here.
VMware usually releases a new version annually. Each successive version of vSphere has offered more scaleability (like the “Monster VM”) and improved performance. For example, VMware’s memory compression feature (see my video on Understanding Memory Compression in vSphere 4.1), which automatically kicks in when memory pressure occurs, was included (and is automatically used) with vSphere 4.1. vSphere 5 introduced new features like storage distributed resource scheduler (SDRS), VMFS5, and virtual hardware 8.And vSphere 5.1 introduced even more!
Bottom line: Running the latest version of vSphere (and all it’s associated options) is a smart move to ensure you can utilize the most recent performance features.
You can’t be sure how your virtual machine utilization will change over time. VMware’s distributed resource scheduler can monitor a cluster’s CPU and memory utilization and automatically balance the virtual machines load, with no downtime. It’s available in the enterprise and enterprise-plus editions of vSphere; they almost magically creates a self-balancing, high performing, resource cluster. VMware’s high availability (vSphere HA) and distributed power manager (DPM) can also easily be incorporated into a DRS-enabled cluster with just a couple of check boxes.
More information about VMware’s DRS is available here.
Depending on the application, how it works, and how much data it accesses, you may be I/O-bound. What that means is that your performance is constrained by I/O resources. No matter the application, you may become IO-bound if you put many virtual machines on a single spindle (or LUN). With solid state disk (SSD) coming down in price, this disk storage that offers consistent I/O performance, no matter the load, is a viable option.
You could use a disk storage array that utilizes SSD instead of standard disk or an array with SSD and intelligent caching (an example is Tegile Zebi).
Another way to utilize SSD with vSphere to improve performance is to utilize the vSphere host-caching feature such that if a host is low on RAM, the host-swapping option is used to swap memory to low-latency disks that you specify (such as SSD). Now, if you use swap to host cache, remember that it’s not the same as placing regular swap files on SSD-backed datastores — the host still needs to create regular swap files. Even so, when you use swap to host cache, the speed of the storage where the host places regular swap files is less important.
One of the great benefits of virtualization is the ability to use VM snapshots. While it’s a great feature to use before an upgrade or configuration change, snapshots should only be used temporarily and then removed. Too many admins think that snapshots are for periodic backup purposes and leave many GB of snapshots on disks. Besides being a waste of disk space, VM snapshots slow down many things such as svMotion of the virtual disk, backups, disaster recovery, and more.
Analyze what snapshots you have in use, remove them, and then only use snapshots when needed (and then remove them). You can learn how to find VMware virtual machine snapshots with PowerCLI here.
vSphere offers a number of advanced features to benefit storage performance on vSphere servers. One of those features is Storage I/O Control (SIOC). Storage I/O Control prevents one ESXi server from monopolizing the storage with an unfair number of I/O requests. You can learn about SIOC in my Petri IT Knowledgbase video on storage I/O control (SIOC) in vSphere 4.1.
Another feature that you should consider is Storage Distributed Resource Scheduler (SDRS). It does for storage what DRS did for CPU and RAM. By creating a storage cluster, the disk availability and storage latency is measured on the storage and virtual machine disk files can be moved to another datastore that offers better performance. SDRS can do its job to balance the storage load automatically and with no downtime to the virtual machines or applications.
You can learn more about SDRS and about enabling and disabling Storage DRS (SDRS) in the vSphere Web Client.
Too many times sysadmins try to get more out of their software when it’s really the hardware that has reached its’ limits. New servers have multiple cores per CPU (4, 6, or more cores per CPU) and 256, 512, or more GB of RAM. Hundreds of virtual machines can easily run on one of these new powerful servers that has many multi-core CPUs and many hundreds of GB of RAM. One example of an amazingly powerful server line is Cisco UCS. Learn more about Cisco UCS in this Petri IT Knowledgebase Cisco UCS overview and check out resources like this video of TrainSignal’s Cisco UCS instructor, Jason Nash.
In many cases, VMs run poorly because they have too little CPU, too little memory, or too few I/Os per second available to them from the storage. In other cases, virtual machines are oversized because an application owner specified how much CPU and RAM they wanted based on old physical server resource requirements. Where underallocation can certainly cause performance issues, overallocation can also cause slowdowns for other VMs that can’t gain access to the memory that they need. By knowing your application and monitoring its performance, over time you’ll better be able to “right-size” the virtual machine that it runs on.
There are numerous guides out there to help you right-size virtual machines. One example is my whitepaper on vCPU sizing. What can help you even more are tools that make right-sizing recommendations for you (see the next section). Which leads us to…
In most cases, when vSphere admins are looking at performance it’s because they have performance issues (or even end-user complains about slowdowns). Yes there are numerous performance-boosting tweaks that you can make but the first question I ask is, “Do you have a performance and capacity analysis tool” ? These types of tools, created just for VMware virtual infrastructures, can, in most cases, predict your capacity and performance problems before those problems ever even occur. And if you already have a capacity bottleneck, these tools can quickly locate it. Examples of tools are vKernel vOPS, VMware vCenter Operations Manager 5 , and Solarwinds Virtualization Manager.
All of these tools are available in free trial versions that can, typically, quickly find your issue (even in trial mode). If you don’t have any long-term budget for such tools, there are even free tools available such as vKernel’s vOPS Server Explorer.