Traffic Shaping With VMware ESX Server

Posted on January 8, 2009 by Daniel Petri in VMware with 0 Comments

Many technicians have personal computers, servers, networks, gadgets etc. located in their homes for various purposes. Reasons usually involve sharpening skills for industry certification, for practical application in the workplace, to facilitate career advancement, or because they have way too much time on their hands.

I’m no exception, however, I’ve taken it a step further. Rather than limiting my lab to my basement in the Midwest United States, I’ve set up multiple sites including a west coast location in sunny San Diego, California with plans to expand to the east coast. I was thinking New York. My San Diego office is connected to my central Minnesota location by router and T1. I plan on a T3 or DS3 WAN link between Minnesota and New York and a 56Kbps link to stretch between San Diego to New York for redundancy. My three year plan includes a satellite office in Tel Aviv, Israel so I can visit with my friend Daniel Petri of, but first I need to learn Hebrew which isn’t going to be easy!

If you’ve made it this far, you may be wondering how or why I would afford such a lab setup. Did I have a banner year in revenue and profit? No, not really. My lab doesn’t generate much revenue or profit, at least none that is really tangible. Am I independently wealthy with Bill Gates or Donald Trump type money? No. Then what’s my secret?

VMware virtualization, and it’s no secret. Virtualization has been around for a little while now. Ok, but what about these WAN links and offices in Minnesota, San Diego, New York, and Tel Aviv? That sounds somewhat like Microsoft Official Curriculum, Microsoft Press, or a Microsoft exam – New York, Seattle, London. Contoso, Fabrikam, etc. That’s right, the similarities are uncanny and by design. But what about the expensive WAN links?! Well, the truth is that they don’t really exist from a real life perspective. They are a figment of my imagination, borrowed from Microsoft and what I have picked up along the way in my certification studies.

Truth be told, the Minnesota and San Diego sites exist in a virtual sense on my network. I’ll probably build out New York and Tel Aviv in the near future. It will only take half a Saturday to complete start to finish. Anyone can build a server and make believe it’s somewhere else, but I’ve taken it to a higher level in order to more accurately simulate sites on different subnets with a router in between. In addition to that, I’ve limited the bandwidth between the sites to my requirements in order to simulate network latency and sites separated by great distances. I accomplish this by using a feature built into VMware ESX called Traffic Shaping. Traffic shaping is quite similar to bandwidth throttling, except VMware’s traffic shaping provides a few more variables that can be tweaked which is why it’s actually called traffic shaping instead of simple bandwidth throttling.

My entire lab, with the exception of one server, is virtual. Nearly 40 virtual machines, most of them servers, populate my network. This provides many benefits: saves time, saves space, saves on utility costs such as cooling and electrical, drastically reduces network and electrical cabling, provides centralized management, etc. I could go on and on. Using VMware ESX Server as the platform for my virtual lab also gives me traffic shaping which is what allows me to simulate sites across the country and eventually around the globe by way of limited bandwidth. Limited bandwidth is a familiar concept but it is not necessarily easy to simulate and thus can be easy to overlook when testing XYZ application in the lab. Later on you find out the hard way in production that XYZ application isn’t bandwidth friendly or works poorly over slower WAN links. Wouldn’t it be nice to find out by testing this in the lab instead of finding out when it’s too late in production? Let’s see how it works!

Here we have a diagram of what the Active Directory architecture looks like. Two existing sites, Fremont (Minnesota) and California. These two sites are on two different subnets connected by a router and a VMware traffic shaped link that allows a maximum of 1.544Mbps of bandwidth each way. The router, named calirouter, is a multihomed Windows Server 2003 host running Routing and Remote Access Services (RRAS). Obiwan is an AD domain controller in the Fremont Site. Bobafett is an AD domain controller in the California site. Both are bridgehead servers in their sites and are connected by an IP site link. For good measure, I’ve thrown in some placeholder sites for future expansion to New York and Tel Aviv.

A cost of 900 has been assigned to the CALI IP SITE LINK. After all, this is just a T1:

A Windows Server 2003 machine named calirouter is running RRAS with RIP enabled on both routing interfaces. This router connects the Fremont and California sites. You will see shortly how this router will be the key to shaping the bandwidth between the two sites.



In the screenshot below, I’ve configured traffic shaping on the calirouter virtual machine using VMware VirtualCenter. I’m simulating a T1 connection by allowing a maximum of 1,544Kbps of outbound traffic on each of its two NICs.

If you don’t have VMware VirtualCenter, the same configuration can be made using the VMware ESX MUI:

I want to pause here for a moment and clarify an important fact about traffic shaping with VMware ESX. Traffic shaping can only be applied to outgoing traffic. Incoming traffic is not shaped or throttled in any way. This is important to remember when designing a traffic shaping strategy with VMware ESX. Outbound traffic can be controlled and inbound traffic can not. This works perfectly on a multihomed virtual machine acting as a router. Although incoming traffic to router interface 1 cannot be slowed down to simulate a slow WAN link, the routed traffic heading outbound on router interface 2 can. The reverse also applies. Traffic inbound on router interface 2 can not be throttled, but the resulting traffic heading out router interface 1 can. In this configuration, we’re shaping bandwidth in both directions.

VMware ESX’s flexibility to dynamically allow adjustments of virtual resources also let’s us change the traffic shaping properties on the fly. If we wanted to quickly increase our T1 pipe to California to a T3, we’d increase the average and peak values of 1,544Kbps to 45,000Kbps. Burst Size can be set to whatever you’d like it to be, however just be aware of what burst size is and how it impacts bandwidth. Burst size is the contiguous amount of data transfer allowed to exceed peak bandwidth before being capped. Don’t allow too much burst to skew your testing results.

In the following screenshot, I’m logged on to the Bobafett console in California copying a 180MB file named Hot Rod.VCD from the Fremont site in Minnesota to the California site. In tandem, I’m also capturing network performance statistics with the protocol analyzer Microsoft Network Monitor. Notice the maximum value “Bytes Per Second” of 208,078 falls right in line with the performance of our T1 circuit. The maximum throughput of a T1 circuit in terms of Bytes Per Second is 193,000. The slight overage can be attributed to bursting. Also notice the length of time to copy this file is around 45 minutes. In typical Explorer shell fashion, the copy time displayed jumped around quite a bit during the file copy.

During the same file copy, here’s a look at the VMware ESX MUI while we examine the network traffic shaping property sheets of both the calirouter and bobafett VMs. For some reason, the reported bandwidth numbers are about half of what they should be. A bug maybe. The Traffic Shaping configuration has also decided to round itself up from 1544Kbps to 2Mbps in the display. Not to worry though, VirtualCenter (not shown) was still showing the traffic shaping setting at 1,544Kbps:

While all the VMs are truly located in my basement, Bobafett and Calirouter are both configured for the west coast Pacific time zone. In addition to simply changing it’s time zone, I have been able to simulate the mileage between sites by using traffic shaping.

This is just one of many uses of traffic shaping with VMware ESX Server. You don’t need slow WAN links to simulate slow WAN links or networks with heavy utilization. A little imagination helps too! Anyone know a good English/Hebrew translation dictionary?

Author Bio:

Jason has worked as a Systems Engineer professionally for over 10 years, the last 8 years with a large bank with 150,000 users. His main areas of interest and expertise are Microsoft Windows (including Active Directory, Group Policy, DNS, IIS, and Exchange), VMware ESX, Citrix, and some Linux.

Jason is a VMware forum moderator on the forums as well as a moderator on the VMware VMTN community discussion forums.  He is the Minnesota area VMware User Group Leader.  The Minnesota VMUG is a VMware, Inc. chartered special interest group consisting of several hundred members who meet quarterly and discuss computer and server virtualization using VMware technology.

Jason holds the following certifications:  MCSE NT4/2000/2003, MCSA 2003, MCP+Internet, MCP, VCP, CCA, A+, and can also be found on other technology forums.