Xperf Rocks Part 1: Troubleshooting Storage Performance Problems

Xperf is part of the Microsoft Windows Performance Toolkit (WPT for short) which includes several other software development tools.  This article will introduce Xperf as a troubleshooting tool, covering the tool installation and data collection process.  It will focus on storage related troubleshooting, but can be leveraged for all aspects of performance issues.

Troubleshooting Windows storage performance issues can be a very challenging endeavor.  You can get a cursory overview of the system performance by using the Task Manager, or a more in depth look by using Perfmon, as discussed in the Windows troubleshooting tools article, which covers a variety of free tools for troubleshooting Windows storage performance problems.  Or you can use PAL, the Performance Analysis of Logs tool, to automate the analysis of Perfmon logs (see Performance Analysis of Logs article) and the generation of PAL reports (see Performance Analysis of Logs Reports article). However, there is one crown jewel from Microsoft’s toolbox treasure trove that has not received much attention, a tool called Xperf.  This tool allows you to dig deeper into performance issues than Perfmon and PAL ever dreamed.

Xperf Installation

Xperf is a free download kit that is part of the Microsoft SDK (Software Development Kit).  It capitalizes on ETW (Event Tracing for Windows) which has been built into the operating system since Windows 2000.  In addition to storage related problems, Xperf can be used to troubleshoot other issues such as CPU bottlenecks, application responsiveness, slow boot times and service startup issues.

Before you can install Xperf, you must have the Microsoft .NET Framework 4 installed.  The next step is to install the Microsoft SDK, but you only need to install part of the kit for WPT (see figure 1).  You can also specify that a redistribution package be created so you can install WPT/Xperf on other servers without having to install the SDK.  WPT (Xperf) will be installed in the folder:

C:\Program Files\Microsoft Windows Performance Toolkit

And the WPT kits will be staged in the folder:

C:\Program Files\Microsoft SDKs\Windows\v7.1\Redist\Windows Performance Toolkit

There is a Microsoft SDK web page to assist you with any problems you may encounter when installing the SDK, such as deleting all the temporary files in the folder \AppData\Local\Temp.  The Windows Performance Toolkit program group will be created as part of the installation with the Performance Analyzer shortcut created for Xperf.

Xperf Installation
Figure 1: SDK Installation Includes the Windows Performance Toolkit

Xperf Traces

Once you install WPT on the problematic server, use the Xperf CLI command to start a trace.  This will collect ETW data in an ETL log file (Event Trace Log file).  There are several pre-canned traces called Kernel Groups (KG) you can choose from depending on what problem you are attempting to fix.  Each of the KGs contains traces from a variety of operating system providers.

You can list all the KGs with the command:  xperf –providers KG as seen in figure 2 below:

Xperf Traces
Figure 2: Xperf Provides Pre-Defined Kernel Trace Groups for Collecting Events

If you are concerned about a storage I/O bottleneck, then you would want to include the DISK_IO provider which is included in many KGs such as Base, Diag, DiagEasy, Latency, FileIO and IOTrace as seen above.  All of the providers are documented with complete online help by accessing the help file, WindowsPerformanceToolkit.chm, in the WPT folder.

For the purpose of example, the following Xperf command uses the FileIO KG to start a trace:

Xperf –on FileIO

While the trace is running, it is storing events into non-paged pool buffers and then flushed to an ETL file.  By default, traces are collected in a trace file called Kernel.ETL (this can be overridden with the –f option to specify a different log file name).

Once you have collected several minutes of data that reflect the storage issue, you can stop the trace.  The following command will stop the current trace collection and merge the data to a log file called tracedata.etl:

Xperf –d tracedata.etl

Summary

You are now ready to analyze the ETL data with Xperf and generate graphs and tables that will help you to pinpoint your storage bottleneck.  Stay tuned for part 2 of “Xperf Rocks” when you will learn how to use Xperf to generate graphs, tables and more.