I recently used Microsft FSCT to load test a new Dell MD3220i iSCSI array. It took a bit of poking around to get going and I wanted to share my experience.
FSCT consists of three components:
- The Server being tested (FSCT-Server)
- The Clients used for testing (FSCT-Client)
- The Controller that manages the client\server interaction (FSCT-Controller)
A FSCT setup requires two networks: the ‘data’ network, and the ‘control’ network. The clients and server must have a minimum of 2 nic’s, one for the data network and one for the control network. The controller requires only 1 nic which must be on the control network. The two networks must reside in different subnets.
I used 192.168.0.xxx for the control network, and 192.168.10.xxx for the data network. Make sure that all devices on the same network can ping each other. I also enabled jumbo frames, flow control, and QoS priority on my nics and switch.
FSCT relies on DNS lookups. Using the hosts file on each system is the easiest way to satisfy this requirement. All client\server\controller systems in the test bed should have the control network IP’s and names of all the other systems in their respective local hosts file.
To install FSCT, simply extract the contents of the downloaded package into a folder on each system. There is no official installation procedure.
Preparing the Server
FSCT formats any volumes used for load testing during its ‘prepare’ stage. Make sure you have a clean volume ready to go with no needed data. In order to properly prepare the server, you must provide the following information.
- Volumes to use during testing (drive letters only, no mount points).
- Maximum number of remote ‘users’ that will be used for testing.
- A password you would like to assign to the ‘users’ that will be created on the server. In order to properly download results, it should also be the local admin password on the client, server, and controller.
- The computer names of all clients connecting (must be in hosts file with control network IP).
An example of the server preparation command is as follows:
fsct prepare server /users 5000 /password a1234567! /clients fsct-client-01 /volumes "E: F: G: H: I: J: K: L:" /workload homeFolders
Preparing the Client
To prepare the client you must provide the following information.
- A password for the users on the server. This must match the password in the step ‘preparing the server’.
- The server’s data network IP (for /SERVER_IP)
- The server’s computer name (must be in hosts file with control network IP)
- The maximum number of users you wish to make available to this client for testing.
An example of the client preparation command is as follows:
fsct prepare client /server fsct-server /password a1234567! /users 2500 /server_ip 192.168.10.10 /workload homeFolders
Preparing the Controller
This is easy; simply run the following command.
fsct prepare controller
The client command is straight-forward. The server command includes the ability to run multiple times with a different number of users per run. To start a single run, set min_users and max_users to the same number with a step of 1. To start a succession of runs, set a higher max_users and increase the step value as needed. The step value indicates how many users to add between runs. For example, if min_users is 1 and max users is 100 with a step of 1, FSCT will run 100 times. In the same example with a step of 50, FSCT will run twice. Duration is in seconds; 10-15 minutes is the recommended run time per Microsoft’s FSCT Users Guide.
- On the client, run the following command:
fsct run client /controller fsct-controller /server fsct-server /password a1234567!
- On the controller, run the following command:
fsct run controller /server fsct-server /password a1234567! /volumes "E: F: G: H: I: J: K: L:" /clients fsct-client-01 /min_users 1250 /max_users 1250 /step 1 /duration 900 /workload homeFolders
Returning the Output
On the controller, run the following command to gather the results. An output directory will be created at the path given.
fsct cleanup controller /backup C:\workingtemp\fsctbackup01
The output of the FSCT_data file should look similar to this:
Users Overload Throughput Errors Errors [%] Duration [ms]
1250 125% 52 44 0% 1055923
*** Test's information
FSCT version: 1.0
Time: 2011/11/02 17:46
*** Performance Counters
1 - \Processor(_Total)\% Processor Time
2 - \PhysicalDisk(_Total)\Disk Write Bytes/sec
3 - \PhysicalDisk(_Total)\Disk Read Bytes/sec
4 - \Memory\Available Mbytes
5 - \Processor(_Total)\% Privileged Time
6 - \Processor(_Total)\% User Time
7 - \System\Context Switches/sec
8 - \System\System Calls/sec
9 - \PhysicalDisk(_Total)\Avg. Disk Queue Length
10 - \TCPv4\Segments Retransmitted/sec
11 - \PhysicalDisk(_Total)\Avg. Disk Bytes/Read
12 - \PhysicalDisk(_Total)\Avg. Disk Bytes/Write
13 - \PhysicalDisk(_Total)\Disk Reads/sec
14 - \PhysicalDisk(_Total)\Disk Writes/sec
15 - \PhysicalDisk(_Total)\Avg. Disk sec/Read
16 - \PhysicalDisk(_Total)\Avg. Disk sec/Write
*** Server resources
Users CPU DiskWrite DiskRead Memory avg( 5) avg( 6) avg( 7) avg( 8) avg( 9) avg(10) avg(11) avg(12) avg(13) avg(14) avg(15) avg(16)
1250 35.0% 9938461.0 12517901.0 955.2 34.7 0.3 7152.0 2176.9 5.4 588.7 35680.3 295838.1 376.5 72.3 0.0 0.0
*** Client Resources (1250 users)
Name CPU DiskWrite DiskRead Memory avg( 5) avg( 6) avg( 7) avg( 8) avg( 9) avg(10) avg(11) avg(12) avg(13) avg(14) avg(15) avg(16)
fsct-client-01 0.0% 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
*** Label descriptions
Overload - server's overload in percent. For example if the return value is 900% it means
that to support the given number of users the server capacity must be increased
by 900% (so if there was 1 machine, 9 more are needed).
Errors [%] - number of errors / number of executed scenarios * 100%.
The value can be greater than 100% because multiple errors can occur during
a single scenario execution.
To ‘clean up’ the client and sever, run the following commands.
fsct cleanup server /users 5000 /clients fsct-client-01 /volumes "H: I: J: K:"
fsct cleanup client /users 2500
Getting Consistent Results
At first, my outputs varied widely. After researching the issue and re-reading TFM, I found this gem in the FAQ: “To achieve repeatable results, you must reformat the data volumes, recreate the file set, restart all of the computers (server, controller, and clients), and run a single iteration per run…You can run multiple iterations to investigate the maximum number for a configuration, but you should redo the testing as indicated to get a repeatable and reportable result “. This honestly makes sense because of the amount of caching involved in a file system.
Here’s a script to help out with prepping the server between runs. This is a destructive script in that it formats volumes without asking and will need modified for your environment. For it to work, the your volumes must be labeled “FSCT”.
First, create a text file named “format-override.txt” with the following contents.
Next, create a file named “prep-server.cmd” with the following contents:
fsct cleanup server /users 1250 /clients fsct-client-01 /volumes "E: F: G: H: I: J: K: L:"
type format-override.txt | format E: /q /X /V:"fsct"
type format-override.txt | format F: /q /X /V:"fsct"
type format-override.txt | format G: /q /X /V:"fsct"
type format-override.txt | format H: /q /X /V:"fsct"
type format-override.txt | format I: /q /X /V:"fsct"
type format-override.txt | format J: /q /X /V:"fsct"
type format-override.txt | format K: /q /X /V:"fsct"
type format-override.txt | format L: /q /X /V:"fsct"
fsct prepare server /users 1250 /password a1234567! /clients fsct-client-01 /volumes "E: F: G: H: I: J: K: L:" /workload homeFolders
shutdown /r /t 00 /f /c "prepping for fsct" /d P:0:0
Good luck and happy performance hunting!