|
Tuesday, February 6, 2007
What we're about to tell you is something nobody has ever told you before. Till now, whenever any magazine (including us) carried any shootout, aspects covered were performance results of products, how the products compared against each other, their pros, cons, etc. There's also a broad mention of the test procedures that were followed by the various features that were compared. More stress is given on the actual product reviews and the test results and less on the actual tests and how they were run. But what we're going to tell you this time is completely different. We'll let you in on the actual test procedures and all the excitement and heartaches we've experienced; and the learning that followed while doing this shootout. Instead of focusing on product evaluations alone and declaring a winner, we'll focus on uncovering how we actually ran the benchmarks on those products. And then, we'll tell you how to interpret their results, and finally we'll reveal the results we got on the NAS boxes. Why are we doing this, you might ask.
For one, you'll be able to better understand the results we received for the NAS boxes. Two, if you ever need to, you can run those tests on your own storage boxes to find out their performance. And three, if you're planning to purchase a new NAS box, then you can even ask the vendors to conduct similar tests on their boxes and show you the results. In other words, we're empowering you with information to help you make a much better purchase decision now and in the future.
We've spent more than three months in testing the NAS boxes we received for evaluation. During this phase, we learnt a lot of new things, not only about performance, but also about features, vendor support issues, pricing models, and much more. So presented in this story therefore, is a complete buying guide to NAS boxes, with our personal learning. With this, you'll not only be able to decide whether or not to buy one of the four NAS boxes we tested, but any NAS box that's out there.
Choosing benchmarks
This is easier said than done. On the face of it, we ran just two benchmarks-Netbench and IOmeter. But when we actually got down to doing it, we ended up creating twelve different test mixes from this duo for testing various aspects of the NAS boxes. NetBench at one time was given away free of cost by Ziff Davis Labs of US. You just had to bear the shipping and handling charges for the CD. Though they've discontinued doing this, you'll still find it available for download at various websites.
The benchmark provides a lot of flexibility and control over the test setup and let's you create a real live scenario for testing the throughput and response time of NAS boxes and servers.
The other benchmark, IOmeter is also free and can be downloaded from http://iometer.org. There are dozens of different metrics to adjust while running this benchmark and it can reveal the true potential of any storage device. It measures the number of I/Os per second of the storage device and also gives the throughput in Mbps. You can also easily run these benchmarks in your setup, besides doing the regular file transfer tests.
 |
The test environment
Just like any other network device testing, this one also requires an isolated network. Many organizations already have a separate test network that they use for testing anything new before incorporating it into the production network. If you don't, then maybe it's time you also created at least a basic setup. What you will need though is a Gigabit Ethernet switch and a few PCs that will act as the client load. The more PCs you have the better. We used 19 PCs in our test setup, and a 20th machine that acted as the controller for the remaining machines. On top of this of course, you could use a KVM over IP switch to manage all the nodes remotely.
| Our observations |
| 1. NetBench throws a lot of I/O requests to the NAS box through multiple clients. It records how quickly is the NAS box able to respond to these requests. At the end, it just maps them on two graphs. One shows the maximum throughput for each client set, and the other shows the associated response time. So you can easily tell not only the peak throughput delivered by a NAS box, but with how many clients did it deliver the same.
2. We tested the NAS boxes with network load balancing enabled and found that the performance actually went down instead of improving. This could possibly be because of the RRDNS process, which is balancing the load.
3. The I/O performance of a NAS box changes based on the volume size you use. As volume size increases, I/O performance reduces.
4. Never run performance tests on a production system. If you must do it, then you must first put it on an isolated network and ensure that there are no users accessing it.
5. If your NAS's performance suddenly deteriorates, check all HDDs. In RAID 5, even if a drive fails, the NAS continues to run, but with slower performance.
6. Rebuilding a RAID with a new disk takes 2 to 3 hours depending upon the RAID array size.
7. Perform network backups from the NAS after office hours, or when the network is free.
|
Understanding NetBench
NetBench uses a number of PCs to generate file I/O requests to a shared location on the NAS. It loads the NAS slowly, starting with one PC and gradually continues to add more as per your test definition until it reaches the maximum defined load. You can also specify how long it should run and many other things (See screenshot). It gives you graphs of the I/O throughput and average response time at various load points. If you've loaded the NAS sufficiently, then there will be one peak throughput point, beyond which the throughput will start declining.
Setting up NetBench
The first thing you need to do is create a shared directory on your NAS box with full read/write access to everyone. Connect it to the Gigabit Ethernet switch. Then install the NetBench Controller program on a PC running Windows 2000/2003 or XP. You will define all benchmarks and control the entire test process from here. Next, you need the PCs that will send requests to the NAS box. These should ideally be of the same spec and preferably P4 machines running Windows XP. Install the NetBench client software on them. You can add up to 60 PCs to this load. Each PC can emulate several PCs, so you can simulate a much bigger load. Note that all machines have to be assigned an IP address manually. No DHCP please! Before starting the test, you need to configure the computer running the controller program. First, change the computer name to “controller”. Then open and modify the client ID file 'client.cdb' from CONTROLLER_DIR>\CLIENTIDS. Here you'll find an entry for each client in your test bed with an associate ID number. Add each client's IP address and a unique ID number and save the file. On all clients, you need to map the shared folder on your NAS to the drive letter f:.
Running the tests
First run NetBench client on all the PCs. Then go to the controller machine and launch the controller program. Here, from the menu, select Clients>Start Log in option to connect all client machines to the controller. Then add the tests you want to run from Suit>Add. Here, open the DM.tst. Next, from the tool bar select the “Go” button to start the test. The default test takes about an hour to execute, but you can modify it as per your requirement. Let's see how to do that.
Modify NetBench tests
From the controller window choose Suites>Create/Edit Test Suites. Open DM.tst. You will see many test mixes (see screenshot on p.103). Each client represents one PC and each engine will run a test mix. So two engines on a client means it's going to run the same test mix twice, i.e. simulating the workload of two PCs. Note that there's no ideal mix. You have to define the right one as per the number of clients in your test setup and the device under test. We, used 7 test mixes, which gradually increased the load on the NAS from one client to 18 clients running 4 engines each, meaning 72 clients. Each mix took about 3 minutes to complete.
Analyzing Results
NetBench automatically creates a full test report with graphs showing throughput and response time for each test mix. From this, you can easily tell the maximum throughput and the number of clients and engines used to achieve it. You can compare this with reports for other NAS boxes to determine which one delivers higher throughput and with how many clients. All test results are converted into an Excel file by NetBench for easier analysis.
NetBench & file compression
Many NAS boxes support compressed folders, wherein any data stored on the NAS is compressed on the fly to give you more storage space. The compression does of course take a toll on the NAS's CPU cycles. What has to be checked is whether this affects the overall performance.
 |
Using IOmeter
IOmeter is a widely used I/O performance analysis benchmark, specifically meant to test storage devices. IOmeter consists of a workload generator called Dynamo to do I/O operations and IOmeter itself to measure the I/O operations .The good thing about this benchmark is that there's a whole range of test mixes you can create. IOmeter is available on a variety of Operating Systems, including Windows, Linux, and even NetWare. You can even test a NAS box by running IOmeter from a single client. Installing IOmeter is fairly simple. You can install it on multiple client machines as well if you want, and in fact you should do that if you really want to stress the storage device. The only problem is that there's no central controller from where you can control all the IOmeter installations. You have to manually go to each machine to run it. But, like in NetBench, you need to create a shared folder on the NAS box and map it to all clients running IOmeter.
When you start IOmeter on both clients, Dynamo is automatically started. Here, look for the topology window and click on all the managers to expand the menu. Here you will see a default manager having with the same name as your client machine. By default it has two workers. This is similar to NetBench's client instances. In our tests we used four workers on every client. You can have up to 100 workers under one manager. Next look for the Disk Target tab. This shows you all available drives, including the mapped drive on the NAS.
Select it for all workers. Here, you can also specify the maximum disk size to be used by IOmeter. By default this value is set to zero meaning
the entire disk will be used by IOmeter.
Simple file copy benchmark
Even if you can't these benchmarks, the easiest thing to do is a simple file transfer. We took about 100 GB of documents, spreadsheets, presentations, PDFs, and even MP3s and graphics files. We copied them to a shared folder on the NAS and measured how long it took. We even created an ISO image of this data and copied it to the NAS. This gave the NAS a continuous stream of data to store, and told us the maximum throughput that the NAS box is capable of delivering to a single machine.
|
