One million-plus results certainly sound impressive, but do these tests provide any tangible value to businesses that simply have work to get done? The bottom line is that any test can be skewed in favor of a vendor's product. The test results, while important, are not as important as how the tests were conducted. The good news is that some of this data can be extrapolated and applied to your environment.
No test environment or benchmark can perfectly simulate your unique environment. The one area of critical importance is to determine whether the vendor's test was designed to show not only a high water mark in performance but also how well their solution fared under less ideal conditions.
For example, in hybrid flash arrays, systems consisting of solid-state drive (SSD) and rotational hard disk drives (HDD), it is very difficult to simulate a "cache miss" (when data is no longer in SSD and has to be retrieved from HDD storage). As a result, most vendor tests do not incorporate a methodology that reproduces what happens to I/O performance during a cache miss.
In fact, some hybrid flash vendors actually go to the other extreme and generate their test results by using an all-flash configuration -- hardly emblematic of how these systems will get deployed into a real-world data center. Consequently, it is important for end users to understand that the conventional disk storage residing behind a front-end flash pool is equally, if not more, critical for sustaining acceptable levels of performance.
Check the specs
If a particular benchmark study did use a combination of flash modules along with HDD, examine the configuration of the drives that were used. For instance, if a very high count of high-RPM disk drives were configured and then short-stroked (configuring the drives to only use the outermost portion of the disk platter) to enhance performance, this could be a non-starter for some buyers because of the inflated costs of such a solution.
To diminish the latencies inherent in a slow disk back end, some vendors have implemented a hybrid array approach, which is a combination of flash and conventional disk. The challenge is that there are significant performance differences between flash and conventional disk drives. Flash-enabled workloads will perform orders of magnitude faster than comparable workloads on conventional disk drives. The question is do vendor benchmark tests point out these differences or do they merely represent a best-case scenario. In other words, how does the array perform when all I/O is being served up by flash storage capacity?
The suggestion is that the user can expect the mean average of the I/O performance of flash and conventional disk. This can be extremely misleading, particularly when a hybrid array is configured with a relatively small cache in front of low-speed 7.5k RPM disk drives. Therefore, it is very important to understand how the hybrid array vendor generated its performance numbers. Avoid performance numbers when they lock the test workload in flash because that doesn't relate to real-world use cases.
Be sure to scrutinize all-flash array vendors' benchmark performance tests before making a purchase decision. Flash array products ship with SLC flash modules (high performance/cost), MLC modules (lower performance/cost) or a combination thereof.
Like the hybrid array example discussed above, it is important to understand how a flash manufacturer arrives at its benchmark test numbers. Make sure the performance benchmarks that you are using to make your decision reflect the same flash configuration you are going to buy. In other words, don't make an MLC flash array purchase based on the benchmark results of an SLC flash array test.
Watch the workload
Another essential area of scrutiny is the read/write mix, especially on flash storage where the performance delta between read and write operations is considerable. Unless your workload is 100% reads, then a 100% sequential read benchmark is of little value to you. By the same token, a 100% write workload is unrealistic. In short, look for benchmarks that provide a range of results and test a variety of workload mixes.
The other aspects of flash vendor performance tests that deserve a closer look are the ways the vendor generated a performance load. Was it a database stress test? Did it emulate end user desktop activity, or did it use a command-line I/O generating utility? None of these are bad; they are just different.
While workload-specific tests like a database stress test or VDI simulator have captured a lot of attention lately, even these tests can be problematic when testing flash workloads. The fact is flash is so fast that sometimes it exposes weakness in the benchmark itself. Both command-line testing tools and application simulators have issues; neither is perfect and their weaknesses need to be understood.
Benchmarks can't replace good design
The good news is that if hybrid and all-flash arrays are properly designed with the right complement of storage, end users can obtain high IOPS while limiting the downside risk of sluggish performance occurring as the result of a cache miss. Therefore, be sure to factor in the performance of each storage tier and to overcompensate for its shortcomings.
In the case of hybrid flash arrays, as the expression goes, you're only as strong as your weakest link. For this reason, the slowest data tier must be composed of a fairly robust disk back-end. For example, in some cases, it may be entirely suitable for the front-end cache to be modestly sized as long as the disk tier uses a large compliment of high-speed 15k RPM disk drives to mitigate performance degradation during a cache miss.
Alternatively, if the HDD tier consists of a few high-capacity drives, then the flash tier should be configured at a higher percentage of overall data. With the cost effectiveness of products from newer storage vendors, 100% of the active data could be stored in flash and deliver cache hit rates in the high 90s.
All-flash arrays are easier to benchmark and evaluate since the variable of multiple tiers is eliminated. In this case, it is important to understand the value of SLC and MLC flash. While MLC is slower than SLC flash, it is important to bear in mind that it is much faster than hard disk drive storage. In some cases, MLC may be the speed boost you need.
A way to get the best of both flash capabilities is to mix them. Some vendors are introducing a mixed MLC/SLC capability. This re-introduces a variable similar to hybrid SSD/HDD, but when used correctly, the two flash types can serve as a perfect complement for each other. Stratifying the array with a small complement of SLC modules for the most performance demanding I/O, like writes and very active reads, with a larger array of MLC flash for the remainder of the data, can be an ideal approach.
The bottom line is that, in IT, it is all about predictable performance. Wide deviations in performance will never be tolerated by end users, so plan for the worst-case situation and architect your storage solution accordingly.
Vendor storage benchmark testing can provide value provided they are properly vetted out and it is clearly understood what they represent. Most test results are designed to show the extreme performance limits of what a given storage platform can sustain, but most tests do not incorporate how well that same platform will manage the I/O latency associated with a data request from a slower storage medium (SLC to MLC or flash to HDD). Vendor benchmarks can add value, but they're just one component in the overall decision-making process.
This was first published in January 2014