This article can also be found in the Premium Editorial Download "Storage magazine: Flash technology debate: All-flash or hybrid?."
Download it now to read this article plus other related content.
Now that solid-state storage prices have dropped, more vendors offer all-flash arrays; but are they really much better than hybrid storage arrays that mix flash with spinning disk?
Comparing technology systems has traditionally relied on a "price-to-performance" analysis that tries to normalize system differences into an apples-to-apples comparison. With respect to storage, the advent of flash (solid-state) drives created a situation where IT buyers had to decide between price and performance. Flash storage offered blazing speeds, but at a very high cost per gigabyte ($/GB). At the other end of the spectrum, multi-terabyte hard disk drives (HDDs) are very economical, but with just around 75 raw IOPS per drive there better not be much work in the workload if that's where your data is located.
HDDs have an advantage in $/GB, while flash has an advantage in $/IOPS.
Hybrid arrays are intended to balance that equation. By adding a thin slice of flash storage to an array (i.e., 2% to 5% of total capacity), available IOPS may double and reduce read latency from 10+ milliseconds (ms) down to 3 ms to 5 ms. Even though the flash is expensive by itself, an overall 10% to 20% increase in array price to yield a 2X performance gain adds a lot of bang to the buck.
As good as 3 ms to 5 ms latency sounds, for an increasing number of applications this variability is unacceptable. All-flash arrays can deliver sub-millisecond read latency with a guaranteed quality of service (QoS). There are no pre-fetch issues with flash and no variability between a cached read and a seek because on flash all I/Os are effectively cache reads. To put it in perspective, improving from 10 ms to 5 ms and then down to 0.5 ms is a 20X performance gain from HDD to all-flash on a latency basis. Although the difference between HDD prices and flash has narrowed considerably, most organizations still don't have the budget to deploy hundreds of terabytes of it. So, if deploying flash technology judiciously is important to an organization, knowing where the breakpoints between hybrid and all-flash are will help managers to make the best decision.
Price/performance of hybrid storage arrays vs. all-flash arrays
With price-to-performance being the major consideration in storage purchases, let's examine those two areas first. When it comes to cost, it's an industry truism that the purchase price is approximately 20% of the three-year cost of ownership for conventional HDD arrays. All-flash storage arrays break this model, however, partly because the purchase price goes up but operating costs go down. For example, Nimbus Data, a maker of all-flash arrays, estimates that its arrays use just 8 watts per TB, whereas HDD arrays may use 80 watts per TB. Moreover, Nimbus advises that users can fully populate a rack with usable space and get up to 90% utilization without performance degradation. This offers the potential for less floor space, cooling and power per TB of storage. High-capacity SATA drives offer gigabyte density, but won't meet even moderate IOPS or latency requirements. Given that flash is following the downward curve of cache memory, it's well worth the time to compare the total cost of ownership (TCO) of all-flash and hybrid storage systems.
Hewlett-Packard (HP) Co. offers a different perspective on cost metrics: $/transaction. This metric is reflective of the true business cost, whereas $/GB and $/IOPS reflect purchase cost only. The $/transaction metric can be applied irrespective of media type, making a "non-denominational" comparison between techs. High-transaction workloads may actually see a lower $/transaction on all-flash arrays, whereas lower transaction environments may have a lower $/transaction on hybrid storage arrays. Either way, the result is calculable and definitive.
Although it may be considered a "soft" cost, IT managers should factor in the value of the user's experience. Justifiably or not, if users perceive an application to be slow, they're more likely to have a negative perception of the IT organization or provider. It's a bit like the adage that when airline passengers find coffee stains on their tray tables, they question the quality of engine maintenance. It may therefore make sense to spend a few extra dollars to improve the user's perception of quality, especially in this era of IT outsourcing.
Deduplication and compression are staples of nearly all storage systems and are frequently used to generate a $/GB figure that's much more favorable than those based on raw capacity. However, a cautionary note is in order. Increasingly, deduplication and single-instance storage (SIS) are taking place at the OS and application layers (i.e., VMware and SIS products for Exchange). Deduplication and compression can't occur twice on the same data, so storage managers may see less reduction than they anticipate as dedupe moves up the stack.
For the purposes of commercial computing applications, storage performance has been all about IOPS until very recently. Perhaps this was because storage arrays inherently couldn't guarantee a specific QoS. All-flash arrays change that game, and SolidFire promotes its flash arrays squarely as storage QoS delivery vehicles. SolidFire allows IOPS to be provisioned on a per-volume basis with minimum, maximum and burst parameters. This throttling, along with dynamic adjustment, gives storage managers an additional mechanism for precisely delivering performance where it's needed. SolidFire also suggests that guaranteed QoS solves the "noisy neighbor" problem. For example, in a typical hybrid array, applications may compete for the flash tier of storage based on data access, resulting in oversubscribed flash and sub-optimal performance for all applications. Allocating IOPS to each application specifically solves this problem.
Either or both?
Established vendors have extended their traditional HDD products to embrace flash technology, often with hybrid devices and all-flash units. IBM, EMC Corp., NetApp Inc. and HP each offer an "all of the above" portfolio, though the specific implementations differ in important ways.
IBM has extended its SAN Volume Controller (SVC) virtualization capability to manage "fit for purpose" all-flash devices such as the FlashSystem family or hybrid arrays in its XIV Storage System, Storwize and DS product lines. Because all devices can be managed through SVC, IBM extends the concept of hybrid to not only arrays, but the enterprise storage ecosystem. The intent is to allow ultimate flexibility in deploying both combinations of media, as well as combinations of arrays for scale and performance where needed.
EMC's VMAX can virtualize hybrid and all-flash systems into a single ecosystem. Its VMAX, VNX, VNXe and Isilon products can also be configured as either hybrid or all-flash. Thus, users can enter at any point and evolve the solution as requirements change over time. In addition, the company's XtremIO arrays offer an all-flash solution from the ground up. Because it doesn't necessarily have to separate its hybrid and all-flash offerings, EMC views the market more horizontally. That is, it can offer traditional HDD arrays to the more budget-conscious buyer, hybrid configurations to those needing more performance and all-flash for situations that demand guaranteed QoS. Which product line will be recommended is determined by Reliability, Availability and Serviceability (RAS) requirements, with customers needing up to "six nines" of availability choosing VMAX, regardless of hybrid or all-flash provisioning.
HP's 3PAR similarly offers everything from all-HDD to hybrid to flash configurations, but with some interesting wrinkles. HP offers both single-level cell and multi-level cell flash in the same arrays. The company recommends using the caching layer for writes and the flash tier for reads. This builds upon its Adaptive Optimization software that enables sub-LUN tiering. 3PAR's OS has a built-in clustered volume manager and virtual memory implementation that seamlessly virtualizes all media types, including new media.
NetApp offers flash across its entire portfolio of FAS, V-series and E-series arrays; the EF540 is an all-flash system. NetApp usually recommends Flash Pools in its arrays with 1% to 2% of total capacity in flash. The company suggests that typical workloads, such as email, Web serving, app dev and collaboration are best served with hybrid configurations. In these use cases, workloads require less than 150,000 IOPS and can tolerate 3 ms to 5 ms of latency. For higher IOPS workloads or where QoS demands sub-ms latency, all-flash is prescribed.
Oracle's ZFS Storage ZS3 are hybrid arrays, but the company says users may get up to a 90% data hit rate using its flash/DRAM architecture and sub-ms latency on those reads. Moreover, the company touts a significant cost advantage over competitive all-flash systems. Thus, users may get near all-flash performance at a hybrid price. Oracle's Hybrid Storage Pool dynamically and automatically moves data across DRAM, read-flash and write-flash to optimize array performance.
Flash devices need to be protected by RAID just like any other storage media. Although most vendors support conventional RAID techniques for flash, RAID use can extract a price in capacity terms as well as processing overhead just as it does for HDDs. IBM addresses this issue with "variable stripe RAID" in its FlashSystem technology where RAID 5 is built into the flash controller. The result is parity-based RAID at line speed with the workload distributed across controllers. NetApp deploys Dynamic Disk Pools in its SANtricity (E-series) software that distributes data, parity and spare capacity across drives. NetApp claims this speeds recovery of failed drives while maintaining greater performance. Nimbus uses a RAID 5 algorithm designed to avoid the wear penalty associated with writing to flash.
Some agreement on guidelines
Despite differences in architectures, the vendors generally agree on some hybrid vs. all-flash guidelines. First, if sub-ms or guaranteed QoS is required, then all-flash arrays are the way to go. Or, in the case of Oracle, a hybrid that can deliver near all-flash performance. QoS application candidates include e-commerce where user experience is paramount and decision support analysis where time is of the essence. Second, if variable and unpredictable workloads are to be serviced, hybrid devices can often serve the need at a lower $/GB. Nimbus, SolidFire and other all-flash vendors may have different answers to this rule of thumb that do indeed have interesting price/performance characteristics and may have a $/IOPS advantage. Application candidates in this area include collaboration, email and anything where data lifecycle issues mean that not all data requires immediate access.
About the author:
Phil Goodwin is a storage consultant and freelance writer.
This was first published in January 2014