Online marketplace eBay Inc. today revealed that it's using 100 TB of Nimbus Data Systems S-class solid-state storage, a deployment Nimbus’ CEO pointed to as evidence that flash storage is rapidly moving into the mainstream.
eBay began deploying Nimbus’ all-solid-state drive (SSD) system last February as the storage connected to its VMware environment. eBay uses Nimbus as part of virtualization pods along with Dell blade servers, Cisco switching and VMware vSphere.
Michael Craft, eBay’s manger of QA systems administration, said his group started with 10 TB “to get our feet wet,” and has gone back for more four times when adding applications to the VM stack.
eBay’s purchases have ranged from 5 TB to 60 TB, and it now has 12 S-Class systems with 100 TB as part of its multiple PB of storage under management.
eBay is using the 2U S-class systems with iSCSI connectivity over 10 Gigabit Ethernet (10 GbE).
“We want something that could scale at a moment’s notice,” Craft said. “Nimbus provides the performance we need, and the physical size lets us have a small data center footprint and keep it small.”
Craft said he would eventually like to have all solid-state storage connected to his virtualized servers, although he acknowledged that it would take a while to reach that goal. eBay’s other storage includes NetApp iSCSI SANs.
“For our virtualization stack, I don’t see a need for spindles,” Craft said. “I want to continue to deploy flash where appropriate and replace spindle arrays. I’m restricted in what I can purchase, so I will have spindles for some time, but I’d like to get rid of them."
Craft said before using Nimbus, he considered cost the main hurdle to solid-state storage, but Nimbus’ price tag put it on par with spinning disk arrays. Nimbus systems cost $24,995 for 2.5 TB, $49,995 for 5 TB and $99,995 for 10 TB systems. That includes its HALO operating system with management and data protection features built-in. The vendor no longer sells arrays with spinning disk.
Craft said he likes the way Nimbus keeps workloads even to take care of wear levels, and its reporting that tells him when he's coming close to his write-limit on a drive.
“We looked at all SSD solutions, even pure SSD systems from other vendors,” Craft said. “We wanted something that we could set and forget across all the LUNs all the time.”
eBay isn’t the only large online company to go big on solid-state storage. When Fusion-io went public this year, it disclosed that Facebook spent approximately $69 million on its PCIe-based flash cards over the last half of 2010 and the first three months of this year.
“We believe this is the largest flash deployment of its kind,” Nimbus CEO Tom Isakovich said of eBay. “It’s not a case of using flash for tiering or caching of hot data. This is production deployment of flash for primary storage. We’re showing that flash is mainstream and the acquisition cost is no longer a barrier to adoption.”
Nimbus today also said it increased the performance of the S-Class systems, adding more processing cores and bandwidth to its non-blocking mid-plane architecture. Processing cores have increased from 16 to 24, and the systems have 24 channels of 6 Gbps SAS interconnects, up from 3 Gbps SAS in the first generation. Nimbus claims its performance has increased to 800,000 IOPS per box and 16 million IOPS per rack. Previously, it topped out at approximately 250,000 IOPS per system.
It also now supports 6 Gbps SAS flash modules, and 8 Gbps Fibre Channel and 40 Gbps InfiniBand connectivity to go with the 10 GbE supported in the first-generation systems. The S-Class scales to 250 TB by stacking systems.
Nimbus manufactures its NAND flash modules and uses enterprise multi-level cell (eMLC) chips from Micron Technology Inc. to keep costs down. Isakovich said Nimbus storage costs $12.65 per gigabyte for hardware, RAID overhead, file system, software licensing and the first year of support.
“That’s not the kind of price that people think of when they think of solid-state storage,” said Jim Handy, chief analyst at semiconductor market research firm Objective Analysis. Handy said because Nimbus manufactures its own SSDs, it can “make sure its system-level stuff and SSD stuff play together.”
Handy said solid-state storage adoption has gone beyond the early adopter stage, and organizations that are more reluctant to try new technologies are starting to implement it as well.
A recent SearchStorage survey of more than 500 storage pros found that 49.6% are using solid-state.
“Some data centers will embrace a new technology, and others will be hesitant to change anything that could cause their systems to go down,” Handy said. “I think we’re at a crossroads now with solid-state.”