Staffers had researched a variety of solid-state storage options, ranging from SSD-only appliances to server-based NAND flash using conventional SAS or SATA interfaces and PCI Express (PCIe)-based solid-state cards. But HP's IO Accelerator proved the natural choice, as Answers.com is an HP BladeSystem server user. The PCIe-based IO Accelerator cards, custom-built for HP by Fusion-io, connect directly into the mezzanine slots in the HP blade servers.
Initial testing of the IO Accelerator cards produced results Marriott called "almost unbelievable." IT staffers configured two read, or slave, database servers -- one with IO Accelerator and another with the blade server's conventional SAS disk drive array. They restored the database to both servers and then timed how long it took each one to catch up on replication with the master, or write, database server. The read server without the card took a little more than six hours, while the server with the IO Accelerator card took 12.5 minutes.
"We rubbed our eyes in disbelief," Marriott said. "We did the test a few times, and we got the same results. We were seriously impressed."
Using IO Accelerator in production since last April, Answers.com has seen an increase of approximately 900% in the number of queries each read database server can process. According to Marriott, the servers delivered 350 queries per second prior to IO Accelerator and 3,500 queries per second after it. As a result, the company has been able to reduce the number of database servers in some of its read server clusters by 75%, lowering capital expenditures as well as operational costs.
"With some clusters, I haven't had to increase the numbers of servers for months and months because I still have so much headroom," he said.
In addition, Answers.com has been able to enhance its disaster recovery (DR) procedures now that the time to restore database servers has dropped from approximately eight hours to less than 60 minutes, thanks to the IO Accelerator cards, Marriott said.
Sound database design a key factor when implementing solid-state storage
But solid-state storage alone isn't a "magic bullet." Marriott noted the importance of sound database design to ensure that query constructs, reads and writes are efficient; CPU is adequate; RAM sizing is appropriate; caching of critical indexes and tables is judicious; and network I/O isn't creating bottlenecks. Although Answers.com didn't do any special work tailored to its solid-state storage, Marriott said the staff is constantly striving to improve database performance.
"We've been in the scalable website game in a big way since 2005, and we've been building and optimizing databases for many years before that," Marriott said. "As we've grown from humble beginnings, we're lucky enough to have some really clever people within our organization, including database experts."
Before moving to HP's IO Accelerator, Answers.com used either internal storage or conventional SAS disk arrays directly attached to its blade servers. The company still uses a direct-attached SAS array for its primary master databases, as well as a storage-area network (SAN) for other storage purposes.
"Of course, we have a SAN, but there's no way that I can afford to put all my database traffic across the SAN. I'm not going to get the performance I need. I don't think a SAN could offer anywhere close to the IOPS that the IO Accelerator can offer me," Marriott said. He noted that lab tests using the open-source Iometer tool confirmed IOPS in the range of 80,000 with multi-level cell (MLC) flash cards.
Overcoming early adopter difficulties with IO Accelerator drivers, performance
Installing IO Accelerator cards into the HP ProLiant BL460c blades was relatively pain-free, taking less than an hour the first time and approximately 10 minutes today, according to Marriott. He said staffers simply popped open the blade, installed the card, booted the Linux-based server, installed the drivers, partitioned and formatted the drives, and loaded the data. The company now uses a script for new installations.
"The driver installation within the Linux environment was just a piece of cake," he said. "It was ridiculously straightforward."
However, some early adopters running Windows and Microsoft Corp. SQL Server reported difficulties with drivers and performance. Rick White, Fusion-io's chief marketing officer, acknowledged that, for early Windows customers, "there was a whole lot of tuning that had to go on to support that product, and the performance was not as good as our Linux-based product." "That's not the case today," he claimed.
White said a "few beta drivers" had been "released out into the wild" prior to the late-summer volume shipments for Windows users. "We had a sales team -- which is how some of this broke out -- that felt like they had to get it in customers' hands so they could play with it because they didn't want to lose a sale," he added.
Answers.com's Marriott said the only problem his firm has encountered since April was a bug in an early firmware release that was corrected in a newer firmware release that it merely needed to install.
"We had no HP IO Accelerator hardware failures and suffered no data loss," he said.
Concerns over wear-out factor?
One concern with NAND flash-based solid-state technology is the wear-out factor associated with the erase/program process to write data to a chip, an issue that is more pronounced in MLC flash than in single-level cell (SLC) flash. Vendors typically build in extra gigabytes of raw capacity that can be substituted if a storage block wears out or fails, and they add mechanisms such as wear leveling, error correction and system checks to give users greater levels of comfort.
Answers.com has 320 GB MLC, 160 GB SLC and 80 GB SLC IO Accelerator cards in production, and has found all of them to be "extremely reliable," Marriott said. He added that he doesn't worry about the more vulnerable MLC cards because they have an extra 80 GB of raw capacity and sophisticated wear leveling. Plus, calculations show the cards should outlast the company's expected three-year replacement cycle for its production servers.
"These are redundant database slave servers that I'm using these IO Accelerator cards in. If one of these servers should die for any reason, where it blows a fuse or has some hardware failure, so what?" Marriott said. "I just replace it."
The IO Accelerator cards don't come cheap, especially in comparison with hard disk drives of similar capacity. The list price of the 320 GB MLC card is approximately $13,200 vs. $520 for a 300 GB 10,000 rpm SAS drive, Marriott said. But for him, the performance benefits make the technology worth the investment.
"I don't think it's suitable for everyone," Marriott said. "But if you're running a high-performance database and you're in the Web 2.0 space, like we are, and you have a large percentage of reads vs. writes, you'd be foolish not to look at PCI-based SSD drives to improve your performance. It's a revolutionary technology in my eyes."
This was first published in March 2010