IBM SSD system scans 10bn files in 43 mins

SAN JOSE, USA: Researchers from IBM demonstrated a storage system that can scan 10 billion files on a single system in just 43 minutes, breaking the previous record of one billion files in three hours by a factor of 37 at the Supercomputing 2007 conference in Reno, NV.

Growing at unprecedented scales, this advance unifies data environments on a single platform, instead of being distributed across several systems that must be separately managed. 

Also Read: IBM's new memory tech 100 times faster than flash

It also reduces and simplifies data management tasks, allowing more information to be stored in the same technology, rather than continuing to buy more and more storage, claims the company.

The system uses IBM's General Parallel File System (GPFS) running on a cluster of 10 eight core systems and solid state storage, taking 43 minutes to perform this selection.

IBM says that GPFS's advanced algorithm makes possible the full use of all processor cores on all of these machines in all phases of the task (data read, sorting and rules evaluation). GPFS exploits the solid state storage appliances with only 6.8 terabytes of capacity for excellent random performance and high data transfer rates for containing the metadata storage.

"Today's demonstration of GPFS scalability will pave the way for new products that address the challenges of a rapidly growing, multi-zettabyte world," said Doug Balog, vice president, storage platforms, IBM. "This has the potential to enable much larger data environments to be unified on a single platform and dramatically reduce and simplify data management tasks such as data placement, aging, backup and migration of individual files."

"Businesses in every industry are looking to the future of storage and data management as we face a problem springing from the very core of our success — managing the massive amounts of data we create on a daily basis," said Bruce Hillsberg, director of storage systems, IBM Research — Almaden. "From banking systems to MRIs and traffic sensors, our day-to-day lives are engulfed in data. But, it can only be useful if it is effectively stored, analyzed and applied, and businesses and governments have relied on smarter technology systems as the means to manage and leverage the constant influx of data and turn it into valuable insights."