Advertisment

Quantcast releases QFS project to open source

author-image
CIOL Bureau
Updated On
New Update

SAN FRANCISCO, USA: Quantcast, the company that's harnessing big data to deliver audience measurement and targeting solutions has announced the release of the Quantcast File System (QFS) to open source.

Advertisment

Evolved from the Kosmos Distributed File System (KFS, also known as CloudStore), QFS offers a higher performance alternative to the Hadoop Data File System (HDFS) for batch data processing, improving data I/O speeds and halving the disk space required to reliably store massive data sets, said a press release.

Fully integrable with Apache Hadoop, QFS has been live at Quantcast for four years, reliably handling petabyte-scale production workloads.

"Quantcast houses one of the world's largest data sets and understands the challenges and opportunities of big data in a way that few organizations can. The sheer size and scale of its data set easily positions the company among the top five data processing organizations in the world, and its success leveraging QFS in production offers a compelling use case," said Ben Woo, principal analyst and managing director at Neuralytix.

Advertisment

KFS was designed to provide high-performance back-end storage infrastructure for batch processing of large data sets. Released to the open source community in 2007, KFS demonstrated the advantages that a native distributed file system implementation similar to HDFS could deliver, but was fundamentally experimental and insufficiently stable for production usage.

"I am thrilled that the file system the Quantcast team and I built has reached this level of maturity. As we discovered, achieving production reliability at a multi-petabyte scale on thousands of nodes is a long, difficult effort," said Sriram Rao, originator of the KFS project and Principal Scientist Lead at Microsoft.

"It's a rare open source project that provides this much value and has such a track record in a tough production environment. I'm delighted the broader community will be able to benefit and continue to evolve it," added Rao

"Quantcast makes use of open source software and by making our own contribution with QFS we're hopeful that others will benefit as we have, and that community collaboration will enable QFS to meet the production demands of big data environments for years to come," said Konrad Feldman, CEO at Quantcast.

tech-news