BANGALORE, INDIA: Old style database management systems (DBMSs) are premised on disk storage: some records (usually the most recently used) are held in cache for faster access, but changes are written through the cache, to permanent storage.

But real-time applications’ growing need to manage large volumes of complex data has ushered in a new technology: the in-memory database system (IMDS), which stores data in main memory. IMDSs never go to disk; as a result, they deliver performance that is an order of magnitude faster than traditional (“on-disk”) database systems.

IMDSs emerged in the past decade. Currently, five to 10 vendors specialize in the technology, which is used most widely in telecom, industrial control, financial trading and other real-time applications. Because IMDSs are new, they are less well understood than conventional DBMSs, even by IT and software professionals.

Difference between IMDSs, and on-disk databases; Storage mode (disk vs. memory) is one difference, but there are others, such as elimination of caching logic and file management in IMDSs.

IMDS vs. database caching: aren’t both “in memory”? Caching only speeds up database “reads” without affecting “writes”; elimination of data transfer within the database system is another key difference.

Can’t the same results be achieved by placing a conventional DBMS in RAM? RAM-disk deployment does, indeed, improve on-disk databases’ responsiveness, but does not remove performance overhead to the extent seen in IMDSs. Benchmark results are available.

“Footprint” implications; Because, IMDSs eliminate multiple processes related to disk, file and cache management, their architectures are often streamlined, requiring less CPU and processing power. This small footprint is a major reason for their adoption in resource-constrained consumer electronics and other embedded systems.

Persistence; Won’t all data in an IMDS disappear in a system crash? Only if no safety measures are in place. IMDS persistence options include transaction logging, replication/clustering, non-volatile RAM (NVRAM), and hybrid (in-memory/on-disk) architectures.

Scalability; The need for performance in enterprise-scale systems coupled with the decreasing RAM cost, have made IMDSs of hundreds of gigabytes, and even terabytes, a reality. One benchmark of a 1.15 terabyte, 15 billion row IMDSs documented performance of more than 87.78 million query transactions per second using a native application programming interface (API) and 28.14 million transactions per second using a SQL ODBC API.