In-Memory Database Survey

To shed light on the state of the in-memory database market, we conducted a survey on the prevalent use cases for in-memory databases. Respondents included software architects, developers, enterprise executives and data scientists1. The results revealed a high demand for real-time capabilities, such as analytics and data capture, as well as a high level of interest in Spark Streaming.

Real-Time Needs for In-Memory Databases

It is no surprise that our survey results highlight real-time analytics as the top use case for in-memory databases. For years, big data was heralded as the future of technology – today, it is a reality for companies big and small. Going real-time is the next phase for big data, and people seek technologies that address real-time data needs above all else. Those who can successfully converge transactional and analytical data processing, see greater efficiency in data management and have an invaluable advantage over their competitors.

The Rise of Spark

Looking at in-memory market trends, we asked our respondents which technologies they plan to evaluate in the upcoming year. Apache Spark takes the cake, with more than 50% of respondents planning to evaluate the data processing framework in the next year.

According to our survey, Spark Streaming is the most widely adopted Spark library. This  shift in popularity from Spark SQL to Spark Streaming is good news for companies like Pinterest, who utilize Spark capabilities to structure real-time data up to the last click. By combining Spark Streaming with other memory-optimized technologies, like MemSQL, Pinterest can measure user engagement and developing trends in real-time.

Challenges and Solutions

In many ways, the purpose of an in-memory database is to provide a fast, persistent solution to big data challenges. The final, open-ended, portion of our survey asked our respondents what kinds of big data challenges are currently causing them the greatest strife. Popular answers include log processing bottlenecks, subpar performance, high costs, data migration from legacy systems, and complex queries for analytics. The utility and value of a database engine depends on its ability to provide solutions these common challenges. At MemSQL, we have spent the past four years iterating to offer an in-memory solution that gives users flexibility, agility, and security, with features like:

JSON Support
As Chris Preimesberger notes in an eWeek article, the MemSQL real-time, big data analytics platform is the first to combine structured and semi-structured data in a single database with JSON analytics. This integration saves users time and dollars spent on other middleware solutions.

MemSQL Loader
The MemSQL Loader is a productivity tool that allows for more efficient and streamlined data ingest, like from Amazon Web Service (AWS) or Hadoop Distributed File System  (HDFS). Data ingestion is often a cumbersome and complex. The MemSQL Loader addresses this challenge by allowing direct streaming from the datastore in just one transfer, and is capable of supporting multiple parallel input streams. The result: increased performance and minimization of repetitive operations.

Spark Connector
The MemSQL Spark Connector, which unites Spark’s in-memory data processing framework with MemSQL, maximizes operational data with highly advanced analytics. Taking advantage of Spark’s capabilities means top notch analytics and high-performance parallel throughput.

Converged Processing
HTAP, or hybrid transaction and analytical processing, enables MemSQL customers to merge transactions and analytics into a single database system. HTAP facilitates analyzing large volumes of data without the need for separate data marts or warehouses. Because transactions and analytics are combined in a single database, users are able to aggregate and report on real-time data in a more advanced manner than with traditional architecture

To download MemSQL Community Edition for free, visit www.memsql.com/download.


1. Market Outlook for In-Memory Databaeses – 67 survey respondents