pipelines

Create a MemSQL Pipeline for Apache Kafka in 5 minutes

Creating an IoT Kafka Pipeline in Under Five Minutes

In a recent MemSQL webcast, we discussed how modern enterprises can easily adopt new data management tools to manage data size, growth, and complexity. Then we demonstrated how to use Apache Kafka and MemSQL to build interactive, real-time data pipelines. Data pipelines capture, process, and serve massive amounts of data to millions of users. During the webcast we also shared how to: Build new data pipelines with modern tools Enable data workflows to support machine learning and predictive...


blog header

MemSQL 201: Tips & Tricks Webcast

In a recent webcast, we shared tips and tricks for understanding MemSQL, best practices for implementation, and demoed MemSQL with a real-world use case. Here are five top tips and tricks we shared: When moving an application to, or creating one on MemSQL, start by thinking about whether rowstore or columnstore storage (or both) is ideal If you have high throughput requirements, consider using MemSQL Pipelines Use EXPLAIN and PROFILE operators to identify query bottlenecks Take advantage of...


1.3 Billion NYC Taxi Rows into MemSQL Cloud

Experience teaches us that when loading data into a database, in whatever form ― normalized, denormalized, schema-less, hierarchical, key-value, document, etc ― the devil is always in the data load. For enterprise companies in the real-time economy, every second saved means improved efficiency, productivity, and profitability. Thankfully, MemSQL Cloud makes your enterprise data fast to load and easy to access. You can spin up a cluster in MemSQL Cloud in minutes and load data very quickly...


Amazon S3 Real-Time Analytics

Turning Amazon S3 Into a Real-Time Analytics Pipeline

MemSQL 5.7 introduces a new pipeline extractor for Amazon Simple Storage Service (S3). Many modern applications interface with Amazon S3 to store data objects into buckets up to 5TB providing a new modern approach for today’s enterprise data lake. Without analytics, the data is just a bunch of files For modern enterprise data warehouses, the challenge is to harness the unlimited nature of S3 for ad-hoc and real-time analytics. For traditional data warehouse applications, extracting data from...


Exactly-Once Semantics

Getting to Exactly-Once Semantics with Apache Kafka and MemSQL Pipelines (Webcast On-Demand)

The urgency for IT leaders to bring real-time analytics to their organizations is stronger than ever. For these organizations, the ability to start with fresh data and combine streaming, transactional, and analytical workloads in a single system can revolutionize their operations. When moving from batch to real time, data architects should carefully consider what type of streaming semantics will optimize their workload. The table below highlights the nuances among different types of streaming...


MemSQL Pipelines

MemSQL Pipelines: Real-Time Data Ingestion with Exactly-Once Semantics

Today we launched MemSQL 5.5 featuring MemSQL Pipelines, a new way to achieve maximum performance for real-time data ingestion at scale. This implementation enables exactly-once semantics when streaming from message brokers such as Apache Kafka. An end-to-end real-time analytics data platform requires real-time analytical queries and real-time ingestion. However, it is rare to find a data platform that satisfies both of these requirements. With the launch of MemSQL Pipelines as a native feature...