Reflections on Event Streaming as Confluent Turns Five – Part 2
When people ask me the very top-level question “why do people use Kafka,” I usually lead with the story in my last post, where I talked about how Apache Kafka® […]
Streaming Data from the Universe with Apache Kafka
You might think that data collection in astronomy consists of a lone astronomer pointing a telescope at a single object in a static sky. While that may be true in […]
Deploying Kafka Streams and KSQL with Gradle – Part 2: Managing KSQL Implementations
In part 1, we discussed an event streaming architecture that we implemented for a customer using Apache Kafka®, KSQL from Confluent, and Kafka Streams. Now in part 2, we’ll discuss […]
Deploying Kafka Streams and KSQL with Gradle – Part 1: Overview and Motivation
Red Pill Analytics was recently engaged by a Fortune 500 e-commerce and wholesale company that is transforming the way they manage inventory. Traditionally, this company has used only a few […]
Kafka Connect Deep Dive – Error Handling and Dead Letter Queues
Kafka Connect is part of Apache Kafka® and is a powerful framework for building streaming pipelines between Kafka and other technologies. It can be used for streaming data into Kafka […]
The Simplest Useful Kafka Connect Data Pipeline in the World…or Thereabouts – Part 3
We saw in the earlier articles (part 1, part 2) in this series how to use the Kafka Connect API to build out a very simple, but powerful and scalable, streaming […]
The Simplest Useful Kafka Connect Data Pipeline in the World…or Thereabouts – Part 2
In the previous article in this blog series I showed how easy it is to stream data out of a database into Apache Kafka®, using the Kafka Connect API. I […]
The Simplest Useful Kafka Connect Data Pipeline in the World…or Thereabouts – Part 1
This short series of articles is going to show you how to stream data from a database (MySQL) into Apache Kafka® and from Kafka into both a text file and Elasticsearch—all […]
Building a Real-Time Streaming ETL Pipeline in 20 Minutes
There has been a lot of talk recently that traditional ETL is dead. In the traditional ETL paradigm, data warehouses were king, ETL jobs were batch-driven, everything talked to everything […]
Creating a Data Pipeline with the Kafka Connect API – from Architecture to Operations
This is a guest blog from Alexandra Wang, Software Engineer, Pandora Media. You can connect with her on LinkedIn. Also special thanks to Lawrence Weikum and Stu Thompson as contributors of the work in this post. […]