Streaming Data from the Universe with Apache Kafka
You might think that data collection in astronomy consists of a lone astronomer pointing a telescope at a single object in a static sky. While that may be true in […]
Deploying Kafka Streams and KSQL with Gradle – Part 2: Managing KSQL Implementations
In part 1, we discussed an event streaming architecture that we implemented for a customer using Apache Kafka®, KSQL from Confluent, and Kafka Streams. Now in part 2, we’ll discuss […]
Deploying Kafka Streams and KSQL with Gradle – Part 1: Overview and Motivation
Red Pill Analytics was recently engaged by a Fortune 500 e-commerce and wholesale company that is transforming the way they manage inventory. Traditionally, this company has used only a few […]
Kafka Connect Deep Dive – Error Handling and Dead Letter Queues
Kafka Connect is part of Apache Kafka® and is a powerful framework for building streaming pipelines between Kafka and other technologies. It can be used for streaming data into Kafka […]
Creating a Data Pipeline with the Kafka Connect API – from Architecture to Operations
This is a guest blog from Alexandra Wang, Software Engineer, Pandora Media. You can connect with her on LinkedIn. Also special thanks to Lawrence Weikum and Stu Thompson as contributors of the work in this post. […]
Streaming data from Oracle using Oracle GoldenGate and the Connect API in Kafka
This is a guest blog from Robin Moffatt. Robin Moffatt is Head of R&D (Europe) at Rittman Mead, and an Oracle ACE. His particular interests are analytics, systems architecture, administration, and […]
Build and monitor Kafka pipelines with Confluent Control Center
On May 24, we announced Confluent Control Center, an application for managing and monitoring a Kafka-based event streaming platform. Control Center has a beautiful user interface, and under the surface we […]
Announcing Kafka Connect: Building large-scale low-latency data pipelines
For a long time, a substantial portion of data processing that companies did ran as big batch jobs — CSV files dumped out of databases, log files collected at the […]
How to Build a Scalable ETL Pipeline with Kafka Connect
Apache Kafka is a high-throughput distributed message system that is being adopted by hundreds of companies to manage their real-time data. Companies use Kafka for many applications (real time stream […]
Ja, Virginia, Sie brauchen wirklich eine Schema Registry
I ran into the schema-management problem while working with my second Hadoop customer. Until then, there was one true database and the database was responsible for managing schemas and pretty […]