[Webinar] Q1 Confluent Cloud Launch Brings You the Latest Features | Register Now

Presentation

Building a Data Subscription Service with Kafka Connect

« Kafka Summit 2020

After migrating much of our tech stack to Kafka Streams and Kafka Connect, some of our newer streaming applications encountered the same problem: while their primary input data was available in Kafka and their static side input data could be ingested using traditional connectors, they still needed access to semi-static data that has historically been acquired through in-line synchronous requests and responses. External service calls made from within a Kafka Streams application are highly discouraged, as they make error-handling difficult and typically slow down the application. To solve this problem, we used Kafka Connect components to build a large-scale data subscription service which subscribes to, polls, and disseminates auxiliary data to Kafka topics for immediate consumption by our streaming applications.

This talk will cover:
• An overview of our subscription service architecture
• A deep-dive into our Kafka Connector and each of its custom components
• Design considerations and various pitfalls that we encountered along the way

Related Links

How Confluent Completes Apache Kafka eBook

Leverage a cloud-native service 10x better than Apache Kafka

Confluent Developer Center

Spend less on Kafka with Confluent, come see how