Project Metamorphosis: Unveiling the next-gen event streaming platform.Learn More
Log Compaction

Log Compaction | Highlights in the Apache Kafka and Stream Processing Community | October 2016

This month the community has been focused on the upcoming release of Apache Kafka 0.10.1.0. Led by the fearless release manager, Jason Gustafson, we voted on a release plan, cut branches and started voting on the first release candidate. Please contribute to the community by downloading the release candidate, testing it out and letting everyone know how it went. If no serious bugs are found, we are hoping to finalize the release by mid-October.

In addition to the vote, we gave our website a quick facelift, contribution of Derrick Or. We appreciated the feedback from the community and issues were quickly addressed.

And as usual, there are several very lively discussions in the community:

  • KIP-74: Proposal to limit not just the amount of data returned by a consumer fetch per partition, but also the amount of data returned for each fetch request overall. This will give users better control over the memory usage of consumers, but even better – this allows consumers to make progress even if a partition contains messages larger than the maximum fetch size. This proposal has been merged and will be part of the 0.10.1.0 release.
  • KIP-79: Proposal to add methods for searching by timestamp to the new consumer was accepted and merged. It will be included in the next release to everyone’s great joy.
  • KIP-82: Proposal for adding headers to Kafka messages. This proposal is very popular because so many organizations are using headers internally. It is also controversial – Kafka project has a long tradition of keeping the message completely unstructured and letting the users and client put whatever structure they need inside the message. Whatever the decision is, it will have serious impact on the Apache Kafka ecosystem.
  • KIP-83: Much welcome proposal that allows to instantiate clients with different security configurations in the same JVM. There are already patches available by Rajini Sivaram and Edurdo Comar and once integrated it will allow us to update MirrorMaker to support different security configurations on source and target clusters.
  • KIP-85: Allowing clients to take JAAS configurations dynamically rather than via a file. This will be huge for those of us implementing microservices in containers – adding files to containers has been very inconvenient.

In addition to ongoing Kafka improvements, there are other interesting news and blogs:

If you are interested in learning all about streaming data platforms, Confluent has released a 6-part online talk series focusing on Apache Kafka. You can view the recordings for the first two talks in the series by Jay Kreps and Jun Rao, and register for the upcoming sessions at www.confluent.io/apache-kafka-talk-series.

Did you like this blog post? Share it now

Subscribe to the Confluent blog

More Articles Like This

Kafka Streams Interactive Queries Go Prime Time

What is stopping you from using Kafka Streams as your data layer for building applications? After all, it comes with fast, embedded RocksDB storage, takes care of redundancy for you, […]

Apache Kafka Needs No Keeper: Removing the Apache ZooKeeper Dependency

Currently, Apache Kafka® uses Apache ZooKeeper™ to store its metadata. Data such as the location of partitions and the configuration of topics are stored outside of Kafka itself, in a […]

From Eager to Smarter in Apache Kafka Consumer Rebalances

Everyone wants their infrastructure to be highly available, and ksqlDB is no different. But crucial properties like high availability don’t come without a thoughtful, rigorous design. We thought hard about […]

Jetzt registrieren

Erhalten Sie in den ersten drei Monaten einen Rabatt von bis zu 50 USD pro Kalendermonat auf Ihre Rechnung.

Nur neue Registrierungen.

By clicking “sign up” above you understand we will process your personal information in accordance with our und bin damit einverstanden.

Indem Sie oben auf „Registrieren“ klicken, akzeptieren Sie die Nutzungsbedingungen und den gelegentlichen Erhalt von Marketing-E-Mails von Confluent. Zudem ist Ihnen bekannt, dass wir Ihre personenbezogenen Daten gemäß unserer und bin damit einverstanden.

Auf einem einzigen Kafka Broker unbegrenzt kostenlos verfügbar
i

Die Software ermöglicht die unbegrenzte Nutzung der kommerziellen Funktionen auf einem einzelnen Kafka Broker. Nach dem Hinzufügen eines zweiten Brokers startet automatisch ein 30-tägiger Timer für die kommerziellen Funktionen, der auch durch ein erneutes Herunterstufen auf einen einzigen Broker nicht zurückgesetzt werden kann.

Wählen Sie den Implementierungstyp aus
Manual Deployment
  • tar
  • zip
  • deb
  • rpm
  • docker
oder
Automatische Implementierung
  • kubernetes
  • ansible

By clicking "download free" above you understand we will process your personal information in accordance with our Datenschutzerklärung zu.

Indem Sie oben auf „kostenlos herunterladen“ klicken, akzeptieren Sie die Confluent-Lizenzvertrag und den gelegentlichen Erhalt von Marketing-E-Mails von Confluent. Zudem erklären Sie sich damit einverstanden, dass wir Ihre personenbezogenen Daten gemäß unserer Datenschutzerklärung zu.

Diese Website verwendet Cookies zwecks Verbesserung der Benutzererfahrung sowie zur Analyse der Leistung und des Datenverkehrs auf unserer Website. Des Weiteren teilen wir Informationen über Ihre Nutzung unserer Website mit unseren Social-Media-, Werbe- und Analytics-Partnern.