Is Kafka overkill?



Kafka is a hot technology. And yes, sometimes it can be overkill depending on what you are trying to do

A lot of developers seem to think they should use Kafka just because companies like Netflix and Uber use it. The reality is Kafka is overkill for most projects that aren't utilizing event based architecture to stream process million plus records a second

I wouldn't use Kafka unless your working for a big company and need to work with real time data processing...


Kafka can be used in a lot of different ways. It's a great log aggregator, message bus, and data prevention tool depending on how you use it...and of course streaming.


According to Kafka official documentation...Kafka is good for the following:

1. messaging

2. website activity tracking

3. metrics

4. log aggregation

5. stream processing

6. event sourcing

7. commit log

When you take these use cases at face value then it seems like everyone and their dog should be using Kafka..but YES Kafka can be overkill if your not working with the volumes of messages that Kafka was built to address.

See Kafka is DESIGNED OUT OF THE BOX to handle horizontal scaling of a distributed cluster of machines. Obviously if you have a website for your bakery this is overkill...but if you have multiple "microservice like" applications that need to talk to one another then use Kafka. This is because Kafka implements messaging better than anything out there...yes even RabbitMQ and JMS.


Kafka is overkill only if you aren't using event based messaging in your application(s)...

If you have multiple applications that need to talk to each other then Kafka is the best way to do this (at the time of this writing anyways)...this is because Kafka does pub/sub better than anything else...and if you plan on growing your applications then this sets you up for success.


I think Kafka is the most overhyped thing since sliced many buzzwords surrounding real time data streaming, event processing, microservice architecture, fault tolerance, etc....these are all just words at the end of the day and don't apply to most of the things we build as indi developers. there are plenty of other solutions out there for event based programming and I wouldn't waste my time with Kafka unless I worked for a large company say Netflix or Uber :)


kafka can be overkill if all you need is basic messaging. Alternatives like RabbitMQ and JMS are simpler if you don't need the scenarios kafka is best known for aka real time data streaming.

if you just want to implement event based communication between your applications and aren't concerned with data storage, the ordering of messages, or fault tolerance then these alternatives are fine. These solutions are also actively maintained and will continue to introduce more kafka related functionality overtime.

if you plan on growing your system to send millions of events, want to process those messages in real time, care about order etc. then go with Kafka. Kafka is deliberately built for big data problems and the movements of PB of data daily etc.

With all of this said, if you use Kafka as a simple messaging service you won't be in a worse position than if you used the alternatives. It would just be like using an xbox to play heads/tails.