What do useful patterns for implementing reliable high-load software systems like CQRS, Event Sourcing, and EDA have in common? All of these patterns are based on working with event streams. Despite the simplicity of this concept, there are surprisingly few ready-made tools for its implementation, with Kafka being the only widely popular one.

Often, people do not distinguish between the concepts of event streams and message queues, which leads to misunderstanding and underestimation of the prospects of using event streams. Let’s get to the point.

What are event streams, and how do they differ from queues? Event streams involve the long-term storage of messages in a continuous sequence. Each message in a stream has its own sequence number. These features of event streams have obvious significant advantages over using message brokers:

  • History Access: You always have the opportunity to process not only current events but also the history of events, which is available automatically without any additional effort on your part.
  • Scalability: Adding new event receivers does not increase the server’s processing load, as all receivers read from the same streams.
  • Reliability: It allows for easy implementation of guaranteed delivery in the correct sequence without loss. This is invaluable for developing reliable software.
  • Traceability: You can always view the history of events and how your microservices reacted to them, and in many cases, even reprocess these events if necessary.
  • Extensibility: It is always possible to add a new microservice or report and feed it the entire event history.

How does this work? How is it guaranteed that delivery is in the correct sequence and that each event is handled exactly once? It’s relatively straightforward if you can retrieve any event by its sequence number. When an event is received, the microservice processes it and, when updating the state in the database, records the event’s sequence number. If something goes wrong and the microservice stops, upon restarting, it retrieves the sequence number of the last processed event from the database and resumes processing from the next sequence number. This ensures that the event does not affect the microservice’s state twice and is not lost.

Of course, there are challenges to address in such systems. For example, handling changes in event structure when releasing new microservice versions, determining how long to store history, or managing incorrect events. However, these questions are not unique to event streams; similar problems arise with any data store, and solutions can be found or adapted from others. I may discuss this topic in future posts.

In conclusion, if you find Kafka too heavy, consider NATS JetStream or EventStoreDb, which are often overlooked. These systems represent the future, and I encourage you to explore them now.