CQRS and Event Sourcing architecture pattern implementation with open source technologies.

Figure 01: CQRS and Event Sourcing implemented solution

The Command and Query Responsibility Segregation (CQRS) pattern separate read and update operations for a data store. So in the case of implementation, it is recommended to keep a separate model for command and query. Implementing CQRS in your application can maximize its performance, scalability, and security. The flexibility created by migrating to CQRS allows a system to better evolve over time and prevents update commands from causing merge conflicts at the domain level.

When we talk about the CQRS pattern most often it is used along with the Event Sourcing pattern. In the Event sourcing pattern, all commands represent a sequence of events. Most of the cases those events are handled by a Message Broker ( MB) solution which can handle a higher number of transactions per second( TPS ) compared to RDBMS.

Let’s consider a typical retail application. The items will be provided by Supplies and item inventory needed to maintain for those items. Also, the sales process operates from each store-level POS system and inventory needs to update according to sales. Also, let’s assume it is required to provide supplier wise daily reports as well as daily sales reports against each store.

In the above scenario if we directly try to update inventory operations which include multiple tables write operations with proper business rules the central Database always becomes a bottleneck for the entire solution. Also if we try to generate some reports from the same database, transaction locks may generate issues for this read operation. Also, some inventory updates may fail due to heavy traffic in the Database environment. Also in this situation, we may have to normalize and denormalize the data to make either the read or write operations efficiently.

In this article, I will try to explain how CQRS architecture with Event sourcing will solve the above traditional issues. Let’s consider the traditional CRUD base datastore pattern first.

Traditional CRUD base datastore pattern

Before discussing CQRS architecture lets consider traditional architectures, the same data model is used to query and update a database. Basically, we treat it as a CRUD datastore. But when we consider complex applications this approach can become unwieldy. For example, on the read side, the application may perform many different queries, returning data transfer objects (DTOs) with different shapes. Object mapping can become complicated. On the write operation, the frontend application needs to wait until the model continues complex business logic as well as database transaction to multiple tables or even multiple databases.

Figure 02: Traditional CRUD base solution

Challenges in the CRUD Pattern.

The following are the few challenges faced in this type of CRUD operation based application.

• There is often a mismatch between the read and write representations of the data, such as additional columns or properties( to provide fast response for a read operation ) that must be updated correctly even though they aren’t required as part of an operation.
• Data contention can occur when operations are performed in parallel on the same set of data.
• The traditional approach can have a negative effect on performance due to the load on the data store and data access layer. In the case of a read operation, there is a possibility to execute complex queries due to the unavailability of properly formatted data for a read operation.
• It is required to normalize and denormalize the data to make either the read or write operations efficiently. However, as a result, other operations can become inefficient.

CQRS pattern

As a solution for the above challenge, the CQRS pattern introduces which separates reads and writes into different models, using commands to update data, and queries to read data.

So when we consider the CQRS pattern we have to consider the below scenarios.

• Commands should be task-based, rather than data-centric. (“Book command in hotel room booking applications is not just “set ReservationStatus to Reserved”. That may include the whole process from data insert into several tables to call different APIs to complete the service).
• Commands may be placed on a queue for asynchronous processing, rather than being processed synchronously to keep proper decouple between application and backend. So that eliminates the waiting time of front end applications.
• Queries never modify the database. In other words, the read database can be configured according to query requirements while keeping the master database according to the model requirement.

Event Sourcing

In the case of the CQRS pattern most often used along with the Event Sourcing pattern. Event sourcing involves modeling the state changes made by applications as an immutable sequence or “log” of events.

Event Sourcing ensures that all changes to the application state are stored as a sequence of events. Not just can we query these events, we can also use the event log to reconstruct past states, and as a foundation to automatically adjust the state to cope with retroactive changes.

Like with everything, there are two sides to this. Having an event sourcing layer can possibly delay the writes and the reader might look at it from that perspective.

On the flip side, I think there are multiple advantages to sourcing the events.

• Events are immutable and can be stored using an append-only operation. The user interface, workflow, or process that initiated an event can continue, and tasks that handle the events can run in the background. This, combined with the fact that there’s no contention during the processing of transactions, can vastly improve performance and scalability for applications, especially for the presentation level or user interface.
• Events are simple objects that describe some action that occurred, together with any associated data required to describe the action represented by the event. Events don’t directly update a data store. They’re simply recorded for handling at the appropriate time. This can simplify implementation and management.
• Event sourcing can help prevent concurrent updates from causing conflicts because it avoids the requirement to directly update objects in the data store. However, the domain model must still be designed to protect itself from requests that might result in an inconsistent state.
• The append-only storage of events provides an audit trail that can be used to monitor actions taken against a data store, regenerate the current state as materialized views or projections by replaying the events at any time, and assist in testing and debugging the system.
• The event store raises events, and tasks perform operations in response to those events. This decoupling of the tasks from the events provides flexibility and extensibility. Tasks know about the type of event and the event data, but not about the operation that triggered the event. In addition, multiple tasks can handle each event. This enables easy integration with other services and systems that only listen to new events raised by the event store.

Sample Use case for CQRS and Event Sourcing

Let’s consider a typical retail application. The items will be provided by Supplies and item inventory needed to maintain for those items. Also, the sales process operates from each store level and inventory needs to update according to sales. Also, let’s assume it is required to provide supplier wise daily reports as well as daily sales reports against each store.

Now let’s consider how the above process will match CQRS and event sourcing architecture. When we consider daily supply details and daily sales details, those are commands to the system. Those commands need to be handled according to the event sourcing pattern. Also, outputs like supplier wise daily reports and store-level reports are query outputs from the system

Figure 03: CQRS and Event Sourcing solution

Now let’s consider possible solutions for components like Event Queue, Event Store, Event handler, and query APIs.

Event Queue accepts incoming commands and it is better to use some message broker solution to implement event queue. Since these events trigger from the supplier’s network or store network it is better to provide some secure APIs to accept these events and pass into the event queue. Then the message broker solution can be used as an event store where the events store temporarily till it is ready for processing.

The next component in this pattern is an event handler where it reads events from the Event queue or a Topic and processes it according to the data model. In this example, the supplier’s items need to be stored on the supplier breakdown information table as well as item inventory. Also in case of sales events, each sales entry needs to be maintained against the store as well as it will update inventory information. So in this event handler component processes all commands in the system and writes it into multiple tables according to business requirements.

Now let’s consider read operation and what is recommended database implementation to read data. The read store can be a read-only replica of the write store, or the read and write stores can have a different structure altogether. Using multiple read-only replicas can increase query performance, especially in distributed scenarios where read-only replicas are located close to the application instances. The read database avoids complex joins or complex ORM mappings. It might even use a different type of data store. For example, the write database might be relational, while the read database is a document database.

So in the above solution, it does not become other operation inefficient due to normalization and denormalization operations.

CQRS and Event Sourcing implementation with open source technologies.

In this section, I would like to discuss sample pattern implementation with open source technologies. I have selected the WSO2 open source middleware Integration platform and Apache ActiveMQ message broker to implement this pattern.

Figure 04: CQRS and Event Sourcing implementation with Open Source solutions.

The above-proposed solution includes following open source product to implement CQRS architecture pattern with Event Sourcing

1.WSO2 API management platform- WSO2 API Manager is a fully open-source full API lifecycle management solution that can be run anywhere. It can be deployed on-prem, on a private cloud, is available as a service on the cloud, or deployed in a hybrid fashion where its components can be distributed and deployed across multiple clouds and on-prem infrastructures. In this solution, it uses API management to expose all commands and Queries to outsiders in a secure way. Also, it provides discoverability to all APIs with proper traffic management.

2. WSO2 Enterprise Integrator (EI)- WSO2 Enterprise Integrator (WSO2 EI) 7.0 is an open-source hybrid integration platform that enables API-centric integration using integration architecture styles such as microservices or centralized ESB. The platform provides a graphical drag-and-drop flow designer and a configuration-driven approach to build low-code integration solutions for cloud- and container-native projects.

In the above solution, it has proposed WSO2 Enterprise Integration for following CQRS components

• Event Queue service: This Event Queue service needs to accept HTTP API requests as commands and need to send to ActiveMQ Queue or a Topic. It uses WSO2 EI which exposes API commands and transforms input as JMS messages. At the end, the message will be sent to the ActiveMQ solution and it immediately responds with a 202 code. Please refer to WSO2 documentation to understand how this type of transformation is implemented with WSO2 EI.
• Event handle Service: This event handler service needs to read events from the ActiveMQ Queue or a Topic and need to process it and store it to the primary database or call different APIs from different subsystems to complete command action. Also, this event data needs to be transformed according to the data model and the data might be needed to write into multiple tables, multiple databases, or different APIs. This Event handler component can be implemented with WSO2 EI. First WSO2 EI needs to read messages from ActiveMQ Queue or Topic and please refer to WSO2 documentation to understand how easy to do that task with WSO2 EI. Later WSO2 EI needs to process the message and need to store it into the primary database or call different APIs. WSO2 EI mediators can be used to implement this type of message transformation with complex event processing. Also, the WSO2 integration studio can be used to design, develop, debug, and test integration solutions.

Also if there is a high volume of data needed to process ( High TPS data) WSO2 Streaming integrator (SI) can be used instated or WSO2 EI to implement event handler service. WSO2 SI allows you to integrate event streams and take action based on it. Also, Its sophisticated, web-based IDE provides an enhanced developer experience.

• Query App( Data services). The next challenge is how to provide query responses to outside as APIs. So in this requirement, it is required to use a read replica or document database to read data and provide responses to queries. WSO2 EI data services can be used to configure this type of data services from RDBMS like MySQL, MSSQL, Oracle, DB2, etc as well as document databases like MongoDB or even a CSV file. The advantage of using this type of data services is that it helps to configure data service without development effort. Also, the person with basic SQL knowledge can configure this type of query output as APIs on demand.

3. Apache ActiveMQ- This solution uses Apache ActiveMQ as an Open source message broker solution. But it can be used any message broker solution in this implementation.

CQRS and Event Sourcing Pros & Cons

Pros:

• Better performance and scalability of your system.
• Better concurrent access handling.
• Less complex domain model and a simple query model.
• No deadlocks.
• Events (facts) are well understood by the business expert.
• Audit trails can be implemented from events.
• we can get an object state at any point in time.
• Easy to test and debug.
• The data model is decoupled from the domain model.
• flexibility — many different domain models can be constructed from the same stream of events.
• We can use this model with reversing events, retroactive events.

Cons:

• Read and Write models must be kept in sync.
• Maintenance and administration costs if you choose two different engines for reading and writing.
• Eventual consistency is not always allowed.
• Querying beyond one aggregate is a bit harder (you have to construct projections for each type of query you want to add to the system).
• Event schema changes are much harder than in the case of the relational model (lack of standard schema migration tools)

Conclusion

CQRS and Event sourcing patterns help you to come out of the traditional CRUD base datastore pattern. But It is required to consider actual architectural expectations before selecting the best pattern for your project. I hope the above Pros and Cons will help you to select the best pattern. Also when we consider modern microservice architecture CQRS and event sourcing patterns help a lot.