Outbox Pattern using dotnet 6
This article showcases how to implement the Transactional Outbox pattern in microservices architecture with sample dotnet code. You can find the sample code used in this article here.
In distributed applications, microservices use messages/events to communicate with each other. So for example take an example of food delivery system (or say just a small part of it) as shown in below diagram.
Here we have one microservice ‘Order’ which upon receiving an order, updates the order details in database and then pushes this message to a message queue(RabbitMQ in this case). Then another microservice called ‘Notifier’ receives these messages and accordingly notifies to different clients(Restaurant and Delivery Agent in this case).
Problem Statement
The straight approach to solve this is to save the data to Order service’s database and then notify the change to other service. Something like shown in below activity diagram.
But what if something goes wrong after saving the state, i.e. saving the order details in database and before pushing the message to RabbitMQ. In this case our system will go in an inconsistent state. Order will be kept within it’s scope and other services(notifier) will not become aware of it. Hence restaurant, delivery agent or fulfilment system will not be aware of an incoming order and that my friend is a bug in the system.
How to tackle this scenario? The answer to that is Transaction Outbox Pattern (or commonly known as Outbox Pattern).
Solution is Transaction Outbox Pattern
What Outbox Pattern says is that while saving the state in database, insert the message/event to an Outbox table. That means saving the state and inserting the message/event to an outbox table should happen in a single transaction.
A separate process reads these messages/events from Outbox and pushes it to the message broker. After pushing to broker, it then removes the entry from Outbox table. Check the below activity diagram which shows how Outbox pattern can be implemented.
This way either both the state and message/event will be stored or none. And if service or message broker becomes unavailable because of some reason, once it is back it will start processing from where it is left. This way systems will be in consistent state and we will not get the bug specified above.
Implementation
For demo purpose, I have created a basic application which consists of:
- 2 microservices: Order and Notifier.
- RabbitMQ as message broker.
- SQL server as storage(database) in Order service.
- InMemory database for Notifier.
- Both services are created using MinimalAPI.
I have used docker desktop to run the application. You can find the docker-compose.yaml as shown below.
2 important aspects while implementing outbox pattern is:
- Saving state and event within a single transaction.
- Reading from the outbox and pushing the message/event to message broker on a separate process.
The sample application is elementary and easily understandable. The focus is on showcasing the outbox pattern, hence ignore other implementation immaturity if any. You can find the complete code on GitHub from below link:
This was a sample use case showing how to implement transactional outbox pattern in microservices. Hope you found this article useful.
For further reading on Transactional Outbox Pattern you can check out https://microservices.io/patterns/data/transactional-outbox.html and https://itnext.io/the-outbox-pattern-in-event-driven-asp-net-core-microservice-architectures-10b8d9923885.
Please do checkout other articles by me and share your feedback(s) if any. Thank you and Happy Coding :)
