Outbox Lib
A library implementing the Outbox Pattern for reliable publication of events to a messaging broker (like Kafka) with an
at-least-once, in-order delivery guarantee of messages.
Therefore, consumers of these messages must be able to handle duplicates — for example, by checking a
message’s sequence-id.
Table of contents:
Outbox
An Outbox solves the problem of the technology gap that occurs, when data needs to be written consistently and atomically to two independent systems - in this case, PostgreSQL and Kafka. To achieve consistency, a shared transaction between both systems would be required - also known as a global transaction. However, since these are independent systems, this is not easily possible.
Transaction Managers
One possible solution is the use of global transactions through a transaction manager.
This manager orchestrates transactions between both systems - provided both support the necessary 2 Phase Commit
Standard.
Using a transaction manager guarantees an exactly-once, in-order delivery of messages.
However, the downside is that the use can be complex, often involves commercial products and significantly reduces
both response time and throughput of both systems.
Outbox Pattern
An alternative solution is the use of the Outbox Pattern. This resolves the technology gap by relying on a single system - in this case, the database. Domain updates and events are stored within a single local transaction in the database. Reading and publishing the events is handled by another service - in the MSD, this is the Outbox-Service.
This service polls the Outbox for new events and publishes them in order per partition per topic. If a crash occurs after the polling while events are being published, the next poll may retrieve events that have already been published. This can lead to duplicates. Therefore, the delivery guarantee of an Outbox is at-least-once, in-order.
About
The Outbox Lib covers both use cases of an Outbox - writing to the Outbox and reading from it. To achieve this, the library extends a service’s database schema with an additional table - the Outbox. Producers can then store the events to be published within a transaction together with all related domain changes.
Installation
To integrate the library into an existing project, several steps must be followed. These steps are described below.
Requirements
The library defines the following requirements for integration. Failure to comply may lead to unexpected conflicts or errors.
- Java 21
- Spring Boot 3.5.9
- Spring Data JPA
- Postgres 17.x
- Flyway (or similar tool)
1. Add to Maven
The library’s packages are hosted in the Gitlab repository’s registry. For this reason, both the registry and the package dependency must be declared in Maven.
First, add the corresponding Gitlab Registry as follows:
<repositories>
<repository>
<id>msd-gitlab-outbox-lib</id>
<url>https://gitlab.com/api/v4/projects/71089152/packages/maven</url>
</repository>
</repositories>
Then, declare the Outbox library as a new dependency:
<dependency>
<groupId>de.microservice-dungeon</groupId>
<artifactId>outbox-lib</artifactId>
<version>7.1.7</version>
</dependency>
2. Enable in Spring
JPA and Spring Data JPA are used for database interaction. However, Spring limits the detection of JPA annotations to the classpath of a project. This means that Spring does not automatically recognize the library’s annotations. To enable this, the following annotations must be added to the main project’s application service.
@SpringBootApplication@EnableJpaRepositories("de.microservicedungeon")@EntityScan("de.microservicedungeon")
For @EnableJpaRepository und @EntityScan both the package path of the project and that of the library must be
specified.
For the library, this path is de.microservicedungeon.
In the following example, the path prefix of both projects is identical.
@SpringBootApplication
// must include both package paths
// here, both projects share the same prefix
@EnableJpaRepositories("de.microservicedungeon")
@EntityScan("de.microservicedungeon")
public class GameServiceApplication {
public static void main(String[] args) {
SpringApplication.run(GameServiceApplication.class, args);
}
}
3. Add Database Schema
The library requires a database schema that must be created initially and migrated during version upgrades.
This schema is located under /database/migration/.
For each schema change, two scripts are provided: a migration script starting with the letter V and a base script
starting with B.
Important: The schema must be created manually; there is no automation for this. Using Flyway or a similar migration tool is recommended.
4. Configure
Each target topic must be configured for the Outbox library with its name in Kafka. The reason is that sharding occurs when events are written to the Outbox. To determine the appropriate partition, the partition size of a topic must be known. This size is automatically retrieved for each configured topic using Spring Kafka-Admin.
The following example shows the configuration of two topics:
# resources/application.yaml
outbox:
topics:
- name: db.player.ecst.v1
- name: bl.player.events.v1
Update
Library updates follow the Semantic Versioning convention. Any change to the database schema is considered a breaking change and results in an increment of the major version. However, all major versions are exactly one version backward compatible to support parallel operation of both schemas. For each major version, the availability of a new schema must be checked.
Usage
Reading and writing to the Outbox is done through two interfaces, whose usage is described below. A Spring Boot autoconfiguration creates a bean for each interface, which can be overwritten with a custom configuration if needed.
Writing to the Outbox
To write to the Outbox, a WriteOutbox instance is required.
One or more OutboxWritableevents can be passed to this instance as arguments.
Important: Domain updates and writes to the Outbox must occur within the same transaction - see the JavaDoc for further details.
Reading the Outbox
To read from the Outbox, a ReadOutbox instance is required.
This instance allows reading the partitions of a topic.
Retrieval follows a read and acknowledge pattern.
Until a acknowledgement succeeds, the same events may be returned repeatedly - see the JavaDoc for further details.
Important: A topic’s partition must not be read concurrently!