Outbox-Service

The Outbox Service is part of the Outbox Pattern. As such, it reads messages from the outbox and publishes them to Kafka.

Table of contents

• Outbox Pattern
• About

Outbox Pattern

An Outbox solves the problem of the technology gap that occurs, when data needs to be written consistently and atomically to two independent systems - in this case, PostgreSQL and Kafka. To achieve consistency, a shared transaction between both systems would be required - also known as a global transaction. However, since these are independent systems, this is not easily possible.

The Outbox Pattern solves this problem for producers by relying on a single system - in this case, the database. Domain updates and events are stored within a single local transaction. Reading and publishing the events is handled by another service - the Outbox-Service. This results in an at-least-once, in-order delivery guarantee, as long as there is only one reader per partition.

The graphic shows the integration of the Outbox-Service.

About

The responsibility of the Outbox-Service is to read messages from the outbox and forward them to Kafka.

The respective topics and partitions must be configured in advance. For each configured topic, the service starts a virtual thread that periodically polls the outbox for new messages. If message is found, it is forwarded to Kafka and then deleted from the outbox. If a crash occurs during this process, duplicate publications may occur.

Important: A partition must be read by only a single Outbox-Service instance. Otherwise, race conditions will occur. There is no leader election mechanism between Outbox-Service instances.