Section 1.1: The Dual Write Challenge

Consider a scenario where an Order service creates an order and a separate Payment service processes the payment. The Order service needs to both create an entry in its database and publish an event to a queue for the Payment service.

Here, we face two distinct systems: a database and a queue, each with its own transaction boundaries. If the queue publishing fails, how do we ensure that the database entry is rolled back?

Subsection 1.1.1: Introducing the Transactional Outbox Pattern

One effective solution is the Transactional Outbox Pattern. Instead of directly publishing to the queue, we create a separate Outbox table within the same database for event storage.

A dedicated Order Event Processing service then reads from this Outbox table and sends events to the queue. This approach guarantees that both the Order table and the Outbox table can be modified in a single transaction, allowing for rollback if necessary. Moreover, if the Event Processing service fails, the event data remains in the Outbox table, ensuring it can be retried later.

Here’s a sample schema for the Outbox table:

CREATE TABLE outbox (

id SERIAL PRIMARY KEY,

event_type VARCHAR(255) NOT NULL,

event_payload JSONB NOT NULL,

status VARCHAR(50) NOT NULL DEFAULT 'PENDING',

created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,

processed_at TIMESTAMP NULL,

retry_count INTEGER NOT NULL DEFAULT 0

);

• id: Unique identifier for each event.
• event_type: A string to identify the type of event (e.g., OrderCreated).
• event_payload: A JSONB column to hold the event data.
• status: Indicates the current status of the event (e.g., PENDING, PROCESSED).
• created_at: Timestamp for when the event was created.
• processed_at: Timestamp for when the event was successfully processed.
• retry_count: Tracks how many times the event has been retried.

Section 1.2: Handling Duplicate Messages

Despite the advantages of the Outbox Pattern, we still face the issue of potential duplicate messages if the Order Event Processing service goes down. To mitigate this risk, we implement message de-duplication at the Payment Service using the Transactional Inbox Pattern.

This pattern involves creating an Inbox table in the Payments database to record received messages along with their timestamps. When a message arrives from the queue, the Payment Service checks if it already exists in the Inbox. If it does, the service can safely ignore it.

Here’s a sample schema for the Inbox table:

CREATE TABLE inbox (

id SERIAL PRIMARY KEY,

event_id UUID NOT NULL,

event_type VARCHAR(255) NOT NULL,

event_payload JSONB NOT NULL,

received_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,

UNIQUE(event_id)

);

• id: Unique identifier for each row.
• event_id: A UUID that uniquely identifies each event, matching the Outbox table.
• event_type: Identifies the type of event (e.g., OrderCreated, PaymentProcessed).
• event_payload: JSONB column for the event data.
• received_at: Timestamp for when the event was received.

Together, these patterns foster a robust architecture capable of navigating the complexities of distributed systems. They allow services to scale independently, maintain loose coupling, and simplify the process of adding or removing services, all while ensuring data consistency and reliability.

The first video, "Reliably Save State & Publish Events (Outbox Pattern)," discusses how the Outbox pattern can ensure event reliability in microservices.

The second video, "What is the Transactional Outbox Pattern? | Designing Event-Driven Microservices," explains the design principles behind the Outbox pattern in event-driven architectures.

“There's no better guarantee of failure than convincing yourself that success is impossible, and therefore never even trying.” — Max Tegmark, Our Mathematical Universe: My Quest for the Ultimate Nature of Reality.