Resilient Service Communication: Sync Calls, Events, Retries, and Idempotency

Inside a single process, a function call either returns or throws. Across a network there's a third outcome, you don't know. The request may have been lost on the way, processed with the response lost, or still be running. Every reliability pattern in microservices exists to deal with that uncertainty.

Synchronous vs. asynchronous: choose deliberately

Synchronous (REST, gRPC)

The caller sends a request and waits for the answer.

Good for:

• Queries where the user needs the answer right now ("what's my balance?")
• Commands that need immediate validation feedback

Costs:

• Temporal coupling. The callee must be up right now for the caller to work.
• Latency stacking. A → B → C adds the latencies together and multiplies the failure probabilities.

gRPC adds strongly typed contracts through Protocol Buffers, efficient binary encoding, and streaming. That makes it a good fit for internal service-to-service traffic. REST with JSON is simpler to debug and remains the default for public APIs.

Asynchronous (events and messages through a broker)

The producer publishes an event and moves on. Consumers process it when they can.

Good for:

• Side effects that don't need to block the user (emails, analytics, search indexing)
• Fan-out to many interested services
• Absorbing traffic spikes, because the queue acts as a buffer

Costs:

• Eventual consistency. Different parts of the system may briefly disagree.
• Harder debugging and tracing, plus the need to handle duplicate and out-of-order messages.

Rule of thumb: use synchronous calls for queries and immediate validations, and events to propagate facts that have already happened.

Timeouts: the pattern everyone forgets

A call without a timeout can wait forever while holding a thread, a connection, and memory. Under load, those stuck calls pile up until the caller falls over too.

• Set a timeout on every outbound call. Many HTTP clients default to no timeout at all.
• Propagate deadlines. If the user request has 2 seconds left, a downstream call shouldn't get a fresh 5-second timeout. gRPC propagates deadlines automatically, and in Go you get this by passing context.Context everywhere.
• Base timeouts on measurements. Start from the callee's p99 latency plus some headroom, not an arbitrary round number.

Retries: helpful until they aren't

Retries fix transient failures, but they multiply load. Retrying at every layer turns one failure into an avalanche. If three layers each retry 3 times, one user request becomes 27 calls to the bottom service, exactly when it is already struggling.

Guidelines:

• Retry only on retryable failures: connection errors, 503, 429, and timeouts. Never retry 400 or 422, because the request is wrong and will stay wrong.
• Use exponential backoff with jitter:

func backoff(attempt int) time.Duration {
    base := 100 * time.Millisecond
    max := 5 * time.Second
    d := base << attempt // 100ms, 200ms, 400ms...
    if d > max {
        d = max
    }
    return time.Duration(rand.Int63n(int64(d))) // full jitter
}

Jitter keeps a thousand clients from retrying at the same moment.

• Cap attempts (usually 2–3) and retry at only one layer, preferably the edge closest to the user.
• Use a retry budget. Allow retries only up to, say, 10% of normal request volume, so retries can't overload a service that's already degraded.

Idempotency: making retries safe

A retry is only safe if doing the operation twice has the same effect as doing it once. Reads are naturally idempotent. Writes such as "charge the card" are not.

The standard fix is an idempotency key: the client generates a unique ID for each logical operation and sends it with every attempt.

POST /payments
Idempotency-Key: 5f2b8a1e-3c4d-4e7f-9a0b-1c2d3e4f5a6b

The server stores the key with the result:

CREATE TABLE idempotency_keys (
    key         TEXT PRIMARY KEY,
    request_hash TEXT NOT NULL,
    response    JSONB,
    created_at  TIMESTAMPTZ NOT NULL DEFAULT now()
);

On each request:

1. Try to insert the key. If it already exists with a stored response, return the stored response without doing the work again.
2. If the key exists but the request body hash differs, reject the request, since the client reused a key by mistake.
3. Otherwise do the work and save the response in the same transaction as the business change.

Event consumers need the same protection. Brokers usually guarantee at-least-once delivery, so every consumer must handle duplicates, typically by recording processed message IDs.

Circuit breakers: failing fast

When a dependency is down, calls that wait until they time out waste resources and slow everything down. A circuit breaker watches failures and short-circuits calls once a threshold is reached:

• Closed: calls flow normally and failures are counted.
• Open: after, say, 50% failures over 20 requests, calls fail immediately for a cooldown period.
• Half-open: after the cooldown, a few trial calls go through. If they succeed, the breaker closes. If they fail, it opens again.

Combine breakers with fallbacks where the product allows it: cached data, a default recommendation list, or a degraded UI. A page without personalized recommendations is much better than a page that doesn't load.

Bulkheads: containing the damage

Give each dependency its own limited pool of connections or concurrent calls. If the slow recommendations service uses up its 20-connection pool, payment calls still have their own pool available. This works like compartments in a ship's hull, where one breach doesn't sink the whole vessel.

The dual-write problem and the outbox pattern

A classic bug looks like this:

db.Save(order)                // succeeds
broker.Publish(OrderPlaced{}) // process crashes before this line

The order now exists, but no downstream service will ever hear about it. Reversing the order doesn't help, because then you can publish events for orders that never got saved.

The transactional outbox solves this:

1. In the same database transaction as the business change, insert the event into an outbox table.
2. A separate relay process reads unpublished rows and publishes them to the broker, marking each row as sent.
3. If the relay crashes, it resumes where it left off. Consumers handle the occasional duplicate through idempotency.

Change-data-capture tools can tail the database log and act as the relay, which removes polling entirely.

Sagas for multi-service workflows

Distributed transactions (two-phase commit) across services are fragile and rarely worth it. A saga models a workflow as a sequence of local transactions, each with a compensating action:

1. Orders creates the order as pending.
2. Payments charges the card, or on failure publishes PaymentFailed.
3. Inventory reserves stock, or on failure triggers a refund compensation.
4. Orders marks the order confirmed.

Sagas can be choreographed, with services reacting to each other's events, or orchestrated, with a coordinator that directs each step. Orchestration is easier to follow once a workflow has more than three or four steps.

Summary

• Choose sync or async per interaction, not per system.
• Put timeouts on everything and propagate deadlines.
• Retry carefully: at one layer, with jittered backoff and a budget.
• Make every write idempotent, and every consumer too.
• Use circuit breakers and bulkheads to contain failures.
• Publish events through an outbox, never through a dual write.

Distributed systems will fail in partial, confusing ways. These patterns keep those failures small, visible, and recoverable.