Designing a Reliable Notification Service¶

A notification service turns a business event into a message that may reach one person through several devices and channels. The difficult part is not calling an email or push API. It is preserving the intent of the notification while recipients change preferences, providers throttle traffic, devices go offline, callbacks arrive late, and workers retry after ambiguous failures.

This chapter designs a multichannel notification platform for a backend product. It focuses on the service boundary, fan-out, delivery state, provider behavior, and operational guarantees. It does not reimplement a message broker or the transactional outbox pattern; those are supporting mechanisms with their own trade-offs.

1. Define what "delivered" means¶

Start by separating states that are often collapsed into one dashboard number:

Accepted by the API: the notification service validated the request and durably recorded responsibility for processing it.
Accepted by a provider: an email, SMS, or push provider accepted a send request.
Delivered to a destination: the provider supplied a delivery receipt, where that channel supports one.
Displayed or seen: the client or recipient gave a further signal, such as an app receipt or an open event.

These states are not interchangeable. A successful APNs response means the provider accepted the request; it does not prove that the device displayed the notification. FCM similarly distinguishes a message accepted by its backend from a message delivered to the app. Email delivery events can indicate acceptance by a recipient mail server, not that a person read the message.

A notification system should therefore expose the strongest state it can actually observe. For channels without reliable delivery receipts, keep the status as provider-accepted or delivery-unknown rather than inventing a delivered state.

Also define whether the product needs:

Best-effort alerts, where an expired notification can be dropped.
Durable business notices, where every accepted notification must reach a terminal state and failures need investigation.
Replaceable state updates, where a newer notification makes an older pending one irrelevant.
Time-sensitive messages, where a late notification is worse than no notification.

Those categories lead to different expiry, priority, retry, and storage policies.

2. Clarify the requirements before choosing components¶

A useful interview starts with a short requirements table.

Area	Questions to settle
Producers	Which services or user actions create notifications?
Recipients	One user, many users, a tenant, or a device group?
Channels	Push, email, SMS, in-app inbox, or a user-selected subset?
Preferences	Which categories are enabled, and what consent is required for each channel?
Timing	Immediate, scheduled, quiet-hours aware, or digest delivery?
Reliability	What does acceptance guarantee? What may expire or be dropped?
Ordering	Must notifications for one recipient arrive in order, or is best-effort sufficient?
Scale	How many logical notifications, recipients, endpoints, and provider sends are expected?
Isolation	Can one tenant's campaign consume another tenant's delivery capacity?
Operations	How are provider outages, invalid endpoints, retries, and manual replay handled?

Do not promise strict ordering or exactly-once delivery unless the product has a concrete reason to pay for those guarantees. Most push providers are best-effort systems. A product can guarantee that it durably accepted an intent and attempted delivery according to policy; it cannot generally guarantee that a mobile operating system displayed a message at an exact time.

3. A reference architecture¶

The service should separate the durable notification intent from the channel-specific delivery attempts.

Product services
      |
      v
Notification API
  - authentication and tenant policy
  - request validation and idempotency
  - preference and consent policy
      |
      v
Notification store + durable handoff
      |
      v
Scheduler and fan-out workers
      |
      +---- priority / tenant / channel queues
                    |
                    +---- push workers ---- APNs / FCM
                    |
                    +---- email workers --- email provider
                    |
                    +---- SMS workers ----- SMS provider
                    |
                    +---- in-app workers -- inbox store
                                      |
Provider callbacks / client receipts
      |
      v
Delivery-event processor
      |
      +---- delivery state and audit history
      +---- metrics, alerts and reconciliation

The API does not wait for every provider to finish. It validates the request, records the notification and its idempotency result, then returns an identifier the caller can use to inspect progress. Fan-out and provider calls happen asynchronously.

The notification store is the source of truth for the product's intent and delivery state. A queue is a work-distribution mechanism, not the only record that a notification exists. If a broker message is lost, delayed, redelivered, or moved to a dead-letter queue, the durable record still allows the system to reconcile what should happen next.

If a producer must atomically commit a business change and the intent to notify, use a durable handoff such as an outbox or an equivalent transactional mechanism. Keep that dual-write problem separate from provider delivery; the notification service still needs its own idempotency and attempt tracking.

4. Design an asynchronous API contract¶

A small API surface is usually enough:

POST /v1/notifications — accept a logical notification.
GET /v1/notifications/{id} — inspect its aggregate state.
POST /v1/notifications/{id}/cancel — cancel work that has not crossed an irreversible provider boundary.
GET /v1/preferences and PUT /v1/preferences — read or update user-controlled channel and category settings.

An example request:

POST /v1/notifications
Authorization: Bearer <service-token>
Idempotency-Key: order-7842-shipped-v1
Content-Type: application/json

{
  "type": "order.status_changed",
  "recipient": { "user_id": "user-42" },
  "channels": ["push", "email"],
  "template": "order-shipped-v3",
  "data": {
    "order_id": "7842",
    "status": "shipped"
  },
  "expires_at": "2030-01-01T00:00:00Z"
}

A successful response can be:

HTTP/1.1 202 Accepted
Content-Type: application/json

{
  "notification_id": "ntf_01J...",
  "status": "accepted",
  "status_url": "/v1/notifications/ntf_01J..."
}

The 202 Accepted response means the service accepted responsibility for processing the request. It does not mean a provider accepted it or that a person received it.

Scope the idempotency key to the authenticated tenant and define its retention period. Store a hash of the normalized request with the key:

Same key and same request: return the original notification identifier and status.
Same key and different request: return a conflict; do not silently mutate the original notification.
New key: create the intent and persist the idempotency result in the same transaction.

The key prevents a caller retrying after a timeout from creating a second logical notification. It does not by itself prevent duplicate provider sends after a worker loses an acknowledgement; that requires delivery-level idempotency and reconciliation.

For large audiences, do not accept an unbounded recipient list and expand it synchronously inside the request. Accept a campaign or audience reference, validate its ownership, and let a controlled fan-out process create recipient deliveries in batches.

5. Model the notification and each delivery separately¶

One logical notification may produce several delivery records: two devices, an email address, and an in-app inbox entry are different destinations with different outcomes.

A simplified PostgreSQL model:

CREATE TABLE notification (
    id                uuid PRIMARY KEY,
    tenant_id         uuid NOT NULL,
    recipient_id      uuid NOT NULL,
    notification_type text NOT NULL,
    template_key      text NOT NULL,
    template_version  integer NOT NULL,
    payload           jsonb NOT NULL,
    idempotency_key   text NOT NULL,
    request_hash      text NOT NULL,
    created_at        timestamptz NOT NULL DEFAULT now(),
    scheduled_at      timestamptz,
    expires_at        timestamptz NOT NULL,
    state             text NOT NULL,
    UNIQUE (tenant_id, id),
    UNIQUE (tenant_id, idempotency_key)
);

CREATE TABLE notification_delivery (
    id                  uuid PRIMARY KEY,
    notification_id     uuid NOT NULL,
    tenant_id           uuid NOT NULL,
    endpoint_id         uuid NOT NULL,
    channel             text NOT NULL,
    provider            text,
    state               text NOT NULL,
    attempt_count       integer NOT NULL DEFAULT 0,
    next_attempt_at     timestamptz NOT NULL DEFAULT now(),
    provider_message_id text,
    accepted_at         timestamptz,
    delivered_at        timestamptz,
    last_error_code     text,
    created_at          timestamptz NOT NULL DEFAULT now(),
    UNIQUE (notification_id, endpoint_id, channel),
    FOREIGN KEY (tenant_id, notification_id)
        REFERENCES notification (tenant_id, id)
);

CREATE INDEX notification_due_idx
    ON notification (COALESCE(scheduled_at, created_at))
    WHERE state IN ('accepted', 'scheduled');

CREATE INDEX delivery_due_idx
    ON notification_delivery (next_attempt_at, created_at)
    WHERE state IN ('queued', 'retry_wait');

This is a starting point, not a complete schema. The same table definitions, tenant-scoped foreign key, idempotency constraint, and due-work indexes are exercised in the PostgreSQL 18 schema smoke test. In a production design, also model user endpoints, category preferences, consent records, provider callback events, and an append-only attempt history. Store device tokens and addresses encrypted, and avoid copying raw addresses into queue messages or logs.

Keep the immutable notification intent separate from mutable delivery state. A template version and a validated data payload make the original request reproducible even after a template changes. The endpoint record identifies the current destination; the delivery record records which endpoint version was targeted.

A useful delivery state machine is:

queued -> attempting -> provider_accepted -> delivered
             |                 |
             |                 +---- delivery_unknown
             |
             +---- retry_wait -> attempting
             |
             +---- failed_permanently
             +---- expired
             +---- suppressed
             +---- cancelled

Record transitions with timestamps and provider event identifiers. A callback can arrive after a retry or after a newer state transition, so make event ingestion idempotent and define which events are allowed to advance or correct a state. Do not let an old callback move a terminal, newer state backwards.

6. Fan out without overwhelming the system¶

There are two broad fan-out strategies:

Fan-out on write: resolve recipients and create delivery work when the event arrives. Reads are simple, but a large audience can create a sudden write and queue burst.
Fan-out on read: store a campaign or audience reference and resolve recipients later. Creation is cheap, but delivery workers must page through a changing audience and decide whether membership is evaluated at campaign time or send time.

For transactional notifications such as an order update, fan-out on write to the user's currently registered endpoints is usually straightforward. For a campaign to millions of recipients, use a durable campaign record and bounded batches. Define whether recipients are snapshotted at campaign creation or evaluated when each batch is processed; silently changing the audience during a long campaign makes results difficult to explain.

Partition work to balance throughput and fairness:

Keep critical transactional alerts separate from bulk campaigns.
Apply per-tenant concurrency and rate budgets so one customer cannot monopolize provider capacity.
Partition by recipient when per-recipient sequencing is required; otherwise use a key that distributes load evenly.
Avoid one global FIFO queue when independent recipients can progress concurrently.
Use bounded batches and bounded worker concurrency. A large queue is a backlog, not additional processing capacity.

A database-backed queue can be a sensible starting point at modest scale. PostgreSQL's FOR UPDATE SKIP LOCKED can let multiple workers claim different due rows without waiting on rows already claimed. PostgreSQL explicitly notes that this produces an inconsistent view and is appropriate for queue-like access, not general-purpose reads. Keep claim transactions short, use leases or visibility deadlines, and reclaim work after a worker crash.

As volume and fan-out grow, a managed queue or broker can isolate producers from consumers and scale workers independently. It adds operational choices around retention, visibility timeouts, redelivery, partitioning, replay, and dead-letter handling. Choose it because the workload needs those properties, not because every system design diagram needs Kafka.

7. Treat providers as independent, rate-limited dependencies¶

Use a provider adapter per channel. The adapter owns authentication, request formatting, provider-specific response classification, rate limits, and callback verification. The rest of the system should work with a small internal result contract rather than interpreting provider-specific status codes everywhere.

A provider router can choose a configured provider using channel, region, tenant policy, capability, cost, and current health. Keep fallback explicit: sending an email because push failed may violate the user's preferences or create a duplicate if the push provider actually accepted the message before the connection timed out.

Provider behavior is not uniform:

APNs: a successful response indicates that APNs accepted the request. A 410 response means the device token is no longer active for that topic; remove or invalidate that endpoint. A 429 can indicate too many consecutive requests for the same device token and should be retried only after a delay. Apple says to wait 15 minutes before retrying server-side 5xx responses, using backoff. Do not retry permanent token errors such as BadDeviceToken, DeviceTokenNotForTopic, ExpiredToken, or Unregistered. APNs may reorder, throttle, store, or fail to deliver notifications.
FCM: an accepted message ID is not proof of device delivery. Set a message time-to-live that matches the product's usefulness. Collapse keys are appropriate for replaceable updates, such as a latest-state refresh, not for independent events that must each be processed. Android high priority is intended for time-sensitive, user-visible messages, not as a default throughput setting.
Email: providers enforce account- and region-specific daily quotas and send rates. Amazon SES, for example, counts recipients rather than message API calls, and its acceptance rate can be lower than the configured maximum. Quota errors require controlled waiting rather than a tight retry loop. Monitor bounce and complaint signals and suppress addresses that should no longer receive mail.
SMS: throughput, sender registration, content rules, and delivery receipts vary by provider and destination country. Keep regional policy and consent checks in the adapter/configuration layer; do not assume one global rate limit.

Maintain independent concurrency and rate budgets for each provider account, region, and channel. A single shared worker pool can let a slow email provider starve urgent push traffic. A circuit breaker should pause or reduce work for a failing provider while allowing healthy providers to continue.

8. Retries, deduplication, expiry, and dead letters¶

Assume at-least-once processing. A worker can send a request, the provider can accept it, and the network can fail before the worker receives the response. Retrying may send the same notification again. A queue's acknowledgement or visibility timeout does not make an external side effect exactly once.

Use a stable delivery identifier across attempts. Pass it as a provider idempotency key or request identifier when the provider supports one. Persist the provider message ID and attempt outcome. If the provider has no idempotency facility, accept that an ambiguous timeout can create a duplicate and make the product payload safe to recognize as a repeated event.

Classify failures before retrying:

Failure	Typical policy
Connection reset or timeout before a conclusive response	Retry carefully; acceptance may be ambiguous
Provider `429` or explicit throttling	Respect provider guidance and reduce send rate
Provider `5xx` or temporary unavailability	Retry with exponential backoff and jitter
Invalid device token or address	Suppress or deactivate the endpoint; do not retry the same destination
Invalid payload or unsupported template	Mark permanent failure and fix the producer/template
Authentication or account configuration failure	Pause the affected provider route and alert; do not retry every delivery indefinitely
Notification past its expiry	Mark expired; do not send stale content

Use bounded exponential backoff with full jitter to spread retry traffic. If a provider supplies Retry-After, treat it as a minimum delay and add a small jitter window after it. Cap attempts and total retry age. The message's expiry time is a hard boundary: a retry that cannot happen before expiry should not be scheduled.

The accompanying Python retry-policy example and deterministic unit tests implement this delay calculation. It is a scheduling helper, not a complete retry engine; the worker still needs persistent attempt state, a provider error classifier, and a durable scheduler.

A dead-letter queue is for work that exhausted its retry policy or needs investigation. Preserve the original notification ID, delivery ID, error class, attempt history, and provider response metadata. Replay only after the cause is understood, with the same logical delivery identity and a controlled rate. Do not blindly redrive a large dead-letter backlog into a provider that is still unhealthy.

A notification that is technically deliverable can still be inappropriate to send. Model preferences by category and channel, not as one global on/off switch. Distinguish product-critical account or security notices from marketing messages, and apply the consent and opt-out rules that govern each channel and region.

Evaluate preferences as close to dispatch as practical. A user may opt out after a campaign is queued but before the worker sends it. The final send check should consider current consent, quiet hours, locale, endpoint verification, tenant policy, and expiry. Record the decision and policy version so the system can explain why a delivery was suppressed.

Other safeguards include:

Encrypt device tokens, email addresses, phone numbers, and sensitive payload fields at rest.
Keep message bodies and personal identifiers out of queue metadata, metrics labels, and routine logs.
Authenticate provider callbacks, verify signatures where available, and deduplicate callback event IDs.
Restrict which producer can send each notification type and which template data it may supply.
Escape or sanitize untrusted template values and validate deep links before placing them in a push payload.
Apply per-tenant quotas and abuse controls to prevent notification flooding.
Keep an audit trail for consent changes, endpoint registration, suppression, and administrative replay.

A provider callback is an external input. Validate its signature, timestamp or replay protection, account identity, and event shape before allowing it to change delivery state.

10. Backpressure, observability, and recovery¶

A notification system can be healthy at the API boundary while its queues grow for hours. Measure the distance between accepting work and processing it.

Track at least:

Queue depth and age of the oldest pending delivery, by channel, priority, and tenant.
Time from API acceptance to first provider attempt.
Provider acceptance, throttling, timeout, and permanent-failure rates.
Delivery receipts and their coverage, separately from provider acceptance.
Attempts per logical delivery, retry age, expiry count, and dead-letter volume.
Suppressed deliveries by preference, consent, invalid endpoint, and quiet-hours reason.
Provider quota usage, active connections, worker saturation, and cost per successful delivery.

Alert on user-visible service objectives, such as the percentage of eligible transactional notifications accepted by a provider within a target time. Do not call that a device-delivery SLO unless the provider supplies reliable device-level evidence.

When the backlog grows, stop adding work blindly. Reduce low-priority campaign admission, slow producers, increase workers only when the provider and downstream systems have capacity, and preserve reserved capacity for critical messages. A bounded retry budget prevents a provider outage from multiplying the load that caused it. The handbook's distributed rate-limiting guide covers shared quotas and admission control; the capacity-estimation guide covers workload and worker-capacity estimates.

Recovery should be testable. Simulate a provider returning throttling responses, a callback arriving twice, a worker crashing after provider acceptance, a token becoming invalid, and a backlog that contains both expired and urgent messages. Verify that the service can pause one provider, continue on another, reconcile ambiguous attempts, and replay selected work without flooding recipients.

11. Important trade-offs¶

Decision	Simpler choice	More controlled choice	Cost of the stronger guarantee
Queue	Database table	Managed queue or broker	More infrastructure and replay/retention configuration
Fan-out	Create all deliveries immediately	Batched campaign expansion	More state and eventual completion
Ordering	Best effort	Per-recipient sequence	Hot partitions and lower parallelism
Retry	Fixed bounded policy	Provider-aware classifier and budgets	More adapter logic and operational tuning
Provider fallback	One provider	Health-aware routing	Duplicate and consent semantics become harder
Delivery status	Provider acceptance only	Receipts and client acknowledgements	Callback processing, retention, and incomplete coverage
Preferences	Check at request time	Re-check before each send	More reads and explicit policy versioning

Do not choose the strongest option in every row. A security alert may justify a high-priority path and a second permitted channel. A feed update may be collapsible and expire quickly. A marketing campaign may be delayed to respect a provider quota and user quiet hours. The requirements should determine the guarantee.

12. Interview questions with reasoned answers¶

Q1. Why should the API return 202 instead of waiting for delivery?¶

Provider calls are slow and independently fail. Returning after durable acceptance keeps the producer's request path bounded and lets the notification system retry asynchronously. The response must clearly state that acceptance is not delivery.

Q2. How do you avoid duplicate notifications when the producer retries?¶

Use a tenant-scoped idempotency key and request hash when accepting the logical notification. Then use a stable delivery ID across worker attempts and provider idempotency support where available. The first protects API creation; the second protects external sends. Neither creates a universal exactly-once guarantee.

Q3. What happens if a provider accepts a send but the worker times out?¶

The result is ambiguous. Persist the attempt as unknown, then reconcile through a provider query or callback if possible. If the provider supports idempotency, retry with the same key. Otherwise, a retry can duplicate the message, so the product must tolerate that possibility.

Q4. Should every notification be sent through every channel?¶

No. Resolve the eligible channels from the notification category, current user preferences, consent, endpoint availability, and product policy. Channel fallback is a deliberate rule, not an automatic reaction to every provider error.

Q5. How would you handle millions of recipients?¶

Create a durable campaign record, resolve recipients in bounded batches, and enqueue delivery intents gradually. Isolate bulk traffic from transactional alerts, enforce tenant and provider budgets, and monitor oldest-message age. Avoid one enormous synchronous fan-out transaction.

Q6. Do you need strict ordering?¶

Usually not for independent alerts. For a sequence where later state must not be shown before earlier state, partition by recipient or aggregate and maintain a sequence number. This reduces parallelism for a hot recipient and still cannot force a mobile operating system to display every push in order.

Q7. When would you use a database queue instead of Kafka or a managed queue?¶

A database queue can be adequate when volume is modest, workers can claim rows safely, and operational simplicity matters. A broker or managed queue is useful when independent consumer scaling, retention, replay, throughput, or producer isolation justify the added system. Choose from measured requirements, not familiarity with a product.

Q8. What belongs in a dead-letter queue?¶

A delivery that exhausted a bounded retry policy or requires operator investigation, together with enough metadata to understand and safely replay it. Permanent errors such as an invalid device token should normally update endpoint state rather than be retried forever.

Q9. How do you keep one tenant from delaying everyone else?¶

Use per-tenant admission and concurrency budgets, separate bulk and transactional queues, and a fair scheduling policy. Monitor queue age by tenant and priority. A single global FIFO queue can let one large campaign create head-of-line blocking.

Q10. What would you monitor to know that the service is healthy?¶

Measure API acceptance separately from provider acceptance and actual delivery receipts. Track queue age, attempt rate, provider throttling, expiry, suppression, dead letters, callback lag, and cost by channel. A low API error rate alone says little about a growing delivery backlog.

13. Design checklist¶

Before presenting or implementing a notification service, verify that you can:

Define acceptance, provider acceptance, delivery, display, and expiry separately.
Explain how the API remains idempotent when callers retry.
Model one logical notification and multiple endpoint/channel deliveries.
Choose fan-out on write or batched fan-out on read from audience size and freshness requirements.
Keep durable intent separate from transient queue messages.
Apply per-provider, per-region, per-channel, and per-tenant budgets.
Classify permanent and transient provider errors.
Bound retries by attempts, age, and notification expiry.
Explain why ambiguous provider timeouts can still produce duplicates.
Re-check preferences and consent immediately before dispatch.
Authenticate and deduplicate provider callbacks.
Measure queue age and delivery evidence rather than relying on API success alone.
Test provider outages, worker crashes, stale endpoints, duplicate callbacks, and replay.

The core design principle is simple: store the user's intent durably, turn it into independently tracked deliveries, and make every external boundary explicit about what it can and cannot guarantee. A reliable notification service is not one that never sees a failed send. It is one that can explain each outcome, stop unsafe retries, preserve eligible work, and recover without surprising recipients.