Skip to content

HTTP API Semantics, Idempotency, and Pagination

Reliable APIs are designed for retries, partial failures, concurrent clients, and changing datasets—not only for the happy path. This guide connects HTTP's standardized method semantics to practical API contracts for errors, duplicate writes, and pagination.

1. Start with HTTP semantics

HTTP methods describe the intended operation on a resource. They are not merely labels for controller functions.

Method Typical use Safe? Idempotent?
GET Read a representation Yes Yes
HEAD Read representation metadata Yes Yes
POST Submit data or trigger resource-specific processing No No, by default
PUT Create or replace the state at a known target URI No Yes
PATCH Apply a partial modification No Not inherently
DELETE Remove the target resource No Yes

These properties come from HTTP semantics, not from whether an implementation happens to produce the same response body every time.

Safe versus idempotent

  • Safe means the client does not request a state change. Incidental server-side effects such as access logging do not change that classification.
  • Idempotent means repeating the same request has the same intended effect on server state as applying it once. The response does not have to be byte-for-byte identical.

For example, deleting an existing resource may return 204 No Content, while a repeated DELETE may return 404 Not Found. The intended state—resource absent—is still the same.

HTTP defines GET, HEAD, OPTIONS, and TRACE as safe. Safe methods are also idempotent. PUT and DELETE are idempotent, while POST is not defined as idempotent by default.

Retry rule: a client can generally retry an idempotent request after an ambiguous connection failure. It should not automatically retry a non-idempotent request unless the application has an additional mechanism that makes the operation safe to repeat.

Do not use GET for operations that mutate business state. Crawlers, prefetchers, caches, and other automated clients assume safe methods do not request state changes.

2. Make POST retries safe with idempotency

Consider a payment API:

POST /v1/payments HTTP/1.1
Content-Type: application/json
Idempotency-Key: "8e03978e-40d5-43e8-bc93-6894a57f9324"

{
  "order_id": "ord_123",
  "amount": 4999,
  "currency": "INR"
}

The server creates the payment, but the connection drops before the client receives the response. The client cannot know whether the payment was created. Retrying without protection can create a second payment.

An idempotency key lets the server recognize retries that represent the same client intent. It is an application-level contract: sending a header alone does not make an endpoint idempotent unless the server implements and documents the behavior.

A robust server-side contract

Define these rules before implementation:

  1. Scope keys to the authenticated principal or tenant and the operation. A key from one customer must not collide with another customer's key.
  2. Require a high-entropy, client-generated key. A UUID is a practical choice.
  3. Bind the key to a request fingerprint. Reusing the same key with a different payload must be rejected, not silently treated as a retry.
  4. Persist the outcome and replay the original result for a completed duplicate.
  5. Define key retention and expiry. The retry guarantee only lasts as long as the record is retained.
  6. Specify what happens when two identical requests arrive concurrently.
  7. Never let a client choose a key that exposes another principal's stored response.

The IETF HTTPAPI working group has an active Internet-Draft for the Idempotency-Key header. As of October 2026, it is still a draft, not a published RFC. Its current syntax uses an HTTP Structured Field string, hence the quoted example above. Treat the header contract as something your API must document, and check the draft's status before claiming standardization.

Database-backed implementation

For operations whose effects are contained in one relational database, keep the idempotency record, business mutation, and stored response in the same database transaction.

A simplified PostgreSQL table:

CREATE TABLE idempotency_records (
    tenant_id       TEXT NOT NULL,
    idempotency_key TEXT NOT NULL,
    request_hash    TEXT NOT NULL,
    response_status INTEGER,
    response_body   JSONB,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    expires_at      TIMESTAMPTZ NOT NULL,
    PRIMARY KEY (tenant_id, idempotency_key)
);

A transaction can follow this sequence:

BEGIN

1. Attempt to insert (tenant_id, key, request_hash).
   - A unique constraint arbitrates concurrent use of the same key.
2. If this transaction owns the new record:
   - Validate business rules.
   - Apply the business mutation.
   - Store the response status and body in the idempotency row.
   - COMMIT.
3. If the key already exists:
   - Read the existing record.
   - If the request hash differs, reject key reuse.
   - If the request hash matches and a result is complete, replay it.
   - If the original operation is still in progress, apply the documented
     concurrent-request policy.

ROLLBACK on failure.

With a single transaction and a unique constraint, a competing insert can wait for the first transaction to commit or roll back. After commit, the duplicate can read the completed result; after rollback, it can attempt the operation itself. Exact locking and conflict behavior should be verified against the chosen database and isolation level.

This pattern is appropriate when the business mutation and idempotency result can commit atomically in the same database. It does not make an external side effect—such as charging a remote payment provider or sending an email—atomic with your database transaction. For those workflows, use the provider's own idempotency mechanism where available, persist a durable operation state, and consider a transactional outbox plus reconciliation. Never claim end-to-end exactly-once execution merely because an idempotency table exists.

Failure cases to design for

Situation Expected behavior
Same key, same payload, first request completed Replay the stored result
Same key, different payload Reject; do not execute the new payload
Same key, concurrent request Wait, return a documented conflict/in-progress response, or use another explicit policy
Application crashes before transaction commit Transaction rolls back; retry can attempt the operation
Application crashes after commit but before response reaches client Retry reads and replays the committed result
Idempotency record expires The guarantee for that key may no longer hold; document the retention window

Do not store only a boolean such as processed = true. A retry needs enough information to return a result consistent with the original operation. Also decide which response headers are safe and useful to replay; do not blindly persist hop-by-hop or request-specific headers.

3. Return machine-readable API errors

Use HTTP status codes for the general class of outcome and a stable response format for actionable details. RFC 9457 defines Problem Details for HTTP APIs and obsoletes RFC 7807.

Example validation response:

HTTP/1.1 422 Unprocessable Content
Content-Type: application/problem+json

{
  "type": "https://api.example.com/problems/validation-error",
  "title": "Request validation failed",
  "status": 422,
  "detail": "One or more fields are invalid.",
  "instance": "/v1/orders",
  "errors": [
    {
      "pointer": "#/quantity",
      "detail": "Must be greater than zero."
    }
  ]
}

Important design points:

  • Use a stable type URI to identify the problem category. Document application-specific problem types.
  • Keep title short and stable for the problem type.
  • Use detail for a safe, human-readable explanation of this occurrence. Do not expose stack traces, SQL, secrets, internal hostnames, or sensitive data.
  • Add documented extension members such as field-level validation errors.
  • Clients should branch on status, problem type, and documented extensions—not parse human-readable detail strings.
  • Do not invent a new error envelope when standard Problem Details fits the API.

Choose status codes according to their semantics. For example, use 401 when valid authentication credentials are absent or invalid, 403 when the authenticated caller is not allowed to perform the action, 404 when the resource is not found or its existence is intentionally concealed, 409 for a conflict with current resource state, and 422 when the content is understood but its instructions cannot be processed. Do not use 200 for failed business operations.

4. Paginate collections deliberately

Returning an unbounded collection creates unpredictable response sizes, memory pressure, database work, and latency. Pagination is part of the API contract.

Offset pagination

GET /v1/orders?limit=20&offset=40

Advantages: - Simple to implement and understand. - Supports jumping to an approximate page number. - Useful for small, relatively static datasets.

Trade-offs: - Large offsets can require the database to scan and discard many rows. - Inserts and deletes between requests can shift page boundaries, causing duplicates or skipped rows. - Results need a deterministic ORDER BY; ordering by a non-unique column alone is not enough.

Cursor/keyset pagination

For frequently changing or large datasets, keyset pagination is often more predictable. Sort by a stable, unique tuple and use the last row from the previous page as the next boundary.

Example for PostgreSQL:

-- First page
SELECT id, created_at, status
FROM orders
WHERE tenant_id = :tenant_id
ORDER BY created_at DESC, id DESC
LIMIT :page_size_plus_one;

-- Following page: use the final row's (created_at, id)
SELECT id, created_at, status
FROM orders
WHERE tenant_id = :tenant_id
  AND (created_at, id) < (:cursor_created_at, :cursor_id)
ORDER BY created_at DESC, id DESC
LIMIT :page_size_plus_one;

The unique id tie-breaker matters: timestamps alone may not uniquely order rows. Fetch one extra row to determine whether another page exists, then return only the requested page size.

A public API should usually expose an opaque cursor rather than raw database values. Encode the ordering values and relevant query context, and authenticate the cursor (for example, with an HMAC) if clients must not be able to tamper with it. Bind it to the filter, sort order, and tenant context; validate all of those again on every request. A cursor is not an authorization token—authorization must still be checked for every page.

Keyset pagination avoids offset drift before the cursor for a stable ordering, but it does not automatically provide a snapshot of the entire dataset. Rows can still change or be deleted between requests. If the product requires a consistent export or snapshot, define a snapshot boundary or use a database-supported snapshot strategy with an explicit lifetime.

Pagination contract checklist

  • Set a server-enforced maximum page size; reject or clamp excessive values consistently.
  • Define a stable default sort and include a unique tie-breaker.
  • Keep tenant and authorization filters in every page query.
  • Preserve filters and sort order in the cursor context.
  • Return a clear next-page indicator, such as next_cursor: null when complete.
  • Document whether totals are exact, estimated, or omitted. Exact counts can be expensive.
  • Define behavior when a cursor is malformed, expired, or used with different filters.

5. Interview questions

  1. Is POST always non-idempotent?
    HTTP does not define POST as idempotent by default. An application can design a particular POST operation to be safely repeatable, but clients need a documented contract or mechanism before retrying it automatically.

  2. Does idempotent mean the response is identical?
    No. It means repeated identical requests have the same intended server-side effect. Status codes and response bodies can differ.

  3. Why is a unique constraint necessary for idempotency?
    Application-level “check then insert” logic races under concurrency. A database uniqueness constraint provides an atomic arbitration point for a key within its defined scope.

  4. Why store the response with the business mutation?
    If the business operation commits but its replay record does not, a retry can execute the operation again. A shared transaction closes that gap for effects in the same database.

  5. Does a database transaction guarantee exactly-once payment processing?
    No. A remote payment provider is outside the local transaction. Use provider-side idempotency, durable workflow state, and reconciliation for cross-system outcomes.

  6. When would you choose keyset pagination over offset pagination?
    For large or frequently changing collections where stable continuation and predictable query cost matter more than jumping directly to arbitrary page numbers.

  7. Why include a unique tie-breaker in cursor ordering?
    Non-unique sort values can make row boundaries ambiguous, producing skipped or repeated rows across pages.

  8. Should clients parse Problem Details detail text?
    No. Use the HTTP status, stable problem type, and documented extension fields. Human-readable detail is not a machine-stable interface.

6. Further reading