ayushdoesdev
← All work

Independent lab · not an employer deliverable

PromptOps

LLM Gateway and Evaluation Studio

Route sample tasks through model profiles, compare prompt versions, and score fixtures without calling a model API.

Jump to interactive demo

Problem

LLM features fail in operations: the wrong model, an unbounded prompt, a schema break, an untested fallback. PromptOps is the control plane Cortex is not—routing, budgets, evaluation, and traces over fixtures.

Interactive demo

Fixture responses and sample rates. No real LLM calls are performed.

PromptOps · LLM gateway studioFixture responses · no model API

Task

Order sku=api-plan amount=49 currency=USD tenant=northwind

Route

Route quality · 65 tok · $0.00078 modeled · 820 ms sample

{"sku":"api-plan","amount":49,"currency":"USD","tenant":"northwind"}

Prompt comparison

v1 passes 6/6. v2 passes 6/6.

  • pass · JSON shapeFixture parsed as JSON.
  • pass · Required fieldsLooks for sku, label, customer, or status on the fixture.
  • pass · Length limitOutput length 68 vs 280-character demo cap.
  • pass · Source referencesv1 summary fixture has no sources line.
  • pass · Policy fixtureCost route + classify v1 is the canned policy failure.
  • pass · Fallback pathPreferred fixture stayed within budget.

Trace

Preferred (quality) → quality

Architecture

A gateway selects a route from policy (quality, latency, cost, fallback). Prompt versions map to deterministic response fixtures. An evaluation suite checks JSON shape, required fields, length, citations, and policy. Token cost uses labelled sample rates. No LLM is invoked.

Task + prompt version → route policy → budget and retry checks → fixture selection → output validation → trace → optional fallback fixture.

Decisions

  • — Keep every displayed completion as a named fixture so nothing looks generated.
  • — Score evaluation from actual check functions on those fixtures.
  • — Make route, budget, and fallback changes alter the trace and selected output.

Trade-offs

  • — Cheaper routes reduce illustrative cost and they can fail schema checks that the quality route passes.
  • — Strict JSON validation catches drift and it rejects slightly malformed but useful text.
  • — Fallbacks improve availability and they can hide that the preferred route is failing.

Failure modes

  • — A token budget below the fixture size forces the fallback route.
  • — A policy fixture fails when the selected prompt version includes a disallowed instruction.
  • — Retries increment the trace without changing the fixture unless fallback is enabled.

Evaluation

Pass/fail counts come from the local evaluation suite. Latency and cost are labelled sample assumptions.

Production extensions

Proposed — not implemented in this portfolio.

  • — Call a real model behind the same route and trace UI.
  • — Store prompt versions and eval sets in a registry.
  • — Add human-review queues for policy failures.

Python · FastAPI · LLM gateway patterns · structured outputs · evaluation · request tracing