An INT8 ONNX model sitting on disk is just a file. To make it useful - something that can handle real traffic, skip redundant computation, and scale without falling over - you need to build the infrastructure around it. This is the story of doing exactly that: a FastAPI gateway, a Redis caching layer, a load classifier, and a dynamic batching engine. Each piece introduced a new class of problem and each problem had a non-obvious fix.

The Cache That Was Slower Than No Cache

The first thing any inference service needs is a cache. Running the same embedding twice is pure waste - so the architecture was simple: hash the input text with SHA256, check Upstash Redis (Mumbai, ap-south-1), return the stored result on a hit, run the model and write to Redis on a miss.

The first benchmark run looked like this:

The model alone ran at 43.8 ms. The cache was adding roughly 580 ms on top of every request - including the hits. Something was catastrophically wrong.

The Culprit: A New TLS Handshake on Every Redis Command

The bug was a single architectural mistake in the Redis client. The original _cmd() method looked like this:

# BAD: creates a brand new HTTPS connection on every call
async def _cmd(self, *args):
    async with httpx.AsyncClient() as client:
        resp = await client.post(self._url, ...)

Every Redis command - every GET, every SET, every cache hit - was spinning up a fresh HTTPS connection. That means a full TLS handshake each time. From Goa to Mumbai, that costs you:

22–42 ms is the actual cost of a Mumbai round-trip. Everything above that was TLS overhead being paid on every single call. The fix is straightforward: create the AsyncClient once at startup and reuse it for the lifetime of the process.

# GOOD: one client, one connection pool, reused forever
class RedisCache:
    def __init__(self, url: str, token: str):
        self._client = httpx.AsyncClient(
            headers={"Authorization": f"Bearer {token}"},
            timeout=5.0,
            limits=httpx.Limits(
                max_connections=10,
                max_keepalive_connections=5,
            ),
        )

    async def _cmd(self, *args):
        resp = await self._client.post(self._url, json=list(args))

One other detail: add await cache.close() to the FastAPI lifespan shutdown handler. Without it, the connection pool doesn't drain cleanly when the server terminates.

After the Fix

The numbers now make physical sense. 42.7 ms on a cache hit ≈ one Mumbai round-trip (~30 ms) plus FastAPI overhead. 112.3 ms on a miss ≈ inference (43.8 ms) + Redis write (~40 ms) + overhead.

Throughput at Scale: Dynamic Batching

A 70% cache hit rate is great. But what happens to the 30% of requests that actually reach the model? Under high concurrency, each cache miss triggers its own ONNX forward pass - a separate tokenisation call, separate matrix operations, separate overhead. At 24 concurrent requests, you're paying that fixed cost 24 times over.

The solution is dynamic batching: group concurrent cache-miss requests together and run them as a single ONNX forward pass. Tokenise once for N items, get back N embeddings, dispatch each one to its waiting caller. The fixed overhead gets amortised across the whole batch.

The Load Classifier

Before batching, you need to know when to batch. A single request should be served immediately - holding it hostage for 50 ms waiting for friends that may never arrive is just adding pointless latency. But under real backpressure, batching is the right call.

The classifier uses queue depth rather than requests-per-second. RPS is a lagging indicator. Queue depth is instantaneous:

class LoadTier(Enum):
    LOW    = "low"
    MEDIUM = "medium"
    HIGH   = "high"

def classify_load(queue_depth: int) -> LoadTier:
    if queue_depth < LOW_THRESHOLD:   # < 3
        return LoadTier.LOW
    if queue_depth < HIGH_THRESHOLD:  # < 10
        return LoadTier.MEDIUM
    return LoadTier.HIGH

The Batching Mechanism

Each cache-miss request calls batcher.submit(text), which drops a BatchItem containing the text and an asyncio.Future into a shared queue, then awaits that Future. The caller is suspended, not polling.

# BatchWorker._run_batch() - one ONNX call for N items
async def _run_batch(self, batch: list[BatchItem]):
    embeddings = await loop.run_in_executor(
        None,
        engine.run_batch,
        [item.text for item in batch],
    )
    for item, emb in zip(batch, embeddings):
        item.future.set_result(emb)  # unblocks each waiting caller

Benchmark Results

Three runs at increasing concurrency, all with unique texts to force cache misses and isolate inference performance:

What I Took Away

• Profile your infrastructure, not just your model: The ONNX model was running at 43.8 ms. The naive Redis client was adding 580 ms on top. Measure the full request path before assuming the model is the bottleneck.
• HTTP connection reuse is not optional at sub-100 ms targets: TLS handshakes cost 100-400 ms in real network conditions. Any client that creates a new connection per request will blow your latency budget instantly.
• Queue depth beats RPS for load classification: RPS tells you what load was. Queue depth tells you what load is. For batching decisions, instantaneous queue depth is vastly superior.
• Batch size plateaus at a hardware sweet spot: Throwing more concurrency at the system didn't grow the batch beyond ~8.65. The 50 ms window filled to the ceiling of what one CPU core could parallelise. Know your hardware's floor.