01 / FastAPI gateway
Validates input, bounds generation, assigns UUIDs, and applies Redis-backed request limits. An optional bearer token protects the backend connection.
THE SYSTEM, EXPLAINED
Every layer. Nothing hidden.
One streaming path, from a browser prompt to GPU execution and back.
INFERENCE DATA PLANE HTTP → SSE
BrowserWAITING
FastAPIWAITING
vLLMWAITING
GPUWAITING
Token streamWAITING
The frontend sends a request through its same-origin API proxy to FastAPI. The gateway validates and forwards it to the private vLLM server. Text deltas stream back along the same path; the final event contains measured request metrics.
02 / Redis sidecar
A side branch of the gateway, not a step before GPU execution. Stores expiring request metadata, deterministic response caches, and atomic rate-limit counters.
03 / vLLM + Hugging Face
The inference server downloads the configured open-weight model from Hugging Face at startup. Continuous batching schedules inference on the GPU.
04 / NVIDIA GPU
Runs prefill and autoregressive decoding. Model weights and attention KV caches occupy VRAM. The GPU endpoint stays on a private network.
05 / Token stream
Server-sent events carry text deltas to the browser. Stream chunks may contain several tokenizer tokens; final usage comes from the inference server.
06 / Observability branch
FastAPI records timing and usage metadata in Redis. NVIDIA tooling samples GPU telemetry when installed on the GPU host. The dashboard polls the collector.
DEPLOYMENT BOUNDARIES
Frontend anywhere. Inference on your GPU.
Browser → Frontend / API proxy → FastAPI → vLLM → NVIDIA GPU
├─ Redis
└─ Metrics collector → Dashboard
Hugging Face ── model download at startup ──→ vLLM
vLLM ── streamed text + final usage ──→ FastAPI ── SSE ──→ BrowserDemo mode is explicit: all scripted inference and simulated readings are labeled. Production disables simulation and reports unavailable telemetry as unavailable. API credentials remain on the server.