← WorkModelCache

ModelCache — Semantic caching backend

Reuse an LLM answer when the next question is the same idea in different words.

People ask the same question in slightly different words. A normal cache treats each wording as new work, so the model runs again.

How a request is decided

ModelCache decides whether a previous answer is close enough to reuse. It embeds the prompt, searches nearby vectors, and checks a similarity floor for that topic. Close enough, and Redis returns the cached response. Not close enough, and GPT-4o answers — then the result is stored.

A prompt, a score, a hit or a miss

The playground is where you watch that decision. You send a prompt. You see the score. You see a hit or a miss.

What got reused
Score, topic, TTL

Each cache entry keeps the score, the topic, and a TTL. Reuse is something you can inspect. A health view watches Redis, FAISS, and OpenAI.

Redis, FAISS, OpenAI

What I built

Embeddings, FAISS vector search, Redis caching, topic and intent classification, GPT-4o fallback, and a Next.js dashboard for query streaming, cache stats, and system health.

What happens to a prompt

The request is classified, embedded, and compared to stored answers. If the score clears the floor for that topic, Redis returns the cached response. If not, GPT-4o answers and the result is stored.

  1. 1.Prompt
  2. 2.Cache key
  3. 3.Topic and intent
  4. 4.Embedding
  5. 5.FAISS search
  6. 6.Similarity check
  7. 7.Hit: return Redis
  8. 8.Miss: stream GPT-4o, then store

Key details

  • Similar prompts can share a response even when the wording differs. FAISS does exact cosine search on L2-normalized vectors.
  • Intents like weather, news, how_to, history, finance, and sports make retrieval easier to inspect.
  • Evergreen, semi-persistent, and ephemeral topics use different similarity floors and Redis TTLs.
  • Prometheus and the dashboard show hit or miss, similarity, latency, TTLs, and Redis / FAISS / OpenAI health. No invented cost or hit-rate claims.

Built with

Python, FastAPI, Redis 7, FAISS IndexFlatIP, OpenAI text-embedding-3-small, OpenAI GPT-4o, Prometheus, Docker Compose, Next.js, TypeScript, Tailwind, shadcn/ui.

How it works

How close is close enough

Each prompt becomes a 1536-dimension vector, L2-normalized. FAISS compares with inner product, which is cosine similarity on those vectors. Long inputs are chunked, embedded, and mean-pooled first.

  • The default floor is 0.85. Evergreen, semi-persistent, and ephemeral topics raise or lower that floor and set Redis TTLs from minutes to 90 days.
  • Finance, weather, dates, and cities do not reuse a near-duplicate if the context could make the old answer wrong.
  • Embeddings started on Qwen, then moved to OpenAI text-embedding-3-small for cost, latency, and a simpler integration.

What ships

FastAPI, Redis, FAISS, Docker Compose, Prometheus, and a Next.js dashboard. This page does not invent hit rates or cost savings.