← WorkModelCache
ModelCache system architecture diagram
Architecture

1 / 7

ModelCache — Semantic caching backend

A semantic caching and retrieval system designed to reduce LLM inference cost and latency by determining when previous responses can be safely reused.

While building LLM applications, I noticed that users often asked the same question in slightly different ways. Traditional caches rely on exact matches, so each variation still triggered another model call.

I built ModelCache to explore semantic caching using embeddings, vector search, and similarity scoring. I designed the retrieval pipeline, cache logic, provider fallback, intent classification, and observability around latency, token usage, and cost.

The project gave me a deeper understanding of retrieval systems, cache design, and the trade-offs involved in deciding when a previous response is safe to reuse.

What I built

Embeddings, FAISS vector search, Redis caching, topic and intent classification, GPT-4o fallback, and a Next.js dashboard for query streaming, cache stats, and system health.

System flow

Each request is keyed, classified, embedded, searched, ranked against a topic threshold, then either served from Redis or fulfilled by GPT-4o and written back.

  1. 1.Prompt
  2. 2.Preprocessing and cache key
  3. 3.Topic and intent classification
  4. 4.Embedding generation (text-embedding-3-small)
  5. 5.FAISS vector search (IndexFlatIP)
  6. 6.Ranking and similarity threshold
  7. 7.Cache hit → return Redis response
  8. 8.Cache miss → stream GPT-4o, store in Redis + FAISS

Key details

  • Semantic cache: similar prompts can share responses even when wording differs, using cosine similarity on L2-normalized vectors via FAISS IndexFlatIP.
  • Intent-aware retrieval: keyword intents (weather, news, how_to, history, finance, sports, general) improve retrieval and make usage analyzable.
  • Topic-based thresholds and TTLs: evergreen, semi-persistent, and ephemeral topics use different similarity floors and Redis TTLs so volatile queries expire faster.
  • Observability: Prometheus metrics and a dashboard for hit/miss, similarity, latency, TTLs, and Redis / FAISS / OpenAI health. No invented cost or hit-rate claims on this page.

Built with

Python, FastAPI, Redis 7, FAISS IndexFlatIP, OpenAI text-embedding-3-small, OpenAI GPT-4o, Prometheus, Docker Compose, Next.js, TypeScript, Tailwind, shadcn/ui.

How it works

How the similarity decision works

Queries are normalized lightly, embedded into 1536-dimensional vectors, L2-normalized, and compared with inner-product search so FAISS can do exact cosine search. Long inputs are chunked, embedded, and mean-pooled before normalization.

  • Default similarity floor is configurable (0.85); evergreen / semi-persistent / ephemeral topics tighten or relax that floor and set Redis TTLs from minutes to 90 days.
  • Guardrails filter context-sensitive categories (finance, weather, dates, cities) so near-duplicate wording does not reuse a wrong answer.
  • Embedding path started on Qwen from Hugging Face rankings, then moved to OpenAI text-embedding-3-small for integration reliability, cost, and latency.

What ships in the stack

FastAPI async API, Redis cache metadata and metrics, FAISS persistence, Docker Compose orchestration, and a Next.js App Router UI for query streaming, cache management, and system health.