Skip to contentSkip to contact
Dennis Khor
Recruiter mode

Writing

How I keep a public AI agent cheap and safe

The guard in front of my portfolio's Claude agent: per-visitor limits, a budget reserved before every call, free replays, and evals published pass or fail.

Author
Dennis Khor, senior full-stack engineer shipping AI agents in Sydney
Published
Reading time
5 min read
Decision records

My portfolio has an agent that answers questions about my work from the site's case studies and decision records, citing every claim. It runs on Claude, so every question costs real money, and the endpoint is public: anyone can call it, as often as they like.

I wanted the real model on without risking my API credit. So the agent sits behind a guard with four layers: limits per visitor, a budget reserved before every call, free replays of fixed questions, and a free answer from the sources whenever the model cannot be used. Each layer is built so no single failure can lift the ceiling.

Every gate runs before the model

A question passes the gates in a fixed order: a size cap on the request body, a JSON content type, a check that refuses cross-site browser requests, a schema check on the question, then the per-visitor limit. All of them run before any model call, so a rejected request costs nothing.

Then the agent takes the cheapest path that can answer. Common questions about pay or private details get a fixed reply, and so do ones that read like a member asking an organisation about their own account. A question with no word in common with the write-ups gets a reply saying so. None of these call a model.

Per-visitor limits that never fail open

Each visitor gets 5 questions per 10 minutes and 20 per UTC day, counted against a salted SHA-256 hash of the IP address, so a raw address is never stored. Without a configured salt, each instance picks a random one, so nobody can match a stored hash by hashing addresses they guess.

The counts live in Upstash Redis, so every serverless instance sees the same numbers. If that store is missing or failing, the limit falls back to an in-memory counter on each instance. That is weaker, since each instance counts alone, but it is never open. In production, the model is not called at all unless the store, the salt, the API key and the private-term list are all set.

A budget reserved before the call

Rate limits stop one visitor. They do not stop a thousand. For that there is a budget: by default 1 US dollar a day and 10 US dollars a month of model spend.

The key is when it is checked. Before every model call, the agent reserves the most that call could cost: every input token at the dearest input rate and a full output cap of 1,500 tokens on the default model, plus the same worst case for the fallback model it may fail over to. It reserves against the day, then the month, each with one atomic Redis increment, so concurrent requests each see a distinct running total. A reservation that would take either total past its limit is refunded and refused, and the question is answered from the sources instead: the best-matching sentences, cited and labelled as answered without a model, at no cost. When an answer finishes, the reservation is replaced with what the call was billed, failed and abandoned attempts included, because their tokens are billed too.

A test fires 20 concurrent reservations of 0.3 against a limit of 1 through a simulated store that applies them in a shuffled order, as separate instances would reach Redis. With four different seeds, exactly 3 are admitted every time.

The questions I wrote replay for free

Most visitors tap a suggested question, or one of the strong points on the home page, instead of typing one. Each of those eight fixed questions has its first clean model answer saved for 7 days, one that stopped normally with nothing withdrawn. The cache is scoped to the deployment, so new sources never replay an answer written from old ones, and only those eight are cached, so nothing a visitor types is stored.

After every deploy I ask the eight once, in two sittings, since my questions meet the same limit as anyone's. Each first answer cost just under one US cent when the model went live, and every later visitor on a deployment gets the same cited answer free.

Citations, and a scan that can only withdraw

Retrieval is BM25 in memory over paragraph-sized chunks. The top 6 sections, at most 3 from any one document, go to the model as search results with citations on, so every cited sentence points back to its paragraph. The server maps each citation through its own list of chunks by index, never by the source string the model repeats back, so a citation can only point at a passage the server sent.

Text streams as it is generated, so the output check can only withdraw. When the answer settles, a scan looks for a marker hidden in the instructions, a phone number, any email address other than the public one, and any term from the private list. A hit replaces the whole answer with a fixed message. The trade-off is that a visitor could see the withdrawn text for a moment: the price of streaming.

Evals published whether they pass or fail

Retrieval is measured on every commit by a unit test with no model call: its recall at 6, the average share of each answerable question's expected sources that reach the top 6, must be at least 0.9, and the committed numbers must equal the live ones. When I last recomputed it, recall at 6 was 0.95. Every number in this post is pinned to its source the same way, as Keeping docs current with Claude Code and tests explains.

The model evals are paid, so they run on demand. They ask all 91 questions in the case file through the same endpoint visitors use: 55 it should answer, 10 its sources do not cover, 10 privacy probes, 10 prompt injections and 6 off-task requests. Fixed checks and a separate judge model grade each answer, and one privacy failure or one leaked marker fails the run. The runner's token that skips the visitor limit is never honoured in production. Every complete run is published, pass or fail, and until the first one the page says so.

What I would copy into any public AI feature

  • Put the ceiling in code, and check it before the call.
  • Make the free path the default whenever a dependency is missing.
  • Cache only the questions you wrote, never what visitors type.
  • Set a spend limit on the provider account too. Code is the first line of defence, not the last.

You can ask the agent yourself on the Ask page, and see how it scores on the eval dashboard.

Open to senior AI and software engineering roles: on site or hybrid in Sydney, or remote for Australian and US teams. Read my CV, or paste a job description into Ask about fit for a verdict with cited evidence.

Ask about fit

Paste a job description. You get a verdict, the evidence that matches it with citations, and what the sources do not show.

Open to senior AI and software engineering roles: on site or hybrid in Sydney, or remote for Australian and US teams.

Email meBook 15 minCV