feat: cut per-turn tokens, add daily token budgets, and enforce safety in code #15

Closed

Jc-965 wants to merge 0 commits from optimization-token-limit into main

Jc-965 commented

2026-08-07 03:01:12 +00:00

(Migrated from codeberg.org)

Summary

Per-turn input cost is down about 60%, with tool-enabled turns dropping from roughly 13k to 5k tokens in live Haiku tests. Each user now has a 1 million-token daily budget, and key safety rules are enforced directly in code instead of relying on the prompt.

The system prompt now uses a compact, names-only tool catalog, and each request binds only the tool groups it likely needs. Selection checks keywords first, then earlier user turns, then falls back to guide-only tools. Ambiguous campus queries still get the full tool set.
Static input for chitchat drops from about 3.4k to 1.6k tokens.
Provider-reported usage is recorded after every model pass in SQLite across both Uvicorn workers. The API returns a 429 before making a model call once a user reaches 1 million tokens in a UTC day.
A deterministic output guard redacts secrets and replaces responses that quote the system prompt. Streamed output passes through a holdback scrubber so leaks are caught before reaching the client.
Expected text such as crisis numbers and the standard refusal format is allowlisted to avoid false positives.
Clear jailbreak attempts receive a fixed refusal without any model or MCP calls.
Transport errors no longer expose internal hosts, and requests are limited to a 256 KB body, a 4,000-character query, and 200 history items.

### Summary Per-turn input cost is down about 60%, with tool-enabled turns dropping from roughly 13k to 5k tokens in live Haiku tests. Each user now has a 1 million-token daily budget, and key safety rules are enforced directly in code instead of relying on the prompt. - The system prompt now uses a compact, names-only tool catalog, and each request binds only the tool groups it likely needs. Selection checks keywords first, then earlier user turns, then falls back to guide-only tools. Ambiguous campus queries still get the full tool set. - Static input for chitchat drops from about 3.4k to 1.6k tokens. - Provider-reported usage is recorded after every model pass in SQLite across both Uvicorn workers. The API returns a 429 before making a model call once a user reaches 1 million tokens in a UTC day. - A deterministic output guard redacts secrets and replaces responses that quote the system prompt. Streamed output passes through a holdback scrubber so leaks are caught before reaching the client. - Expected text such as crisis numbers and the standard refusal format is allowlisted to avoid false positives. - Clear jailbreak attempts receive a fixed refusal without any model or MCP calls. - Transport errors no longer expose internal hosts, and requests are limited to a 256 KB body, a 4,000-character query, and 200 history items.