Put the context budget in your pipeline
Send a configuration — your sub-agent and skill definitions, your rule and command
files, the MCP config (.mcp.json), and the top-level project-convention file your
assistant reads on every turn
— together with your own measurement of what each file costs, and
get back one JSON object: every component classified always, sometimes or
rarely needed against the project it belongs to, the issues that matter with what each
one costs in tokens, and a ranked list of trims with the tokens each recovers. That makes it a
CI gate ("fail the build when the setup passes 20% of the window"), a pre-commit check on a
shared .claude directory, a nightly report across every repo in the org, or a
one-shot answer to why a session feels slow. Everything the app does goes through the SkillSafe
App API — plain JSON over HTTPS. One thing is different from most APIs here, and step 3 is
entirely about it: you measure the files, not the model, and your measurements
are the authority the reply is checked against. Every code step below is shown in cURL, Python,
JavaScript, Go, Java, Ruby, PHP and C#; pick a language once and the whole page follows.
Basics
Base URL: https://api.skillsafe.ai/v1/app-api, app slug
context-lens. Every request sends
Authorization: Bearer <token> and JSON bodies with
Content-Type: application/json. Responses are wrapped in an envelope:
{"data": …} on success, {"error": {"code", "message"}} on failure.
The audit is produced by the gpt-terra model. Estimates are free; runs are
metered against your credit balance. There is a single run task — one configuration in,
one audit out, no follow-up calls and no session state to carry.
| Status | Meaning |
|---|---|
401 | Missing or expired token — create a new session. |
402 | Not enough credits — top up at skillsafe.ai/account/credits. |
403 | The token isn't allowed to do this (e.g. a guest submitting a very large inventory). |
404 | Unknown job or record id. |
5xx | Transient platform error — retry with backoff. |
Browsers enforce CORS for this API, so run these examples from a server, script or terminal — not from another website's frontend.
Step 0 — A tiny client
Every task below is a single HTTP call, so start with a short helper that adds the auth
header, sends JSON and unwraps the data envelope. The later steps reuse it.
export API="https://api.skillsafe.ai/v1/app-api"
export TOKEN="YOUR_TOKEN" # see step 1
# every call looks like:
# curl -s "$API/..." -H "Authorization: Bearer $TOKEN" [-d '{json}']
# jq is used below to pull fields out of the {"data": ...} envelope,
# and again in step 3 to build the inventory itselfimport json, requests
API = "https://api.skillsafe.ai/v1/app-api"
TOKEN = "YOUR_TOKEN" # see step 1 — read it from your shell environment in real code
def api(method, path, body=None, **headers):
res = requests.request(method, API + path, json=body,
headers={"Authorization": f"Bearer {TOKEN}", **headers})
payload = res.json()
if not res.ok:
raise RuntimeError(payload.get("error", {}).get("message", res.reason))
return payload["data"]// Node 18+ (built-in fetch)
const API = "https://api.skillsafe.ai/v1/app-api";
const TOKEN = "YOUR_TOKEN"; // see step 1 — read it from your shell environment in real code
async function api(method, path, body, extraHeaders = {}) {
const res = await fetch(API + path, {
method,
headers: { Authorization: `Bearer ${TOKEN}`, "Content-Type": "application/json", ...extraHeaders },
body: body === undefined ? undefined : JSON.stringify(body),
});
const json = await res.json();
if (!res.ok) throw new Error(json.error?.message ?? res.statusText);
return json.data;
}package main
import (
"bytes"
"encoding/json"
"fmt"
"net/http"
"os"
)
const API = "https://api.skillsafe.ai/v1/app-api"
var token = os.Getenv("SKILLSAFE_TOKEN") // see step 1
func call(method, path string, body, out any) error {
var buf bytes.Buffer
if body != nil {
json.NewEncoder(&buf).Encode(body)
}
req, _ := http.NewRequest(method, API+path, &buf)
req.Header.Set("Authorization", "Bearer "+token)
req.Header.Set("Content-Type", "application/json")
res, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
defer res.Body.Close()
var env struct {
Data json.RawMessage `json:"data"`
Error *struct{ Message string `json:"message"` } `json:"error"`
}
json.NewDecoder(res.Body).Decode(&env)
if res.StatusCode >= 400 {
return fmt.Errorf("api %s %s: %s", method, path, env.Error.Message)
}
if out == nil {
return nil
}
return json.Unmarshal(env.Data, out)
}// Java 17+, no dependencies. Pair with your JSON library (Jackson, Gson…)
// to build the inventory in step 3 and read fields out of the envelope.
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
public class SkillSafe {
static final String API = "https://api.skillsafe.ai/v1/app-api";
static final String TOKEN = System.getenv("SKILLSAFE_TOKEN"); // see step 1
static final HttpClient HTTP = HttpClient.newHttpClient();
static String api(String method, String path, String jsonBody) throws Exception {
var req = HttpRequest.newBuilder(URI.create(API + path))
.header("Authorization", "Bearer " + TOKEN)
.header("Content-Type", "application/json")
.method(method, jsonBody == null
? HttpRequest.BodyPublishers.noBody()
: HttpRequest.BodyPublishers.ofString(jsonBody))
.build();
var res = HTTP.send(req, HttpResponse.BodyHandlers.ofString());
if (res.statusCode() >= 400) throw new RuntimeException(res.body());
return res.body(); // envelope: {"data": …}
}
}require "net/http"
require "json"
API = "https://api.skillsafe.ai/v1/app-api"
TOKEN = ENV.fetch("SKILLSAFE_TOKEN") # see step 1
def api(method, path, body = nil, extra = {})
uri = URI(API + path)
req = Net::HTTP.const_get(method.capitalize).new(uri)
req["Authorization"] = "Bearer #{TOKEN}"
req["Content-Type"] = "application/json"
extra.each { |k, v| req[k] = v }
req.body = body.to_json if body
res = Net::HTTP.start(uri.host, uri.port, use_ssl: true) { |h| h.request(req) }
payload = JSON.parse(res.body)
raise (payload.dig("error", "message") || res.message) unless res.is_a?(Net::HTTPSuccess)
payload["data"]
end<?php
const API = "https://api.skillsafe.ai/v1/app-api";
$TOKEN = getenv("SKILLSAFE_TOKEN"); // see step 1
function api(string $method, string $path, ?array $body = null, array $extra = []): mixed {
global $TOKEN;
$ch = curl_init(API . $path);
curl_setopt_array($ch, [
CURLOPT_CUSTOMREQUEST => $method,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_HTTPHEADER => array_merge([
"Authorization: Bearer $TOKEN",
"Content-Type: application/json",
], $extra),
CURLOPT_POSTFIELDS => $body === null ? null : json_encode($body),
]);
$payload = json_decode(curl_exec($ch), true);
$status = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
curl_close($ch);
if ($status >= 400) {
throw new Exception($payload["error"]["message"] ?? "HTTP $status");
}
return $payload["data"];
}// .NET 8+
using System.Net.Http.Json;
using System.Text.Json;
static class SkillSafe
{
const string Api = "https://api.skillsafe.ai/v1/app-api";
static readonly HttpClient Http = new();
static SkillSafe() =>
Http.DefaultRequestHeaders.Authorization =
new("Bearer", Environment.GetEnvironmentVariable("SKILLSAFE_TOKEN")); // see step 1
public static async Task<JsonElement> ApiAsync(HttpMethod method, string path, object? body = null)
{
var req = new HttpRequestMessage(method, Api + path);
if (body != null) req.Content = JsonContent.Create(body);
var res = await Http.SendAsync(req);
var json = await res.Content.ReadFromJsonAsync<JsonElement>();
if (!res.IsSuccessStatusCode)
throw new Exception(json.GetProperty("error").GetProperty("message").GetString());
return json.GetProperty("data");
}
}Step 1 — Get a token
POST /guest
A guest token lets you check balances and estimate costs for free. For metered audits billed to
your own account, use your personal token: open the
token page, sign in with SkillSafe, and press
Copy shell export — it puts export SKILLSAFE_TOKEN="…" on your
clipboard, which every example below reads. Treat the token like a password: it can spend
your credits. For fully headless scripts, POST /guest mints a guest token with
no browser involved.
curl -s -X POST "$API/guest" \
-H "Content-Type: application/json" \
-d '{"slug":"context-lens"}' | jq -r '.data.token'token = api("POST", "/guest", {"slug": "context-lens"})["token"]const { token } = await api("POST", "/guest", { slug: "context-lens" });var guest struct{ Token string `json:"token"` }
err := call("POST", "/guest", map[string]string{"slug": "context-lens"}, &guest)String envelope = api("POST", "/guest", """
{"slug":"context-lens"}""");
// token is at data.token in the returned JSONtoken = api("POST", "/guest", { slug: "context-lens" })["token"]$token = api("POST", "/guest", ["slug" => "context-lens"])["token"];var guest = await SkillSafe.ApiAsync(HttpMethod.Post, "/guest",
new { slug = "context-lens" });
var token = guest.GetProperty("token").GetString();
The app stores this browser's token under the localStorage key
skillsafe_app_token:context-lens, on the app's own origin. The
token page reads and manages it for you — you never need
to open developer tools, and nobody should ever be instructed to.
Step 2 — Check who you are and your balance
GET /me
Returns subject_type ("user" or "guest"),
subject_id and your credits balance. Worth checking before you loop
over forty repositories.
curl -s "$API/me" -H "Authorization: Bearer $TOKEN" | jq '.data'me = api("GET", "/me")
print(me["subject_type"], me["credits"])const me = await api("GET", "/me");
console.log(me.subject_type, me.credits);var me struct {
SubjectType string `json:"subject_type"`
Credits int64 `json:"credits"`
}
err := call("GET", "/me", nil, &me)String envelope = api("GET", "/me", null);
// data.subject_type, data.creditsme = api("GET", "/me")
puts "#{me["subject_type"]}: #{me["credits"]} credits"$me = api("GET", "/me");
echo "{$me['subject_type']}: {$me['credits']} credits\n";var me = await SkillSafe.ApiAsync(HttpMethod.Get, "/me");
Console.WriteLine($"{me.GetProperty("subject_type")}: {me.GetProperty("credits")} credits");Step 3 — Build the inventory
This is the step that makes this app different, so it is worth a paragraph before the code.
In the browser, a scanner (ctxscan.js) measures every dropped file
before anything is sent, and those measurements travel with the run as
inventory. The model is instructed that they are authoritative: it may not
contradict a measured number, may not claim a saving larger than the measured
tokens of the component it proposes to remove, and may not name a component that
is not in the inventory. The app then re-checks all three in the browser and shows the user
every place the reply disagrees. As an API caller you are the scanner —
there is no server-side fallback that will measure the files for you, so an empty or invented
inventory produces an audit grounded in nothing.
The good news is that all of it is arithmetic, and none of it needs a tokenizer. Here are the rules, exactly as the browser applies them.
| Measurement | Rule |
|---|---|
| Token estimate | Walk the file line by line. A line whose first non-space characters are ``` or ~~~ toggles fenced mode and counts as code. Inside a fence, or on any non-blank line indented by four spaces or a tab, the line counts as code at len(line) + 1 characters. Every other line is prose, contributing its whitespace-separated word count. Then tokens = round(code_chars / 4 + prose_words * 1.3). code_ratio is code characters over total characters, to two decimals — it is what lets the report say why a file is expensive. |
| Component kind — by filename | Resolution order is path, then filename, then a content sniff. Taking filenames first: .mcp.json, mcp.json and any *.mcp.json are kind mcp. A file named SKILL.md is kind skill. Kind memory covers the top-level project-convention files, under whichever of the four well-known names your assistant uses. |
| Component kind — by directory | A path segment decides the rest. Anything below a directory named agent or its plural is kind agent; below skill or skills is kind skill; below rule or rules is kind rule; below command or commands is kind command. |
| Component kind — loose files | For a file with no directory to go on: frontmatter carrying both name and description makes it a skill when the description matches use (this) skill or the body opens with a # Skill heading, otherwise an agent; a .json containing "mcpServers" is mcp; any other .md or .mdc is a rule; everything else is other. |
| Frontmatter | A leading --- block, scalar keys only, plus the folded and literal block forms (description: >, description: |, >-, |-) whose indented continuation lines are joined with single spaces — that form is exactly where the bloated descriptions hide. One leading and one trailing quote are stripped. description_words is the whitespace word count of the result; over 30 words is a flag, because that text sits in the context of every Task call whether or not the agent is ever invoked. name falls back to the base name with its extension removed. |
| Duplicates | Hash each file's exact text. The first occurrence is counted: true; every later one gets counted: false and duplicate_of pointing at the first path, and its tokens are excluded from the totals. (The browser uses FNV-1a for this; any content hash groups the files identically.) |
| MCP servers | Parse the config and read mcpServers, mcp_servers or servers. Per server: command is command joined with args (or the url); tools comes from tools.length, toolCount or tool_count, and when none of those exist it is 8 with assumed: true; tokens = tools * 500. wraps is the first CLI name (gh, git, npm, docker, psql, aws, terraform, filesystem, …) found as a whole word in the server's name, command or URL. A server name declared twice keeps its first definition. |
| Overlaps | Between always-on files only — the rule and memory ones, where redundancy is permanently resident. Lowercase the text, drop fenced blocks and punctuation, take word trigrams, and compare each pair by Jaccard similarity. A score at or above 0.25 is an overlap; sort descending and keep the top eight. |
| Totals | file_tokens is the sum over counted components, mcp_tokens the sum over servers, overhead_tokens their sum. pct is overhead / window * 100 to one decimal, available_tokens is max(0, window - overhead), and by_kind maps each kind to {count, tokens}, with an mcp entry added when servers exist. |
| Excerpts | The model still needs to read the files to judge relevance, but a real setup is hundreds of kilobytes. Give each counted component a share of a character budget (the app uses 44,000) weighted by (1 + 2 if memory + 1 if rule + 1.5 if flagged) * sqrt(chars), floor 400. A file inside its share is sent whole; a larger one is sent head-and-tail — 70% head, 30% tail, with a marker naming how many characters of the middle were withheld. Never a blind cut, and never silence about it: excerpt_note states what was profiled locally versus sent, and is shown to the user verbatim. |
The thresholds that turn measurements into prescan_facts.flags are fixed, and every
id you send comes back in coverage_check exactly once — addressed by part of
the audit, or set aside with a reason. That makes the flag list the thing to assert on in an
automated check.
| Flag id | Fires when |
|---|---|
bloated-desc:{n} | A component's frontmatter description is over 30 words. {n} is its 1-based position in components. |
no-frontmatter:{n} | An agent or skill has no frontmatter at all, so it can be neither selected on demand nor described. |
heavy-agent:{n} | An agent file is over 200 lines. |
heavy-skill:{n} | A skill file is over 400 lines. |
heavy-rule:{n} | A rule file is over 100 lines — and rules are always on. |
memory-bloat:1 | The combined memory files are over 300 lines. Always id 1; there is only ever one. |
duplicate:{n} | Component {n} is byte-identical to an earlier one (the classic mirrored skills directory). |
mcp-oversubscribed:{i} | Server {i} (1-based in mcp.servers) exposes more than 20 tools. |
mcp-oversubscribed:servers | More than 10 servers are connected at once. Note the literal servers suffix rather than a number. |
cli-wrapper:{i} | Server {i} wraps a command-line tool the agent could simply run. Usually the highest-leverage finding in the whole audit. |
overlap:{i} | The {i}-th ranked always-on pair scoring at or above 0.25. |
Below is a runnable inventory builder that walks a directory and produces the complete request
body. Python and JavaScript implement the whole thing, duplicates, overlaps,
flags, excerpts and all. cURL, Go, Java, Ruby, PHP and C# implement the reduced
form — token estimate, kind, frontmatter, MCP costing and totals — and send
overlaps and prescan_facts.flags empty with
duplicate_count: 0. Everything they do send is measured by the same rules, so the
audit is still grounded and still checkable; you simply lose the duplicate and overlap findings
and the coverage_check rows they would have produced. Port the Python loop when you
want them back.
#!/usr/bin/env bash
# ctxscan.sh — reduced form: kind + token estimate + MCP costing + totals.
# Needs jq. Duplicates, overlaps and flags are left to the richer clients.
set -euo pipefail
ROOT=".claude"
WINDOW=200000
NAME="Harborline monorepo"
TYPE="TypeScript monorepo — Next.js web app and a Go service"
kind_of() {
p=$(printf '%s' "$1" | tr 'A-Z' 'a-z'); b=${p##*/}
case "$b" in mcp.json|.mcp.json|*.mcp.json) echo mcp; return;; esac
case "$b" in claude.md|agents.md|gemini.md|cursor.md) echo memory; return;; esac
case "$p" in */agent/*|*/agents/*|agent/*|agents/*) echo agent; return;; esac
case "$b" in skill.md) echo skill; return;; esac
case "$p" in */skill/*|*/skills/*|skill/*|skills/*) echo skill; return;; esac
case "$p" in */rule/*|*/rules/*|rule/*|rules/*) echo rule; return;; esac
case "$p" in */command/*|*/commands/*|command/*|commands/*) echo command; return;; esac
case "$b" in *.md|*.mdc) echo rule; return;; esac
echo other
}
# fenced or 4-space-indented lines at chars/4, everything else at words*1.3
est_tokens() {
awk '
/^[ \t]*(```|~~~)/ { fence = !fence; code += length($0) + 1; next }
{
if (fence || ($0 ~ /^([ ]{4,}|\t)/ && $0 !~ /^[ \t]*$/)) { code += length($0) + 1 }
else { n = split($0, w, /[ \t]+/); for (i = 1; i <= n; i++) if (w[i] != "") words++ }
}
END { printf "%d\n", int(code / 4 + words * 1.3 + 0.5) }' "$1"
}
: > components.jsonl
cd "$(dirname "$ROOT")"
find "$(basename "$ROOT")" -type f \( -name '*.md' -o -name '*.mdc' -o -name '*.json' \) |
while read -r f; do
k=$(kind_of "$f")
[ "$k" = "mcp" ] && continue
jq -n --arg path "$f" --arg kind "$k" \
--arg name "$(basename "$f" | sed 's/\.[^.]*$//')" \
--arg excerpt "$(head -c 3000 "$f")" \
--argjson lines "$(wc -l < "$f")" \
--argjson chars "$(wc -c < "$f")" \
--argjson tokens "$(est_tokens "$f")" \
'{path:$path, kind:$kind, name:$name, description:"", description_words:0,
has_frontmatter:false, lines:$lines, chars:$chars, tokens:$tokens,
code_ratio:0, flags:[], _x:$excerpt, _t:($chars > 3000)}' >> components.jsonl
done
# .mcp.json -> servers, 500 tokens per tool, 8 tools assumed when absent
jq '[(.mcpServers // .mcp_servers // .servers // {}) | to_entries[]
| ((.value.tools | if type == "array" then length else null end)
// .value.toolCount // .value.tool_count) as $t
| {name: .key,
command: (([.value.command] + (.value.args // [])) | map(select(.)) | join(" ")),
tools: ($t // 8), assumed: ($t == null), wraps: null,
tokens: (($t // 8) * 500)}]' "$ROOT/.mcp.json" > servers.json
jq -s --slurpfile mcp servers.json --arg name "$NAME" --arg type "$TYPE" \
--argjson window "$WINDOW" '
. as $c
| ($mcp[0]) as $s
| (($c | map(.tokens) | add) // 0) as $ft
| (($s | map(.tokens) | add) // 0) as $mt
| ($ft + $mt) as $oh
| {config_name: $name, project_type: $type, window_tokens: $window,
inventory: {
totals: {overhead_tokens: $oh, file_tokens: $ft, mcp_tokens: $mt,
window_tokens: $window,
pct: (((($oh / $window) * 1000) | round) / 10),
available_tokens: ([$window - $oh, 0] | max),
component_count: ($c | length), duplicate_count: 0,
by_kind: ($c | group_by(.kind)
| map({key: .[0].kind,
value: {count: length, tokens: (map(.tokens) | add)}})
| from_entries)},
components: ($c | map(del(._x, ._t))),
mcp: {servers: $s, tool_count: (($s | map(.tools) | add) // 0), tokens: $mt,
assumed: ($s | map(.assumed) | any), error: ""},
overlaps: []},
excerpts: ($c | map({path: .path, excerpt: ._x, truncated: ._t})),
excerpt_note: ("Measured locally with ctxscan.sh; each file sent head-only, "
+ "first 3,000 chars."),
prescan_facts: {
components: ($c | map({id: ("component:" + (.path | split("/") | last)),
label: ((.path | split("/") | last) + " (" + .kind + ", "
+ (.tokens | tostring) + "t)")})),
flags: []}}' components.jsonl > input.json
jq '.inventory.totals' input.json"""ctxscan.py — the browser's measurement, in Python. Complete form."""
import hashlib, json, math, os, re
LIMITS = {"agent_lines": 200, "skill_lines": 400, "rule_lines": 100,
"memory_lines": 300, "desc_words": 30, "server_tools": 20,
"servers": 10, "overlap": 0.25}
TOKENS_PER_TOOL = 500
DEFAULT_TOOLS_PER_SERVER = 8
EXCERPT_BUDGET = 44000 # chars of file body sent with the run
CLI_WRAPPERS = ["gh", "git", "npm", "pnpm", "yarn", "bun", "docker", "kubectl",
"supabase", "vercel", "netlify", "aws", "gcloud", "azure", "heroku",
"psql", "sqlite", "mysql", "terraform", "shell", "bash", "zsh",
"filesystem", "fs", "make", "cargo", "go", "pip", "poetry",
"curl", "rg", "ripgrep"]
FENCE = re.compile(r"^\s*(```|~~~)")
INDENT = re.compile(r"^(\s{4,}|\t)")
FM = re.compile(r"^\ufeff?---\r?\n(.*?)\r?\n---\s*(?:\r?\n|$)", re.S)
KEY = re.compile(r"^([A-Za-z_][A-Za-z0-9_-]*)\s*:\s*(.*)$")
def estimate_tokens(text):
code_chars = prose_chars = prose_words = 0
in_fence = False
for line in text.split("\n"):
if FENCE.match(line):
in_fence = not in_fence
code_chars += len(line) + 1
continue
if in_fence or (INDENT.match(line) and line.strip()):
code_chars += len(line) + 1
else:
prose_chars += len(line) + 1
prose_words += len(line.split())
total = code_chars + prose_chars
# int(x + 0.5), not round(): the browser's Math.round is half-up.
return {"tokens": int(code_chars / 4 + prose_words * 1.3 + 0.5),
"code_ratio": round(code_chars / total, 2) if total else 0}
def frontmatter(text):
m = FM.match(text)
if not m:
return {"found": False, "name": "", "description": "", "body": text}
out = {"found": True, "name": "", "description": "", "body": text[m.end():]}
lines = m.group(1).split("\n")
i = 0
while i < len(lines):
km = KEY.match(lines[i].rstrip("\r"))
if km:
key, val = km.group(1).lower(), km.group(2)
if val in (">", "|", ">-", "|-", ""): # folded / literal block scalar
acc, j = [], i + 1
while j < len(lines):
if re.match(r"^\s+\S", lines[j]):
acc.append(lines[j].strip()); i = j
elif not lines[j].strip():
acc.append(""); i = j
else:
break
j += 1
val = " ".join(acc).strip()
val = re.sub(r'^["\']', "", val)
val = re.sub(r'["\']$', "", val).strip()
if key == "name":
out["name"] = val
elif key == "description":
out["description"] = val
i += 1
return out
def classify(path, text):
p = path.replace("\\", "/").lower()
base = p.rsplit("/", 1)[-1]
if re.search(r"(^|/)\.?[a-z0-9_-]*mcp(\.[a-z]+)?\.json$", p) or base in ("mcp.json", ".mcp.json"):
return "mcp"
if base in ("claude.md", "agents.md", "gemini.md", "cursor.md"):
return "memory"
if re.search(r"(^|/)agents?/", p):
return "agent"
if re.search(r"(^|/)skills?/", p) or base == "skill.md":
return "skill"
if re.search(r"(^|/)rules?/", p):
return "rule"
if re.search(r"(^|/)commands?/", p):
return "command"
fm = frontmatter(text) # loose file: sniff the content
if fm["found"] and fm["name"] and fm["description"]:
if (re.search(r"\buse\s+(this\s+)?skill\b|^skill\b", fm["description"], re.I)
or re.search(r"^#\s*skill", fm["body"], re.I | re.M)):
return "skill"
return "agent"
if base.endswith(".json"):
return "mcp" if re.search(r'"mcpservers"', text, re.I) else "other"
if re.search(r"\.mdc?$", base):
return "rule"
return "other"
def parse_mcp(text):
data = json.loads(text)
table = data.get("mcpServers") or data.get("mcp_servers") or data.get("servers") or {}
servers = []
for name, cfg in table.items():
parts = [cfg.get("command")] + list(cfg.get("args") or [])
cmd = " ".join(str(x) for x in parts if x)
probe = f'{name} {cmd} {cfg.get("url") or ""}'.lower()
wraps = next((w for w in CLI_WRAPPERS
if re.search(r"(^|[^a-z])" + w + r"([^a-z]|$)", probe)), None)
tools = None
if isinstance(cfg.get("tools"), list):
tools = len(cfg["tools"])
elif isinstance(cfg.get("toolCount"), int):
tools = cfg["toolCount"]
elif isinstance(cfg.get("tool_count"), int):
tools = cfg["tool_count"]
assumed = tools is None
if assumed:
tools = DEFAULT_TOOLS_PER_SERVER
servers.append({"name": name, "command": cmd or str(cfg.get("url") or ""),
"tools": tools, "assumed": assumed, "wraps": wraps,
"tokens": tools * TOKENS_PER_TOOL})
return servers
def trigrams(text):
t = re.sub(r"```.*?```", " ", text.lower(), flags=re.S)
t = re.sub(r"\s+", " ", re.sub(r"[^a-z0-9\s]", " ", t)).strip()
w = t.split(" ") if t else []
return {" ".join(w[i:i + 3]) for i in range(len(w) - 2)}
def jaccard(a, b):
if not a or not b:
return 0.0
inter = len(a & b)
union = len(a) + len(b) - inter
return inter / union if union else 0.0
def read_files(root):
out = []
for dirpath, dirnames, filenames in os.walk(root):
dirnames[:] = [d for d in dirnames if d not in (".git", "node_modules")]
for fn in sorted(filenames):
if not (fn.endswith((".md", ".mdc", ".markdown", ".json", ".txt"))):
continue
full = os.path.join(dirpath, fn)
with open(full, encoding="utf-8", errors="replace") as fh:
out.append({"path": os.path.relpath(full, root).replace(os.sep, "/"),
"text": fh.read()})
return out
def build_input(root, config_name="", project_type="", window_tokens=200000, mcp_tools=None):
components, servers, seen, mcp_error = [], [], {}, ""
for f in read_files(root):
kind = classify(f["path"], f["text"])
if kind == "mcp":
try:
for s in parse_mcp(f["text"]):
if not any(x["name"] == s["name"] for x in servers):
servers.append(s)
except ValueError as exc: # keep going, report it
mcp_error = mcp_error or f"not valid JSON ({exc})"
continue
fm = frontmatter(f["text"])
est = estimate_tokens(f["text"])
digest = hashlib.sha1(f["text"].encode("utf-8")).hexdigest()
dup = seen.get(digest)
seen.setdefault(digest, f["path"])
base = os.path.basename(f["path"])
components.append({
"path": f["path"], "kind": kind,
"name": fm["name"] or re.sub(r"\.(md|mdc|markdown|txt|json)$", "", base, flags=re.I),
"description": fm["description"],
"description_words": len(fm["description"].split()),
"has_frontmatter": fm["found"],
"lines": f["text"].count("\n") + 1 if f["text"] else 0,
"chars": len(f["text"]), "tokens": est["tokens"],
"code_ratio": est["code_ratio"],
"duplicate_of": dup, "counted": dup is None,
"flags": [], "_text": f["text"],
})
# A typed total from /mcp beats the per-server assumption; spread it proportionally.
if mcp_tools is not None and servers:
raw = sum(s["tools"] for s in servers)
for s in servers:
s["tools"] = max(1, int(mcp_tools * s["tools"] / raw + 0.5)) if raw else 0
s["assumed"] = False
s["tokens"] = s["tools"] * TOKENS_PER_TOOL
tool_count = sum(s["tools"] for s in servers)
mcp_tokens = sum(s["tokens"] for s in servers)
flags = []
def flag(fid, label, comp=None):
flags.append({"id": fid, "label": label})
if comp is not None:
comp["flags"].append(fid)
memory_lines = 0
for idx, c in enumerate(components, start=1):
base = os.path.basename(c["path"])
if c["kind"] == "memory":
memory_lines += c["lines"]
if not c["counted"]:
flag(f"duplicate:{idx}", f'{base} duplicates {c["duplicate_of"]}', c)
continue
if c["description_words"] > LIMITS["desc_words"]:
flag(f"bloated-desc:{idx}", f'{c["name"]} — {c["description_words"]}-word description', c)
if c["kind"] in ("agent", "skill") and not c["has_frontmatter"]:
flag(f"no-frontmatter:{idx}", f"{base} has no frontmatter", c)
if c["kind"] == "agent" and c["lines"] > LIMITS["agent_lines"]:
flag(f"heavy-agent:{idx}", f'{c["name"]} — {c["lines"]} lines (~{c["tokens"]:,} tokens)', c)
if c["kind"] == "skill" and c["lines"] > LIMITS["skill_lines"]:
flag(f"heavy-skill:{idx}", f'{c["name"]} — {c["lines"]} lines (~{c["tokens"]:,} tokens)', c)
if c["kind"] == "rule" and c["lines"] > LIMITS["rule_lines"]:
flag(f"heavy-rule:{idx}", f'{c["name"]} — {c["lines"]} lines (~{c["tokens"]:,} tokens)', c)
if memory_lines > LIMITS["memory_lines"]:
flag("memory-bloat:1", f"CLAUDE.md chain is {memory_lines} lines")
if len(servers) > LIMITS["servers"]:
flag("mcp-oversubscribed:servers", f"{len(servers)} MCP servers connected")
for i, s in enumerate(servers, start=1):
if s["tools"] > LIMITS["server_tools"]:
flag(f"mcp-oversubscribed:{i}", f'{s["name"]} exposes {s["tools"]} tools (~{s["tokens"]:,} tokens)')
if s["wraps"]:
flag(f"cli-wrapper:{i}", f'{s["name"]} wraps the {s["wraps"]} CLI (~{s["tokens"]:,} tokens)')
always_on = [c for c in components if c["counted"] and c["kind"] in ("rule", "memory")]
grams = [trigrams(c["_text"]) for c in always_on]
overlaps = []
for a in range(len(always_on)):
for b in range(a + 1, len(always_on)):
score = jaccard(grams[a], grams[b])
if score >= LIMITS["overlap"]:
overlaps.append({"a": os.path.basename(always_on[a]["path"]),
"b": os.path.basename(always_on[b]["path"]),
"score": round(score, 2)})
overlaps.sort(key=lambda o: -o["score"])
for i, ov in enumerate(overlaps[:8], start=1):
flag(f"overlap:{i}", f'{ov["a"]} and {ov["b"]} overlap {round(ov["score"] * 100)}%')
counted = [c for c in components if c["counted"]]
file_tokens = sum(c["tokens"] for c in counted)
overhead = file_tokens + mcp_tokens
by_kind = {}
for c in counted:
slot = by_kind.setdefault(c["kind"], {"count": 0, "tokens": 0})
slot["count"] += 1
slot["tokens"] += c["tokens"]
if servers:
by_kind["mcp"] = {"count": len(servers), "tokens": mcp_tokens}
# Excerpts: head-and-tail inside a weighted share, never a blind cut.
weights = [(1 + (2 if c["kind"] == "memory" else 0) + (1 if c["kind"] == "rule" else 0)
+ (1.5 if c["flags"] else 0)) * math.sqrt(max(1, c["chars"])) for c in counted]
wsum = sum(weights) or 1
excerpts, sent_chars = [], 0
for c, w in zip(counted, weights):
share = max(400, int(EXCERPT_BUDGET * w / wsum))
body = c["_text"]
if len(body) <= share:
text = body
else:
head = -(-share * 7 // 10) # ceil(share * 0.7)
text = (body[:head] + f"\n[... {len(body) - share:,} chars of the middle withheld — "
"the head carries the frontmatter and opening instructions, the tail the "
"closing ones ...]\n" + (body[len(body) - (share - head):] if share > head else ""))
sent_chars += len(text)
excerpts.append({"path": c["path"], "excerpt": text, "truncated": len(body) > share})
total_chars = sum(c["chars"] for c in counted)
def slim(c):
keys = (("path", "kind", "name", "description", "description_words", "has_frontmatter",
"lines", "chars", "tokens", "code_ratio", "flags") if c["counted"]
else ("path", "kind", "name", "lines", "chars", "tokens",
"duplicate_of", "counted", "flags"))
return {k: c[k] for k in keys}
note = (f"Profiled {len(counted)} component{'' if len(counted) == 1 else 's'} "
f"({total_chars:,} chars) locally; sent {sent_chars:,} chars of excerpts"
+ (f", withholding {total_chars - sent_chars:,} chars from the middles of large files"
if total_chars > sent_chars else " (nothing withheld)") + ".")
return {
"config_name": config_name,
"project_type": project_type,
"window_tokens": window_tokens,
"inventory": {
"totals": {
"overhead_tokens": overhead, "file_tokens": file_tokens,
"mcp_tokens": mcp_tokens, "window_tokens": window_tokens,
"pct": round(overhead / window_tokens * 100, 1) if window_tokens else 0,
"available_tokens": max(0, window_tokens - overhead),
"component_count": len(counted),
"duplicate_count": len(components) - len(counted),
"by_kind": by_kind,
},
"components": [slim(c) for c in sorted(components, key=lambda c: not c["counted"])],
"mcp": {"servers": servers, "tool_count": tool_count, "tokens": mcp_tokens,
"assumed": any(s["assumed"] for s in servers), "error": mcp_error},
"overlaps": overlaps,
},
"excerpts": excerpts,
"excerpt_note": note,
"prescan_facts": {
"components": [{"id": "component:" + os.path.basename(c["path"]),
"label": f'{os.path.basename(c["path"])} ({c["kind"]}, {c["tokens"]}t)'}
for c in counted],
"flags": flags,
},
}
payload = build_input(".claude",
config_name="Harborline monorepo",
project_type="TypeScript monorepo — Next.js web app and a Go service",
window_tokens=200000)
t = payload["inventory"]["totals"]
print(payload["excerpt_note"])
print(f'{t["overhead_tokens"]:,} tokens ({t["pct"]}% of the window) before the first turn')
print("flags:", [f["id"] for f in payload["prescan_facts"]["flags"]])// ctxscan.mjs — the browser's measurement, in Node 18+. Complete form.
import { createHash } from "node:crypto";
import { readFile, readdir } from "node:fs/promises";
import { join, basename, relative, sep } from "node:path";
const LIMITS = { agentLines: 200, skillLines: 400, ruleLines: 100, memoryLines: 300,
descWords: 30, serverTools: 20, servers: 10, overlap: 0.25 };
const TOKENS_PER_TOOL = 500;
const DEFAULT_TOOLS_PER_SERVER = 8;
const EXCERPT_BUDGET = 44000;
const CLI_WRAPPERS = ["gh", "git", "npm", "pnpm", "yarn", "bun", "docker", "kubectl",
"supabase", "vercel", "netlify", "aws", "gcloud", "azure", "heroku", "psql", "sqlite",
"mysql", "terraform", "shell", "bash", "zsh", "filesystem", "fs", "make", "cargo",
"go", "pip", "poetry", "curl", "rg", "ripgrep"];
export function estimateTokens(text) {
let codeChars = 0, proseChars = 0, proseWords = 0, inFence = false;
for (const line of String(text).split("\n")) {
if (/^\s*(```|~~~)/.test(line)) { inFence = !inFence; codeChars += line.length + 1; continue; }
const indented = /^(\s{4,}|\t)/.test(line) && line.trim().length > 0;
if (inFence || indented) codeChars += line.length + 1;
else {
proseChars += line.length + 1;
const w = line.trim();
if (w) proseWords += w.split(/\s+/).length;
}
}
const total = codeChars + proseChars;
return { tokens: Math.round(codeChars / 4 + proseWords * 1.3),
codeRatio: total ? Math.round((codeChars / total) * 100) / 100 : 0 };
}
export function frontmatter(text) {
const t = String(text);
const m = /^\uFEFF?---\r?\n([\s\S]*?)\r?\n---\s*(?:\r?\n|$)/.exec(t);
if (!m) return { found: false, name: "", description: "", body: t };
const out = { found: true, name: "", description: "", body: t.slice(m[0].length) };
const lines = m[1].split(/\r?\n/);
for (let i = 0; i < lines.length; i++) {
const km = /^([A-Za-z_][A-Za-z0-9_-]*)\s*:\s*(.*)$/.exec(lines[i]);
if (!km) continue;
const key = km[1].toLowerCase();
let val = km[2];
if (val === ">" || val === "|" || val === ">-" || val === "|-" || val === "") {
const acc = [];
for (let j = i + 1; j < lines.length; j++) {
if (/^\s+\S/.test(lines[j])) { acc.push(lines[j].trim()); i = j; }
else if (/^\s*$/.test(lines[j])) { acc.push(""); i = j; }
else break;
}
val = acc.join(" ").trim();
}
val = val.replace(/^["']/, "").replace(/["']$/, "").trim();
if (key === "name") out.name = val;
else if (key === "description") out.description = val;
}
return out;
}
export function classify(path, text) {
const p = String(path).replace(/\\/g, "/").toLowerCase();
const base = p.split("/").pop();
if (/(^|\/)\.?[a-z0-9_-]*mcp(\.[a-z]+)?\.json$/.test(p) || base === "mcp.json" || base === ".mcp.json") return "mcp";
if (["claude.md", "agents.md", "gemini.md", "cursor.md"].includes(base)) return "memory";
if (/(^|\/)agents?\//.test(p)) return "agent";
if (/(^|\/)skills?\//.test(p) || base === "skill.md") return "skill";
if (/(^|\/)rules?\//.test(p)) return "rule";
if (/(^|\/)commands?\//.test(p)) return "command";
const fm = frontmatter(text);
if (fm.found && fm.name && fm.description) {
if (/\buse\s+(this\s+)?skill\b|^skill\b/i.test(fm.description) || /^#\s*skill/im.test(fm.body)) return "skill";
return "agent";
}
if (/\.json$/.test(base)) return /"mcpservers"/i.test(String(text)) ? "mcp" : "other";
if (/\.mdc?$/.test(base)) return "rule";
return "other";
}
export function parseMcp(text) {
const data = JSON.parse(text);
const table = data.mcpServers || data.mcp_servers || data.servers || {};
return Object.entries(table).map(([name, cfg = {}]) => {
const cmd = [cfg.command, ...(Array.isArray(cfg.args) ? cfg.args : [])].filter(Boolean).join(" ");
const probe = `${name} ${cmd} ${cfg.url ?? ""}`.toLowerCase();
const wraps = CLI_WRAPPERS.find((w) => new RegExp(`(^|[^a-z])${w}([^a-z]|$)`).test(probe)) ?? null;
let tools = null;
if (Array.isArray(cfg.tools)) tools = cfg.tools.length;
else if (typeof cfg.toolCount === "number") tools = cfg.toolCount;
else if (typeof cfg.tool_count === "number") tools = cfg.tool_count;
const assumed = tools === null;
if (assumed) tools = DEFAULT_TOOLS_PER_SERVER;
return { name, command: cmd || String(cfg.url ?? ""), tools, assumed, wraps,
tokens: tools * TOKENS_PER_TOOL };
});
}
const trigrams = (text) => {
const words = String(text).toLowerCase().replace(/```[\s\S]*?```/g, " ")
.replace(/[^a-z0-9\s]/g, " ").replace(/\s+/g, " ").trim().split(" ").filter(Boolean);
const set = new Set();
for (let i = 0; i + 2 < words.length; i++) set.add(words.slice(i, i + 3).join(" "));
return set;
};
const jaccard = (a, b) => {
if (!a.size || !b.size) return 0;
let inter = 0;
for (const g of a) if (b.has(g)) inter++;
const union = a.size + b.size - inter;
return union ? inter / union : 0;
};
async function readFiles(root) {
const out = [];
const walk = async (dir) => {
for (const e of await readdir(dir, { withFileTypes: true })) {
if (e.name === ".git" || e.name === "node_modules") continue;
const full = join(dir, e.name);
if (e.isDirectory()) await walk(full);
else if (/\.(md|mdc|markdown|json|txt)$/i.test(e.name)) {
out.push({ path: relative(root, full).split(sep).join("/"),
text: await readFile(full, "utf8") });
}
}
};
await walk(root);
return out.sort((a, b) => a.path.localeCompare(b.path));
}
export async function buildInput(root, opts = {}) {
const windowTokens = opts.windowTokens ?? 200000;
const components = [], servers = [], seen = new Map();
let mcpError = "";
for (const f of await readFiles(root)) {
const kind = classify(f.path, f.text);
if (kind === "mcp") {
try {
for (const s of parseMcp(f.text)) {
if (!servers.some((x) => x.name === s.name)) servers.push(s);
}
} catch (e) { mcpError = mcpError || `not valid JSON (${e.message})`; }
continue;
}
const fm = frontmatter(f.text);
const est = estimateTokens(f.text);
const digest = createHash("sha1").update(f.text).digest("hex");
const dup = seen.get(digest) ?? null;
if (!dup) seen.set(digest, f.path);
const base = basename(f.path);
components.push({
path: f.path, kind, base,
name: fm.name || base.replace(/\.(md|mdc|markdown|txt|json)$/i, ""),
description: fm.description,
description_words: fm.description ? fm.description.trim().split(/\s+/).length : 0,
has_frontmatter: fm.found,
lines: f.text ? f.text.split("\n").length : 0,
chars: f.text.length, tokens: est.tokens, code_ratio: est.codeRatio,
duplicate_of: dup, counted: !dup, flags: [], text: f.text,
});
}
if (typeof opts.mcpTools === "number" && servers.length) {
const raw = servers.reduce((a, s) => a + s.tools, 0);
for (const s of servers) {
s.tools = raw ? Math.max(1, Math.round((opts.mcpTools * s.tools) / raw)) : 0;
s.assumed = false;
s.tokens = s.tools * TOKENS_PER_TOOL;
}
}
const toolCount = servers.reduce((a, s) => a + s.tools, 0);
const mcpTokens = servers.reduce((a, s) => a + s.tokens, 0);
const flags = [];
const flag = (id, label, comp) => { flags.push({ id, label }); if (comp) comp.flags.push(id); };
let memoryLines = 0;
components.forEach((c, i) => {
const n = i + 1;
if (c.kind === "memory") memoryLines += c.lines;
if (!c.counted) return flag(`duplicate:${n}`, `${c.base} duplicates ${c.duplicate_of}`, c);
if (c.description_words > LIMITS.descWords)
flag(`bloated-desc:${n}`, `${c.name} — ${c.description_words}-word description`, c);
if ((c.kind === "agent" || c.kind === "skill") && !c.has_frontmatter)
flag(`no-frontmatter:${n}`, `${c.base} has no frontmatter`, c);
if (c.kind === "agent" && c.lines > LIMITS.agentLines)
flag(`heavy-agent:${n}`, `${c.name} — ${c.lines} lines`, c);
if (c.kind === "skill" && c.lines > LIMITS.skillLines)
flag(`heavy-skill:${n}`, `${c.name} — ${c.lines} lines`, c);
if (c.kind === "rule" && c.lines > LIMITS.ruleLines)
flag(`heavy-rule:${n}`, `${c.name} — ${c.lines} lines`, c);
});
if (memoryLines > LIMITS.memoryLines) flag("memory-bloat:1", `CLAUDE.md chain is ${memoryLines} lines`);
if (servers.length > LIMITS.servers) flag("mcp-oversubscribed:servers", `${servers.length} MCP servers connected`);
servers.forEach((s, i) => {
if (s.tools > LIMITS.serverTools)
flag(`mcp-oversubscribed:${i + 1}`, `${s.name} exposes ${s.tools} tools`);
if (s.wraps) flag(`cli-wrapper:${i + 1}`, `${s.name} wraps the ${s.wraps} CLI`);
});
const alwaysOn = components.filter((c) => c.counted && (c.kind === "rule" || c.kind === "memory"));
const grams = alwaysOn.map((c) => trigrams(c.text));
const overlaps = [];
for (let a = 0; a < alwaysOn.length; a++) {
for (let b = a + 1; b < alwaysOn.length; b++) {
const score = jaccard(grams[a], grams[b]);
if (score >= LIMITS.overlap)
overlaps.push({ a: alwaysOn[a].base, b: alwaysOn[b].base, score: Math.round(score * 100) / 100 });
}
}
overlaps.sort((x, y) => y.score - x.score);
overlaps.slice(0, 8).forEach((ov, i) =>
flag(`overlap:${i + 1}`, `${ov.a} and ${ov.b} overlap ${Math.round(ov.score * 100)}%`));
const counted = components.filter((c) => c.counted);
const fileTokens = counted.reduce((a, c) => a + c.tokens, 0);
const overhead = fileTokens + mcpTokens;
const byKind = {};
for (const c of counted) {
byKind[c.kind] = byKind[c.kind] || { count: 0, tokens: 0 };
byKind[c.kind].count++;
byKind[c.kind].tokens += c.tokens;
}
if (servers.length) byKind.mcp = { count: servers.length, tokens: mcpTokens };
const weights = counted.map((c) =>
(1 + (c.kind === "memory" ? 2 : 0) + (c.kind === "rule" ? 1 : 0) + (c.flags.length ? 1.5 : 0))
* Math.sqrt(Math.max(1, c.chars)));
const wsum = weights.reduce((a, b) => a + b, 0) || 1;
let sentChars = 0;
const excerpts = counted.map((c, i) => {
const share = Math.max(400, Math.floor((EXCERPT_BUDGET * weights[i]) / wsum));
const body = c.text;
let text = body;
if (body.length > share) {
const head = Math.ceil(share * 0.7);
text = body.slice(0, head) +
`\n[... ${(body.length - share).toLocaleString()} chars of the middle withheld — ` +
"the head carries the frontmatter and opening instructions, the tail the closing ones ...]\n" +
body.slice(body.length - (share - head));
}
sentChars += text.length;
return { path: c.path, excerpt: text, truncated: body.length > share };
});
const totalChars = counted.reduce((a, c) => a + c.chars, 0);
const slim = (c) => c.counted
? { path: c.path, kind: c.kind, name: c.name, description: c.description,
description_words: c.description_words, has_frontmatter: c.has_frontmatter,
lines: c.lines, chars: c.chars, tokens: c.tokens, code_ratio: c.code_ratio, flags: c.flags }
: { path: c.path, kind: c.kind, name: c.name, lines: c.lines, chars: c.chars,
tokens: c.tokens, duplicate_of: c.duplicate_of, counted: false, flags: c.flags };
return {
config_name: opts.configName ?? "",
project_type: opts.projectType ?? "",
window_tokens: windowTokens,
inventory: {
totals: {
overhead_tokens: overhead, file_tokens: fileTokens, mcp_tokens: mcpTokens,
window_tokens: windowTokens,
pct: windowTokens ? Math.round((overhead / windowTokens) * 1000) / 10 : 0,
available_tokens: Math.max(0, windowTokens - overhead),
component_count: counted.length,
duplicate_count: components.length - counted.length,
by_kind: byKind,
},
components: [...counted, ...components.filter((c) => !c.counted)].map(slim),
mcp: { servers, tool_count: toolCount, tokens: mcpTokens,
assumed: servers.some((s) => s.assumed), error: mcpError },
overlaps,
},
excerpts,
excerpt_note: `Profiled ${counted.length} component${counted.length === 1 ? "" : "s"} ` +
`(${totalChars.toLocaleString()} chars) locally; sent ${sentChars.toLocaleString()} chars of excerpts` +
(totalChars > sentChars
? `, withholding ${(totalChars - sentChars).toLocaleString()} chars from the middles of large files`
: " (nothing withheld)") + ".",
prescan_facts: {
components: counted.map((c) => ({ id: `component:${c.base}`,
label: `${c.base} (${c.kind}, ${c.tokens}t)` })),
flags,
},
};
}
const payload = await buildInput(".claude", {
configName: "Harborline monorepo",
projectType: "TypeScript monorepo — Next.js web app and a Go service",
windowTokens: 200000,
});
const t = payload.inventory.totals;
console.log(payload.excerpt_note);
console.log(`${t.overhead_tokens.toLocaleString()} tokens (${t.pct}% of the window) before the first turn`);
console.log("flags:", payload.prescan_facts.flags.map((f) => f.id));// Reduced form: token estimate, kind, frontmatter name/description, MCP costing
// and totals. No duplicate, overlap or flag detection — prescan_facts.flags and
// inventory.overlaps go out empty. Port the Python loop when you want them.
package main
import (
"encoding/json"
"io/fs"
"math"
"os"
"path/filepath"
"regexp"
"strings"
)
const tokensPerTool = 500
const defaultToolsPerServer = 8
var (
reFence = regexp.MustCompile("^[ \t]*(```|~~~)")
reIndent = regexp.MustCompile(`^(\s{4,}|\t)`)
reFM = regexp.MustCompile(`(?s)\A---\r?\n(.*?)\r?\n---\s*(\r?\n|$)`)
reKey = regexp.MustCompile(`^([A-Za-z_][A-Za-z0-9_-]*)\s*:\s*(.*)$`)
cliWraps = []string{"gh", "git", "npm", "pnpm", "yarn", "bun", "docker", "kubectl",
"supabase", "vercel", "netlify", "aws", "gcloud", "azure", "heroku", "psql",
"sqlite", "mysql", "terraform", "shell", "bash", "zsh", "filesystem", "fs",
"make", "cargo", "go", "pip", "poetry", "curl", "rg", "ripgrep"}
)
func estimateTokens(text string) (int, float64) {
codeChars, proseChars, proseWords, inFence := 0, 0, 0, false
for _, line := range strings.Split(text, "\n") {
if reFence.MatchString(line) {
inFence = !inFence
codeChars += len(line) + 1
continue
}
indented := reIndent.MatchString(line) && strings.TrimSpace(line) != ""
if inFence || indented {
codeChars += len(line) + 1
} else {
proseChars += len(line) + 1
proseWords += len(strings.Fields(line))
}
}
total := codeChars + proseChars
ratio := 0.0
if total > 0 {
ratio = math.Round(float64(codeChars)/float64(total)*100) / 100
}
return int(math.Round(float64(codeChars)/4 + float64(proseWords)*1.3)), ratio
}
// name and description only — enough for description_words, which is the field
// that costs on every Task call. Folded block scalars are joined with spaces.
func frontmatter(text string) (bool, string, string) {
m := reFM.FindStringSubmatch(text)
if m == nil {
return false, "", ""
}
lines := strings.Split(m[1], "\n")
name, desc := "", ""
for i := 0; i < len(lines); i++ {
km := reKey.FindStringSubmatch(strings.TrimRight(lines[i], "\r"))
if km == nil {
continue
}
key, val := strings.ToLower(km[1]), km[2]
if val == ">" || val == "|" || val == ">-" || val == "|-" || val == "" {
var acc []string
for j := i + 1; j < len(lines); j++ {
if strings.TrimSpace(lines[j]) == "" || strings.HasPrefix(lines[j], " ") || strings.HasPrefix(lines[j], "\t") {
acc = append(acc, strings.TrimSpace(lines[j]))
i = j
} else {
break
}
}
val = strings.TrimSpace(strings.Join(acc, " "))
}
val = strings.Trim(strings.TrimSpace(val), `"'`)
switch key {
case "name":
name = val
case "description":
desc = val
}
}
return true, name, desc
}
func classify(path string) string {
p := strings.ToLower(filepath.ToSlash(path))
base := p[strings.LastIndex(p, "/")+1:]
switch {
case strings.HasSuffix(base, "mcp.json"):
return "mcp"
case base == "claude.md" || base == "agents.md" || base == "gemini.md" || base == "cursor.md":
return "memory"
case strings.Contains("/"+p, "/agents/") || strings.Contains("/"+p, "/agent/"):
return "agent"
case strings.Contains("/"+p, "/skills/") || strings.Contains("/"+p, "/skill/") || base == "skill.md":
return "skill"
case strings.Contains("/"+p, "/rules/") || strings.Contains("/"+p, "/rule/"):
return "rule"
case strings.Contains("/"+p, "/commands/") || strings.Contains("/"+p, "/command/"):
return "command"
case strings.HasSuffix(base, ".md") || strings.HasSuffix(base, ".mdc"):
return "rule"
}
return "other"
}
type server struct {
Name string `json:"name"`
Command string `json:"command"`
Tools int `json:"tools"`
Assumed bool `json:"assumed"`
Wraps any `json:"wraps"`
Tokens int `json:"tokens"`
}
func parseMCP(text []byte) []server {
var doc struct {
McpServers map[string]struct {
Command string `json:"command"`
Args []string `json:"args"`
URL string `json:"url"`
Tools []any `json:"tools"`
ToolCount *int `json:"toolCount"`
} `json:"mcpServers"`
}
if json.Unmarshal(text, &doc) != nil {
return nil
}
out := []server{}
for name, cfg := range doc.McpServers {
cmd := strings.TrimSpace(cfg.Command + " " + strings.Join(cfg.Args, " "))
probe := strings.ToLower(name + " " + cmd + " " + cfg.URL)
var wraps any
for _, w := range cliWraps {
if regexp.MustCompile(`(^|[^a-z])`+w+`([^a-z]|$)`).MatchString(probe) {
wraps = w
break
}
}
tools, assumed := defaultToolsPerServer, true
if len(cfg.Tools) > 0 {
tools, assumed = len(cfg.Tools), false
} else if cfg.ToolCount != nil {
tools, assumed = *cfg.ToolCount, false
}
out = append(out, server{name, cmd, tools, assumed, wraps, tools * tokensPerTool})
}
return out
}
func buildInput(root, name, projectType string, window int) map[string]any {
components := []map[string]any{}
excerpts := []map[string]any{}
servers := []server{}
filepath.WalkDir(root, func(p string, d fs.DirEntry, err error) error {
if err != nil || d.IsDir() {
return nil
}
ext := strings.ToLower(filepath.Ext(p))
if ext != ".md" && ext != ".mdc" && ext != ".json" && ext != ".txt" {
return nil
}
raw, _ := os.ReadFile(p)
rel := filepath.ToSlash(strings.TrimPrefix(strings.TrimPrefix(p, root), "/"))
kind := classify(rel)
if kind == "mcp" {
servers = append(servers, parseMCP(raw)...)
return nil
}
text := string(raw)
tokens, ratio := estimateTokens(text)
found, fmName, fmDesc := frontmatter(text)
base := filepath.Base(rel)
if fmName == "" {
fmName = strings.TrimSuffix(base, filepath.Ext(base))
}
components = append(components, map[string]any{
"path": rel, "kind": kind, "name": fmName, "description": fmDesc,
"description_words": len(strings.Fields(fmDesc)), "has_frontmatter": found,
"lines": len(strings.Split(text, "\n")), "chars": len(text),
"tokens": tokens, "code_ratio": ratio, "flags": []string{},
})
body := text
truncated := false
if len(body) > 3000 { // head-only excerpt; see the Python version for head-and-tail
body, truncated = body[:3000], true
}
excerpts = append(excerpts, map[string]any{"path": rel, "excerpt": body, "truncated": truncated})
return nil
})
fileTokens, mcpTokens, toolCount, assumed := 0, 0, 0, false
byKind := map[string]map[string]int{}
for _, c := range components {
fileTokens += c["tokens"].(int)
k := c["kind"].(string)
if byKind[k] == nil {
byKind[k] = map[string]int{"count": 0, "tokens": 0}
}
byKind[k]["count"]++
byKind[k]["tokens"] += c["tokens"].(int)
}
for _, s := range servers {
mcpTokens += s.Tokens
toolCount += s.Tools
assumed = assumed || s.Assumed
}
if len(servers) > 0 {
byKind["mcp"] = map[string]int{"count": len(servers), "tokens": mcpTokens}
}
overhead := fileTokens + mcpTokens
available := window - overhead
if available < 0 {
available = 0
}
facts := []map[string]string{}
for _, c := range components {
base := filepath.Base(c["path"].(string))
facts = append(facts, map[string]string{"id": "component:" + base, "label": base})
}
return map[string]any{
"config_name": name, "project_type": projectType, "window_tokens": window,
"inventory": map[string]any{
"totals": map[string]any{
"overhead_tokens": overhead, "file_tokens": fileTokens, "mcp_tokens": mcpTokens,
"window_tokens": window,
"pct": math.Round(float64(overhead)/float64(window)*1000) / 10,
"available_tokens": available, "component_count": len(components),
"duplicate_count": 0, "by_kind": byKind,
},
"components": components,
"mcp": map[string]any{"servers": servers, "tool_count": toolCount,
"tokens": mcpTokens, "assumed": assumed, "error": ""},
"overlaps": []any{},
},
"excerpts": excerpts,
"excerpt_note": "Measured locally in Go (reduced form: no duplicate or overlap detection); each file sent head-only.",
"prescan_facts": map[string]any{"components": facts, "flags": []any{}},
}
}
var payload = buildInput(".claude", "Harborline monorepo",
"TypeScript monorepo — Next.js web app and a Go service", 200000)// Reduced form: token estimate, kind, frontmatter, MCP costing and totals.
// No duplicate, overlap or flag detection — prescan_facts.flags and
// inventory.overlaps go out empty. Serialize the maps with Jackson or Gson.
import java.nio.file.*;
import java.util.*;
import java.util.regex.*;
import java.util.stream.*;
public class CtxScan {
static final int TOKENS_PER_TOOL = 500, DEFAULT_TOOLS = 8;
static final Pattern FENCE = Pattern.compile("^\\s*(```|~~~)");
static final Pattern INDENT = Pattern.compile("^(\\s{4,}|\t)");
static final Pattern FM = Pattern.compile("\\A---\\r?\\n(.*?)\\r?\\n---\\s*(\\r?\\n|$)", Pattern.DOTALL);
static final Pattern KEY = Pattern.compile("^([A-Za-z_][A-Za-z0-9_-]*)\\s*:\\s*(.*)$");
record Est(int tokens, double codeRatio) {}
static Est estimateTokens(String text) {
int code = 0, prose = 0, words = 0;
boolean fence = false;
for (String line : text.split("\n", -1)) {
if (FENCE.matcher(line).find()) { fence = !fence; code += line.length() + 1; continue; }
boolean indented = INDENT.matcher(line).find() && !line.isBlank();
if (fence || indented) code += line.length() + 1;
else {
prose += line.length() + 1;
if (!line.isBlank()) words += line.trim().split("\\s+").length;
}
}
int total = code + prose;
double ratio = total == 0 ? 0 : Math.round((double) code / total * 100) / 100.0;
return new Est((int) Math.round(code / 4.0 + words * 1.3), ratio);
}
/** {found, name, description} — description_words is what costs on every Task call. */
static String[] frontmatter(String text) {
Matcher m = FM.matcher(text);
if (!m.find()) return new String[] {"false", "", ""};
String[] lines = m.group(1).split("\r?\n", -1);
String name = "", desc = "";
for (int i = 0; i < lines.length; i++) {
Matcher km = KEY.matcher(lines[i]);
if (!km.find()) continue;
String key = km.group(1).toLowerCase(), val = km.group(2);
if (val.isEmpty() || val.equals(">") || val.equals("|") || val.equals(">-") || val.equals("|-")) {
var acc = new ArrayList<String>();
int j = i + 1;
for (; j < lines.length; j++) {
if (lines[j].isBlank() || lines[j].startsWith(" ") || lines[j].startsWith("\t")) acc.add(lines[j].trim());
else break;
}
i = j - 1;
val = String.join(" ", acc).trim();
}
val = val.replaceAll("^[\"']", "").replaceAll("[\"']$", "").trim();
if (key.equals("name")) name = val;
else if (key.equals("description")) desc = val;
}
return new String[] {"true", name, desc};
}
static String classify(String rel) {
String p = rel.replace('\\', '/').toLowerCase();
String base = p.substring(p.lastIndexOf('/') + 1);
if (base.endsWith("mcp.json")) return "mcp";
if (List.of("claude.md", "agents.md", "gemini.md", "cursor.md").contains(base)) return "memory";
String q = "/" + p;
if (q.contains("/agents/") || q.contains("/agent/")) return "agent";
if (q.contains("/skills/") || q.contains("/skill/") || base.equals("skill.md")) return "skill";
if (q.contains("/rules/") || q.contains("/rule/")) return "rule";
if (q.contains("/commands/") || q.contains("/command/")) return "command";
if (base.endsWith(".md") || base.endsWith(".mdc")) return "rule";
return "other";
}
public static Map<String, Object> buildInput(Path root, String name, String projectType, int window) throws Exception {
var components = new ArrayList<Map<String, Object>>();
var excerpts = new ArrayList<Map<String, Object>>();
var servers = new ArrayList<Map<String, Object>>();
try (var walk = Files.walk(root)) {
for (Path p : walk.filter(Files::isRegularFile).sorted().toList()) {
String rel = root.relativize(p).toString().replace('\\', '/');
if (!rel.matches("(?i).*\\.(md|mdc|markdown|json|txt)")) continue;
String text = Files.readString(p);
String kind = classify(rel);
if (kind.equals("mcp")) { servers.addAll(parseMcp(text)); continue; }
Est est = estimateTokens(text);
String[] fm = frontmatter(text);
String base = rel.substring(rel.lastIndexOf('/') + 1);
String cname = fm[1].isEmpty() ? base.replaceFirst("\\.[^.]*$", "") : fm[1];
components.add(new LinkedHashMap<>(Map.of(
"path", rel, "kind", kind, "name", cname, "description", fm[2],
"description_words", fm[2].isBlank() ? 0 : fm[2].trim().split("\\s+").length,
"has_frontmatter", Boolean.parseBoolean(fm[0]),
"lines", text.split("\n", -1).length, "chars", text.length(),
"tokens", est.tokens(), "code_ratio", est.codeRatio())));
String body = text.length() > 3000 ? text.substring(0, 3000) : text;
excerpts.add(Map.of("path", rel, "excerpt", body, "truncated", text.length() > 3000));
}
}
int fileTokens = components.stream().mapToInt(c -> (int) c.get("tokens")).sum();
int mcpTokens = servers.stream().mapToInt(s -> (int) s.get("tokens")).sum();
int toolCount = servers.stream().mapToInt(s -> (int) s.get("tools")).sum();
int overhead = fileTokens + mcpTokens;
var byKind = new LinkedHashMap<String, Map<String, Integer>>();
for (var c : components) {
var slot = byKind.computeIfAbsent((String) c.get("kind"),
k -> new LinkedHashMap<>(Map.of("count", 0, "tokens", 0)));
slot.put("count", slot.get("count") + 1);
slot.put("tokens", slot.get("tokens") + (int) c.get("tokens"));
}
if (!servers.isEmpty()) byKind.put("mcp", Map.of("count", servers.size(), "tokens", mcpTokens));
var facts = components.stream().map(c -> {
String b = ((String) c.get("path")).replaceAll(".*/", "");
return Map.of("id", "component:" + b, "label", b + " (" + c.get("kind") + ")");
}).collect(Collectors.toList());
return Map.of(
"config_name", name, "project_type", projectType, "window_tokens", window,
"inventory", Map.of(
"totals", Map.of("overhead_tokens", overhead, "file_tokens", fileTokens,
"mcp_tokens", mcpTokens, "window_tokens", window,
"pct", Math.round((double) overhead / window * 1000) / 10.0,
"available_tokens", Math.max(0, window - overhead),
"component_count", components.size(), "duplicate_count", 0,
"by_kind", byKind),
"components", components,
"mcp", Map.of("servers", servers, "tool_count", toolCount, "tokens", mcpTokens,
"assumed", servers.stream().anyMatch(s -> (boolean) s.get("assumed")), "error", ""),
"overlaps", List.of()),
"excerpts", excerpts,
"excerpt_note", "Measured locally in Java (reduced form); each file sent head-only.",
"prescan_facts", Map.of("components", facts, "flags", List.of()));
}
/** Parse .mcp.json with your JSON library: for every entry under mcpServers,
* tools = tools.length | toolCount | tool_count, else 8 with assumed = true,
* tokens = tools * 500, wraps = the CLI name found in name/command/url. */
static List<Map<String, Object>> parseMcp(String json) { /* … */ return List.of(); }
}# Reduced form: token estimate, kind, frontmatter, MCP costing and totals.
# No duplicate, overlap or flag detection — prescan_facts.flags and
# inventory.overlaps go out empty. Port the Python loop when you want them.
require "json"
require "pathname"
TOKENS_PER_TOOL = 500
DEFAULT_TOOLS = 8
CLI_WRAPPERS = %w[gh git npm pnpm yarn bun docker kubectl supabase vercel netlify aws
gcloud azure heroku psql sqlite mysql terraform shell bash zsh
filesystem fs make cargo go pip poetry curl rg ripgrep]
def estimate_tokens(text)
code = prose = words = 0
fence = false
text.split("\n", -1).each do |line|
if line =~ /\A\s*(```|~~~)/
fence = !fence
code += line.length + 1
next
end
if fence || (line =~ /\A(\s{4,}|\t)/ && !line.strip.empty?)
code += line.length + 1
else
prose += line.length + 1
words += line.split(/\s+/).reject(&:empty?).length
end
end
total = code + prose
{ tokens: (code / 4.0 + words * 1.3).round,
code_ratio: total.zero? ? 0 : (code.to_f / total).round(2) }
end
# name / description only, including the folded block form where the bloat hides.
def frontmatter(text)
m = /\A\xEF\xBB\xBF?---\r?\n(.*?)\r?\n---\s*(?:\r?\n|\z)/m.match(text)
return { found: false, name: "", description: "" } unless m
out = { found: true, name: "", description: "" }
lines = m[1].split(/\r?\n/, -1)
i = 0
while i < lines.length
if (km = /\A([A-Za-z_][A-Za-z0-9_-]*)\s*:\s*(.*)\z/.match(lines[i]))
key = km[1].downcase
val = km[2]
if ["", ">", "|", ">-", "|-"].include?(val)
acc = []
j = i + 1
while j < lines.length && (lines[j].strip.empty? || lines[j] =~ /\A\s+\S/)
acc << lines[j].strip
i = j
j += 1
end
val = acc.join(" ").strip
end
val = val.sub(/\A["']/, "").sub(/["']\z/, "").strip
out[:name] = val if key == "name"
out[:description] = val if key == "description"
end
i += 1
end
out
end
def classify(rel)
p = rel.downcase
base = File.basename(p)
return "mcp" if base.end_with?("mcp.json")
return "memory" if %w[claude.md agents.md gemini.md cursor.md].include?(base)
return "agent" if p =~ %r{(\A|/)agents?/}
return "skill" if p =~ %r{(\A|/)skills?/} || base == "skill.md"
return "rule" if p =~ %r{(\A|/)rules?/}
return "command" if p =~ %r{(\A|/)commands?/}
return "rule" if base =~ /\.mdc?\z/
"other"
end
def parse_mcp(text)
data = JSON.parse(text)
table = data["mcpServers"] || data["mcp_servers"] || data["servers"] || {}
table.map do |name, cfg|
cmd = ([cfg["command"]] + (cfg["args"] || [])).compact.join(" ")
probe = "#{name} #{cmd} #{cfg["url"]}".downcase
wraps = CLI_WRAPPERS.find { |w| probe =~ /(\A|[^a-z])#{w}([^a-z]|\z)/ }
tools = cfg["tools"].is_a?(Array) ? cfg["tools"].length : (cfg["toolCount"] || cfg["tool_count"])
assumed = tools.nil?
tools ||= DEFAULT_TOOLS
{ "name" => name, "command" => cmd, "tools" => tools, "assumed" => assumed,
"wraps" => wraps, "tokens" => tools * TOKENS_PER_TOOL }
end
end
def build_input(root, config_name, project_type, window_tokens = 200_000)
components = []
excerpts = []
servers = []
Pathname.new(root).find.sort.each do |path|
next unless path.file? && path.to_s =~ /\.(md|mdc|markdown|json|txt)\z/i
rel = path.relative_path_from(Pathname.new(root)).to_s
text = path.read(encoding: "UTF-8")
kind = classify(rel)
if kind == "mcp"
servers.concat(parse_mcp(text)) rescue nil
next
end
fm = frontmatter(text)
est = estimate_tokens(text)
components << {
"path" => rel, "kind" => kind,
"name" => (fm[:name].empty? ? File.basename(rel).sub(/\.[^.]*\z/, "") : fm[:name]),
"description" => fm[:description],
"description_words" => fm[:description].split(/\s+/).reject(&:empty?).length,
"has_frontmatter" => fm[:found], "lines" => text.split("\n", -1).length,
"chars" => text.length, "tokens" => est[:tokens],
"code_ratio" => est[:code_ratio], "flags" => []
}
excerpts << { "path" => rel, "excerpt" => text[0, 3000], "truncated" => text.length > 3000 }
end
file_tokens = components.sum { |c| c["tokens"] }
mcp_tokens = servers.sum { |s| s["tokens"] }
overhead = file_tokens + mcp_tokens
by_kind = components.group_by { |c| c["kind"] }.transform_values do |list|
{ "count" => list.length, "tokens" => list.sum { |c| c["tokens"] } }
end
by_kind["mcp"] = { "count" => servers.length, "tokens" => mcp_tokens } unless servers.empty?
{ "config_name" => config_name, "project_type" => project_type,
"window_tokens" => window_tokens,
"inventory" => {
"totals" => {
"overhead_tokens" => overhead, "file_tokens" => file_tokens,
"mcp_tokens" => mcp_tokens, "window_tokens" => window_tokens,
"pct" => (overhead.to_f / window_tokens * 1000).round / 10.0,
"available_tokens" => [0, window_tokens - overhead].max,
"component_count" => components.length, "duplicate_count" => 0,
"by_kind" => by_kind
},
"components" => components,
"mcp" => { "servers" => servers, "tool_count" => servers.sum { |s| s["tools"] },
"tokens" => mcp_tokens, "assumed" => servers.any? { |s| s["assumed"] },
"error" => "" },
"overlaps" => []
},
"excerpts" => excerpts,
"excerpt_note" => "Measured locally in Ruby (reduced form); each file sent head-only.",
"prescan_facts" => {
"components" => components.map { |c|
b = File.basename(c["path"])
{ "id" => "component:#{b}", "label" => "#{b} (#{c["kind"]}, #{c["tokens"]}t)" }
},
"flags" => []
} }
end
payload = build_input(".claude", "Harborline monorepo",
"TypeScript monorepo — Next.js web app and a Go service")
puts "#{payload["inventory"]["totals"]["overhead_tokens"]} tokens before the first turn"<?php
// Reduced form: token estimate, kind, frontmatter, MCP costing and totals.
// No duplicate, overlap or flag detection — prescan_facts.flags and
// inventory.overlaps go out empty.
const TOKENS_PER_TOOL = 500;
const DEFAULT_TOOLS = 8;
const CLI_WRAPPERS = ["gh","git","npm","pnpm","yarn","bun","docker","kubectl","supabase",
"vercel","netlify","aws","gcloud","azure","heroku","psql","sqlite","mysql","terraform",
"shell","bash","zsh","filesystem","fs","make","cargo","go","pip","poetry","curl","rg","ripgrep"];
function estimate_tokens(string $text): array {
$code = $prose = $words = 0;
$fence = false;
foreach (explode("\n", $text) as $line) {
if (preg_match('/^\s*(```|~~~)/', $line)) {
$fence = !$fence;
$code += strlen($line) + 1;
continue;
}
$indented = preg_match('/^(\s{4,}|\t)/', $line) && trim($line) !== "";
if ($fence || $indented) {
$code += strlen($line) + 1;
} else {
$prose += strlen($line) + 1;
$words += count(preg_split('/\s+/', trim($line), -1, PREG_SPLIT_NO_EMPTY));
}
}
$total = $code + $prose;
return ["tokens" => (int) round($code / 4 + $words * 1.3),
"code_ratio" => $total ? round($code / $total, 2) : 0];
}
function frontmatter(string $text): array {
if (!preg_match('/\A\x{FEFF}?---\r?\n(.*?)\r?\n---\s*(?:\r?\n|\z)/su', $text, $m)) {
return ["found" => false, "name" => "", "description" => ""];
}
$out = ["found" => true, "name" => "", "description" => ""];
$lines = preg_split('/\r?\n/', $m[1]);
for ($i = 0; $i < count($lines); $i++) {
if (!preg_match('/^([A-Za-z_][A-Za-z0-9_-]*)\s*:\s*(.*)$/', $lines[$i], $km)) continue;
$key = strtolower($km[1]);
$val = $km[2];
if (in_array($val, ["", ">", "|", ">-", "|-"], true)) { // folded block scalar
$acc = [];
for ($j = $i + 1; $j < count($lines); $j++) {
if (trim($lines[$j]) === "" || preg_match('/^\s+\S/', $lines[$j])) {
$acc[] = trim($lines[$j]);
$i = $j;
} else break;
}
$val = trim(implode(" ", $acc));
}
$val = trim(preg_replace('/["\']$/', "", preg_replace('/^["\']/', "", $val)));
if ($key === "name") $out["name"] = $val;
elseif ($key === "description") $out["description"] = $val;
}
return $out;
}
function classify(string $rel): string {
$p = strtolower(str_replace("\\", "/", $rel));
$base = basename($p);
if (str_ends_with($base, "mcp.json")) return "mcp";
if (in_array($base, ["claude.md", "agents.md", "gemini.md", "cursor.md"], true)) return "memory";
if (preg_match('#(^|/)agents?/#', $p)) return "agent";
if (preg_match('#(^|/)skills?/#', $p) || $base === "skill.md") return "skill";
if (preg_match('#(^|/)rules?/#', $p)) return "rule";
if (preg_match('#(^|/)commands?/#', $p)) return "command";
if (preg_match('/\.mdc?$/', $base)) return "rule";
return "other";
}
function parse_mcp(string $text): array {
$data = json_decode($text, true) ?: [];
$table = $data["mcpServers"] ?? $data["mcp_servers"] ?? $data["servers"] ?? [];
$out = [];
foreach ($table as $name => $cfg) {
$cmd = trim(implode(" ", array_filter(array_merge([$cfg["command"] ?? null], $cfg["args"] ?? []))));
$probe = strtolower("$name $cmd " . ($cfg["url"] ?? ""));
$wraps = null;
foreach (CLI_WRAPPERS as $w) {
if (preg_match('/(^|[^a-z])' . $w . '([^a-z]|$)/', $probe)) { $wraps = $w; break; }
}
$tools = is_array($cfg["tools"] ?? null) ? count($cfg["tools"])
: ($cfg["toolCount"] ?? $cfg["tool_count"] ?? null);
$assumed = $tools === null;
$tools = $tools ?? DEFAULT_TOOLS;
$out[] = ["name" => $name, "command" => $cmd, "tools" => $tools,
"assumed" => $assumed, "wraps" => $wraps,
"tokens" => $tools * TOKENS_PER_TOOL];
}
return $out;
}
function build_input(string $root, string $name, string $type, int $window = 200000): array {
$components = $excerpts = $servers = [];
$it = new RecursiveIteratorIterator(new RecursiveDirectoryIterator($root, FilesystemIterator::SKIP_DOTS));
foreach ($it as $file) {
if (!$file->isFile() || !preg_match('/\.(md|mdc|markdown|json|txt)$/i', $file->getFilename())) continue;
$rel = ltrim(str_replace($root, "", $file->getPathname()), "/");
$text = file_get_contents($file->getPathname());
$kind = classify($rel);
if ($kind === "mcp") { $servers = array_merge($servers, parse_mcp($text)); continue; }
$fm = frontmatter($text);
$est = estimate_tokens($text);
$components[] = [
"path" => $rel, "kind" => $kind,
"name" => $fm["name"] !== "" ? $fm["name"] : preg_replace('/\.[^.]*$/', "", basename($rel)),
"description" => $fm["description"],
"description_words" => $fm["description"] === "" ? 0
: count(preg_split('/\s+/', $fm["description"], -1, PREG_SPLIT_NO_EMPTY)),
"has_frontmatter" => $fm["found"],
"lines" => substr_count($text, "\n") + 1, "chars" => strlen($text),
"tokens" => $est["tokens"], "code_ratio" => $est["code_ratio"], "flags" => [],
];
$excerpts[] = ["path" => $rel, "excerpt" => substr($text, 0, 3000),
"truncated" => strlen($text) > 3000];
}
$fileTokens = array_sum(array_column($components, "tokens"));
$mcpTokens = array_sum(array_column($servers, "tokens"));
$overhead = $fileTokens + $mcpTokens;
$byKind = [];
foreach ($components as $c) {
$byKind[$c["kind"]] ??= ["count" => 0, "tokens" => 0];
$byKind[$c["kind"]]["count"]++;
$byKind[$c["kind"]]["tokens"] += $c["tokens"];
}
if ($servers) $byKind["mcp"] = ["count" => count($servers), "tokens" => $mcpTokens];
return [
"config_name" => $name, "project_type" => $type, "window_tokens" => $window,
"inventory" => [
"totals" => [
"overhead_tokens" => $overhead, "file_tokens" => $fileTokens,
"mcp_tokens" => $mcpTokens, "window_tokens" => $window,
"pct" => round($overhead / $window * 100, 1),
"available_tokens" => max(0, $window - $overhead),
"component_count" => count($components), "duplicate_count" => 0,
"by_kind" => $byKind,
],
"components" => $components,
"mcp" => ["servers" => $servers, "tool_count" => array_sum(array_column($servers, "tools")),
"tokens" => $mcpTokens,
"assumed" => (bool) array_filter(array_column($servers, "assumed")),
"error" => ""],
"overlaps" => [],
],
"excerpts" => $excerpts,
"excerpt_note" => "Measured locally in PHP (reduced form); each file sent head-only.",
"prescan_facts" => [
"components" => array_map(fn($c) => [
"id" => "component:" . basename($c["path"]),
"label" => basename($c["path"]) . " ({$c['kind']}, {$c['tokens']}t)",
], $components),
"flags" => [],
],
];
}
$payload = build_input(".claude", "Harborline monorepo",
"TypeScript monorepo — Next.js web app and a Go service");
echo $payload["inventory"]["totals"]["overhead_tokens"] . " tokens before the first turn\n";// Reduced form: token estimate, kind, frontmatter, MCP costing and totals.
// No duplicate, overlap or flag detection — prescan_facts.flags and
// inventory.overlaps go out empty.
using System.Text.Json;
using System.Text.RegularExpressions;
static class CtxScan
{
const int TokensPerTool = 500, DefaultTools = 8;
static readonly Regex Fence = new(@"^\s*(```|~~~)");
static readonly Regex Indent = new(@"^(\s{4,}|\t)");
static readonly Regex Fm = new(@"\A---\r?\n(.*?)\r?\n---\s*(\r?\n|$)", RegexOptions.Singleline);
static readonly Regex Key = new(@"^([A-Za-z_][A-Za-z0-9_-]*)\s*:\s*(.*)$");
static readonly string[] CliWrappers = {
"gh","git","npm","pnpm","yarn","bun","docker","kubectl","supabase","vercel","netlify",
"aws","gcloud","azure","heroku","psql","sqlite","mysql","terraform","shell","bash","zsh",
"filesystem","fs","make","cargo","go","pip","poetry","curl","rg","ripgrep" };
public static (int Tokens, double CodeRatio) EstimateTokens(string text)
{
int code = 0, prose = 0, words = 0;
bool fence = false;
foreach (var line in text.Split('\n'))
{
if (Fence.IsMatch(line)) { fence = !fence; code += line.Length + 1; continue; }
var indented = Indent.IsMatch(line) && line.Trim().Length > 0;
if (fence || indented) code += line.Length + 1;
else
{
prose += line.Length + 1;
if (line.Trim().Length > 0)
words += line.Trim().Split((char[]?)null, StringSplitOptions.RemoveEmptyEntries).Length;
}
}
var total = code + prose;
return ((int)Math.Round(code / 4.0 + words * 1.3),
total == 0 ? 0 : Math.Round((double)code / total, 2));
}
public static (bool Found, string Name, string Description) Frontmatter(string text)
{
var m = Fm.Match(text);
if (!m.Success) return (false, "", "");
var lines = Regex.Split(m.Groups[1].Value, "\r?\n");
string name = "", desc = "";
for (var i = 0; i < lines.Length; i++)
{
var km = Key.Match(lines[i]);
if (!km.Success) continue;
var key = km.Groups[1].Value.ToLowerInvariant();
var val = km.Groups[2].Value;
if (val is "" or ">" or "|" or ">-" or "|-") // folded / literal block scalar
{
var acc = new List<string>();
var j = i + 1;
for (; j < lines.Length; j++)
{
if (lines[j].Trim().Length == 0 || Regex.IsMatch(lines[j], @"^\s+\S")) acc.Add(lines[j].Trim());
else break;
}
i = j - 1;
val = string.Join(" ", acc).Trim();
}
val = Regex.Replace(Regex.Replace(val, "^[\"']", ""), "[\"']$", "").Trim();
if (key == "name") name = val;
else if (key == "description") desc = val;
}
return (true, name, desc);
}
public static string Classify(string rel)
{
var p = rel.Replace('\\', '/').ToLowerInvariant();
var b = p[(p.LastIndexOf('/') + 1)..];
if (b.EndsWith("mcp.json")) return "mcp";
if (b is "claude.md" or "agents.md" or "gemini.md" or "cursor.md") return "memory";
if (Regex.IsMatch(p, "(^|/)agents?/")) return "agent";
if (Regex.IsMatch(p, "(^|/)skills?/") || b == "skill.md") return "skill";
if (Regex.IsMatch(p, "(^|/)rules?/")) return "rule";
if (Regex.IsMatch(p, "(^|/)commands?/")) return "command";
if (Regex.IsMatch(b, @"\.mdc?$")) return "rule";
return "other";
}
public static List<Dictionary<string, object?>> ParseMcp(string json)
{
var servers = new List<Dictionary<string, object?>>();
using var doc = JsonDocument.Parse(json);
if (!doc.RootElement.TryGetProperty("mcpServers", out var table)) return servers;
foreach (var s in table.EnumerateObject())
{
var cmd = s.Value.TryGetProperty("command", out var c) ? c.GetString() ?? "" : "";
if (s.Value.TryGetProperty("args", out var args))
cmd = (cmd + " " + string.Join(" ", args.EnumerateArray().Select(a => a.GetString()))).Trim();
var probe = $"{s.Name} {cmd}".ToLowerInvariant();
var wraps = CliWrappers.FirstOrDefault(w => Regex.IsMatch(probe, $"(^|[^a-z]){w}([^a-z]|$)"));
int? tools = s.Value.TryGetProperty("tools", out var t) && t.ValueKind == JsonValueKind.Array
? t.GetArrayLength()
: s.Value.TryGetProperty("toolCount", out var tc) ? tc.GetInt32() : null;
var assumed = tools is null;
var n = tools ?? DefaultTools;
servers.Add(new() { ["name"] = s.Name, ["command"] = cmd, ["tools"] = n,
["assumed"] = assumed, ["wraps"] = wraps, ["tokens"] = n * TokensPerTool });
}
return servers;
}
public static Dictionary<string, object?> BuildInput(
string root, string name, string projectType, int window = 200000)
{
var components = new List<Dictionary<string, object?>>();
var excerpts = new List<object>();
var servers = new List<Dictionary<string, object?>>();
foreach (var full in Directory.EnumerateFiles(root, "*", SearchOption.AllDirectories).Order())
{
if (!Regex.IsMatch(full, @"\.(md|mdc|markdown|json|txt)$", RegexOptions.IgnoreCase)) continue;
var rel = Path.GetRelativePath(root, full).Replace('\\', '/');
var text = File.ReadAllText(full);
var kind = Classify(rel);
if (kind == "mcp") { servers.AddRange(ParseMcp(text)); continue; }
var (tokens, ratio) = EstimateTokens(text);
var (found, fmName, fmDesc) = Frontmatter(text);
var baseName = Path.GetFileName(rel);
components.Add(new()
{
["path"] = rel, ["kind"] = kind,
["name"] = fmName.Length > 0 ? fmName : Path.GetFileNameWithoutExtension(baseName),
["description"] = fmDesc,
["description_words"] = fmDesc.Split((char[]?)null, StringSplitOptions.RemoveEmptyEntries).Length,
["has_frontmatter"] = found,
["lines"] = text.Split('\n').Length, ["chars"] = text.Length,
["tokens"] = tokens, ["code_ratio"] = ratio, ["flags"] = Array.Empty<string>(),
});
excerpts.Add(new { path = rel, excerpt = text.Length > 3000 ? text[..3000] : text,
truncated = text.Length > 3000 });
}
var fileTokens = components.Sum(c => (int)c["tokens"]!);
var mcpTokens = servers.Sum(s => (int)s["tokens"]!);
var overhead = fileTokens + mcpTokens;
var byKind = components.GroupBy(c => (string)c["kind"]!).ToDictionary(
g => g.Key, g => new { count = g.Count(), tokens = g.Sum(c => (int)c["tokens"]!) });
return new()
{
["config_name"] = name, ["project_type"] = projectType, ["window_tokens"] = window,
["inventory"] = new Dictionary<string, object?>
{
["totals"] = new
{
overhead_tokens = overhead, file_tokens = fileTokens, mcp_tokens = mcpTokens,
window_tokens = window,
pct = Math.Round((double)overhead / window * 100, 1),
available_tokens = Math.Max(0, window - overhead),
component_count = components.Count, duplicate_count = 0,
by_kind = byKind,
},
["components"] = components,
["mcp"] = new { servers, tool_count = servers.Sum(s => (int)s["tools"]!),
tokens = mcpTokens, assumed = servers.Any(s => (bool)s["assumed"]!),
error = "" },
["overlaps"] = Array.Empty<object>(),
},
["excerpts"] = excerpts,
["excerpt_note"] = "Measured locally in C# (reduced form); each file sent head-only.",
["prescan_facts"] = new
{
components = components.Select(c => new
{
id = "component:" + Path.GetFileName((string)c["path"]!),
label = $"{Path.GetFileName((string)c["path"]!)} ({c["kind"]}, {c["tokens"]}t)",
}),
flags = Array.Empty<object>(),
},
};
}
}
var payload = CtxScan.BuildInput(".claude", "Harborline monorepo",
"TypeScript monorepo — Next.js web app and a Go service");
Two habits keep an API-built inventory honest. First, if you cannot get real MCP tool counts,
leave assumed: true on the servers rather than inventing a number — the model
is instructed to say so in assumptions, and a loud guess beats a silent one. Run
/mcp in the agent, count the tools, and pass the real total when you can. Second,
never send an excerpt that is a blind middle slice: head-and-tail with the gap
marked is what lets the model say "this file is truncated and I will not recommend deleting it"
instead of guessing.
Step 4 — Estimate the cost
POST /estimate
Send exactly the body you would send to /run; the response's
hold_credits is the worst-case cost. Nothing is charged and no job is created, so
estimating is free — which makes it the right thing to call in a loop before you audit
forty repositories. The reply carries model, model_alias,
markup_bps, hold_credits, min_credits and
sponsor_enabled.
| Field | Meaning |
|---|---|
model | The concrete model that will run the audit. |
model_alias | The alias this app is bound to. Asserting model_alias == "gpt-terra" is how you check the binding — if it ever comes back as something else, your pipeline is not talking to the app you think it is. |
markup_bps | Platform markup in basis points, applied to the metered cost. |
hold_credits | Worst case: what is reserved before the run. You are charged only what the run uses. |
min_credits | The floor below which a run will not start at all — check it against /me before you submit rather than after a 402. |
sponsor_enabled | True while a sponsored allowance is covering guest runs. |
# input.json came out of step 3
curl -s -X POST "$API/estimate" \
-H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
-d @input.json | jq '.data | {model, model_alias, hold_credits, min_credits}'
# fail loudly if the app is not bound to the model you expect
curl -s -X POST "$API/estimate" \
-H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
-d @input.json | jq -e '.data.model_alias == "gpt-terra"' > /dev/null \
|| { echo "unexpected model binding"; exit 1; }est = api("POST", "/estimate", payload) # payload from step 3
assert est["model_alias"] == "gpt-terra", f'unexpected model: {est["model_alias"]}'
print("worst case:", est.get("hold_credits", est.get("credits")), "credits",
"| floor:", est.get("min_credits"))const est = await api("POST", "/estimate", payload); // payload from step 3
if (est.model_alias !== "gpt-terra") throw new Error(`unexpected model: ${est.model_alias}`);
console.log("worst case:", est.hold_credits ?? est.credits, "credits | floor:", est.min_credits);var est struct {
Model string `json:"model"`
ModelAlias string `json:"model_alias"`
MarkupBps int `json:"markup_bps"`
HoldCredits int64 `json:"hold_credits"`
MinCredits int64 `json:"min_credits"`
Sponsored bool `json:"sponsor_enabled"`
}
if err := call("POST", "/estimate", payload, &est); err != nil {
log.Fatal(err)
}
if est.ModelAlias != "gpt-terra" {
log.Fatalf("unexpected model binding: %s", est.ModelAlias)
}
fmt.Printf("worst case: %d credits (floor %d)\n", est.HoldCredits, est.MinCredits)String jsonPayload = toJson(CtxScan.buildInput(Path.of(".claude"),
"Harborline monorepo", "TypeScript monorepo — Next.js web app and a Go service", 200000));
String envelope = api("POST", "/estimate", jsonPayload);
// data.model_alias must equal "gpt-terra"; data.hold_credits is the worst case,
// data.min_credits the floor below which a run will not start.est = api("POST", "/estimate", payload) # payload from step 3
raise "unexpected model: #{est["model_alias"]}" unless est["model_alias"] == "gpt-terra"
puts "worst case: #{est["hold_credits"] || est["credits"]} credits (floor #{est["min_credits"]})"$est = api("POST", "/estimate", $payload); // $payload from step 3
if (($est["model_alias"] ?? "") !== "gpt-terra") {
throw new Exception("unexpected model: " . ($est["model_alias"] ?? "?"));
}
echo "worst case: " . ($est["hold_credits"] ?? $est["credits"]) .
" credits (floor {$est['min_credits']})\n";var est = await SkillSafe.ApiAsync(HttpMethod.Post, "/estimate", payload); // step 3
var alias = est.GetProperty("model_alias").GetString();
if (alias != "gpt-terra") throw new Exception($"unexpected model: {alias}");
Console.WriteLine($"worst case: {est.GetProperty("hold_credits")} credits " +
$"(floor {est.GetProperty("min_credits")})");
A large inventory is not automatically an expensive run: the excerpt budget in step 3 is what
bounds the input, and hold_credits moves with it. If an estimate surprises you,
check excerpt_note — it tells you how many characters actually went over the
wire.
Step 5 — Run the audit and wait for the result
POST /run
GET /jobs/{job_id}
/run takes the same body as /estimate, places a credit hold and
returns a job_id. Poll /jobs/{job_id} every 1–2 seconds until
status is succeeded or failed (a run typically takes
40–120 s, since the reply classifies every component and every MCP server).
The audit is in output — usually nested as output.output, and as
a JSON string, so parse defensively.
Always send an Idempotency-Key header, and derive it from the input.
A dropped connection during a metered run is the one failure that costs money twice: retry with
a fresh key and you pay for a second audit of the same files. The app builds its key as
context-lens:{hash of the inventory and settings}:a{attempt}, which has exactly the
property you want — retrying the same configuration reuses the key and returns the
original job, while genuinely re-auditing a changed configuration produces a new key and a new
run. Hash the serialized body; do not use a timestamp or a random UUID unless you are certain
you never retry.
# key derived from the body, so a retry of the same input cannot double-bill
KEY="context-lens:$(shasum -a 256 input.json | cut -c1-16):a1"
JOB_ID=$(curl -s -X POST "$API/run" \
-H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
-H "Idempotency-Key: $KEY" \
-d @input.json | jq -r '.data.job_id')
while :; do
JOB=$(curl -s "$API/jobs/$JOB_ID" -H "Authorization: Bearer $TOKEN")
STATUS=$(echo "$JOB" | jq -r '.data.status')
[ "$STATUS" = "succeeded" ] || [ "$STATUS" = "failed" ] && break
sleep 2
done
# unwrap the reply once, then read it
echo "$JOB" | jq -r '.data.output.output' > audit.json
jq -r '
"\(.config_name) [\(.posture)]: \(.verdict)",
"",
"BUDGET \(.budget.overhead_tokens) of \(.budget.window_tokens) tokens (\(.budget.pct)%) " +
"-> \(.budget.after_trim_tokens) after trim (\(.budget.after_trim_pct)%)",
"",
"RECOMMENDATIONS",
(.recommendations[] | " #\(.rank) \(.action) [-\(.saves_tokens) tokens: \(.components | join(", "))]"),
"",
"ISSUES",
(.issues[] | " [\(.severity)] \(.component): \(.problem)"),
"",
"CLASSIFICATION",
(.classification[] | " \(.tier | ascii_upcase) \(.component) — \(.why)"),
"",
"COVERAGE",
(.coverage_check[] | " \(.id): \(if .addressed then "ok" else "SET ASIDE" end) - \(.note)"),
"",
"NEXT",
(.next_steps[] | " - \(.)")' audit.json
# a CI gate: fail the build when the setup is over budget
jq -e '.posture != "over-budget"' audit.json > /dev/null \
|| { echo "context budget exceeded"; exit 1; }import hashlib, time
# Derived from the input: a retry after a blip reuses it and cannot double-bill.
key = "context-lens:" + hashlib.sha256(
json.dumps(payload, sort_keys=True).encode("utf-8")).hexdigest()[:16] + ":a1"
job_id = api("POST", "/run", payload, **{"Idempotency-Key": key})["job_id"]
while True:
job = api("GET", f"/jobs/{job_id}")
if job["status"] in ("succeeded", "failed"):
break
time.sleep(1.5)
if job["status"] == "failed":
raise RuntimeError(job.get("error", "run failed"))
raw = job["output"]
if isinstance(raw, dict) and "output" in raw:
raw = raw["output"]
report = json.loads(raw) if isinstance(raw, str) else raw
b = report["budget"]
print(f'{report["config_name"]} [{report["posture"]}]: {report["verdict"]}')
print(f' {b["overhead_tokens"]:,} of {b["window_tokens"]:,} tokens ({b["pct"]}%) '
f'-> {b["after_trim_tokens"]:,} after trim ({b["after_trim_pct"]}%)')
for r in report["recommendations"]:
print(f' #{r["rank"]} {r["action"]} [-{r["saves_tokens"]:,} tokens: {", ".join(r["components"])}]')
if r["risk"]:
print(f' risk: {r["risk"]}')
for i in report["issues"]:
print(f' [{i["severity"]:>6}] {i["component"]}: {i["problem"]}')
print(f' fix: {i["fix"]} (~{i["saves_tokens"]:,} tokens)')
for c in report["classification"]:
print(f' {c["tier"]:<9} {c["component"]} — {c["why"]}')
for c in report["coverage_check"]:
print(f' {c["id"]}: {"ok" if c["addressed"] else "SET ASIDE"} - {c["note"]}')
for k in report["keep"]:
print(" keep:", k)
for n in report["next_steps"]:
print(" next:", n)
with open("audit.json", "w", encoding="utf-8") as fh:
json.dump(report, fh, indent=2)
if report["posture"] == "over-budget":
raise SystemExit("context budget exceeded")import { createHash } from "node:crypto";
import { writeFileSync } from "node:fs";
// Derived from the input: a retry after a blip reuses it and cannot double-bill.
const key = "context-lens:" +
createHash("sha256").update(JSON.stringify(payload)).digest("hex").slice(0, 16) + ":a1";
const { job_id } = await api("POST", "/run", payload, { "Idempotency-Key": key });
let job;
do {
await new Promise((r) => setTimeout(r, 1500));
job = await api("GET", `/jobs/${job_id}`);
} while (job.status !== "succeeded" && job.status !== "failed");
if (job.status === "failed") throw new Error(job.error ?? "run failed");
const raw = job.output?.output ?? job.output;
const report = typeof raw === "string" ? JSON.parse(raw) : raw;
const b = report.budget;
console.log(`${report.config_name} [${report.posture}]: ${report.verdict}`);
console.log(` ${b.overhead_tokens} of ${b.window_tokens} tokens (${b.pct}%) ` +
`-> ${b.after_trim_tokens} after trim (${b.after_trim_pct}%)`);
for (const r of report.recommendations) {
console.log(` #${r.rank} ${r.action} [-${r.saves_tokens} tokens: ${r.components.join(", ")}]`);
if (r.risk) console.log(` risk: ${r.risk}`);
}
for (const i of report.issues) {
console.log(` [${i.severity}] ${i.component}: ${i.problem}`);
console.log(` fix: ${i.fix} (~${i.saves_tokens} tokens)`);
}
for (const c of report.classification) console.log(` ${c.tier} ${c.component} — ${c.why}`);
for (const c of report.coverage_check) {
console.log(` ${c.id}: ${c.addressed ? "ok" : "SET ASIDE"} - ${c.note}`);
}
for (const n of report.next_steps) console.log(` next: ${n}`);
writeFileSync("audit.json", JSON.stringify(report, null, 2));
if (report.posture === "over-budget") process.exitCode = 1;// Idempotency key derived from the body — a retry cannot start a second run.
body, _ := json.Marshal(payload)
sum := sha256.Sum256(body)
key := fmt.Sprintf("context-lens:%x:a1", sum[:8])
req, _ := http.NewRequest("POST", API+"/run", bytes.NewReader(body))
req.Header.Set("Authorization", "Bearer "+token)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", key)
res, err := http.DefaultClient.Do(req)
if err != nil {
log.Fatal(err)
}
var startEnv struct {
Data struct {
JobID string `json:"job_id"`
} `json:"data"`
}
json.NewDecoder(res.Body).Decode(&startEnv)
res.Body.Close()
var job struct {
Status string `json:"status"`
Error string `json:"error"`
Output json.RawMessage `json:"output"`
}
for {
if err := call("GET", "/jobs/"+startEnv.Data.JobID, nil, &job); err != nil {
log.Fatal(err)
}
if job.Status == "succeeded" || job.Status == "failed" {
break
}
time.Sleep(1500 * time.Millisecond)
}
// job.Output is {"output": "<json string>"} — unwrap, then unmarshal:
type Report struct {
ConfigName string `json:"config_name"`
Posture string `json:"posture"`
Verdict string `json:"verdict"`
ExecSummary string `json:"exec_summary"`
Assumptions []string `json:"assumptions"`
OpenQuestions []string `json:"open_questions"`
Classification []struct {
Component, Tier, Why string
} `json:"classification"`
Issues []struct {
ID, Severity, Component, Problem, Fix string
SavesTokens int `json:"saves_tokens"`
} `json:"issues"`
Recommendations []struct {
Rank int `json:"rank"`
Action string `json:"action"`
Components []string `json:"components"`
SavesTokens int `json:"saves_tokens"`
Risk string `json:"risk"`
} `json:"recommendations"`
Budget struct {
OverheadTokens int `json:"overhead_tokens"`
WindowTokens int `json:"window_tokens"`
Pct float64 `json:"pct"`
AfterTrimTokens int `json:"after_trim_tokens"`
AfterTrimPct float64 `json:"after_trim_pct"`
} `json:"budget"`
Keep []string `json:"keep"`
CoverageCheck []struct {
ID, Note string
Addressed bool
} `json:"coverage_check"`
NextSteps []string `json:"next_steps"`
Summary string `json:"summary"`
}
var wrapper struct{ Output string `json:"output"` }
json.Unmarshal(job.Output, &wrapper)
var report Report
json.Unmarshal([]byte(wrapper.Output), &report)
fmt.Printf("%s [%s]: %s\n", report.ConfigName, report.Posture, report.Verdict)
fmt.Printf(" %d of %d tokens (%.1f%%) -> %d after trim\n",
report.Budget.OverheadTokens, report.Budget.WindowTokens,
report.Budget.Pct, report.Budget.AfterTrimTokens)
for _, r := range report.Recommendations {
fmt.Printf(" #%d %s [-%d tokens: %s]\n", r.Rank, r.Action, r.SavesTokens,
strings.Join(r.Components, ", "))
}
for _, c := range report.Classification {
fmt.Printf(" %-9s %s — %s\n", c.Tier, c.Component, c.Why)
}
os.WriteFile("audit.json", []byte(wrapper.Output), 0o644)
if report.Posture == "over-budget" {
os.Exit(1)
}// Idempotency key derived from the body — a retry cannot start a second run.
var digest = java.security.MessageDigest.getInstance("SHA-256")
.digest(jsonPayload.getBytes(java.nio.charset.StandardCharsets.UTF_8));
var key = "context-lens:" + java.util.HexFormat.of().formatHex(digest).substring(0, 16) + ":a1";
var req = HttpRequest.newBuilder(URI.create(API + "/run"))
.header("Authorization", "Bearer " + TOKEN)
.header("Content-Type", "application/json")
.header("Idempotency-Key", key)
.POST(HttpRequest.BodyPublishers.ofString(jsonPayload))
.build();
String started = HTTP.send(req, HttpResponse.BodyHandlers.ofString()).body();
String jobId = /* data.job_id via your JSON library */;
while (true) {
String job = api("GET", "/jobs/" + jobId, null);
String status = /* data.status */;
if (status.equals("succeeded") || status.equals("failed")) break;
Thread.sleep(1500);
}
// The audit is at data.output.output as a JSON string — parse it again, then read
// config_name, posture, verdict, exec_summary, assumptions[], open_questions[],
// classification[] (component/tier/why — one row per counted component AND per MCP server),
// issues[] (id/severity/component/problem/fix/saves_tokens),
// recommendations[] (rank/action/components[]/saves_tokens/risk) — ranked best-first,
// budget {overhead_tokens, window_tokens, pct, after_trim_tokens, after_trim_pct},
// keep[], coverage_check[] (id/addressed/note), next_steps[] and summary.
// Then: Files.writeString(Path.of("audit.json"), reportJson);
// and gate your build on posture — "over-budget" should fail it.require "digest"
# Derived from the input: a retry after a blip reuses it and cannot double-bill.
key = "context-lens:#{Digest::SHA256.hexdigest(JSON.generate(payload))[0, 16]}:a1"
started = api("POST", "/run", payload, { "Idempotency-Key" => key })
job = nil
loop do
job = api("GET", "/jobs/#{started["job_id"]}")
break if %w[succeeded failed].include?(job["status"])
sleep 1.5
end
raise (job["error"] || "run failed") if job["status"] == "failed"
raw = job["output"].is_a?(Hash) ? job["output"].fetch("output", job["output"]) : job["output"]
report = raw.is_a?(String) ? JSON.parse(raw) : raw
b = report["budget"]
puts "#{report["config_name"]} [#{report["posture"]}]: #{report["verdict"]}"
puts " #{b["overhead_tokens"]} of #{b["window_tokens"]} tokens (#{b["pct"]}%) " \
"-> #{b["after_trim_tokens"]} after trim (#{b["after_trim_pct"]}%)"
report["recommendations"].each do |r|
puts " ##{r["rank"]} #{r["action"]} [-#{r["saves_tokens"]} tokens: #{r["components"].join(", ")}]"
puts " risk: #{r["risk"]}" unless r["risk"].to_s.empty?
end
report["issues"].each { |i| puts " [#{i["severity"]}] #{i["component"]}: #{i["problem"]}" }
report["classification"].each { |c| puts " #{c["tier"]} #{c["component"]} — #{c["why"]}" }
report["coverage_check"].each { |c| puts " #{c["id"]}: #{c["addressed"] ? "ok" : "SET ASIDE"}" }
report["next_steps"].each { |n| puts " next: #{n}" }
File.write("audit.json", JSON.pretty_generate(report))
exit 1 if report["posture"] == "over-budget"// Derived from the input: a retry after a blip reuses it and cannot double-bill.
$key = "context-lens:" . substr(hash("sha256", json_encode($payload)), 0, 16) . ":a1";
$started = api("POST", "/run", $payload, ["Idempotency-Key: $key"]);
do {
sleep(2);
$job = api("GET", "/jobs/" . $started["job_id"]);
} while (!in_array($job["status"], ["succeeded", "failed"]));
if ($job["status"] === "failed") {
throw new Exception($job["error"] ?? "run failed");
}
$raw = is_array($job["output"]) ? ($job["output"]["output"] ?? $job["output"]) : $job["output"];
$report = is_string($raw) ? json_decode($raw, true) : $raw;
$b = $report["budget"];
echo "{$report['config_name']} [{$report['posture']}]: {$report['verdict']}\n";
echo " {$b['overhead_tokens']} of {$b['window_tokens']} tokens ({$b['pct']}%) "
. "-> {$b['after_trim_tokens']} after trim ({$b['after_trim_pct']}%)\n";
foreach ($report["recommendations"] as $r) {
echo " #{$r['rank']} {$r['action']} [-{$r['saves_tokens']} tokens: "
. implode(", ", $r["components"]) . "]\n";
if ($r["risk"] !== "") {
echo " risk: {$r['risk']}\n";
}
}
foreach ($report["issues"] as $i) {
echo " [{$i['severity']}] {$i['component']}: {$i['problem']}\n";
}
foreach ($report["classification"] as $c) {
echo " {$c['tier']} {$c['component']} — {$c['why']}\n";
}
foreach ($report["coverage_check"] as $c) {
echo " {$c['id']}: " . ($c["addressed"] ? "ok" : "SET ASIDE") . "\n";
}
foreach ($report["next_steps"] as $n) {
echo " next: $n\n";
}
file_put_contents("audit.json", json_encode($report, JSON_PRETTY_PRINT));
if ($report["posture"] === "over-budget") {
exit(1);
}using System.Security.Cryptography;
using System.Text;
// Derived from the input: a retry after a blip reuses it and cannot double-bill.
var bodyText = JsonSerializer.Serialize(payload);
var key = "context-lens:" +
Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(bodyText)))[..16].ToLowerInvariant() + ":a1";
var runReq = new HttpRequestMessage(HttpMethod.Post, "https://api.skillsafe.ai/v1/app-api/run")
{
Content = new StringContent(bodyText, Encoding.UTF8, "application/json"),
};
runReq.Headers.Add("Idempotency-Key", key);
var startEnv = await (await new HttpClient().SendAsync(runReq)).Content.ReadFromJsonAsync<JsonElement>();
var jobId = startEnv.GetProperty("data").GetProperty("job_id").GetString();
JsonElement job;
while (true)
{
job = await SkillSafe.ApiAsync(HttpMethod.Get, $"/jobs/{jobId}");
var status = job.GetProperty("status").GetString();
if (status is "succeeded" or "failed") break;
await Task.Delay(1500);
}
var rawText = job.GetProperty("output").GetProperty("output").GetString();
using var doc = JsonDocument.Parse(rawText!);
var report = doc.RootElement;
var b = report.GetProperty("budget");
Console.WriteLine($"{report.GetProperty("config_name")} [{report.GetProperty("posture")}]: " +
$"{report.GetProperty("verdict")}");
Console.WriteLine($" {b.GetProperty("overhead_tokens")} of {b.GetProperty("window_tokens")} tokens " +
$"({b.GetProperty("pct")}%) -> {b.GetProperty("after_trim_tokens")} after trim");
foreach (var r in report.GetProperty("recommendations").EnumerateArray())
{
var comps = string.Join(", ", r.GetProperty("components").EnumerateArray().Select(x => x.GetString()));
Console.WriteLine($" #{r.GetProperty("rank")} {r.GetProperty("action")} " +
$"[-{r.GetProperty("saves_tokens")} tokens: {comps}]");
}
foreach (var c in report.GetProperty("classification").EnumerateArray())
Console.WriteLine($" {c.GetProperty("tier")} {c.GetProperty("component")} — {c.GetProperty("why")}");
await File.WriteAllTextAsync("audit.json", rawText!);
if (report.GetProperty("posture").GetString() == "over-budget") Environment.Exit(1);
The model is asked for one JSON object and nothing else, but a stray code fence or preamble is
always possible. Strip a leading ```json fence, take the text between the first
{ and the last }, and only then parse — that is what the app does
before it falls back to a retry_note reformat run. If you build that fallback
yourself, bump the attempt counter in the idempotency key (:a2) so the retry is a
genuinely new run rather than a replay of the malformed one.
Step 6 — Stream the audit as it is written
POST /run-stream
/run-stream takes exactly the same body as /run but answers with
server-sent events, so you can show progress instead of a spinner — useful here because
a classification row for every component makes for a long reply. This app's own progress panel
is this endpoint: it advances its step list by watching for the "config_name",
"classification", "issues", "recommendations",
"budget", "coverage_check", "next_steps" and
"summary" keys as they arrive. Events are separated by a blank line; each has an
event: line and a data: line carrying JSON.
| Event | Payload | Meaning |
|---|---|---|
job | {job_id, status} | Sent once, when the job is accepted — show "starting". |
delta | {text} | A chunk of the reply, in order. Append it; the accumulated length is your only progress signal, since the total is not known in advance. |
done | {job_id, status, charged_credits, output} | The final, authoritative result — read the audit from output.output rather than trusting concatenated deltas, and the settled price from charged_credits. |
error | {code, message} | Replaces done when the run fails. |
# -N disables buffering so events print as they arrive
curl -N -s -X POST "$API/run-stream" \
-H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
-H "Idempotency-Key: $KEY" \
-d @input.json
# event: job
# data: {"job_id":"job_...","status":"running"}
#
# event: delta
# data: {"text":"{\"config_name\":\"Harborline monorepo"}
# ...
# event: done
# data: {"job_id":"job_...","status":"succeeded","charged_credits":512,"output":{"output":"{...}"}}import json, requests
result = None
with requests.post(
API + "/run-stream",
headers={"Authorization": f"Bearer {TOKEN}", "Idempotency-Key": key},
json=payload,
stream=True,
) as r:
r.raise_for_status()
event = None
for line in r.iter_lines(decode_unicode=True):
if not line:
continue
if line.startswith("event:"):
event = line[len("event:"):].strip()
elif line.startswith("data:"):
data = json.loads(line[len("data:"):].strip())
if event == "delta":
print(".", end="", flush=True) # live progress
elif event == "done":
result = data
elif event == "error":
raise RuntimeError(data.get("message", "run failed"))
report = json.loads(result["output"]["output"]) # authoritative
print("\ncharged:", result["charged_credits"], "-", report["config_name"])
print("posture:", report["posture"])
for r_ in report["recommendations"]:
print(f' #{r_["rank"]} {r_["action"]} (-{r_["saves_tokens"]:,} tokens)')
with open("audit.json", "w", encoding="utf-8") as fh:
json.dump(report, fh, indent=2)const res = await fetch(API + "/run-stream", {
method: "POST",
headers: {
Authorization: `Bearer ${TOKEN}`,
"Content-Type": "application/json",
"Idempotency-Key": key,
},
body: JSON.stringify(payload),
});
const reader = res.body.getReader();
const decoder = new TextDecoder();
let buf = "", done = null;
for (;;) {
const chunk = await reader.read();
if (chunk.done) break;
buf += decoder.decode(chunk.value, { stream: true });
const frames = buf.split("\n\n");
buf = frames.pop();
for (const frame of frames) {
const name = /^event:\s*(.+)$/m.exec(frame)?.[1];
const bodyLine = /^data:\s*(.+)$/m.exec(frame)?.[1];
if (!name || !bodyLine) continue;
const data = JSON.parse(bodyLine);
if (name === "delta") console.write?.(".") ?? 0; // live progress
if (name === "done") done = data;
if (name === "error") throw new Error(data.message ?? "run failed");
}
}
const report = JSON.parse(done.output.output);
console.log(`\n${done.charged_credits} credits - ${report.config_name} [${report.posture}]`);
for (const r of report.recommendations) {
console.log(` #${r.rank} ${r.action} (-${r.saves_tokens} tokens)`);
}
writeFileSync("audit.json", JSON.stringify(report, null, 2));body, _ := json.Marshal(payload)
req, _ := http.NewRequest("POST", API+"/run-stream", bytes.NewReader(body))
req.Header.Set("Authorization", "Bearer "+token)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", key)
res, err := http.DefaultClient.Do(req)
if err != nil {
log.Fatal(err)
}
defer res.Body.Close()
var event string
var final map[string]any
sc := bufio.NewScanner(res.Body)
sc.Buffer(make([]byte, 0, 64*1024), 4*1024*1024)
for sc.Scan() {
line := sc.Text()
switch {
case strings.HasPrefix(line, "event:"):
event = strings.TrimSpace(strings.TrimPrefix(line, "event:"))
case strings.HasPrefix(line, "data:"):
var data map[string]any
json.Unmarshal([]byte(strings.TrimPrefix(line, "data:")), &data)
switch event {
case "delta":
fmt.Print(".") // live progress
case "done":
final = data
case "error":
log.Fatal(data["message"])
}
}
}
// final["output"].(map[string]any)["output"].(string) is the audit JSON —
// unmarshal it into the Report struct from step 5, then write it to audit.json.// Java 17+ — read the stream line by line instead of buffering the body.
var req = HttpRequest.newBuilder(URI.create(API + "/run-stream"))
.header("Authorization", "Bearer " + TOKEN)
.header("Content-Type", "application/json")
.header("Idempotency-Key", key)
.POST(HttpRequest.BodyPublishers.ofString(jsonPayload))
.build();
var res = HTTP.send(req, HttpResponse.BodyHandlers.ofLines());
String event = null, done = null;
for (String line : (Iterable<String>) res.body()::iterator) {
if (line.startsWith("event:")) {
event = line.substring(6).trim();
} else if (line.startsWith("data:")) {
String data = line.substring(5).trim();
if ("delta".equals(event)) System.out.print("."); // live progress
else if ("done".equals(event)) done = data;
else if ("error".equals(event)) throw new RuntimeException(data);
}
}
// parse `done`, then parse data.output.output again — it is a JSON string holding
// config_name, posture, verdict, classification[], issues[], recommendations[],
// budget{}, keep[], coverage_check[], next_steps[] and summary.require "net/http"
require "json"
uri = URI(API + "/run-stream")
req = Net::HTTP::Post.new(uri)
req["Authorization"] = "Bearer #{TOKEN}"
req["Content-Type"] = "application/json"
req["Idempotency-Key"] = key
req.body = payload.to_json
event = nil
done = nil
Net::HTTP.start(uri.host, uri.port, use_ssl: true) do |http|
http.request(req) do |res|
res.read_body do |chunk|
chunk.each_line do |line|
line = line.strip
if line.start_with?("event:")
event = line.delete_prefix("event:").strip
elsif line.start_with?("data:")
data = JSON.parse(line.delete_prefix("data:").strip)
case event
when "delta" then print "." # live progress
when "done" then done = data
when "error" then raise (data["message"] || "run failed")
end
end
end
end
end
end
report = JSON.parse(done["output"]["output"])
puts "\n#{done["charged_credits"]} credits - #{report["config_name"]} [#{report["posture"]}]"
report["recommendations"].each { |r| puts " ##{r["rank"]} #{r["action"]} (-#{r["saves_tokens"]} tokens)" }
File.write("audit.json", JSON.pretty_generate(report))$event = null;
$done = null;
$ch = curl_init(API . "/run-stream");
curl_setopt_array($ch, [
CURLOPT_POST => true,
CURLOPT_HTTPHEADER => [
"Authorization: Bearer $TOKEN",
"Content-Type: application/json",
"Idempotency-Key: $key",
],
CURLOPT_POSTFIELDS => json_encode($payload),
CURLOPT_WRITEFUNCTION => function ($ch, $chunk) use (&$event, &$done) {
foreach (explode("\n", $chunk) as $line) {
$line = trim($line);
if (str_starts_with($line, "event:")) {
$event = trim(substr($line, 6));
} elseif (str_starts_with($line, "data:")) {
$data = json_decode(trim(substr($line, 5)), true);
if ($event === "delta") { echo "."; } // live progress
elseif ($event === "done") { $done = $data; }
elseif ($event === "error") { throw new Exception($data["message"] ?? "run failed"); }
}
}
return strlen($chunk);
},
]);
curl_exec($ch);
curl_close($ch);
$report = json_decode($done["output"]["output"], true);
echo "\n{$done['charged_credits']} credits - {$report['config_name']} [{$report['posture']}]\n";
foreach ($report["recommendations"] as $r) {
echo " #{$r['rank']} {$r['action']} (-{$r['saves_tokens']} tokens)\n";
}
file_put_contents("audit.json", json_encode($report, JSON_PRETTY_PRINT));var req = new HttpRequestMessage(HttpMethod.Post,
"https://api.skillsafe.ai/v1/app-api/run-stream")
{
Content = new StringContent(bodyText, Encoding.UTF8, "application/json"),
};
req.Headers.Add("Idempotency-Key", key);
using var res = await Http.SendAsync(req, HttpCompletionOption.ResponseHeadersRead);
using var reader = new StreamReader(await res.Content.ReadAsStreamAsync());
string? evt = null, done = null;
while (await reader.ReadLineAsync() is { } line)
{
if (line.StartsWith("event:")) evt = line[6..].Trim();
else if (line.StartsWith("data:"))
{
var data = line[5..].Trim();
if (evt == "delta") Console.Write("."); // live progress
else if (evt == "done") done = data;
else if (evt == "error") throw new Exception(data);
}
}
using var final = JsonDocument.Parse(done!);
var text = final.RootElement.GetProperty("output").GetProperty("output").GetString();
using var reportDoc = JsonDocument.Parse(text!);
var streamed = reportDoc.RootElement;
Console.WriteLine($"{streamed.GetProperty("config_name")} [{streamed.GetProperty("posture")}]");
foreach (var r in streamed.GetProperty("recommendations").EnumerateArray())
Console.WriteLine($" #{r.GetProperty("rank")} {r.GetProperty("action")} " +
$"(-{r.GetProperty("saves_tokens")} tokens)");
await File.WriteAllTextAsync("audit.json", text!);
In a browser, the native EventSource only speaks GET, and this endpoint is a POST
— read the fetch response body incrementally, as the JavaScript sample above
does. The Go, Java, Ruby, PHP and C# samples have no dedicated SSE client either; they simply
read the response body line by line and switch on the event: prefix, which is all
the protocol requires here. On an idempotent replay the server may answer with a plain JSON
envelope instead of an event stream; check the Content-Type before you start
parsing frames.
Step 7 — Verify the reply against your own measurements
The audit is judgement, and judgement about arithmetic can drift. The browser therefore never
renders the reply as fact until it has re-checked it against the inventory — that is what
reconcile() in ctxscan.js does, and you should do the
same. There are exactly three cross-checks.
- No unknown components. Every
classification[].component,issues[].componentand entry ofrecommendations[].componentsmust resolve to something you sent — a component'spath, its base name, its frontmattername, or anmcp.servers[].name. Match case-insensitively and tolerateagents/planner.mdvsplanner.mdvsplanner. Anything that does not resolve is an invented row. - No overclaimed savings. An issue's
saves_tokensmay not exceed the measuredtokensof the component it names. A recommendation'ssaves_tokensmay not exceed the sum of its named components' tokens — checked only when every named component resolved, since a partial match would compare against an incomplete pool. - Budget within 10%. The reported
budget.overhead_tokensmust be within 10% of your measuredinventory.totals.overhead_tokens. A different tokenizer and a little rounding will never agree exactly; 10% is the band inside which a disagreement is not worth alarming anyone about, and outside which something is wrong.
A fourth thing is worth tracking alongside them, though it is coverage rather than arithmetic:
every id you sent in prescan_facts.flags must appear in coverage_check
exactly once. Missing ids mean a flag was silently dropped. The three checks are shown below in
cURL, Python and JavaScript; the Go, Java, Ruby, PHP and C# ports are the same three
assertions over the same two documents — build the lookup index, walk the three
component-bearing arrays, compare the totals.
cURL
# input.json (step 3) and audit.json (step 5) side by side
jq -n --slurpfile inv input.json --slurpfile rep audit.json '
($inv[0].inventory) as $i | ($rep[0]) as $r
# every name you sent, lowercased: path, base name, frontmatter name, server name
| ([ $i.components[] | select(.counted != false)
| (.path, (.path | split("/") | last), (.path | split("/") | last | sub("\\.[^.]*$";"")), .name) ]
+ [ $i.mcp.servers[].name ] | map(ascii_downcase) | unique) as $known
# measured tokens per resolvable label
| ([ ($i.components[] | select(.counted != false)
| {key: (.path | split("/") | last | ascii_downcase), value: .tokens}),
($i.mcp.servers[] | {key: (.name | ascii_downcase), value: .tokens}) ] | from_entries) as $cost
| {
unknown: [ (($r.classification[].component), ($r.issues[].component),
($r.recommendations[].components[]))
| select((ascii_downcase | IN($known[])) | not) ] | unique,
overclaimed: [ ($r.issues[]
| select(.saves_tokens > ($cost[(.component | split("/") | last | ascii_downcase)] // 1e9))
| {where: "issue", label: .component, claimed: .saves_tokens,
actual: $cost[(.component | split("/") | last | ascii_downcase)]}),
($r.recommendations[]
| . as $rec
| ([ $rec.components[] | $cost[(. | split("/") | last | ascii_downcase)] ]) as $pool
| select(($pool | map(select(. != null)) | length) == ($rec.components | length))
| select($rec.saves_tokens > ($pool | add))
| {where: "recommendation", label: ($rec.components | join(", ")),
claimed: $rec.saves_tokens, actual: ($pool | add)}) ],
budget_delta: (($r.budget.overhead_tokens - $i.totals.overhead_tokens) as $d
| if ($d | fabs) > ($i.totals.overhead_tokens * 0.1)
then {claimed: $r.budget.overhead_tokens,
measured: $i.totals.overhead_tokens, delta: $d}
else null end),
missing_flags: [ $inv[0].prescan_facts.flags[].id
| select((IN($r.coverage_check[].id)) | not) ]
}' | tee crosscheck.json
# refuse to act on an audit that disagrees with the measurements
jq -e '.unknown == [] and .overclaimed == [] and .budget_delta == null' crosscheck.json > /dev/null \
|| { echo "audit disagrees with the local measurements — review before acting"; exit 1; }Python
import os
def reconcile(report, inventory, prescan_flags):
"""The three cross-checks the browser runs, plus flag coverage."""
index = {}
for c in inventory["components"]:
if c.get("counted") is False:
continue
base = os.path.basename(c["path"])
for label in (c["path"], base, os.path.splitext(base)[0], c.get("name")):
if label:
index[label.lower()] = c["tokens"]
for s in inventory["mcp"]["servers"]:
index[s["name"].lower()] = s["tokens"]
def lookup(label):
k = (label or "").strip().lower()
if k in index:
return index[k]
base = os.path.basename(k) # agents/planner.md -> planner.md
if base in index:
return index[base]
return index.get(os.path.splitext(base)[0]) # -> planner
unknown, overclaimed = [], []
# 1. no component named that is not in the inventory
named = ([("classification", c["component"]) for c in report["classification"]]
+ [("issues", i["component"]) for i in report["issues"]]
+ [("recommendations", lbl) for r in report["recommendations"] for lbl in r["components"]])
for where, label in named:
if label and lookup(label) is None and label not in [u["label"] for u in unknown]:
unknown.append({"where": where, "label": label})
# 2. no saving larger than the measured cost of what it names
for i in report["issues"]:
cost = lookup(i["component"])
if cost is not None and i["saves_tokens"] > cost:
overclaimed.append({"where": "issue", "label": i["component"],
"claimed": i["saves_tokens"], "actual": cost})
for r in report["recommendations"]:
costs = [lookup(lbl) for lbl in r["components"]]
if costs and all(c is not None for c in costs) and r["saves_tokens"] > sum(costs):
overclaimed.append({"where": "recommendation", "label": ", ".join(r["components"]),
"claimed": r["saves_tokens"], "actual": sum(costs)})
# 3. the reported overhead is within 10% of the measured overhead
measured = inventory["totals"]["overhead_tokens"]
claimed = report["budget"]["overhead_tokens"]
delta = claimed - measured if claimed else 0
budget_delta = ({"claimed": claimed, "measured": measured, "delta": delta}
if measured and abs(delta) > measured * 0.1 else None)
seen = {c["id"] for c in report["coverage_check"]}
missing = [f["id"] for f in prescan_flags if f["id"] not in seen]
return {"unknown": unknown, "overclaimed": overclaimed,
"budget_delta": budget_delta, "missing_flags": missing}
check = reconcile(report, payload["inventory"], payload["prescan_facts"]["flags"])
for u in check["unknown"]:
print(f' INVENTED {u["label"]} (named in {u["where"]}, not in the inventory)')
for o in check["overclaimed"]:
print(f' OVERCLAIM {o["label"]}: claims {o["claimed"]:,}, measured {o["actual"]:,}')
if check["budget_delta"]:
d = check["budget_delta"]
print(f' BUDGET reported {d["claimed"]:,} vs measured {d["measured"]:,}')
for m in check["missing_flags"]:
print(f" UNCOVERED {m} never appeared in coverage_check")
if check["unknown"] or check["overclaimed"] or check["budget_delta"]:
raise SystemExit("audit disagrees with the local measurements — review before acting")JavaScript
function reconcile(report, inventory, prescanFlags) {
const base = (p) => String(p).split("/").pop();
const stem = (p) => base(p).replace(/\.(md|mdc|json|txt)$/i, "");
const index = new Map();
for (const c of inventory.components) {
if (c.counted === false) continue;
for (const label of [c.path, base(c.path), stem(c.path), c.name]) {
if (label) index.set(String(label).toLowerCase(), c.tokens);
}
}
for (const s of inventory.mcp.servers) index.set(s.name.toLowerCase(), s.tokens);
const lookup = (label) => {
const k = String(label ?? "").trim().toLowerCase();
if (!k) return undefined;
return index.get(k) ?? index.get(base(k)) ?? index.get(stem(k));
};
const unknown = [], overclaimed = [];
// 1. no component named that is not in the inventory
const named = [
...report.classification.map((c) => ["classification", c.component]),
...report.issues.map((i) => ["issues", i.component]),
...report.recommendations.flatMap((r) => r.components.map((l) => ["recommendations", l])),
];
for (const [where, label] of named) {
if (label && lookup(label) === undefined && !unknown.some((u) => u.label === label)) {
unknown.push({ where, label });
}
}
// 2. no saving larger than the measured cost of what it names
for (const i of report.issues) {
const cost = lookup(i.component);
if (cost !== undefined && i.saves_tokens > cost) {
overclaimed.push({ where: "issue", label: i.component, claimed: i.saves_tokens, actual: cost });
}
}
for (const r of report.recommendations) {
const costs = r.components.map(lookup);
if (costs.length && costs.every((c) => c !== undefined)) {
const pool = costs.reduce((a, b) => a + b, 0);
if (r.saves_tokens > pool) {
overclaimed.push({ where: "recommendation", label: r.components.join(", "),
claimed: r.saves_tokens, actual: pool });
}
}
}
// 3. the reported overhead is within 10% of the measured overhead
const measured = inventory.totals.overhead_tokens;
const claimed = report.budget.overhead_tokens || 0;
const delta = claimed ? claimed - measured : 0;
const budgetDelta = measured && Math.abs(delta) > measured * 0.1
? { claimed, measured, delta } : null;
const seen = new Set(report.coverage_check.map((c) => c.id));
const missingFlags = prescanFlags.filter((f) => !seen.has(f.id)).map((f) => f.id);
return { unknown, overclaimed, budgetDelta, missingFlags };
}
const check = reconcile(report, payload.inventory, payload.prescan_facts.flags);
for (const u of check.unknown) console.log(` INVENTED ${u.label} (named in ${u.where})`);
for (const o of check.overclaimed) {
console.log(` OVERCLAIM ${o.label}: claims ${o.claimed}, measured ${o.actual}`);
}
if (check.budgetDelta) {
console.log(` BUDGET reported ${check.budgetDelta.claimed} vs measured ${check.budgetDelta.measured}`);
}
for (const m of check.missingFlags) console.log(` UNCOVERED ${m} never appeared in coverage_check`);
if (check.unknown.length || check.overclaimed.length || check.budgetDelta) {
process.exitCode = 1; // do not act on an audit that disagrees with the measurements
}A failed cross-check is not necessarily a failed audit — the app shows the disagreement above the results rather than throwing the whole reply away, because the prose reasoning may still be sound even when one arithmetic claim is not. What it must never do, and what your pipeline must never do, is present an unverified number as a measurement. If you automate a trim, gate it on a clean cross-check.
Input reference
The complete request body for /estimate, /run and
/run-stream — they all take the same object.
| Field | Type | Meaning |
|---|---|---|
config_name | string | Your label for this setup, e.g. Harborline monorepo. May be an empty string — one is then derived from the files. It is also what a second audit is compared against, so keep it stable across runs of the same configuration. |
project_type | string, optional | What the project is, in your own words: TypeScript monorepo — Next.js web app and a Go service. This is what relevance is judged against — a Perl conventions file is load-bearing in a Perl repo and dead weight in this one. Leave it empty and the reply says in assumptions that relevance was judged from file contents alone. |
window_tokens | number | The context window to budget against: 200000 (the app's default), 1000000, 128000 or 64000. Every percentage in the reply is computed against this, so sending the window your agent actually runs with matters more than it looks. |
inventory | object, required | Your measurements, and the authority for the whole run. Four keys: totals, components, mcp and overlaps, each detailed below. The model may not contradict anything in here. |
inventory.totals | object | {overhead_tokens, file_tokens, mcp_tokens, window_tokens, pct, available_tokens, component_count, duplicate_count, by_kind}. overhead_tokens and window_tokens are copied verbatim into budget, not recomputed. by_kind maps each kind to {count, tokens}. |
inventory.components | array | One entry per file. Counted entries carry {path, kind, name, description, description_words, has_frontmatter, lines, chars, tokens, code_ratio, flags}; duplicates carry {path, kind, name, lines, chars, tokens, duplicate_of, counted: false, flags} and are excluded from the totals. kind is one of memory, agent, skill, rule, command, other. |
inventory.mcp | object | {servers, tool_count, tokens, assumed, error}. Each server is {name, command, tools, assumed, wraps, tokens}, costed at 500 tokens per tool. When assumed is true the counts are an estimate of 8 tools per server rather than a measurement, and the reply says so in assumptions. error carries a parse failure verbatim instead of silently dropping the config. |
inventory.overlaps | array | {a, b, score} for always-on file pairs at or above 0.25 trigram Jaccard. a and b are base names. Send [] if you skip overlap detection. |
excerpts | array | {path, excerpt, truncated} — the file bodies, budgeted. A truncated file is one the model must not recommend deleting on the strength of what it can see; it will classify it sometimes and raise the question instead. |
excerpt_note | string | How much was profiled locally versus sent. Shown to the user verbatim, so write it as a sentence, not a debug dump. |
prescan_facts | object | {components: [{id, label}], flags: [{id, label}]}. Component ids look like component:planner.md; flag ids are the deterministic checks that fired — see the flag table in step 3. Every flag id you send comes back in coverage_check exactly once, addressed or explicitly set aside, which makes it the field to assert on. Send the two empty arrays if you skip flag detection. |
retry_note | string, optional | Only set by an automatic reformat retry when a first reply was not valid JSON; it is obeyed exactly. Leave it out of a first attempt. |
Output reference
One JSON object, always the same shape. Every array is present. Every component named in it
exists in the inventory you sent, every saves_tokens is bounded by the measured
cost of what it names, and budget.overhead_tokens is your figure, copied rather
than recomputed — those are the three things step 7 checks.
| Field | Type | Meaning |
|---|---|---|
config_name | string | A short name for this setup — yours if you sent one, otherwise derived from the files. Defaults to Untitled setup if the model omits it. |
posture | string | healthy | trim-recommended | over-budget. See the table below. This is the field to gate a build on. |
verdict | string | One sentence naming the headline number and the single biggest lever. |
exec_summary | string | Two or three short paragraphs separated by blank lines: what the setup costs before the first turn, where the weight sits, and what changes it. |
assumptions | string[] | Inferences that had to be made — an assumed MCP tool count, a truncated file judged from its head and tail, relevance judged without a project_type. Read these first: a wrong assumption invalidates the recommendation built on it. |
open_questions | string[] | Questions whose answers would change a recommendation. Where a component that looks removable might not be, it says so here rather than guessing. |
classification | array | {component, tier, why} — one row for every counted component and every MCP server, none omitted. tier is always | sometimes | rarely (anything else normalizes to sometimes), and why ties the tier to this project rather than to the file in the abstract. |
issues | array | {id, severity, component, problem, fix, saves_tokens}. id is the matching prescan_facts.flags id when the issue is one of them, otherwise new:1, new:2, … severity is high (over 5% of the window on its own, or duplicated outright) | medium (a threshold breach with real but bounded cost) | low (hygiene). problem states the cost in tokens; fix is specific to that file. Empty only under posture healthy. |
recommendations | array | The core deliverable — {rank, action, components, saves_tokens, risk}, 1–8 entries ranked best-first by tokens recovered. action is imperative ("Disconnect the github MCP server and use the gh CLI"), components names what it touches, and risk is what you give up, or the empty string when nothing. Two recommendations touching the same component do not both claim its tokens. |
budget | object | {overhead_tokens, window_tokens, pct, after_trim_tokens, after_trim_pct}. The first two are copied from inventory.totals; pct is overhead / window * 100 to one decimal; after_trim_tokens is overhead_tokens minus the sum of every recommendations[].saves_tokens, never below zero, and assumes you take all of them. |
keep | string[] | Components that must stay, each with its one-line reason. The counterweight to a page of cuts — useful when a script would otherwise trim by tier alone. |
coverage_check | array | {id, addressed, note} — one entry per prescan_facts.flags id you sent, each exactly once. addressed: true means an issue or a recommendation resolves it and note says which; addressed: false means it was deliberately set aside and note gives the reason (a threshold that fired on something genuinely load-bearing). Nothing you flag is silently dropped, which is what makes this the field to assert on. |
next_steps | string[] | The concrete next actions, in order — e.g. "Move the Deployment section of CLAUDE.md into rules/deploy.md". |
summary | string | Closing paragraph: the recommendation in one breath. |
The three posture values:
| posture | What it means |
|---|---|
healthy | Overhead is under about 15% of the window and nothing structural is wrong. issues may be empty; recommendations still usually names the cheapest available win. |
trim-recommended | There is real waste but the setup is workable. The normal outcome, and the one where the ranked list earns its keep. |
over-budget | Overhead passes roughly 30% of the window, or MCP alone outweighs everything else, or duplicates and overlaps mean the agent is reading the same instructions twice. This is the value to fail a build on. |
A complete, realistic reply for a seven-file setup with two MCP servers:
{
"config_name": "Harborline monorepo — Claude Code setup",
"posture": "trim-recommended",
"verdict": "31,550 tokens (15.8% of the window) go before the first turn, and 41% of that is
one MCP server wrapping a CLI the agent already has.",
"exec_summary": "Seven configuration files and two MCP servers spend 31,550 tokens before you
type anything, leaving 168,450 for the actual work. That is not a crisis, but
it is roughly one long file review's worth of window given away every session.
The weight is lopsided. The github server's 26 tool schemas cost 13,000 tokens
on their own — more than every agent, skill, rule and command in the repo
combined — and everything it exposes is reachable through the gh CLI this
project already uses in CI. Behind it, CLAUDE.md has grown to 420 lines,
about a third of which is a deployment runbook that belongs in a skill loaded
on demand rather than in the file that reloads on every turn.
Taking the four recommendations below brings overhead to 13,380 tokens, or
6.7% of the window, without removing anything the project actually depends
on.",
"assumptions": [
"MCP tool counts were measured from the configuration, not assumed.",
"rules/typescript.md was sent whole; nothing in this audit rests on a truncated file."
],
"open_questions": [
"Does any workflow depend on github MCP resources rather than tools? The gh CLI replaces
the tools, not the resource endpoints.",
"Is skills/deploy-runbook/SKILL.md still current, or has the deployment section of
CLAUDE.md already superseded it?"
],
"classification": [
{ "component": "CLAUDE.md", "tier": "always",
"why": "Project conventions, reloaded every turn — but a third of it is not conventions." },
{ "component": "planner.md", "tier": "sometimes",
"why": "Spawned for multi-step feature work, which is a minority of sessions here." },
{ "component": "reviewer.md", "tier": "sometimes",
"why": "Runs on PRs; 980 tokens is a fair price for that." },
{ "component": "SKILL.md", "tier": "sometimes",
"why": "Deploy runbook, loaded on demand — the cheap kind of context." },
{ "component": "typescript.md", "tier": "always",
"why": "Matches the actual stack; every line of it is load-bearing in this repo." },
{ "component": "style.md", "tier": "always",
"why": "Always on, and 31% of it repeats typescript.md rather than adding to it." },
{ "component": "ship.md", "tier": "rarely",
"why": "A release command invoked a few times a month, at 240 tokens — leave it." },
{ "component": "github", "tier": "rarely",
"why": "Everything it exposes is reachable through the gh CLI this project already uses." },
{ "component": "postgres", "tier": "sometimes",
"why": "Read-only query access the shell genuinely cannot provide against the staging DB." }
],
"issues": [
{ "id": "cli-wrapper:1", "severity": "high", "component": "github",
"problem": "26 tool schemas at ~500 tokens each keep 13,000 tokens resident in every
session — 41% of the whole budget — for operations the agent can perform by
running gh in the shell.",
"fix": "Remove the github entry from .mcp.json and let the agent call gh directly.",
"saves_tokens": 13000 },
{ "id": "memory-bloat:1", "severity": "high", "component": "CLAUDE.md",
"problem": "420 lines, ~5,400 tokens, reloaded on every turn. Roughly 140 of those lines
are a deployment runbook that is only relevant during a release.",
"fix": "Move the Deployment and Rollback sections into skills/deploy-runbook/SKILL.md and
leave a one-line pointer in CLAUDE.md.",
"saves_tokens": 3200 },
{ "id": "heavy-agent:2", "severity": "medium", "component": "planner.md",
"problem": "268 lines, ~2,310 tokens, re-read into context on every Task call that spawns
it — and ~90 of those lines are an inline review checklist.",
"fix": "Move the checklist into a skill the planner loads when it needs it, and cut the
64-word frontmatter description to about 12.",
"saves_tokens": 1110 },
{ "id": "overlap:1", "severity": "medium", "component": "style.md",
"problem": "31% trigram overlap with typescript.md: naming conventions and import ordering
are stated twice, in ~860 tokens of always-on text, and the two copies have
already drifted on default exports.",
"fix": "Fold style.md's naming and import sections into typescript.md and delete the file;
keep the formatting rules, which typescript.md does not cover.",
"saves_tokens": 860 }
],
"recommendations": [
{ "rank": 1, "action": "Disconnect the github MCP server and use the gh CLI instead",
"components": ["github"], "saves_tokens": 13000,
"risk": "Loses MCP-native pagination helpers; gh covers the same operations with more typing." },
{ "rank": 2, "action": "Move the deployment runbook out of CLAUDE.md into the deploy skill",
"components": ["CLAUDE.md"], "saves_tokens": 3200,
"risk": "The agent needs one extra turn to load the skill during a release." },
{ "rank": 3, "action": "Split planner.md's inline checklist into a skill and trim its description",
"components": ["planner.md"], "saves_tokens": 1110,
"risk": "" },
{ "rank": 4, "action": "Fold style.md into typescript.md and delete the duplicate sections",
"components": ["style.md"], "saves_tokens": 860,
"risk": "One file to edit instead of two; the formatting rules must survive the merge." }
],
"budget": {
"overhead_tokens": 31550,
"window_tokens": 200000,
"pct": 15.8,
"after_trim_tokens": 13380,
"after_trim_pct": 6.7
},
"keep": [
"typescript.md — 1,120 tokens of conventions that match the actual stack; this is what
always-on context is for.",
"postgres — the only component here that does something the shell cannot.",
"ship.md — 240 tokens for the release command is not worth the churn of removing it."
],
"coverage_check": [
{ "id": "memory-bloat:1", "addressed": true,
"note": "Raised as a high-severity issue and recommendation 2." },
{ "id": "heavy-agent:2", "addressed": true,
"note": "Recommendation 3 splits the inline checklist out." },
{ "id": "bloated-desc:2", "addressed": true,
"note": "Folded into recommendation 3 — the 64-word description is cut with the split." },
{ "id": "heavy-rule:5", "addressed": false,
"note": "142 lines of TypeScript conventions in a TypeScript monorepo is load-bearing.
The threshold fired, but the cost is earned and nothing here should be cut." },
{ "id": "overlap:1", "addressed": true,
"note": "Recommendation 4 folds style.md into typescript.md." },
{ "id": "cli-wrapper:1", "addressed": true,
"note": "Top issue and recommendation 1." },
{ "id": "mcp-oversubscribed:1", "addressed": true,
"note": "Same server as cli-wrapper:1; recommendation 1 removes the whole 13,000 tokens." }
],
"next_steps": [
"Delete the github block from .mcp.json, restart the session, and confirm a PR review still
completes using gh.",
"Move the Deployment and Rollback sections of CLAUDE.md into
skills/deploy-runbook/SKILL.md, leaving a pointer line.",
"Re-run the audit afterwards and compare — the header states the delta against this one."
],
"summary": "One MCP server and one overgrown memory file are 51% of this setup's context cost;
disconnect github, move the runbook into a skill, and the configuration drops from
15.8% of the window to 6.7% without losing a capability the project uses."
}
This is AI-generated judgement over measurements you supplied, not a guarantee. The token
figures are estimates — a real tokenizer differs by a few percent, though the ranking of
what is expensive, which is what you act on, is stable. Run step 7 before you automate a trim,
read assumptions and open_questions before you delete anything, and
remember that after_trim_tokens assumes every recommendation is taken.