The Context Window Is the Constraint
As enterprises move from chat assistants to production agent systems, a quiet architectural truth is surfacing: the binding constraint on an agent is not the intelligence of the model. It is the context window — and the context window is denominated in tokens.
Every rule, definition, and governance constraint an agent must operate under has to enter the model as tokens. There is no side channel. This piece traces a single causal chain that most knowledge-management strategies ignore: how text becomes tokens, why tokens govern the real economics of inference through the key-value cache, and why the format you choose to represent enterprise knowledge is therefore both an efficiency decision and a reasoning-quality decision — not a documentation one. It is the argument behind SIGN™ (Sigil Intelligence Graph Notation), an open specification originated at Career Highways, illustrated here with measurements against our full ~200-document canon.
Tokens are the unit of consumption
A language model cannot operate on letters or words directly; it operates on numbers. Before any text reaches the model it is segmented into tokens — typically subword chunks — and each token maps to an integer ID drawn from the model’s fixed vocabulary. Common words often become a single token; rarer strings are assembled from smaller pieces. In English, a token averages roughly three-quarters of a word, though this varies substantially with language, code, and numeric content and should be treated as a rule of thumb rather than a constant.
The consequence is that structure is not free. Punctuation, delimiters, and repeated field names are all tokenized exactly like meaningful content. A model is, mechanically, a next-token predictor rolling forward one vocabulary entry at a time — and it pays, in every sense, for each token it must carry. The question ‘how do we represent knowledge for an agent?’ is, underneath, the question ‘how many tokens does that knowledge cost, on every single call?’
The KV cache: why token count governs cost, latency, and memory
To understand why token count is so consequential, look at how autoregressive generation actually runs. The core operation of attention projects each token into a query, a key, and a value. To generate the next token, the model compares the current query against the keys of every prior token and blends their values accordingly. Naively, producing a sequence of length N would recompute every token’s projections at every step — work that scales with the square of the sequence length.
The standard optimization is the key-value cache. A token’s key and value never change once it is in the sequence, so they are computed once and stored. Each new step then processes only the single new token and reads the rest from cache, collapsing per-step work and making generation practical. Essentially every production inference stack relies on it.
The catch is where that cache lives and how it grows. The KV cache must sit in fast accelerator memory for the entire lifetime of a request, and its size grows linearly with the number of tokens in context and linearly with the number of concurrent requests. Model weights are a fixed, one-time cost; the KV cache is a per-request, per-token cost that stacks. On long contexts under real concurrency, it can rival or exceed the memory footprint of the weights themselves, and it is frequently the true reason a deployment runs out of memory or slows down — not model size, but the aggregate weight of the caches.
This reframes what a token in the context window costs. It is not merely input billed once. Each token is persistent accelerator memory held for the duration of the request, and it contributes to latency. And critically, every token spent on knowledge overhead is a token unavailable for the model’s own reasoning — a point we return to below. In an agentic enterprise running many calls per day, the representation of knowledge becomes a first-order driver of cost, throughput, hardware ceiling, and the room left to think.
The representation tax
If tokens are the currency and the KV cache is the reason they are expensive, then the encoding of enterprise knowledge is an efficiency lever hiding in plain sight. The formats teams reach for were each designed for a different consumer, and none for the agent context window as the primary unit of consumption.
| Approach | Limitation for agent knowledge |
|---|---|
| JSON | Machine-readable but token-heavy. A large share of tokens is structural noise — braces, quotes, and field names repeated on every record — carrying no domain meaning. Weak at expressing relationships, constraints, or inference. |
| Raw markdown | Token-efficient but structurally untyped. An agent cannot reliably distinguish a hard constraint from a property or a description, and there is no relationship model. |
| RDF / OWL | Semantically rigorous but adoption-hostile: verbose syntax, heavy toolchain, and sparse presence in model training data, so models are not fluent in it. |
| Prompt engineering | Fast but ungoverned — unversioned, unauditable, and non-reusable. It does not survive scale. |
The gap is not a syntax preference. It is that typed, governed knowledge and low token cost have been treated as mutually exclusive. Structure-rich formats are expensive; cheap formats are unstructured. That trade-off is the tax.
What it costs across a real corpus
SIGN is a knowledge-contract notation designed against the agent context window as its target. Its central move is the use of sigils — compact, single-glyph markers — to carry type and structure densely, so that entities, properties, relationships, constraints, inference rules, and provenance can be expressed with the fidelity of a knowledge format without the structural-noise premium a serialization format like JSON imposes.
Rather than lean on a single hand-picked document, we measured the effect across our entire canon — 198 documents with one-to-one markdown-to-SIGN coverage. Encoding the corpus in SIGN rather than markdown reduces it from roughly 372,000 tokens to roughly 275,000: about a 26% reduction against a format that is already lean, and a far larger reduction against the JSON many teams would otherwise inject.
| Corpus | Markdown | SIGN | Savings |
|---|---|---|---|
| Canon core (44 docs) | 109,600 | 87,900 | 20% |
| Tenant canon (154 docs) | 262,200 | 187,100 | 29% |
| Combined (198 docs) | 371,700 | 275,000 | 26% |
The corpus-level average hides a more useful signal: SIGN’s™ savings track structural density. Prose-dominant material compresses least — general commons and domain documents land around 85% of their markdown size — because there is little structural overhead to remove. Highly structured, governance-heavy material compresses most: our top-level governance charter falls to roughly 45% of its markdown size, since that is exactly the content where typed declarations, relationships, and constraints would otherwise carry the heaviest scaffolding. The rule of thumb for builders: the more typed and relational your knowledge, the more a purpose-built notation returns.
A note on measurement
These figures are estimates from modern subword tokenizers, not official counts from a specific production model. We bracketed the corpus with two independent tokenizers; they agree to within about 0.6%, and tokenizer choice does not move the comparison. As a known property of this class of tokenizer, absolute counts tend to run somewhat below a frontier model’s own tokenizer on prose — plausibly on the order of 10–20% higher in the real model, more on heavily structured text — so treat the absolute totals as a floor and the ratios as the durable result. What is invariant under any tokenizer is the comparison itself: the format runs about three-quarters the size of markdown overall, and the savings concentrate in structured content.
From token savings to reasoning headroom
Efficiency is only half the story, and arguably the smaller half. The more consequential effect is on how well an agent can reason over its knowledge — and this arrives through two distinct channels.
The first is representational. Markdown gives an agent text to retrieve; a typed notation gives it structure to reason over. When a constraint is explicitly marked as a constraint, a relationship as a governed predicate, and a fact as asserted-versus-inferred with its provenance attached, the agent is not left inferring the shape of the knowledge from prose formatting. It can distinguish a rule it must enforce from a property it may use, traverse declared relationships rather than pattern-match across paragraphs, and weight a fact by its derivation rather than by how confidently it happens to be phrased. Each of these removes a class of misread that untyped text invites — the failure mode where an agent treats a hard mutex as a soft suggestion, or an inferred claim as ground truth.
The second channel is budgetary, and it ties directly back to the KV cache. Modern reasoning models do their best work by spending tokens to think — intermediate reasoning, self-checking, working through constraints. That reasoning competes for the same finite context window, and the same cache memory, as the knowledge you inject. Every token reclaimed from knowledge overhead is a token returned to the model’s reasoning budget. A roughly one-quarter reduction in the resident cost of the canon is not merely cheaper; it widens the headroom in which the model can actually reason before it hits the wall — and it does so on every invocation, so the effect compounds with agent volume rather than being paid once.
We want to be precise about the nature of this claim. The representational and budgetary arguments are mechanistic — they follow from how typed knowledge and finite context windows work — not from a controlled reasoning-quality benchmark, which we have not yet run. The honest next step is measurement: paired evaluations that hold the model and task fixed while varying only the knowledge encoding, scoring constraint adherence, correct inference, and error rate. The token economics are measured; the reasoning uplift is, for now, a well-grounded hypothesis we intend to test rather than a number we will quote.
Implications for builders
For CTOs and researchers standing up agent platforms, the practical takeaway is to treat knowledge representation as an infrastructure decision measured in tokens, not a formatting choice — and to recognize that the same decision governs how much room the model has to think. Instrument the token cost of your injected knowledge, treat that cost as recurring KV-cache pressure rather than a one-time input charge, and prefer representations that hold typing and governance without the structural-noise premium.
SIGN™ is one answer, published as an open specification under the Apache 2.0 license so it can be evaluated, measured, and adopted on its merits. Whatever notation a team lands on, the underlying discipline is the same: in the agentic enterprise, the context window is the constraint, tokens are how you spend against it, and the format of your knowledge determines both what you pay and how well your agents can reason within what remains.
