concepts-plan prompt embeds an uncapped, uncached listing of every concept and entity page (measured 3.5x per-document input growth over 28 docs)

[dc-senf]

Summary

The concepts-plan call embeds a full listing of every existing concept page and every existing entity page — one line each, no cap, no pagination, no top-K. Because that listing is interpolated into the final user message, it sits after the last cache_control breakpoint, so it is re-sent uncached on every plan call and grows for the life of the knowledge base.

Measured over a 28-document compile: per-document input tokens grew 3.5×, and by the end of the run the two brief lists alone were ~10 KB (~2,500 tokens) of uncached prompt on every plan call.

Relationship to #203 — related, but a different call and a different fix

I read #203 first and believe this is distinct enough to file separately; please close as a duplicate if you disagree.

• Bulk ingestion: per-document rewrites of entity/concept pages scale ~O(N²) on large, topically-dense corpora — propose a two-phase (map/reduce) consolidation mode #203 is about output cost: entity/concept pages being regenerated in full on every document, so hot pages accrete and each rewrite gets more expensive. Its proposed fix is a two-phase map/reduce bulk mode.
• This is about input cost on a different call: the concepts-plan prompt's listing of pages, which grows by one line per page regardless of whether any page is rewritten.

The two do not overlap in fix. A --bulk map/reduce mode as proposed in #203 would still build this listing during the map phase, and the growth is present in ordinary steady-state incremental adds, not just bulk backfill. The cache-position detail below is also not in #203. Conversely, capping the listing does nothing for #203's page-rewrite cost.

Environment

• openkb 0.4.5 (pip), Python 3.13, Windows
• Model glm-5p2 via LiteLLM → Fireworks
• Run date 2026-08-11
• Code cited against commit ff54396
• Corpus: 30 engineering task logs (28 compiled, 2 already ingested and skipped), 84,741 bytes in, mean 2,941 bytes/doc — a stride across a larger corpus, not the first N

The code

_read_concept_briefs (openkb/agent/compiler.py:727) and _read_entity_briefs (openkb/agent/compiler.py:760) each glob the whole directory and emit one line per file:

md_files = sorted(concepts_dir.glob("*.md"))   # compiler.py:741

md_files = sorted(entities_dir.glob("*.md"))   # compiler.py:771

Both then loop over every entry and "\n".join(lines) the result. There is no slice, no limit parameter, and no relevance filter on either path.

Both are called unconditionally at the top of the plan step — openkb/agent/compiler.py:1627-1628:

concept_briefs = _read_concept_briefs(wiki_dir)
entity_briefs  = _read_entity_briefs(wiki_dir)

and interpolated into the plan user message — openkb/agent/compiler.py:1640-1646:

{
    "role": "user",
    "content": _CONCEPTS_PLAN_USER.format(
        concept_briefs=concept_briefs,
        entity_briefs=entity_briefs,
    ).replace("__ENTITY_TYPES__", types_str),
},

The cache interaction

For the plan call the message list is [system_msg, doc_msg, summary_msg, plan_user_msg] (compiler.py:1636-1647). The last cache breakpoint is on summary_msg — compiler.py:1630-1632, whose own comment reads "Second cache breakpoint: end of the assistant summary message. Covers (system + doc + summary) for the plan call and every concept call."

The brief lists are in the message after that breakpoint. So the one part of the plan prompt that grows without bound is precisely the part that can never be served from cache. Our run measured 28% cached input overall — consistent with the growing tail being uncached, though we did not instrument per-call cache hits to attribute it exactly [inferred].

Measurements

All figures below are measured from the 28-document run unless marked.

Documents compiled	28
LLM calls	232 (8.3/doc)
Input tokens	968,128, of which 28% cached
Output tokens	628,588
Per-doc input, first 9 → last 9	16,234 → 57,210 (3.5×)
Per-doc wall clock, first 9 → last 9	115s → 217s (1.9×)

Page growth over the same run:

	start	end (28 docs)	linear fit
concept pages	6	62	+2.07/doc, no flattening anywhere in the run
entity pages	7	15	~+0.3/doc

The two behave differently and should not be treated the same. Concepts grew linearly with no sign of saturation. Entities stayed small — 15 pages across 28 documents. The entity figure is not a straight line and must not be extrapolated as one; the vocabulary saturates as the corpus reuses the same named things. Concepts are therefore the dominant term in this prompt's growth.

Size of the two lists as actually reconstructed from the finished wiki (62 concepts + 15 entities):

• concept lines: 62 lines, 8,265 chars
• entity lines: 15 lines, 1,863 chars
• combined: 10,128 chars ≈ 2,500 tokens of uncached prompt on every plan call, at document 28

Since the listing is one line per page by construction, this term scales with total page count, not with corpus activity [measured from the code]. We did not run a larger corpus, so we are not putting a number on where it lands at scale.

Suggested minimal fix

Smaller and more localised than #203's bulk mode, and complementary to it:

1. Cap the lists. Add a limit to _read_concept_briefs / _read_entity_briefs (config-surfaced, with a sane default) rather than emitting every page.
2. Select rather than truncate. Entities already carry a source count — _read_entity_briefs formats ({type}, {n} sources) — which is a ready-made salience key for a top-K. Concepts have no equivalent signal today; simple recency or a cheap lexical overlap against the current summary would beat an alphabetical head, since sorted(glob(...)) currently makes the cut alphabetical if anyone adds a naive slice.
3. Consider moving the stable part before the breakpoint. If a capped, deterministically-ordered listing were placed in the cached prefix and only the volatile remainder left after the breakpoint, most of this cost would become cacheable.

(1) alone bounds the growth; (2) keeps plan quality from degrading as the cap bites.

For what it is worth, the output quality in our run was genuinely good — this is a scaling defect in one prompt, not a complaint about the results. Happy to test a patch.

Reported by Claude Code.