Design note: v1.0 compilation state and C++ options - #1254
Draft
jgabry wants to merge 55 commits into
Draft
Conversation
Records the contracts behind #1228, #1234, #1019, #1237 and #1238. These have not been independent defects: each was rediscovered by being violated, because the rules they violate were never written down anywhere. Describes what is recorded about an executable and when, what a configuration means once it reaches make, when that record is validated, and what can be known about an executable cmdstanr did not build. Two decisions reverse earlier ones: options become one-shot at cmdstan_model(), and deferred compilation is removed. The note is a draft for discussion and is deliberately ahead of the tracker. Several issues still assert decisions it supersedes, #1248 most of all, so it lists them explicitly rather than leaving someone to read a stale issue as current. Updating those issues is held until the design settles.
The architecture is unchanged. This resolves contracts that were internally
inconsistent or underspecified.
Validation becomes a pure freshness assessment with two caller behaviours
rather than one rule: cmdstan_model() rebuilds on a trigger, and every
operation that runs or derives state from the binary errors. Stating both as
a single contract read as a contradiction between sections 5 and 6. The error
no longer advises force_recompile after source or configuration changes, since
the constructor detects those on its own; that advice is reserved for corrupt
records, artifact mismatches and explicit distrust.
Executable-only models split into two cases. One produced by compile_stan_file()
and then adopted has a valid hash-bound record, and treating every adopted
executable as unprovenanced discarded information the package itself wrote.
Raw NAME+=value and its siblings are classified as assignments rather than
opaque arguments. Verified against make: every operator collapses to = with
command-line origin, so list("FOO+=x") and list(foo = "x") describe the same
build and must compare equal. Include re-resolution invokes stanc rather than
reimplementing its rules, since stanc --info measures 29.9 ms against a 30-90
second compile and reproducing those rules imperfectly would reintroduce the
silent-stale-binary problem.
provenance_complete becomes known_untracked_dependencies. A regex can show
that a gap exists but never that none does, and the note already warned
against exactly this reasoning for reported_features.
The stages reorder so the deferred-compilation lifecycle is removed before the
record drives any decision, which avoids implementing transitional behaviour
the final design does not specify.
Fourth review round. No architectural change; these are implementation contracts that were underspecified or that the new choices made inconsistent. The introspection snapshot is captured eagerly. $variables() parses from disk on first call, so an edit made before that call would describe the new source while claiming to describe the built one, violating the contract by the mechanism meant to implement it. The assessment already invokes stanc --info for include resolution and the same output carries the variables, so the constructor commits it after a successful build. $format(overwrite_file = TRUE) no longer refreshes the caches: formatting makes the object stale rather than updating it. Include comparison drops the recorded spelling, search roots and selected path in favour of the included_files vector stanc --info already returns, verified to come back fully resolved. Re-resolution invokes stanc from the recorded builder rather than whichever installation is currently selected, and builder identity is checked first so a mismatch is reported without re-resolving. The tri-state reported_features contract gains the consumer policy it was missing. Unknown status errors when an operation requires the feature, scoped to runtime arguments that depend on a build feature so that permanently unreportable options like CXXFLAGS do not error on everything. assert_valid_threads() changes rather than being preserved: it currently stops when a threaded binary has no threads argument but merely warns and discards the argument in the converse case, and both are the same mismatch. The API change and the decision engine become one stage. Separating them leaves a window where an existing unthreaded executable is reused while $compile(), the only escape route, is already gone.
Fifth review round, and the last one: approved after this. The tri-state consumer table was doing two jobs. It now covers one case explicitly — a runtime argument asking for a build feature — where known disabled and unknown both error. The converse, an artifact carrying a feature nobody asked to use, is stated as its own policy rather than an instance of the table, because it is not a mismatch at all. That policy keeps today's error for a threading-enabled binary run without a threads argument, on the grounds that building with threading and not using it is more likely a mistake than an intention. Two things make that conservative rather than new: it has five assertion sites in test-threads.R plus snapshots, and it is already reachable for threading inherited from make/local, since $cpp_options() has merged executable metadata on the construction and no-op paths for some time. #1235 extends that merge to the fresh-compile path, making the behaviour uniform rather than introducing it. The cost is now stated: a user with STAN_THREADS=true in make/local must pass a threads argument every run. Path normalisation is settled rather than open. Normalised absolute paths, and relocating a project rebuilds. Relocatable records would require defining roots, symlink behaviour and out-of-project paths for little benefit, and the case where rebuilding is impossible is already covered by executable-only models.
Two gaps found while checking the note against a summary written from it. The record had no format_version. The third draft moved the field enumeration into the vocabulary section and dropped it, leaving the forward-compatibility rule with nothing to check. It is restored, along with known_untracked_dependencies, which had the same problem: specified in the rebuild section but absent from the list of what a record holds. The rebuild trigger list was source-side only. A replaced or corrupt executable, an unreadable record, and an executable predating records are all reasons to rebuild, and omitting them left the canonical list disagreeing with the sections that describe them. A record whose format_version is newer than we understand is deliberately not among them: rebuilding would install a replacement over a record written by something that knows more, which is what the forward-compatibility rule exists to prevent. Unreadable and readable-but-newer look alike and are now stated as distinct, since conflating them is how the rule gets broken.
A reader currently passes about a hundred lines of purpose and history before reaching a concrete decision. This gives the shape in one screen: what the API becomes, what triggers a rebuild, and what the record holds. It is explicitly orientation rather than specification, so the sections below stay the single place the contract lives. It also takes over some of the orienting work the history section does, which is due to be removed once the tracker catches up.
The directory is developer documentation, not package content, so R CMD check would otherwise flag it as a non-standard top-level file. PR #1235 adds the same line on its own branch; this makes it independent of that PR's merge order.
This was referenced Aug 27, 2026
Closed
Open
The note carried two things whose only job was to survive the gap between the design settling and the issues catching up: a narrative of the two superseded drafts, and a list of issues that would mislead a reader by still asserting decisions this reverses. Both are now false rather than merely unnecessary. #1247, #1248 and #1252 are closed with their reasoning, #1238, #1250 and #1253 are rescoped, and #1255, #1256 and #1257 carry the new work. The trust direction goes back to normal: the issues are the specification, this note is the reasoning behind it. The rejection of persistent options survives, distilled into section 2. It is the most tempting alternative in this design and the one most likely to be proposed again, so the argument for it and the reason it fails belong with the contract rather than in a history section.
…model() An earlier version said to export it only if a consumer committed to it, on the grounds that citing instantiate as motivation was speculative. That was the wrong bar. The argument is parity rather than demand: cmdstanpy already has compile_stan_file, and exporting format_stan_file() and check_syntax_stan_file() while withholding the compile step is arbitrary — with compile = FALSE gone there would be no way to build without constructing an R6 object. Both entry points call one internal, which returns the executable path plus the record, the stanc --info output and the generated C++. Returning only a path would make cmdstan_model() re-read the record and re-run stanc, which is duplication in its most wasteful form; the src_info is what feeds the eager introspection snapshot, and the presence or absence of hpp_code is what answers the generated-C++ question in #1245. dry_run stays on the internal, which is the only argument the public wrapper omits. compile_stan_file() performs the same up-to-date check rather than always compiling, and writes the record, so adopting its output later carries provenance. force_recompile keeps cmdstanr's spelling rather than cmdstanpy's force. Matching the function name is what makes the two APIs teachable together; matching every argument at the cost of internal consistency is not.
jgabry
marked this pull request as ready for review
August 27, 2026 22:13
Member
Author
|
@andrjohns when you have a chance take a peek at my list above (you don't need to read the full document in the PR that Claude, it's just more details on all the items that I wrote in that list above). This redesign of the compilation/build process for 1.0 came out of discussion with @WardBrian and @SteveBronder. I think it's a much cleaner design than what we currently do (and actually simpler in many ways, despite the additional dep file) and it replaces the previous half-done C++ options work that never got finished. You can ignore all the issues that have been opened lately, they're just based off of this list and I'll close them as I go through the implementation. I'm hoping to start working on this ASAP. |
Section 9 gave the ordering but said nothing about execution. Adds the release candidate as a third constraint on the order: stages 0-4 must all be in it, because the API removal is the breaking change downstream packages need to see, while stage 5 only adds a function and can follow. The candidate period is also the real use stage 5 was already waiting on. One constraint falls out of that. The repo-wide formatting and linting work (#1153, #1172) has to land before stage 1 or after 1.0, never between stage 4 and the candidate, where a reformatting diff on top of the API removal would hide what actually broke. Adds a note on how the stages are run: one pull request each, stage 4 built as a tested pure engine before the wiring and the API removal, and only one compiling task at a time, since make/local and the precompiled headers live in the CmdStan installation rather than the checkout and separate checkouts do not separate them. Drops the joint cmdstanpy naming process. Where cmdstanpy already has a name we copy it, and otherwise we pick one and they can copy it; nothing here needs to wait on that. Also brings the issue-consolidation note up to date. It still described the work as pending and pointed at a section that has since been removed.
JSON, named <model>.cmdstanr.json beside the executable. jsonlite is already an import, so the format costs nothing; the name stays clear of .dep and .d, which make and the C++ toolchain already claim in that directory. Stage 2 no longer has to settle this, but portability and the git-ignore story are still open and still have to be answered before anything writes a file. Choosing JSON adds a third way to get the tri-state fields wrong, so the note in section 10 now says that unknown has to round-trip as distinct from both absent and false, and that this is a property to test rather than assume.
32 tasks
Section 6 treats an executable predating records as a rebuild trigger. Since 0.9 stays installable from GitHub, that transition could be tested in CI rather than waited for in the wild. Recorded as a possibility for stage 4 to weigh, not as a commitment; building such an executable by hand when it is needed may well be enough. Also corrects stage 4's issue references. It still said the stage closes #1252, which the consolidation already closed, and pointed at #1019 rather than the #1255 and #1256 that were opened to carry this work.
Air's one-time whole-repo format goes last, immediately before 1.0. It is whitespace-only and deterministic, so shipping it after the candidate is cheap, and by then nothing is left for it to conflict with. Its pull request review action is a separate matter and is better landed early, while stages 2 to 4 are writing the code it would otherwise reformat afterwards. Jarl is not the same kind of change. Adopting it is additive, but acting on its findings is semantic editing, and that cannot follow the candidate without 1.0 shipping code in a form nobody tested. Those are ordinary reviewed changes. The previous note offered "before stage 1" as an option. That was never really available, with #1235 and #1254 both open.
Refusing to replace a record written by a newer cmdstanr protected nothing. The record is entirely derived data, so every field in it can be regenerated by rebuilding, and what the rule was guarding cost thirty to ninety seconds. It paid for that with an error in the one situation where it actually comes up. That situation is also not the one the rule was written for. Passing an executable between machines barely works, since the binary links TBB at an absolute path inside the installation that built it. The realistic case is one person downgrading cmdstanr on one machine: same CmdStan, same make/local, same paths, so nothing else triggers and this was the only thing in the way. The real hazard was never the record but the executable, which a newer cmdstanr may have built under option semantics this version no longer implements. That argues for saying loudly what happened, not for refusing, so a forward-version record now rebuilds with the reason printed, like an executable predating records. Unreadable and forward-version remain separate checks. Same behaviour, different diagnosis: one could not be parsed, the other reports which format version was found and which is understood.
format(overwrite_file = TRUE) rewrites the Stan file and then reassigns stan_code_ and clears variables_, so both accessors end up describing a source the executable was never built from. That is #1228's failure in the opposite direction. The refresh was added deliberately in #1235, under the older contract where code() meant the file as it is now, and the accessors have since been redefined without it catching up. Deleting those lines is the whole fix, so the method stays and keeps overwriting. A reviewer proposed removing it; rewriting the file is the useful part and is not what breaks anything. Afterwards the file changes, the snapshot keeps describing the built source, and the next operation that runs the binary errors and points at cmdstan_model(). Reformatting therefore forces a recompile, which is correct: the bytes changed, and whether the build is unaffected cannot be known without doing it. Also adds a NEWS reconciliation pass before the release candidate. The unreleased section already carries fifteen-plus entries about $compile(), compile = FALSE and dry_run that stage 4 deletes, and one describing the format() refresh this commit reverses. Such entries are removed rather than annotated, since a user upgrading from 0.9 never saw the intermediate behaviour.
The leading dot keeps the record out of `ls`, not out of git, so `git add -A` will commit it silently. A record in version control is worse than none for whoever checks it out: it names an executable hash and a CmdStan installation path that do not exist on their machine, so it fails validation and rebuilds anyway, having produced a diff on every rebuild along the way. `.Rbuildignore` needs the same entry and is the easier one to miss. `R CMD build` filters hidden files by a fixed list rather than by leading dot, so a package author who compiles in a source tree ships records describing their own machine. That is the path that reaches instantiate-style packages. cmdstanr writes neither file. Compiling a model should not edit a user's repository configuration, so this is documentation rather than behaviour. Stage 2 no longer claims portability is undecided, which section 6 settled when dependencies moved to content identity.
rethinking is the third priority alongside brms and instantiate, not for fan-out but for reach: it is how most people first meet cmdstanr. It hard-depends on cmdstanr and reaches it at four places, all `cmdstan_model()` construction, and never reads build state back. Removing deferred compilation is the only thing that touches it. In `ulam()` the argument is already hardcoded true, so the fix is a deleted line. `cstan()` is the one place across all three packages where the removal reaches end users, since `compile` is rethinking's own documented argument passed straight through. Being distributed from GitHub, rethinking can require the new cmdstanr outright rather than guarding on version. Also records a threading fix worth sending independently. Every ulam model is built with threading enabled whether or not it is used, and the guard cannot be restored on its own without warning on every default call, because `threads_per_chain` is passed unconditionally. Both halves have to move together, and `cstan()` already shows the shape. Notes that the neighbouring recompile workaround is inert and should be left alone: `compile = TRUE` never forced a rebuild, and `tempdir()` is what prevents an executable from surviving the session.
The unconditional `stan_threads` in `ulam()` and the inert recompile workaround beside it are rethinking's own concerns, not consequences of anything in 1.0, and they are being fixed in a separate pull request against that package. Keeping the diagnosis here would leave this document owning a fix it does not make. What the survey still needs is the part that bears on 1.0: rethinking satisfies the threading policy unchanged, because it supplies `threads_per_chain` on every call.
Reverses the content-only rule. A dependency matches only when its normalised absolute path and its content hash both match the record, so relocating a project rebuilds it once. The content-only rule assumed the executable is location-independent. It is not. stanc embeds the absolute path of the Stan file and of every resolved include into the generated C++ locations_array__ — six occurrences for a twenty-line model against 2.39 — and those strings are compiled in and surface in every runtime exception message. A relocated executable computes the same answers and reports them against a location that no longer exists, indefinitely, until something else triggers a rebuild. The variant that keeps relocation free by storing an immutable built_from path alongside content identity is coherent and is rejected on cost: one fewer field comparison in exchange for six things to explain, five of which exist only to justify why something that looks wrong is fine. Also scopes the lost-record claim. A missing record costs one compile only where source is available; for an executable-only model there is nothing to rebuild from, so it costs provenance instead.
Builder identity is the normalised installation path and the version, not the version alone. Switching installations with set_cmdstan_path() rebuilds even when both are the same version. Under version-only identity that selection had no effect: the executable stayed the one the old installation built, still linked against its TBB at an absolute path inside it, while validation ran the new installation's stanc. The same TBB linkage used elsewhere to justify requiring the recorded installation to still exist argues just as directly for the path being identity. The case for dropping it was that nothing nameable separates two same-version installations. That is nearly true, and it cuts the other way: make/local lives inside the installation and is hashed separately, so a switch usually rebuilds on that alone. Path identity therefore adds a rebuild only when both installations are configured identically, which is exactly when the user's explicit choice would otherwise be ignored without a word.
The table named twenty-five members and claimed twenty-seven by counting cmdstan_model(), which is not an R6 member. CmdStanModel has twenty-seven public methods and one public field. Missing were $initialize(), $clone() and $functions. The "thirteen in the class body, fourteen attached" sentence is gone. It disguised the gap, adds nothing a reader needs, and goes stale the moment a method moves. Neither new method is guarded. $initialize() is reached through cmdstan_model(), nothing calls it on a live object, and defending against a direct second call is error handling for a case nobody arrives at. $clone() copies state and derives nothing from the binary. $clone() does get a note. functions is an environment and there is no deep_clone method, so a clone shares it with the original — which reads as a defect until you notice that section 8 makes models immutable after construction, so the two can never describe different executables and a shared exposure is valid for both.
The design said what happens to a record written by a newer cmdstanr and left the older direction to assumption. Old-format records keep appearing: a downgrade regenerates them, so v3 writes format 3, v2 rebuilds and writes format 2, and v3 then meets a format-2 record. The rule is now "rebuild on any version this cmdstanr does not read", in either direction. At 1.0 the readable set is a single version, so any mismatch rebuilds, and a later release can widen the set without reopening the contract. The part that needed saying is what format_version versions. It is the build interpretation contract, not the JSON shape: if canonicalisation of cpp_options ever changes, identical bytes in a request field mean something different, and reading an old record with new rules could report "unchanged" for a build that would differ. A change in option or build semantics therefore obliges a bump. Executable-only models are the exception, since section 7 forbids rebuilding them. An unreadable version there joins missing, corrupt and hash-mismatched in leaving the executable unprovenanced, and the record stays where it is. Also carries the installation path into the rebuild-trigger list, which the previous commit changed in the identity rules but not in the summary.
The previous version recommended ignoring the record universally, and justified it with a claim that is false for part of the design: that a missing record costs a single rebuild. Section 7 forbids rebuilding an executable-only model, so for the compile_stan_file() to cmdstan_model(exe_file=) flow a lost record costs provenance permanently. Replaced with a rule that covers every case instead of enumerating them: whatever ignores the executable ignores the record, and wherever the executable goes the record goes with it. The gitignore and Rbuildignore advice now follows from that rather than standing on its own, which keeps it correct for the uncommon cases where an executable is deliberately kept — a CI artifact, a container layer, a shared build directory. Notes that instantiate is not one of those cases. It compiles on the user's machine at install time, so its records are generated locally beside a binary that was never in git or in the tarball. Also states that losing a record breaks nothing, so that "permanently loses provenance" is not read as an executable becoming unusable and fixed by machinery nobody needs.
The document said cmdstan_model() is the only place a build is configured and then defined an exported compile_stan_file() taking the same configuration arguments and driving the same engine. The claim was removed from the pull request body earlier and the document did not follow. The rule is that cmdstan_model() and compile_stan_file() are the public build entry points, and that no operation on an existing model object compiles. That is what section 8 actually establishes; "only cmdstan_model()" was a stronger statement than the design ever supported. The validation table repeated the same claim in its rebuild row, so it is corrected there too.
The guidance reserved the flag for a corrupt or missing record, an executable/record mismatch, or explicit distrust. The first two are rebuild triggers, so the constructor already detects and fixes them and naming them is circular. For an executable-only model the flag cannot help either, since there is no source to rebuild from. What is left is the honest list: explicit distrust of the artifact, and a change to one of the untracked dependencies — toolchain drift, CmdStan or Stan Math modified in place, headers reached transitively through USER_HEADER, a make/local that includes another makefile. Those are the cases where the assessment is correct that nothing it tracks has changed and wrong about the conclusion, which is precisely when a user needs a way to override it.
The $format(overwrite_file = TRUE) snapshot rule was stated twice about twenty lines apart, from two editing passes. The fuller passage is kept, since it carries why the method survives at all and why reformatting forces a recompile; the second copy is removed. Its line citation was also narrower than the fix, naming only the two assignments and leaving the conditional around them. $exe_file() was settled in the validation table and still described as open in the prose above it. The accessor stays and never errors. A public form of the assessment may still be added for callers wanting a fuller answer, which would not change the accessor's contract — the two were never alternatives. An unprovenanced executable reported nothing at all, which threw away information already in hand. Provenance is unavailable; the reported features are not. The four STAN_* flags and the version from <exe> info are still returned, which is precisely what separating the request from the reported features in section 1 was for.
The passage still argued that content comparison delivers what relative paths were meant to, which stopped being true when dependencies went back to path and content together. The conclusion is unchanged and the reason is simpler: under path identity there is no root, symlink policy or out-of-project rule to define, because a moved project just rebuilds once.
stan_package_model() is called inside the user-facing model function, not from configure — instantiate's own example package wraps it and $sample() together in run_bernoulli_model(). Adoption therefore happens on every fit, so anything printed there reaches every user of every package built that way, on every call, through a function that does not look like it touches a compiler. Unknown provenance is a standing property and stan_build_info() is how to ask about it. States the underlying line once, since this is the third place it decides something: say something when the user asks for what we cannot deliver, say nothing about what merely is. The same rule already governs known_untracked_dependencies and the threading policy. Also records that instantiate's .gitignore template will not pick the record up by accident. It ignores everything under inst/stan, re-includes anything with a dot so .stan files survive, then re-ignores Windows binaries — a rule built on "extensionless means binary", which the record is not. Verified with git check-ignore: the executable is ignored and .bernoulli.cmdstanr.json is not. Updating that template joins the instantiate pull request.
The rule was stated as "say nothing about what merely is", which section 6 contradicts a few hundred lines earlier: known_untracked_dependencies is a standing property and is surfaced at construction, correctly, because a make/local that includes another makefile is something the user neither chose nor can find out any other way. The line that actually separates the cases is what the caller's own action implies. Passing exe_file is itself a statement that cmdstanr did not build this, so reporting back that we do not know how it was built adds nothing — and the threading policy is the same call for the same reason, since STAN_THREADS in make/local is deliberate. An unmet request is different again: nothing the user did implies that their cpp_options cannot be applied to an adopted executable.
The previous justification said a make/local that includes another makefile is something the user did not choose. That is wrong: someone wrote the line, and make/local.example ships it as a suggestion, so it is as deliberate as any other configuration. What separates the cases is whether the consequence follows from the action. Passing exe_file already says cmdstanr did not build this, so answering that we do not know how it was built repeats the caller back to themselves; STAN_THREADS in make/local produces the threaded binary that was asked for. Writing an -include line, by contrast, tells nobody that our staleness detection stops at the first file. That is our limitation, invisible from where the user stands, and no amount of deliberateness on their part reveals it.
instantiate adopts executable-only and forwards include_paths into a branch with no source, which arrives from #1094 by two independent changes: dropping stan_file when compile = FALSE, following the workaround suggested in the cmdstanr issue, and adding an include_paths argument. #1094 is an artifact of deferred compilation. Its fix is precompile_include_paths_, a second include-path variable existing only so a model constructed without compiling can carry paths it has not used yet, with a dirname(stan_file) guess when none were supplied. Removing compile = FALSE removes the reason that variable exists, so the mechanism and the bug class go together rather than the latter needing a further patch. The shape to aim for is therefore stan_file and exe_file together, where include_paths is part of the request and compared against the record, and where $variables() and $check_syntax() come back to a package's models. For a genuine executable-only model include_paths remains inert, which is the case that gets rejected rather than ignored. Records the property the instantiate pull request has to demonstrate: with source registered, validation runs on every fit, and it must never conclude rebuild or a compile lands inside a user-facing function.
It restated every contract, and drifted from them in two consecutive review rounds: claiming only cmdstan_model() builds after compile_stan_file() became a co-equal entry point, and claiming the CmdStan installation path is not compared after it became part of builder identity. Both were caught by a reviewer rather than by us, which is what a second copy of a specification reliably costs. It is now a table of which section answers which question, plus the two facts a reader needs before opening any of them. The argument this document makes for being canonical over the issues applies to it internally too.
Section 2's title and the raw-assignment rejection both named cmdstan_model()
specifically, which left compile_stan_file() outside two contracts it shares the
implementation for. The rejection was the consequential one: as written,
compile_stan_file(cpp_options = list("STAN_THREADS=true")) bypassed it entirely,
so the rule could be avoided by choosing the other entry point.
The scope qualifier was added to protect cmdstan_make_local(), which passes
unnamed entries into a file where += is real. That distinction is between
configuring a build and writing make/local, not between the two functions that
build, and it now says so.
Three summaries still described the mechanism as recognising records from a newer cmdstanr, which was the rule before it became bidirectional: the field list in section 1 defined format_version as the schema's own version, stage 2 tested "forward versions", and section 10 singled out newer records. All three now say a version this cmdstanr does not support, in either direction, and the field list says what the version actually versions — the build interpretation contract, with a bump obliged when option or build semantics change. Also separates two words that were being used interchangeably. Unreadable means the file could not be parsed; unsupported means a version was found that we do not interpret. Keeping them apart matters because a supported record can also be corrupt, and the messages differ.
"Losing a record never breaks anything" and "the model still samples" contradicted section 5, which errors when an already-constructed source-backed model finds its record gone. Only executable-only adoption keeps fitting without one. The three cases are now separate: source-backed construction rebuilds, guarded operations on an existing source-backed object error and point back at cmdstan_model(), and executable-only adoption continues with reduced provenance. What survives from the original claim is narrower and still worth stating — losing a record never corrupts the executable. Also drops the two stray .hpp files from the test-resources observation. They were scratch from a reviewer's stanc probe rather than a standing property of the branch, and the working tree is clean. The four missing executable entries are real and carry the argument for pattern-based ignores on their own.
"Say something when the user asks for what we cannot deliver" was not an implementable contract — it named neither the response nor the conditions. With no stan_file, explicitly supplied cpp_options, stanc_options, include_paths, user_header or force_recompile is now an error. None of them can configure an artifact that will not be rebuilt, and a valid record exists to be inspected rather than overridden. The check is on whether the argument was supplied, not on its value, and force_recompile is why: its default comes from a global option, so a check on the resolved value would error for every adoption performed by anyone who has that option set, including every instantiate fit, from inside a package the user never chose to look at. Explicit argument errors, option-sourced value is ignored, and the option's help page says it has no effect on executable-only models. A NULL sentinel is preferred to missing(), which is fragile through the wrapper layers instantiate reaches us through. Also states that force_recompile never enters the record. It changes whether we build, not what we build, so two identical builds must produce identical records whether or not one was forced. The rebuild reason should name the option when the value came from there, since "force_recompile = TRUE" means nothing to someone who set it in .Rprofile and passed nothing.
Surfacing it at every construction collided with the instantiate finding, where construction happens on every fit — a user with the suggested make/local include pattern would have received the same note forever. Keying on the record being written fixes it without state: it is the moment the information is new, adoption never writes, so the note reaches whoever ran the untrackable build and nobody else. Once per session would also work but needs a cache of what has been said, and section 5 prohibits stored verdicts for reasons that apply to any sibling. Separately, the clone note claimed the two objects cannot diverge, a few lines after acknowledging that a second $initialize() can retarget state. Qualified to "through the supported API", which preserves the decision not to guard it without claiming the unguarded case is impossible.
Purpose and scope still said the installation path is not compared. That was the version-only rule, and it survived the at-a-glance rewrite because it sits in a different section — the same reviewer finding, in a place the restructure did not reach.
A dependency now matches on its content hash wherever it lives. The absolute path it had at build time is recorded as built_from, which is provenance: never compared, never rewritten, and needing no mechanism since records are replaced whole on every build. Moving a project no longer rebuilds it. Path-and-content identity had a real argument behind it. stanc bakes absolute source paths into locations_array__, so two builds at two paths genuinely produce different binaries. It was rejected because the difference is one string in an exception message that still carries the correct line and column, while the everyday case of renaming a working directory pays a full recompile for a benefit its user never receives. The exotic cases do not rescue it: cross-machine scenarios rebuild on builder identity anyway, so path was only ever the sole trigger for a folder rename. include_paths and the user header's path are recorded but likewise not compared as spellings. Comparing them would have reintroduced path sensitivity through the request for every model with an include or a header, defeating the change for the population it serves. An include path's whole effect is the set of files it resolved, and that set is compared by content and re-resolved through stanc --info. instantiate keeps executable-only adoption, but on new grounds. The previous reasoning was that staged installation builds under a 00LOCK path that no longer exists, forcing a rebuild on every first fit; content identity dissolves that. The durable reason is that registering source makes a compile inside a fit representable at all, which is the promise instantiate cannot break. An edited file in the package library or a second .libPaths() entry both reach it. Corrects the account of #1094. It was not a deferred $compile() losing include paths but $sample() reaching $variables() and invoking stanc with none, so include_paths configures source resolution rather than the build. Two traps follow: the dirname(stan_file) default is load-bearing behaviour rather than scaffolding, since stanc fails outright without it and instantiate's install path supplies no include paths at all; and whatever resolves that default must do so before the request is recorded, or include re-resolution reproduces #1094 at validation time. Adds --filename-in-msg to stage 4. Every runtime exception from every cmdstanr model currently names a deleted tempfile, because compilation runs against a copy rather than the source. Also removes every reference to earlier drafts of this document. The note is a specification and should read as current truth; where a rejected alternative carried an argument worth keeping, the argument stays and the history goes.
jgabry
marked this pull request as draft
August 29, 2026 04:53
Round-five removed include_paths from the rebuild triggers but left section 6 re-resolving includes with the paths held in the record. With the comparison gone, that made changing include_paths a no-op: stanc would be pointed at the old directories, find the old files, report matching hashes, and reuse a binary the caller did not ask for. Re-resolution now uses the effective paths of the current call; the recorded ones are for replay and provenance. The user header needs the opposite treatment and now matches on normalised path as well as content. CmdStan feeds it in as -include, and a quoted include inside it resolves relative to that header's own directory, so two byte-identical headers in different directories are different translation units. A worked example is in section 6: the same top-level hash, implementations returning 0.3261 and -1.0000, and every compared field matching. This is not an exception to content identity but the same rule where our information is incomplete -- a Stan program's include closure is fully visible through stanc, a C++ header's is not, so the directory stands in for the part we cannot hash. Section 6 already does this once, for make/local. Both of those were derived wrongly at least once because the record's rules were spread across three sections and never written down as rules. Section 4 now carries a per-field recorded/compared table and is the authority on both; prose elsewhere refers to it rather than restating it. Recorded-but-not-compared turns out to be the ordinary case, not a list of exceptions, and that reframing settles the injected stanc options: cmdstanr's own additions are recorded and never compared, so toggling pedantic does not force a recompile. Checking for rules stated twice immediately found two stale ones in section 3. Includes are compared as an ordered sequence rather than a set. That is exact where a set is not, and it is also the simpler implementation, since position gives the error message its file name without a second pass. Also here: variables() must be captured at construction, since parsing on first call makes the answer depend on whether anyone asked before an edit; the standalone family all take include_paths, without which format_stan_file() could not format any program containing an include; filename-in-msg precedence is settled rather than left open; stan_build_info() reports built_from and whether it still exists, because it receives only an executable and cannot resolve a current location; and the .libPaths() example is gone, since instantiate derives the executable from the source it just found. The 30-90 second compile figure was never measured. Replaced with 6.7s and 13.8s for bernoulli.stan with and without precompiled headers, stated as a floor, and the one argument that leaned on the number re-grounded on the caller not having asked to build at all.
The user header's compared path was framed as an exception to content identity, which left the -I directories a user can put in cpp_options looking like a second exception needing its own rule. They are not. Both are directories that decide how a quoted #include resolves, both are already compared -- the header's path as a dependency, the flags inside request.cpp_options -- and neither has its contents tracked. Stating it once as a general rule removes the question rather than answering it again, and the untracked-content gap reaches its existing message through the #include in the header, not through the flag, so nothing changes about when that fires. Section 9 restated the stage plan that #1258 also holds, and the two had already diverged: four of #1258's stage 4 items were absent here. Say plainly that #1258 is the work list and this section is only the reasoning behind the order, then drop the enumerations that duplicated it. Also fixes two positional references that had gone stale or wrong -- the rebuild triggers are now grouped the way #1255 groups them instead of being counted -- and replaces a fourth surviving copy of the unmeasured 30-90 second compile figure with a pointer to the measured one.
The pure decision function was the first half of stage 4, so section 6's contract stayed prose until the riskiest change in the plan was ready to ship. It depends only on stage 2's fixtures, compiles nothing and is called by nothing, so it separates cleanly and can be worked alongside stage 3. Separating it is what makes this section checkable. A rebuild trigger written as a decision-table row with a fixture fails loudly when it contradicts another rule; the same two rules written as paragraphs eight lines apart survived a full review round, which is how re-resolution came to use the recorded include paths while the section above said the spelling was not compared. The argument that the API change and the decision engine ship together is unaffected. It is about the engine being live while $compile() is gone, and an unwired function changes nothing a user can observe.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
dev-notes/compilation-state.md. No code changes.This is the current plan for v1.0, developed based on conversation with @SteveBronder and @WardBrian.
AI disclosure: the compilation-state.md file is being written by Claude and Codex (with plenty of em dashes and Claude-isms) based on the list below, which is my own summary of the planned changes. This is somewhat of an experiment, and if it goes poorly I may end up writing the document myself from scratch.
New API
When do we recompile
If the user sets force_recompile = TRUE or when any of these change:
Or when we can't trust what we recorded:
If more than one of these applies we report all of them.
One exception:
The new dependency file
The current plan is a file
.<model-name>.cmdstanr.jsonthat is written next to the executable. It contains:Tracking issue: #1258