From 2673d6f3c2af5745d4f8f5cb5de15232af04cb06 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Tue, 26 May 2026 21:07:24 +0200 Subject: [PATCH 01/51] Add LLM coding-agent REPL and goal-printing flags Introduce an interactive REPL for LLM coding agents driving EasyCrypt (`easycrypt llm`) using a line-oriented protocol over stdin/stdout, plus two CLI flags for goal inspection: - `-upto ` compile up to a given position and print the goals - `-lastgoals` print the last unproven goals at end-of-file REPL protocol (see `doc/llm/CLAUDE.md` for the full guide): - LOAD "file.ec" [LINE[:COL]] -- compile, optionally up to a position - UNDO / REVERT -- navigate proof state - GOALS / GOALS ALL -- inspect current or all subgoals - CHECKPOINT -- named bookmarks for branching - SEARCH -- lemma search - QUIET ON/OFF -- suppress goal display for bulk input - Direct EasyCrypt input (tactics, declarations, search, print, ...) Responses use a typed envelope (OK/ERROR with uuid) terminated by an `` sentinel for reliable parsing. LOAD reports the last processed line in the response tag; error messages include the offending source text; only the current subgoal is shown by default with a remaining count. --- doc/llm/CLAUDE.md | 272 +++++++++++++++++++++---- src/ec.ml | 493 +++++++++++++++++++++++++++++++++++++++++++--- src/ecOptions.ml | 48 ++--- src/ecOptions.mli | 4 +- 4 files changed, 708 insertions(+), 109 deletions(-) diff --git a/doc/llm/CLAUDE.md b/doc/llm/CLAUDE.md index 0cc20c5a3..c33dc1825 100644 --- a/doc/llm/CLAUDE.md +++ b/doc/llm/CLAUDE.md @@ -6,59 +6,246 @@ computations, program logics (Hoare logic, probabilistic Hoare logic, probabilistic relational Hoare logic), and ambient mathematical reasoning. -## Using the `llm` command +## Using the `llm` interactive mode -The `llm` subcommand is designed for non-interactive, LLM-friendly -batch compilation. It produces no progress bar and no `.eco` cache -files. +The `llm` subcommand provides an interactive REPL with a +machine-friendly protocol designed for LLM agents. The LLM sends +commands over stdin and receives structured responses on stdout. ``` -easycrypt llm [OPTIONS] FILE.ec +easycrypt llm [OPTIONS] ``` -### Options +Standard loader and prover options (`-I`, `-timeout`, `-p`, etc.) are +available. Use `-help` to print this guide and exit: -- `-upto LINE` or `-upto LINE:COL` — Compile up to (but not - including) the given location, then print the current goal state to - stdout and exit with code 0. Use this to inspect the proof state at - a specific point in a file. +``` +easycrypt llm -help +``` -- `-lastgoals` — On failure, print the goal state (as it was just - before the failing command) to stdout, then print the error to - stderr, and exit with code 1. Use this to understand what the - failing tactic was supposed to prove. +### Protocol -Standard loader and prover options (`-I`, `-timeout`, `-p`, etc.) are -also available. +**Startup.** EasyCrypt prints a `READY` message and waits for input: + +``` +READY [uuid:0] + +``` + +**Responses.** Every response has a typed envelope and an `` +sentinel: + +``` +OK [uuid:N] + + +``` + +``` +ERROR [uuid:N] + + +``` + +The `uuid` is a monotonically increasing integer identifying the proof +engine state. It increments with each successful command. + +### Meta-commands -### Output conventions +These are protocol-level commands, not EasyCrypt syntax: -- **Goals** are printed to **stdout**. -- **Errors** are printed to **stderr**. -- **Exit code 0** means success (or `-upto` reached its target). -- **Exit code 1** means a command failed. -- If there is no active proof at the point where goals are requested, - stdout will contain: `No active proof.` +| Command | Description | +|---------|-------------| +| `LOAD "file.ec" [LINE[:COL]] [-nosmt]` | Reset state, compile file (optionally skip SMT) | +| `UNDO` | Undo the last proof step | +| `REVERT ` | Revert to a specific state (by uuid or checkpoint name) | +| `GOALS` | Print the current goal (first subgoal only, with remaining count) | +| `GOALS ALL` | Print all subgoals | +| `CHECKPOINT ` | Save current uuid under a name for later `REVERT` | +| `SEARCH ` | Search for lemmas matching a pattern | +| `QUIET ON` / `QUIET OFF` | Suppress/enable automatic goal display after tactics | +| `` / `` | Delimit multi-line EasyCrypt input | +| `HELP` | Print this guide | +| `QUIT` | Exit | -### Workflow for writing and debugging proofs +### EasyCrypt commands -1. Try to write a pen-and-paper proof first. +Any line that is not a meta-command is parsed as EasyCrypt input. +This covers tactics, declarations, `search`, `print`, `require`, +etc. The line must be a complete EasyCrypt statement ending with `.` -2. Write the `.ec` file with your proof attempt. For a large proof, - write down skeleton and `admit` subgoals first, and then detail - the proof. +``` +smt(). +rewrite H1 H2. +search (%/). +print mulzK. +``` + +For multi-line statements, wrap with `` and ``: + +``` + +lemma test : + 0 <= n => + 0 < n + 1. + +``` + +### Workflow + +**1. Load a file up to the proof point:** -3. Run `easycrypt llm -lastgoals FILE.ec` to check the full file. - - If it succeeds (exit 0), you are done. - - If it fails (exit 1), read the error from stderr and the goal - state from stdout to understand what went wrong. +``` +LOAD "myfile.ec" 42 +``` + +This compiles the file through line 42 (processing any command whose +end is on or before that line). The response includes where it +stopped: + +``` +OK [uuid:15] [loaded:myfile.ec:42] +Current goal +... + +``` -4. Use `-upto LINE` to inspect the proof state at a specific point - without running the rest of the file. This is useful for - incremental proof development. +For large files, use `-nosmt` to skip SMT calls during prefix +compilation (safe when the prefix was already verified): -5. Fix the proof and repeat from step 2. The ultimate proof should - not contain `admit` or `admitted`. +``` +LOAD "myfile.ec" 436 -nosmt +``` + +**2. Try tactics, using REVERT to restart:** + +The uuid returned by LOAD is a revertible state. Use `REVERT` to +return to it after failed experiments — this is instant, unlike +re-doing LOAD which recompiles the prefix. + +``` +LOAD "myfile.ec" 42 +→ OK [uuid:15] [loaded:myfile.ec:42] + +smt(). ← fails, state unchanged +rewrite H1. smt(). ← succeeds (uuid:17) +rewrite H2. ← wrong direction +REVERT 17 ← back to after the successful smt() +``` + +To restart the proof from scratch, revert to the LOAD uuid: + +``` +REVERT 15 ← back to the state right after LOAD +``` + +Always note the LOAD uuid so you can return to it. + +**3. Use checkpoints for branching exploration:** + +``` +CHECKPOINT before_split +split. +smt(). ← fails +REVERT before_split +apply H. ← try a different approach +``` + +**4. Use QUIET mode to save tokens during bulk tactic application:** + +``` +QUIET ON +rewrite H1. +rewrite H2. +rewrite H3. +QUIET OFF +GOALS +``` + +**5. Search for lemmas using patterns:** + +EasyCrypt `search` uses pattern syntax, not keywords. Use `_` as +wildcard: + +``` +search (fdom _). ← lemmas involving fdom +search (_ %/ _). ← integer division lemmas +search (card (_ `|` _)). ← card of union +search (mu _ _) (_ <= _). ← mu lemmas with inequalities +``` + +The SEARCH meta-command is a shorthand that adds `search`/`.`: + +``` +SEARCH (fdom _) +SEARCH (_ %/ _) +``` + +## EasyCrypt proof strategy + +### General approach + +- Start with a pen-and-paper proof plan before writing tactics. +- Use `smt()` aggressively. Try it first — if it fails, add hints: + `smt(lemma1 lemma2)`. +- Build proofs with `have` assertions. Establish intermediate facts + as named hypotheses, then combine with `smt()`. Avoid long rewrite + chains. +- Case split early: `case (n = 0) => [->|hn0].` Base cases often + close by computation. +- Provide specific instances of lemmas to smt: + `have h := lemma arg1 arg2.` SMT works much better with ground + instances than with universally quantified axioms. + +### Integer division (`%/`) + +- `divzK`: `d %| m => m %/ d * d = m` — recovering from exact + division +- `mulzK`: `d <> 0 => m * d %/ d = m` — canceling a known factor +- `divzMpl`: `0 < p => p * m %/ (p * d) = m %/ d` — simplifying + common factors +- To prove `a %/ d = x`, establish `a = x * d` (with `d %| a`), + then use `mulzK`. +- Don't try to rewrite inside `%/` expressions directly. Instead, + prove the equality as a `have` and use it. + +### What works, what doesn't + +- `ring` solves polynomial equalities over integers but treats + abstract ops (like `fact`) as opaque. It **cannot** simplify + `fact(n-1+1)` to `fact(n)`. +- `smt()` can do linear arithmetic and combine hypotheses, but + struggles with nonlinear integer division. Pre-compute key facts + with `have` and `divzK`/`mulzK`, then let smt combine them. +- `rewrite {k}h` rewrites the k-th occurrence only. Essential when a + term appears on both sides of an equation. +- For induction on naturals: `elim/natind: n` gives base (`n ≤ 0`) + and step (`0 ≤ n → P n → P (n+1)`). + +### SMT usage + +`smt()` and `/#` are equivalent — both call external SMT solvers. + +- Use `smt()` **only** on goals that are pure arithmetic or pure + propositional logic. If the goal contains abstract operators, + FMap terms, or `if-then-else`, reduce it first with `rewrite`, + `case`, or `have` before calling `smt()`. +- If `smt()` takes more than 1 second, the goal is too complex. + Simplify with interactive tactics instead of increasing the + timeout. + +### Common pitfalls + +- `rewrite (factS n) //` generates a side goal `0 <= n`. Use + `first smt()` or provide the precondition explicitly. +- `by` closes **all** remaining subgoals. If it fails, the error + refers to the first unclosed goal, which may not be the intended + one. +- When a tactic generates multiple subgoals, each subgoal must be + closed in order. Use `GOALS ALL` to see them all. +- `rewrite lemma in H` modifies hypothesis `H` in place (it does + not consume it). If you need to preserve the original, copy it + first: `have H' := H; rewrite lemma in H'`. ## EasyCrypt language overview @@ -91,8 +278,6 @@ proof. by ring. qed. ### Common tactics - - - `trivial` — solve trivial goals - `smt` / `smt(lemmas...)` — call SMT solvers, optionally with hints - `auto` — automatic reasoning @@ -141,9 +326,10 @@ proof. by ring. qed. ### Guidelines -* Use SMT solver only in direct mode (smt() or /#) on simple goals (arithmetic goals, pure logical goals). +* Use SMT solver only in direct mode (smt() or /#) on simple goals + (arithmetic goals, pure logical goals). * Refrain from unfolding operator definitions unless necessary. - If you need more properties on an operator, state this property in a dedicated lemma, - but avoid unfolding definitions in higher level proofs. - + If you need more properties on an operator, state this property + in a dedicated lemma, but avoid unfolding definitions in higher + level proofs. diff --git a/src/ec.ml b/src/ec.ml index f3bc3467b..bfb802faf 100644 --- a/src/ec.ml +++ b/src/ec.ml @@ -438,6 +438,469 @@ let main () = (* Register user messages printers *) begin let open EcUserMessages in register () end; + (* -------------------------------------------------------------------- *) + (* LLM interactive mode *) + (* -------------------------------------------------------------------- *) + + let llm_guide_path () = + let (module Sites) = EcRelocate.sites in + match EcRelocate.sourceroot with + | Some root -> + Filename.concat (Filename.concat root "doc/llm") "CLAUDE.md" + | None -> + Filename.concat Sites.doc "llm-guide.md" + in + + let print_llm_guide () = + let path = llm_guide_path () in + try + let ic = open_in path in + begin try while true do + print_char (input_char ic) + done with End_of_file -> () end; + close_in ic + with Sys_error e -> + Printf.eprintf "cannot read LLM guide: %s\n%!" e + in + + let run_llm_repl (llmopts : llm_option) = + if llmopts.llmo_help then begin + print_llm_guide (); + exit 0 + end; + + let prvopts = llmopts.llmo_provers in + (* Initialize PRNG *) + Random.self_init (); + + (* Connect to external Why3 server if requested *) + prvopts.prvo_why3server |> oiter (fun server -> + try + Why3.Prove_client.connect_external server + with Why3.Prove_client.ConnectionError e -> + Format.eprintf + "cannot connect to Why3 server `%s': %s" server e; + exit 1); + + (* Add current directory to load path *) + (match relocdir with + | None -> EcCommands.addidir Filename.current_dir_name + | Some pwd -> EcCommands.addidir pwd); + + (* Proof engine configuration *) + let checkmode = { + EcCommands.cm_checkall = prvopts.prvo_checkall; + EcCommands.cm_timeout = odfl 3 prvopts.prvo_timeout; + EcCommands.cm_cpufactor = odfl 1 prvopts.prvo_cpufactor; + EcCommands.cm_nprovers = odfl 4 prvopts.prvo_maxjobs; + EcCommands.cm_provers = prvopts.prvo_provers; + EcCommands.cm_quorum = prvopts.prvo_quorum; + EcCommands.cm_profile = prvopts.prvo_profile; + } in + + (* Notice buffer: collects messages during command processing *) + let notices = Buffer.create 256 in + + let notifier (_ : EcGState.loglevel) (lazy msg) = + Buffer.add_string notices msg; + Buffer.add_char notices '\n' + in + + let initialized = ref false in + + let do_initialize () = + EcCommands.initialize + ~restart:!initialized ~undo:true + ~boot:ldropts.ldro_boot ~checkmode ~checkproof:true; + initialized := true; + (try + List.iter EcCommands.apply_pragma prvopts.prvo_pragmas + with EcCommands.InvalidPragma x -> + EcScope.hierror "invalid pragma: `%s'\n%!" x); + EcCommands.addnotifier notifier; + oiter (fun ppwidth -> + let gs = EcEnv.gstate (EcScope.env (EcCommands.current ())) in + EcGState.setvalue "PP:width" (`Int ppwidth) gs) + prvopts.prvo_ppwidth + in + + (* Error formatting *) + let format_error ?(src="") e = + let base = match e with + | EcScope.TopError (loc, e) -> + let msg = String.strip (EcPException.tostring e) in + if loc = EcLocation._dummy then msg + else Format.asprintf "%s: %s" (EcLocation.tostring loc) msg + | e -> + String.strip (EcPException.tostring e) + in + if src = "" then base + else Printf.sprintf "%s\nsource: %s" base src + in + + (* Output helpers *) + let goals_to_string ?(all=false) () = + let buf = Buffer.create 256 in + let fmt = Format.formatter_of_buffer buf in + EcCommands.pp_current_goal_or_noproof ~all fmt; + Format.pp_print_flush fmt (); + Buffer.contents buf + in + + let quiet = ref false in + + let checkpoints : (string, int) Hashtbl.t = Hashtbl.create 16 in + + let reply_ok ?(tag="") body = + let n = Buffer.contents notices in + Printf.printf "OK [uuid:%d]%s\n" (EcCommands.uuid ()) tag; + if n <> "" then print_string n; + if body <> "" then begin + print_string body; + let len = String.length body in + if len > 0 && body.[len - 1] <> '\n' then + print_char '\n' + end; + Printf.printf "\n%!"; + Buffer.clear notices + in + + let reply_ok_goals ?(all=false) () = + if !quiet then reply_ok "" + else reply_ok (goals_to_string ~all ()) + in + + let reply_error msg = + let goals = goals_to_string () in + Printf.printf "ERROR [uuid:%d]\n%s\n" (EcCommands.uuid ()) msg; + if goals <> "" then begin + print_string goals; + let len = String.length goals in + if len > 0 && goals.[len - 1] <> '\n' then + print_char '\n' + end; + Printf.printf "\n%!"; + Buffer.clear notices + in + + (* Process a single EasyCrypt command, respecting gl_fail *) + let process_action ~src (p : EP.global) = + let loc = p.EP.gl_action.EcLocation.pl_loc in + let succeeded = ref false in + begin try + ignore (EcCommands.process ~src p.EP.gl_action : float option); + succeeded := true + with + | EcCommands.Restart -> raise EcCommands.Restart + | _ when p.EP.gl_fail -> () + | e -> raise (EcScope.toperror_of_exn ~gloc:loc e) + end; + if !succeeded && p.EP.gl_fail then + raise (EcScope.toperror_of_exn ~gloc:loc + (EcScope.HiScopeError (None, + "this command is expected to fail"))) + in + + (* Process EasyCrypt input from a string (one parsed program) *) + let process_ec_input input = + Buffer.clear notices; + let reader = EcIo.from_string input in + let last_src = ref "" in + begin try + let (src, prog) = EcIo.xparse reader in + let src = String.strip src in + last_src := src; + begin match EcLocation.unloc prog with + | EP.P_Prog (commands, _) -> + List.iter (process_action ~src) commands; + reply_ok_goals () + | EP.P_Undo i -> + EcCommands.undo i; + reply_ok_goals () + | EP.P_Exit -> + EcIo.finalize reader; exit 0 + | EP.P_DocComment doc -> + EcCommands.doc_comment doc; + reply_ok "" + end + with + | EcCommands.Restart -> + do_initialize (); + reply_ok "Session restarted" + | e -> + reply_error (format_error ~src:!last_src e) + end; + EcIo.finalize reader + in + + (* Handle LOAD "file.ec" [LINE[:COL]] *) + let handle_load args = + Buffer.clear notices; + let args = String.strip args in + let last_src = ref "" in + + try + (* Parse quoted or unquoted filename *) + let filename, rest = + if String.length args > 0 && args.[0] = '"' then + let close = + try String.index_from args 1 '"' + with Not_found -> + failwith "LOAD: unterminated filename" + in + let fn = String.sub args 1 (close - 1) in + let rest = String.strip ( + String.sub args (close + 1) + (String.length args - close - 1)) in + (fn, rest) + else + match String.split_on_char ' ' args with + | [] -> failwith "LOAD: missing filename" + | [f] -> (f, "") + | f :: rest -> (f, String.concat " " rest) + in + + (* Parse optional LINE[:COL] and flags (-nosmt) *) + let upto, nosmt = + if rest = "" then (None, false) + else + let words = String.split_on_char ' ' rest in + let words = List.filter (fun s -> s <> "") words in + let nosmt = List.mem "-nosmt" words in + let words = List.filter (fun s -> s <> "-nosmt") words in + let upto = match words with + | [] -> None + | [w] -> + begin match String.split_on_char ':' w with + | [line] -> + Some (int_of_string line, None) + | [line; col] -> + Some (int_of_string line, Some (int_of_string col)) + | _ -> failwith "LOAD: invalid LINE[:COL] format" + end + | _ -> failwith "LOAD: unexpected arguments" + in + (upto, nosmt) + in + + (* Validate file extension *) + begin try + ignore (EcLoader.getkind + (Filename.extension filename) : EcLoader.kind) + with EcLoader.BadExtension ext -> + failwith (Format.sprintf + "unknown file extension: %s" ext) + end; + + (* Reset proof engine and process file *) + do_initialize (); + Hashtbl.clear checkpoints; + EcCommands.addidir (Filename.dirname filename); + + let reader = EcIo.from_file filename in + + let past_upto (loc : EcLocation.t) = + match upto with + | None -> false + | Some (line, col) -> + let (el, ec) = loc.loc_end in + el > line || (el = line && match col with + | None -> false + | Some c -> ec > c) + in + + let last_loc = ref None in + + (* In -nosmt mode, admit all SMT calls during prefix loading *) + if nosmt then EcCommands.pragma_check `WeakCheck; + + begin try while true do + let (src, prog) = EcIo.xparse reader in + let src = String.strip src in + last_src := src; + match EcLocation.unloc prog with + | EP.P_Prog (commands, locterm) -> + List.iter (fun p -> + let loc = p.EP.gl_action.EcLocation.pl_loc in + if past_upto loc then raise Exit; + process_action ~src p; + last_loc := Some loc + ) commands; + if locterm then raise Exit + | EP.P_Undo i -> + EcCommands.undo i + | EP.P_Exit -> + raise Exit + | EP.P_DocComment doc -> + EcCommands.doc_comment doc + done with + | Exit | End_of_file -> () + | e -> + EcIo.finalize reader; + if nosmt then EcCommands.pragma_check `Check; + raise e + end; + + EcIo.finalize reader; + + (* Restore full SMT checking for interactive tactics *) + if nosmt then EcCommands.pragma_check `Check; + + let tag = + match !last_loc with + | None -> "" + | Some loc -> + let (el, _) = loc.EcLocation.loc_end in + Printf.sprintf " [loaded:%s:%d]" filename el + in + reply_ok ~tag (goals_to_string ()) + + with + | EcCommands.Restart -> + do_initialize (); + Hashtbl.clear checkpoints; + reply_ok "Session restarted" + | Failure s -> + reply_error s + | e -> + reply_error (format_error ~src:!last_src e) + in + + (* Initialize proof engine *) + do_initialize (); + + (* Signal ready *) + Printf.printf "READY [uuid:%d]\n\n%!" + (EcCommands.uuid ()); + + (* Main REPL loop *) + let multi_buf = Buffer.create 256 in + let in_multi = ref false in + + begin try while true do + let line = input_line stdin in + let line = String.strip line in + + (* Multi-line input: starts, flushes *) + if line = "" then begin + Buffer.clear multi_buf; + in_multi := true + end + else if line = "" && !in_multi then begin + let input = Buffer.contents multi_buf in + Buffer.clear multi_buf; + in_multi := false; + if input <> "" then process_ec_input input + end + else if !in_multi then begin + if Buffer.length multi_buf > 0 then + Buffer.add_char multi_buf ' '; + Buffer.add_string multi_buf line + end + + else if line = "" then + () + else if line = "QUIT" then + exit 0 + else if line = "HELP" then begin + Buffer.clear notices; + let buf = Buffer.create 4096 in + let path = llm_guide_path () in + begin try + let ic = open_in path in + begin try while true do + Buffer.add_char buf (input_char ic) + done with End_of_file -> () end; + close_in ic; + reply_ok (Buffer.contents buf) + with Sys_error e -> + reply_error (Printf.sprintf "cannot read guide: %s" e) + end + end + else if line = "UNDO" then begin + Buffer.clear notices; + let uuid = EcCommands.uuid () in + if uuid > 0 then begin + EcCommands.undo (uuid - 1); + reply_ok_goals () + end else + reply_error "nothing to undo" + end + else if line = "GOALS ALL" then begin + Buffer.clear notices; + reply_ok (goals_to_string ~all:true ()) + end + else if line = "GOALS" then begin + Buffer.clear notices; + reply_ok (goals_to_string ()) + end + else if String.starts_with line "CHECKPOINT " then begin + Buffer.clear notices; + let name = String.strip ( + String.sub line 11 (String.length line - 11)) in + if name = "" then + reply_error "CHECKPOINT: missing name" + else begin + Hashtbl.replace checkpoints name (EcCommands.uuid ()); + reply_ok (Printf.sprintf + "checkpoint '%s' set at uuid %d" name (EcCommands.uuid ())) + end + end + else if String.starts_with line "REVERT " then begin + Buffer.clear notices; + let n = String.strip ( + String.sub line 7 (String.length line - 7)) in + let target = + try Some (int_of_string n) + with Failure _ -> Hashtbl.find_opt checkpoints n + in + begin match target with + | None -> + reply_error (Printf.sprintf + "REVERT: '%s' is not a valid uuid or checkpoint name" n) + | Some target -> + let uuid = EcCommands.uuid () in + if target < 0 || target > uuid then + reply_error (Printf.sprintf + "REVERT: uuid %d out of range [0, %d]" target uuid) + else begin + EcCommands.undo target; + reply_ok_goals () + end + end + end + else if line = "QUIET ON" then begin + Buffer.clear notices; + quiet := true; + reply_ok "" + end + else if line = "QUIET OFF" then begin + Buffer.clear notices; + quiet := false; + reply_ok "" + end + else if String.starts_with line "SEARCH " then begin + let query = String.strip ( + String.sub line 7 (String.length line - 7)) in + let query = + if String.ends_with query "." + then String.sub query 0 (String.length query - 1) + else query + in + process_ec_input (Printf.sprintf "search %s." query) + end + else if String.starts_with line "LOAD " then + handle_load (String.sub line 5 (String.length line - 5)) + else + (* Treat as EasyCrypt input *) + process_ec_input line + done with + | End_of_file -> () + end; + + exit 0 + in + (* Initialize I/O + interaction module *) let module State = struct type t = { @@ -569,34 +1032,8 @@ let main () = end - | `Llm llmopts -> begin - let name = llmopts.llmo_input in - - begin try - let ext = Filename.extension name in - ignore (EcLoader.getkind ext : EcLoader.kind) - with EcLoader.BadExtension ext -> - Format.eprintf "do not know what to do with %s@." ext; - exit 1 - end; - - let lastgoals = llmopts.llmo_lastgoals in - let terminal = - lazy (T.from_channel ~name ~progress:`Silent ~lastgoals (open_in name)) - in - - { prvopts = llmopts.llmo_provers - ; input = Some name - ; terminal = terminal - ; interactive = false - ; eco = true - ; gccompact = None - ; docgen = false - ; outdirp = None - ; upto = llmopts.llmo_upto - ; trace = None } - - end + | `Llm llmopts -> + run_llm_repl llmopts | `Runtest _ -> (* Eagerly executed *) diff --git a/src/ecOptions.ml b/src/ecOptions.ml index a78822aaf..ba47648ae 100644 --- a/src/ecOptions.ml +++ b/src/ecOptions.ml @@ -49,10 +49,8 @@ and doc_option = { } and llm_option = { - llmo_input : string; llmo_provers : prv_options; - llmo_lastgoals : bool; - llmo_upto : (int * int option) option; + llmo_help : bool; } and prv_options = { @@ -381,11 +379,10 @@ let specs = { `Spec ("trace" , `Flag , "Save all goals & messages in .eco"); `Spec ("compact", `Int , "")]); - ("llm", "LLM-friendly batch compilation", [ + ("llm", "LLM-friendly interactive mode", [ `Group "loader"; `Group "provers"; - `Spec ("lastgoals" , `Flag , "Print last unproved goals on failure"); - `Spec ("upto" , `String, "Compile up to LINE or LINE:COL and print goals")]); + `Spec ("help", `Flag, "Print the LLM agent guide and exit")]); ("cli", "Run EasyCrypt top-level", [ `Group "loader"; @@ -574,26 +571,9 @@ let doc_options_of_values values input = { doco_input = input; doco_outdirp = get_string "outdir" values; } -let parse_upto values = - get_string "upto" values |> Option.map (fun s -> - let invalid () = - raise (Arg.Bad (Printf.sprintf - "invalid -upto format: expected LINE or LINE:COL, got %S" s)) in - match String.split_on_char ':' s with - | [line] -> - let line = try int_of_string line with Failure _ -> invalid () in - (line, None) - | [line; col] -> - let line = try int_of_string line with Failure _ -> invalid () in - let col = try int_of_string col with Failure _ -> invalid () in - (line, Some col) - | _ -> invalid ()) - -let llm_options_of_values ini values input = - { llmo_input = input; - llmo_provers = prv_options_of_values ini values; - llmo_lastgoals = get_flag "lastgoals" values; - llmo_upto = parse_upto values; } +let llm_options_of_values ini values = + { llmo_provers = prv_options_of_values ini values; + llmo_help = get_flag "help" values; } (* -------------------------------------------------------------------- *) let parse getini argv = @@ -666,16 +646,14 @@ let parse getini argv = raise (Arg.Bad "this command takes a single input file as argument") end - | "llm" -> begin - match anons with - | [input] -> - let ini = getini (Some input) in - let cmd = `Llm (llm_options_of_values ini values input) in - (cmd, ini, true) + | "llm" -> + if not (List.is_empty anons) then + raise (Arg.Bad "this command does not take arguments"); - | _ -> - raise (Arg.Bad "this command takes a single argument") - end + let ini = getini None in + let cmd = `Llm (llm_options_of_values ini values) in + + (cmd, ini, true) | _ -> assert false diff --git a/src/ecOptions.mli b/src/ecOptions.mli index 0fb1fc3c2..e5c4b5f04 100644 --- a/src/ecOptions.mli +++ b/src/ecOptions.mli @@ -45,10 +45,8 @@ and doc_option = { } and llm_option = { - llmo_input : string; llmo_provers : prv_options; - llmo_lastgoals : bool; - llmo_upto : (int * int option) option; + llmo_help : bool; } and prv_options = { From 97ef14ea5c1f63fcba76872902d373663a01b399 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Wed, 27 May 2026 05:01:33 +0200 Subject: [PATCH 02/51] [llm] add LOAD ... -trace for before/after sentence inspection Add a -trace flag to the LOAD REPL command. When set, LOAD compiles the prefix exactly as today (using the existing LINE[:COL] argument, or up to EOF if omitted), but defers the last sentence and runs it under goal capture, then returns a response body with four delimited blocks: === BEFORE: line L (col C) === === TACTIC (lines L:C - L':C') === === AFTER: line L (col C) === === SUMMARY === open goals: N1 -> N2 Adapted from PR #1018 (-trace LINE[:COL] for batch mode): same delimiters and the same new-or-modified-head filtering for AFTER. The position is taken from LOAD's existing LINE[:COL] argument; the tag is the regular [loaded:file:LINE]. If the deferred sentence is outside a proof context, or there is no sentence to trace, the reply uses the ERROR envelope with a clear message. If the sentence fails, the BEFORE/TACTIC blocks are still delivered, AFTER carries a marker, and the formatted exception is appended. Expose EcCommands.in_proof so the REPL can check the pre-execution proof status without inspecting scope internals. --- doc/llm/CLAUDE.md | 23 +++++- src/ec.ml | 188 ++++++++++++++++++++++++++++++++++++++------- src/ecCommands.ml | 5 +- src/ecCommands.mli | 1 + 4 files changed, 186 insertions(+), 31 deletions(-) diff --git a/doc/llm/CLAUDE.md b/doc/llm/CLAUDE.md index c33dc1825..7918f21fe 100644 --- a/doc/llm/CLAUDE.md +++ b/doc/llm/CLAUDE.md @@ -56,7 +56,7 @@ These are protocol-level commands, not EasyCrypt syntax: | Command | Description | |---------|-------------| -| `LOAD "file.ec" [LINE[:COL]] [-nosmt]` | Reset state, compile file (optionally skip SMT) | +| `LOAD "file.ec" [LINE[:COL]] [-nosmt] [-trace]` | Reset state, compile file (optionally skip SMT or trace last sentence) | | `UNDO` | Undo the last proof step | | `REVERT ` | Revert to a specific state (by uuid or checkpoint name) | | `GOALS` | Print the current goal (first subgoal only, with remaining count) | @@ -117,6 +117,27 @@ compilation (safe when the prefix was already verified): LOAD "myfile.ec" 436 -nosmt ``` +Add `-trace` to a LOAD to inspect the proof state around the last +loaded sentence. The reply body contains four delimited blocks: + +``` +LOAD "myfile.ec" 42 -trace + +=== BEFORE: line 42 (col 0) === + +=== TACTIC (lines 42:0 - 42:10) === + +=== AFTER: line 42 (col 0) === + +=== SUMMARY === +open goals: N1 -> N2 +``` + +The position comes from the existing `LINE[:COL]` argument; omit it to +trace the file's last sentence. On tactic failure the reply uses the +`ERROR` envelope and still includes the BEFORE/TACTIC blocks plus an +`` marker in the AFTER block. + **2. Try tactics, using REVERT to restart:** The uuid returned by LOAD is a revertible state. Use `REVERT` to diff --git a/src/ec.ml b/src/ec.ml index bfb802faf..2d54b09b0 100644 --- a/src/ec.ml +++ b/src/ec.ml @@ -638,6 +638,8 @@ let main () = Buffer.clear notices; let args = String.strip args in let last_src = ref "" in + let trace_prefix = ref "" in + let exception Trace_failed of exn in try (* Parse quoted or unquoted filename *) @@ -660,27 +662,32 @@ let main () = | f :: rest -> (f, String.concat " " rest) in - (* Parse optional LINE[:COL] and flags (-nosmt) *) - let upto, nosmt = - if rest = "" then (None, false) - else - let words = String.split_on_char ' ' rest in - let words = List.filter (fun s -> s <> "") words in - let nosmt = List.mem "-nosmt" words in - let words = List.filter (fun s -> s <> "-nosmt") words in - let upto = match words with - | [] -> None - | [w] -> - begin match String.split_on_char ':' w with - | [line] -> - Some (int_of_string line, None) - | [line; col] -> - Some (int_of_string line, Some (int_of_string col)) - | _ -> failwith "LOAD: invalid LINE[:COL] format" - end - | _ -> failwith "LOAD: unexpected arguments" - in - (upto, nosmt) + (* Parse optional LINE[:COL] and flags (-nosmt, -trace) *) + let upto, nosmt, trace = + let words = + String.split_on_char ' ' rest + |> List.filter (fun s -> s <> "") + in + let nosmt = List.mem "-nosmt" words in + let trace = List.mem "-trace" words in + let words = + List.filter + (fun s -> s <> "-nosmt" && s <> "-trace") + words + in + let upto = match words with + | [] -> None + | [w] -> + begin match String.split_on_char ':' w with + | [line] -> + Some (int_of_string line, None) + | [line; col] -> + Some (int_of_string line, Some (int_of_string col)) + | _ -> failwith "LOAD: invalid LINE[:COL] format" + end + | _ -> failwith "LOAD: unexpected arguments" + in + (upto, nosmt, trace) in (* Validate file extension *) @@ -711,27 +718,65 @@ let main () = let last_loc = ref None in + (* For -trace: lazy whole-file bytes, used to slice the exact + source text of a sentence by byte offsets. *) + let input_bytes = lazy ( + let ic = open_in_bin filename in + let n = in_channel_length ic in + let b = Bytes.create n in + really_input ic b 0 n; + close_in ic; + Bytes.unsafe_to_string b) + in + let sentence_source (loc : EcLocation.t) = + let s = Lazy.force input_bytes in + let lo = max 0 loc.EcLocation.loc_bchar in + let hi = min (String.length s) loc.EcLocation.loc_echar in + if hi <= lo then "" else String.sub s lo (hi - lo) + in + + (* For -trace: defer execution of the last sentence within the + prefix so we can capture goals before and after it. *) + let pending : (string * EP.global) option ref = ref None in + let flush_pending () = + match !pending with + | None -> () + | Some (src, p) -> + last_src := src; + process_action ~src p; + last_loc := Some p.EP.gl_action.EcLocation.pl_loc; + pending := None + in + let step src p = + let loc = p.EP.gl_action.EcLocation.pl_loc in + if past_upto loc then raise Exit; + if trace then begin + flush_pending (); + pending := Some (src, p) + end else begin + last_src := src; + process_action ~src p; + last_loc := Some loc + end + in + (* In -nosmt mode, admit all SMT calls during prefix loading *) if nosmt then EcCommands.pragma_check `WeakCheck; begin try while true do let (src, prog) = EcIo.xparse reader in let src = String.strip src in - last_src := src; match EcLocation.unloc prog with | EP.P_Prog (commands, locterm) -> - List.iter (fun p -> - let loc = p.EP.gl_action.EcLocation.pl_loc in - if past_upto loc then raise Exit; - process_action ~src p; - last_loc := Some loc - ) commands; + List.iter (step src) commands; if locterm then raise Exit | EP.P_Undo i -> + last_src := src; EcCommands.undo i | EP.P_Exit -> raise Exit | EP.P_DocComment doc -> + last_src := src; EcCommands.doc_comment doc done with | Exit | End_of_file -> () @@ -746,6 +791,88 @@ let main () = (* Restore full SMT checking for interactive tactics *) if nosmt then EcCommands.pragma_check `Check; + (* If -trace is set, the last in-prefix sentence is still + pending. Run it with goal capture before and after, and + build the BEFORE/TACTIC/AFTER/SUMMARY response body. *) + let body = + if not trace then + goals_to_string () + else + let pre_state = + match !pending with + | None -> `Nothing + | Some _ when not (EcCommands.in_proof ()) -> `NotInProof + | Some (src, p) -> `Ready (src, p) + in + match pre_state with + | `Nothing -> failwith "trace: nothing to trace" + | `NotInProof -> + failwith + "trace: target sentence is not in a proof context" + | `Ready (src, p) -> + let loc = p.EP.gl_action.EcLocation.pl_loc in + let (sl, sc) = loc.EcLocation.loc_start in + let (el, ec) = loc.EcLocation.loc_end in + let before_goals = EcCommands.pp_all_goals () in + let n1 = List.length before_goals in + let buf = Buffer.create 1024 in + let fmt = Format.formatter_of_buffer buf in + Format.fprintf fmt + "=== BEFORE: line %d (col %d) ===@\n" sl sc; + EcCommands.pp_current_goal_or_noproof ~all:false fmt; + Format.fprintf fmt + "@\n=== TACTIC (lines %d:%d - %d:%d) ===@\n%s@\n@\n" + sl sc el ec (sentence_source loc); + last_src := src; + begin + try + process_action ~src p; + last_loc := Some loc; + pending := None; + let after_goals = EcCommands.pp_all_goals () in + let n2 = List.length after_goals in + Format.fprintf fmt + "=== AFTER: line %d (col %d) ===@\n" sl sc; + let before_set = + List.fold_left + (fun s g -> EcMaps.Sstr.add g s) + EcMaps.Sstr.empty before_goals + in + (* The new focused goal always counts as "modified" + (its focus status changed even if its text matches + an old sibling); the rest are printed only if + they didn't appear in BEFORE. *) + let to_print = + match after_goals with + | [] -> [] + | head :: tl -> + head :: + List.filter + (fun g -> not (EcMaps.Sstr.mem g before_set)) + tl + in + begin match to_print with + | [] -> Format.fprintf fmt "(no open goals)@\n" + | _ -> + List.iteri (fun i g -> + if i > 0 then Format.fprintf fmt "@\n"; + Format.fprintf fmt "%s@\n" g) + to_print + end; + Format.fprintf fmt + "@\n=== SUMMARY ===@\nopen goals: %d -> %d@\n" n1 n2; + Format.pp_print_flush fmt (); + Buffer.contents buf + with e -> + Format.fprintf fmt + "=== AFTER: line %d (col %d) ===@\n@\n" + sl sc; + Format.pp_print_flush fmt (); + trace_prefix := Buffer.contents buf; + raise (Trace_failed e) + end + in + let tag = match !last_loc with | None -> "" @@ -753,13 +880,16 @@ let main () = let (el, _) = loc.EcLocation.loc_end in Printf.sprintf " [loaded:%s:%d]" filename el in - reply_ok ~tag (goals_to_string ()) + reply_ok ~tag body with | EcCommands.Restart -> do_initialize (); Hashtbl.clear checkpoints; reply_ok "Session restarted" + | Trace_failed e -> + let msg = format_error ~src:!last_src e in + reply_error (!trace_prefix ^ msg) | Failure s -> reply_error s | e -> diff --git a/src/ecCommands.ml b/src/ecCommands.ml index 3e08fb640..eb7f51e2f 100644 --- a/src/ecCommands.ml +++ b/src/ecCommands.ml @@ -1136,8 +1136,11 @@ let pp_current_goal ?(all = false) stream = end (* -------------------------------------------------------------------- *) +let in_proof () = + Option.is_some (S.xgoal (current ())) + let pp_current_goal_or_noproof ?(all = false) stream = - if Option.is_some (S.xgoal (current ())) then + if in_proof () then pp_current_goal ~all stream else Format.fprintf stream "No active proof.@\n%!" diff --git a/src/ecCommands.mli b/src/ecCommands.mli index 8a1220ae0..88c119efb 100644 --- a/src/ecCommands.mli +++ b/src/ecCommands.mli @@ -64,6 +64,7 @@ val pp_current_goal : ?all:bool -> Format.formatter -> unit val pp_current_goal_or_noproof : ?all:bool -> Format.formatter -> unit val pp_maybe_current_goal : Format.formatter -> unit val pp_all_goals : unit -> string list +val in_proof : unit -> bool (* -------------------------------------------------------------------- *) val pragma_verbose : bool -> unit From c6d82770b8edae40764cd0eec0be08d719fa8930 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Sat, 30 May 2026 06:58:45 +0200 Subject: [PATCH 03/51] [llm] relax +strict_bullets in REPL, add TREE and focus tag Three small additions to make REPL-driven proof exploration pleasant without weakening +strict_bullets for saved scripts. 1. Bullet relaxation for REPL input. EcCommands.disable_repl_bullets is called at every REPL phrase; it drops pm_strict_bullets and clears puc_bullets on the active proof so REPL-typed tactics are not rejected for missing bullets. Files loaded via LOAD still respect their own pragma; only direct REPL input is relaxed. 2. TREE / TREE ALL meta-commands. List all open subgoals as a flat numbered enumeration with the focused goal marked. TREE shows a one-line conclusion per goal; TREE ALL shows the full goal bodies. Backed by EcCommands.pp_tree on top of EcCoreGoal.all_opened. 3. [focus: k/N] reply tag. When more than one subgoal is open, both tactic replies and the LOAD response carry [focus: 1/N] alongside any other tag, so the caller knows the next tactic targets goal #1 of N. Supporting plumbing: EcScope.set_xgoal exposes a way to swap the active proof_uc without going through the tactic engine. --- doc/llm/CLAUDE.md | 11 +++++++- src/ec.ml | 66 +++++++++++++++++++++++++++++++++++++++++----- src/ecCommands.ml | 53 +++++++++++++++++++++++++++++++++++++ src/ecCommands.mli | 2 ++ src/ecScope.ml | 4 +++ src/ecScope.mli | 1 + 6 files changed, 129 insertions(+), 8 deletions(-) diff --git a/doc/llm/CLAUDE.md b/doc/llm/CLAUDE.md index 7918f21fe..1b7a3b066 100644 --- a/doc/llm/CLAUDE.md +++ b/doc/llm/CLAUDE.md @@ -61,6 +61,8 @@ These are protocol-level commands, not EasyCrypt syntax: | `REVERT ` | Revert to a specific state (by uuid or checkpoint name) | | `GOALS` | Print the current goal (first subgoal only, with remaining count) | | `GOALS ALL` | Print all subgoals | +| `TREE` | List open subgoals as `[N] `, marking the focused one | +| `TREE ALL` | Same as `TREE`, but with full goal bodies | | `CHECKPOINT ` | Save current uuid under a name for later `REVERT` | | `SEARCH ` | Search for lemmas matching a pattern | | `QUIET ON` / `QUIET OFF` | Suppress/enable automatic goal display after tactics | @@ -263,7 +265,14 @@ SEARCH (_ %/ _) refers to the first unclosed goal, which may not be the intended one. - When a tactic generates multiple subgoals, each subgoal must be - closed in order. Use `GOALS ALL` to see them all. + closed in order. Use `GOALS ALL` or `TREE` to see them all. +- When more than one subgoal is open, replies carry a + `[focus: k/N]` tag (e.g. `OK [uuid:42] [focus: 1/3]`) so you know + which one the next tactic will hit. +- `pragma +strict_bullets` does **not** apply to REPL input. Files + loaded via `LOAD` still respect their own pragmas, but tactics typed + at the REPL prompt are never rejected for missing bullets — the + REPL is the focus mechanism. - `rewrite lemma in H` modifies hypothesis `H` in place (it does not consume it). If you need to preserve the original, copy it first: `have H' := H; rewrite lemma in H'`. diff --git a/src/ec.ml b/src/ec.ml index 2d54b09b0..ea2bc2d56 100644 --- a/src/ec.ml +++ b/src/ec.ml @@ -547,6 +547,38 @@ let main () = Buffer.contents buf in + (* Render the focus-tree of open subgoals. [all=false] gives a + one-line digest per goal (conclusion truncated); [all=true] + gives the full goal body for each. *) + let tree_to_string ?(all=false) () = + let entries = EcCommands.pp_tree ~all () in + match entries with + | [] -> "No active proof.\n" + | _ -> + let buf = Buffer.create 256 in + let one_line s = + let s = + match String.index_opt s '\n' with + | None -> s + | Some k -> String.sub s 0 k + in + let limit = 80 in + if String.length s > limit + then String.sub s 0 (limit - 1) ^ "…" + else s + in + List.iter (fun (i, focused, text) -> + let marker = if focused then " <- focused" else "" in + if all then + Buffer.add_string buf + (Printf.sprintf "[%d]%s\n%s\n" i marker text) + else + Buffer.add_string buf + (Printf.sprintf "[%d] %s%s\n" i (one_line text) marker) + ) entries; + Buffer.contents buf + in + let quiet = ref false in let checkpoints : (string, int) Hashtbl.t = Hashtbl.create 16 in @@ -565,9 +597,17 @@ let main () = Buffer.clear notices in + let focus_tag () = + match EcCommands.pp_tree () with + | _ :: _ :: _ as entries -> + Printf.sprintf " [focus: 1/%d]" (List.length entries) + | _ -> "" + in + let reply_ok_goals ?(all=false) () = - if !quiet then reply_ok "" - else reply_ok (goals_to_string ~all ()) + let tag = focus_tag () in + if !quiet then reply_ok ~tag "" + else reply_ok ~tag (goals_to_string ~all ()) in let reply_error msg = @@ -604,6 +644,7 @@ let main () = (* Process EasyCrypt input from a string (one parsed program) *) let process_ec_input input = Buffer.clear notices; + EcCommands.disable_repl_bullets (); let reader = EcIo.from_string input in let last_src = ref "" in begin try @@ -874,11 +915,14 @@ let main () = in let tag = - match !last_loc with - | None -> "" - | Some loc -> - let (el, _) = loc.EcLocation.loc_end in - Printf.sprintf " [loaded:%s:%d]" filename el + let loaded = + match !last_loc with + | None -> "" + | Some loc -> + let (el, _) = loc.EcLocation.loc_end in + Printf.sprintf " [loaded:%s:%d]" filename el + in + loaded ^ focus_tag () in reply_ok ~tag body @@ -964,6 +1008,14 @@ let main () = Buffer.clear notices; reply_ok (goals_to_string ()) end + else if line = "TREE ALL" then begin + Buffer.clear notices; + reply_ok (tree_to_string ~all:true ()) + end + else if line = "TREE" then begin + Buffer.clear notices; + reply_ok (tree_to_string ()) + end else if String.starts_with line "CHECKPOINT " then begin Buffer.clear notices; let name = String.strip ( diff --git a/src/ecCommands.ml b/src/ecCommands.ml index eb7f51e2f..b5389ce00 100644 --- a/src/ecCommands.ml +++ b/src/ecCommands.ml @@ -996,6 +996,28 @@ let push_context scope context = ct_stack = context.ct_stack |> omap (fun st -> context.ct_current :: st); } +(* -------------------------------------------------------------------- *) +(* Disable bullet enforcement for REPL-driven phrases. Drops the global + pragma so newly-opened proofs have no bullet stack, and clears the + stack on any currently active proof so REPL phrases are not checked + against it. Idempotent. Does not advance the undo level. *) +let disable_repl_bullets () = + pragma_strict_bullets false; + match !context with + | None -> () + | Some ctxt -> + match EcScope.xgoal ctxt.ct_current with + | None -> () + | Some puc -> + match puc.EcScope.puc_active with + | None -> () + | Some (pac, _) when pac.EcScope.puc_bullets = None -> () + | Some (pac, pct) -> + let pac = { pac with EcScope.puc_bullets = None } in + let puc = { puc with EcScope.puc_active = Some (pac, pct) } in + let scope = EcScope.set_xgoal ctxt.ct_current puc in + context := Some { ctxt with ct_current = scope } + (* -------------------------------------------------------------------- *) let initialize ~restart ~undo ~boot ~checkmode ~checkproof = assert (restart || EcUtils.is_none !context); @@ -1178,3 +1200,34 @@ let pp_all_goals () = end | _ -> [] + +(* -------------------------------------------------------------------- *) +(* Render the open-subgoals tree. Each entry is (index, is_focused, + text). [index] is 1-based, [is_focused] marks the focused goal + (always at index 1 with EC's current focus model), and [text] is + either a one-line conclusion digest (when [~all = false]) or the + full goal body (when [~all = true]). *) +let pp_tree ?(all = false) () : (int * bool * string) list = + let scope = current () in + match S.xgoal scope with + | Some { S.puc_active = Some ({ puc_jdg = S.PSCheck pf }, _) } -> begin + match EcCoreGoal.opened pf with + | None -> [] + | Some _ -> + let ppe = EcPrinting.PPEnv.ofenv (S.env scope) in + let goals = EcCoreGoal.all_opened pf in + List.mapi (fun i { EcCoreGoal.g_hyps; EcCoreGoal.g_concl } -> + let text = + if all then + let buf = Buffer.create 256 in + let hc = (EcEnv.LDecl.tohyps g_hyps, g_concl) in + Format.fprintf + (Format.formatter_of_buffer buf) + "%a@?" (EcPrinting.pp_goal1 ppe) hc; + Buffer.contents buf + else + Format.asprintf "%a" (EcPrinting.pp_form ppe) g_concl + in + (i + 1, i = 0, text)) goals + end + | _ -> [] diff --git a/src/ecCommands.mli b/src/ecCommands.mli index 88c119efb..1e744cb5f 100644 --- a/src/ecCommands.mli +++ b/src/ecCommands.mli @@ -65,6 +65,8 @@ val pp_current_goal_or_noproof : ?all:bool -> Format.formatter -> unit val pp_maybe_current_goal : Format.formatter -> unit val pp_all_goals : unit -> string list val in_proof : unit -> bool +val disable_repl_bullets : unit -> unit +val pp_tree : ?all:bool -> unit -> (int * bool * string) list (* -------------------------------------------------------------------- *) val pragma_verbose : bool -> unit diff --git a/src/ecScope.ml b/src/ecScope.ml index 8f0e27f87..960d48fbd 100644 --- a/src/ecScope.ml +++ b/src/ecScope.ml @@ -489,6 +489,10 @@ let goal (scope : scope) = let xgoal (scope : scope) = scope.sc_pr_uc +(* -------------------------------------------------------------------- *) +let set_xgoal (scope : scope) (puc : proof_uc) = + { scope with sc_pr_uc = Some puc } + (* -------------------------------------------------------------------- *) let dump_why3 (scope : scope) (filename : string) = try EcSmt.dump_why3 (env scope) filename diff --git a/src/ecScope.mli b/src/ecScope.mli index d73ed66d7..5aeae3e3b 100644 --- a/src/ecScope.mli +++ b/src/ecScope.mli @@ -87,6 +87,7 @@ val env : scope -> EcEnv.env val attop : scope -> bool val goal : scope -> proof_auc option val xgoal : scope -> proof_uc option +val set_xgoal : scope -> proof_uc -> scope (* Creates a scope that is identical to the supplied one except * that the environment and required theories are reset to the ones From 452244b52c185c72d658fbd981b96a2a2ab7f994 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Sat, 30 May 2026 07:11:13 +0200 Subject: [PATCH 04/51] [llm] add FOCUS N and NEXT for explicit goal selection The REPL relies on EC's "first open goal is the focused one" convention. Until now the only way to work on a non-first goal was to discharge the earlier ones; the proving agent often wants to inspect or skip a particular sibling without that. Add two meta-commands: FOCUS N rotate the open-goal list so the goal at index N (from TREE) becomes the focused one, preserving cyclic order. NEXT shorthand for FOCUS 2 (rotate one step). Backed by a new EcCoreGoal.rotate_focus that splits and recombines pr_opened. Going through the tactic engine doesn't work for standalone rotation: tcenv1_of_proof tc_down's the siblings out of view, so Protate (the `first last` parsed form) has nothing to rotate at the top level. EcCommands.focus_goal wraps rotate_focus, applies it via the same scope-mutation path disable_repl_bullets uses, and pushes a new context so UNDO/REVERT can roll the change back. --- doc/llm/CLAUDE.md | 2 ++ src/ec.ml | 27 +++++++++++++++++++++++++++ src/ecCommands.ml | 34 ++++++++++++++++++++++++++++++++++ src/ecCommands.mli | 1 + src/ecCoreGoal.ml | 10 ++++++++++ src/ecCoreGoal.mli | 6 ++++++ 6 files changed, 80 insertions(+) diff --git a/doc/llm/CLAUDE.md b/doc/llm/CLAUDE.md index 1b7a3b066..28c8d049a 100644 --- a/doc/llm/CLAUDE.md +++ b/doc/llm/CLAUDE.md @@ -63,6 +63,8 @@ These are protocol-level commands, not EasyCrypt syntax: | `GOALS ALL` | Print all subgoals | | `TREE` | List open subgoals as `[N] `, marking the focused one | | `TREE ALL` | Same as `TREE`, but with full goal bodies | +| `FOCUS N` | Rotate focus so subgoal `[N]` (from `TREE`) becomes the focused goal | +| `NEXT` | Rotate focus to the next subgoal (equivalent to `FOCUS 2`) | | `CHECKPOINT ` | Save current uuid under a name for later `REVERT` | | `SEARCH ` | Search for lemmas matching a pattern | | `QUIET ON` / `QUIET OFF` | Suppress/enable automatic goal display after tactics | diff --git a/src/ec.ml b/src/ec.ml index ea2bc2d56..ce5b1cfe1 100644 --- a/src/ec.ml +++ b/src/ec.ml @@ -1016,6 +1016,33 @@ let main () = Buffer.clear notices; reply_ok (tree_to_string ()) end + else if String.starts_with line "FOCUS " || line = "NEXT" then begin + Buffer.clear notices; + let request = + if line = "NEXT" then `Next + else + let arg = String.strip ( + String.sub line 6 (String.length line - 6)) in + try `At (int_of_string arg) + with Failure _ -> `Bad arg + in + match request with + | `Bad arg -> + reply_error (Printf.sprintf "FOCUS: not an integer: %s" arg) + | _ -> + let entries = EcCommands.pp_tree () in + let n = List.length entries in + let target = + match request with + | `Next -> if n <= 1 then 1 else 2 + | `At k -> k + | `Bad _ -> 1 + in + begin match EcCommands.focus_goal target with + | Ok _ -> reply_ok_goals () + | Error msg -> reply_error msg + end + end else if String.starts_with line "CHECKPOINT " then begin Buffer.clear notices; let name = String.strip ( diff --git a/src/ecCommands.ml b/src/ecCommands.ml index b5389ce00..f00f5c859 100644 --- a/src/ecCommands.ml +++ b/src/ecCommands.ml @@ -997,6 +997,40 @@ let push_context scope context = |> omap (fun st -> context.ct_current :: st); } (* -------------------------------------------------------------------- *) +(* Rotate the focus of the currently active proof so that the goal at + 1-based index [k] becomes the focused one. The change is persisted + in the context with a new uuid so UNDO/REVERT can roll it back. + Returns the new number of open goals on success, or an error + message on failure. *) +let focus_goal (k : int) : (int, string) result = + match !context with + | None -> Error "no active context" + | Some ctxt -> + match EcScope.xgoal ctxt.ct_current with + | None -> Error "no active proof" + | Some puc -> + match puc.EcScope.puc_active with + | None -> Error "no active proof" + | Some (pac, pct) -> + match pac.EcScope.puc_jdg with + | EcScope.PSNoCheck -> Error "proof is in no-check mode" + | EcScope.PSCheck pf -> + let n = List.length (EcCoreGoal.all_hd_opened pf) in + if n = 0 then Error "no open goals" + else if k < 1 || k > n then + Error (Printf.sprintf + "focus: index %d out of range (1..%d)" k n) + else if k = 1 then Ok n + else begin + let pf = EcCoreGoal.rotate_focus k pf in + let pac = { pac with EcScope.puc_jdg = EcScope.PSCheck pf } in + let puc = + { puc with EcScope.puc_active = Some (pac, pct) } in + let scope = EcScope.set_xgoal ctxt.ct_current puc in + context := Some (push_context scope ctxt); + Ok n + end + (* Disable bullet enforcement for REPL-driven phrases. Drops the global pragma so newly-opened proofs have no bullet stack, and clears the stack on any currently active proof so REPL phrases are not checked diff --git a/src/ecCommands.mli b/src/ecCommands.mli index 1e744cb5f..5248ef8b9 100644 --- a/src/ecCommands.mli +++ b/src/ecCommands.mli @@ -67,6 +67,7 @@ val pp_all_goals : unit -> string list val in_proof : unit -> bool val disable_repl_bullets : unit -> unit val pp_tree : ?all:bool -> unit -> (int * bool * string) list +val focus_goal : int -> (int, string) result (* -------------------------------------------------------------------- *) val pragma_verbose : bool -> unit diff --git a/src/ecCoreGoal.ml b/src/ecCoreGoal.ml index 728824b4d..1df78f2c7 100644 --- a/src/ecCoreGoal.ml +++ b/src/ecCoreGoal.ml @@ -1030,6 +1030,16 @@ let all_opened (pf : proof) = (* -------------------------------------------------------------------- *) let closed (pf : proof) = List.is_empty pf.pr_opened +(* -------------------------------------------------------------------- *) +let rotate_focus (k : int) (pf : proof) = + let n = List.length pf.pr_opened in + if k < 1 || k > n then + invalid_arg "EcCoreGoal.rotate_focus"; + if k = 1 then pf + else + let pre, post = List.split_at (k - 1) pf.pr_opened in + { pf with pr_opened = post @ pre } + (* -------------------------------------------------------------------- *) module Exn = struct let recast pe _hyps f x = diff --git a/src/ecCoreGoal.mli b/src/ecCoreGoal.mli index 2f1b51740..2b6e59074 100644 --- a/src/ecCoreGoal.mli +++ b/src/ecCoreGoal.mli @@ -207,6 +207,12 @@ val all_opened : proof -> pregoal list (* Check if a proof is done *) val closed : proof -> bool +(* Rotate the list of opened goals at the top level. [rotate_focus k pf] + makes the goal currently at 1-based index [k] the new focused goal, + preserving the cyclic order of the others. Raises [Invalid_argument] + if [k] is out of range. *) +val rotate_focus : int -> proof -> proof + (* -------------------------------------------------------------------- *) val tc_error : proofenv -> ?catchable:bool -> ?loc:EcLocation.t -> ?who:string From 675d1acb44bec3da9eb18055afead0fc65e28529 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Sat, 30 May 2026 07:39:17 +0200 Subject: [PATCH 05/51] [llm] add COMMIT to emit a +strict_bullets-friendly proof body REPL phrases are recorded with (uuid, source, parent_handle, opens), where parent_handle is the focused goal right before the phrase ran. COMMIT replays them against the proof DAG to recover the bullet structure. The DAG edge is now explicit. EcCoreGoal's proofenv carries a pr_parent : handle ID.Map.t populated by FApi.newgoal (the single choke-point where every child handle is created); EcCoreGoal exposes children_of_handle / parent_of_handle on top of it. This avoids the older approach of reading children out of g_validation, which only worked for VIntros / VConv / VLConv / VRewrite / VExtern -- not for VApply, whose subgoals are added to the tcenv state outside the validation record. Algorithm: - For each phrase, walk the subtree rooted at its parent handle, registering each multi-child split's children in [sibling_depth] at the right depth. Single-child links are continuations and do not bump depth. - To decide whether a phrase needs a bullet, walk upward via parent_of from its recorded parent until hitting a registered sibling ancestor; if found, emit the bullet for that depth and consume the registration. Bullet tokens are chosen per depth from PR 1017's lexer order (-, +, *, --, ++, **, ---, +++, *** ...), skipping any token already in scope from the LOAD prefix's puc_bullets stack. The stack is snapshotted at the moment REPL input takes over (the new return value of disable_repl_bullets) so COMMIT can avoid token collisions with frames opened by the prefix. Tested patterns: simple split, nested split, case-split, multi-tactic-per-sibling, compound first phrase (move=> hp hq; split.), pHL seq N chain, list induction, have introducing a side goal, [split; split.] producing 4 goals in one phrase, UNDO/REVERT trimming, LOAD mid-proof continuation, and LOAD prefix already using bullet tokens that COMMIT must avoid. All round-trip through `ec.exe compile` under `pragma +strict_bullets`. --- doc/llm/CLAUDE.md | 1 + src/ec.ml | 206 +++++++++++++++++++++++++++++++++++++++++++-- src/ecCommands.ml | 55 +++++++++--- src/ecCommands.mli | 5 +- src/ecCoreGoal.ml | 47 +++++++++-- src/ecCoreGoal.mli | 6 ++ 6 files changed, 295 insertions(+), 25 deletions(-) diff --git a/doc/llm/CLAUDE.md b/doc/llm/CLAUDE.md index 28c8d049a..a06725467 100644 --- a/doc/llm/CLAUDE.md +++ b/doc/llm/CLAUDE.md @@ -65,6 +65,7 @@ These are protocol-level commands, not EasyCrypt syntax: | `TREE ALL` | Same as `TREE`, but with full goal bodies | | `FOCUS N` | Rotate focus so subgoal `[N]` (from `TREE`) becomes the focused goal | | `NEXT` | Rotate focus to the next subgoal (equivalent to `FOCUS 2`) | +| `COMMIT` | Emit recorded REPL phrases as a bulleted proof body (works under `+strict_bullets`) | | `CHECKPOINT ` | Save current uuid under a name for later `REVERT` | | `SEARCH ` | Search for lemmas matching a pattern | | `QUIET ON` / `QUIET OFF` | Suppress/enable automatic goal display after tactics | diff --git a/src/ec.ml b/src/ec.ml index ce5b1cfe1..ba874f49c 100644 --- a/src/ec.ml +++ b/src/ec.ml @@ -583,6 +583,170 @@ let main () = let checkpoints : (string, int) Hashtbl.t = Hashtbl.create 16 in + (* Transcript of REPL-typed phrases that succeeded. Each entry is + (uuid_before, src, parent, opens_at_entry), where: + - [parent] is the handle that was focused right before the + phrase ran ([None] iff outside a proof); + - [opens_at_entry] is the full open-handle list at entry time + (focused-first), used by COMMIT to seed the sibling map for + continuations whose first phrase already starts inside a + frame opened by the LOAD prefix. + Trimmed by UNDO/REVERT; cleared on LOAD/Restart. *) + let transcript : + (int * string * EcCoreGoal.handle option + * EcCoreGoal.handle list) list ref = ref [] in + + (* The bullet stack of the active proof at the moment REPL input + took over. Captured the first time [disable_repl_bullets] + actually clears a non-empty stack; subsequent (idempotent) + calls return [None] and leave this snapshot unchanged. Used by + [COMMIT] to pick bullet characters that don't collide with + tokens already in scope from the LOAD prefix. Cleared on + LOAD/Restart along with the transcript. *) + let prior_bullets : EcBullets.stack option ref = ref None in + + let transcript_trim target = + transcript := + List.filter + (fun (uuid_before, _, _, _) -> uuid_before < target) + !transcript + in + + (* Render the recorded transcript as a +strict_bullets-friendly + proof body. The algorithm reads the proof DAG via + [EcCommands.children_of]/[parent_of] (backed by [pr_parent], + which records the parent handle of every child at + [FApi.newgoal] time). Strategy: + - For each phrase, walk the subtree rooted at its parent + handle, finding every multi-child split, and register each + such child in [sibling_depth] at the corresponding depth. + Single-child links are continuations and don't bump depth. + - To decide whether a phrase needs a bullet, walk upward via + [parent_of] from its recorded parent until hitting a + registered sibling ancestor; if found, emit the bullet for + that depth and consume the registration. + Bullet tokens are chosen per depth, skipping any token + already in the LOAD prefix's [puc_bullets] stack (snapshotted + at the moment REPL input took over) so we never collide. *) + (* Token order matches PR 1017's lexer: -, +, *, --, ++, **, + ---, +++, *** ... *) + let token_at_index i = + let chars = [| "-"; "+"; "*" |] in + let rep = i / 3 + 1 in + let chr = chars.(i mod 3) in + String.concat "" (List.init rep (fun _ -> chr)) + in + + let commit_proof_text () = + let entries = List.rev !transcript in + let buf = Buffer.create 1024 in + let emit_indent depth = + for _ = 1 to depth do Buffer.add_string buf " " done + in + let module Hmap = + Map.Make (struct + type t = EcCoreGoal.handle + let compare = compare + end) + in + let sibling_depth : int Hmap.t ref = ref Hmap.empty in + let current_depth = ref 0 in + (* Pick a bullet token for each depth, skipping tokens already + in scope from the LOAD prefix's bullet stack (so we never + collide with frames the prefix opened). State is per-COMMIT + so each invocation starts from a clean slate. *) + let in_use_tokens = + match !prior_bullets with + | None -> [] + | Some stack -> + List.map (fun (f : EcBullets.frame) -> f.bf_token) stack + in + let depth_cache : (int, string) Hashtbl.t = Hashtbl.create 8 in + let next_tok_idx = ref 0 in + let assigned_tokens = ref [] in + let bullet_for_depth d = + match Hashtbl.find_opt depth_cache d with + | Some t -> t + | None -> + let rec pick () = + let t = token_at_index !next_tok_idx in + incr next_tok_idx; + if List.mem t in_use_tokens || List.mem t !assigned_tokens + then pick () + else t + in + let t = pick () in + assigned_tokens := t :: !assigned_tokens; + Hashtbl.add depth_cache d t; + t + in + (* Seed: if the first recorded phrase entered a state with + multiple open goals, the LOAD prefix opened a frame whose + siblings are still pending. Register all of them as depth-1 + pending siblings so the first phrase's parent gets a bullet. *) + (match entries with + | (_, _, Some _, (_ :: _ :: _ as opens)) :: _ -> + List.iter + (fun h -> sibling_depth := Hmap.add h 1 !sibling_depth) + opens + | _ -> ()); + List.iter (fun (_uuid, src, parent_opt, _opens) -> + match parent_opt with + | None -> + Buffer.add_string buf src; + Buffer.add_char buf '\n' + | Some parent -> + (* Walk upward through pr_parent until we find a registered + sibling ancestor, or run out. If found, emit a bullet at + that depth and consume the registration. *) + let rec find_ancestor h = + match Hmap.find_opt h !sibling_depth with + | Some d -> Some (h, d) + | None -> + match EcCommands.parent_of h with + | Some p -> find_ancestor p + | None -> None + in + (match find_ancestor parent with + | Some (h, d) -> + emit_indent (d - 1); + Buffer.add_string buf (bullet_for_depth d); + Buffer.add_char buf ' '; + current_depth := d; + sibling_depth := Hmap.remove h !sibling_depth + | None -> + emit_indent !current_depth); + Buffer.add_string buf src; + Buffer.add_char buf '\n'; + (* A phrase can chain multiple sub-validations internally + (e.g. [move=> hp hq; split.] is VIntros -> VApply(split) + on a single recorded phrase). Walk single-child + validations until we hit a real split (>=2 children) or + an open leaf. *) + (* Walk the entire subtree rooted at [parent], finding every + multi-child node, and register its children at the + corresponding nesting depth. A compound phrase like + [split; split.] can produce nested splits within a single + phrase; both levels of children need to be registered. + Single-child links don't bump the depth (continuations); + multi-child links do. *) + let rec walk h d = + match EcCommands.children_of h with + | [c] -> walk c d + | (_ :: _ :: _) as cs -> + List.iter + (fun c -> + sibling_depth := + Hmap.add c d !sibling_depth; + walk c (d + 1)) + cs + | [] -> () + in + walk parent (!current_depth + 1) + ) entries; + Buffer.contents buf + in + let reply_ok ?(tag="") body = let n = Buffer.contents notices in Printf.printf "OK [uuid:%d]%s\n" (EcCommands.uuid ()) tag; @@ -623,9 +787,20 @@ let main () = Buffer.clear notices in - (* Process a single EasyCrypt command, respecting gl_fail *) - let process_action ~src (p : EP.global) = + (* Process a single EasyCrypt command, respecting gl_fail. When + [~record:true], capture the parent handle (the focused goal + before the phrase ran) and append a transcript entry on + success. COMMIT will use [EcCommands.children_of] on each + parent to walk the proof DAG and recover bullet structure. *) + let process_action ?(record=false) ~src (p : EP.global) = let loc = p.EP.gl_action.EcLocation.pl_loc in + let pre_uuid = EcCommands.uuid () in + let opens_pre = + if record then EcCommands.open_handles () else [] + in + let parent = + match opens_pre with h :: _ -> Some h | [] -> None + in let succeeded = ref false in begin try ignore (EcCommands.process ~src p.EP.gl_action : float option); @@ -638,13 +813,21 @@ let main () = if !succeeded && p.EP.gl_fail then raise (EcScope.toperror_of_exn ~gloc:loc (EcScope.HiScopeError (None, - "this command is expected to fail"))) + "this command is expected to fail"))); + if record && !succeeded && not p.EP.gl_fail then + transcript := (pre_uuid, src, parent, opens_pre) :: !transcript in (* Process EasyCrypt input from a string (one parsed program) *) let process_ec_input input = Buffer.clear notices; - EcCommands.disable_repl_bullets (); + (* On the first REPL phrase of each proof, capture the bullet + stack that the LOAD prefix left so COMMIT can avoid token + collisions with it. Subsequent calls return [None] and don't + clobber the snapshot. *) + (match EcCommands.disable_repl_bullets () with + | None -> () + | Some _ as snapshot -> prior_bullets := snapshot); let reader = EcIo.from_string input in let last_src = ref "" in begin try @@ -653,10 +836,11 @@ let main () = last_src := src; begin match EcLocation.unloc prog with | EP.P_Prog (commands, _) -> - List.iter (process_action ~src) commands; + List.iter (process_action ~record:true ~src) commands; reply_ok_goals () | EP.P_Undo i -> EcCommands.undo i; + transcript_trim i; reply_ok_goals () | EP.P_Exit -> EcIo.finalize reader; exit 0 @@ -667,6 +851,8 @@ let main () = with | EcCommands.Restart -> do_initialize (); + transcript := []; + prior_bullets := None; reply_ok "Session restarted" | e -> reply_error (format_error ~src:!last_src e) @@ -743,6 +929,8 @@ let main () = (* Reset proof engine and process file *) do_initialize (); Hashtbl.clear checkpoints; + transcript := []; + prior_bullets := None; EcCommands.addidir (Filename.dirname filename); let reader = EcIo.from_file filename in @@ -930,6 +1118,8 @@ let main () = | EcCommands.Restart -> do_initialize (); Hashtbl.clear checkpoints; + transcript := []; + prior_bullets := None; reply_ok "Session restarted" | Trace_failed e -> let msg = format_error ~src:!last_src e in @@ -996,6 +1186,7 @@ let main () = let uuid = EcCommands.uuid () in if uuid > 0 then begin EcCommands.undo (uuid - 1); + transcript_trim (uuid - 1); reply_ok_goals () end else reply_error "nothing to undo" @@ -1016,6 +1207,10 @@ let main () = Buffer.clear notices; reply_ok (tree_to_string ()) end + else if line = "COMMIT" then begin + Buffer.clear notices; + reply_ok (commit_proof_text ()) + end else if String.starts_with line "FOCUS " || line = "NEXT" then begin Buffer.clear notices; let request = @@ -1074,6 +1269,7 @@ let main () = "REVERT: uuid %d out of range [0, %d]" target uuid) else begin EcCommands.undo target; + transcript_trim target; reply_ok_goals () end end diff --git a/src/ecCommands.ml b/src/ecCommands.ml index f00f5c859..012a77110 100644 --- a/src/ecCommands.ml +++ b/src/ecCommands.ml @@ -1034,23 +1034,30 @@ let focus_goal (k : int) : (int, string) result = (* Disable bullet enforcement for REPL-driven phrases. Drops the global pragma so newly-opened proofs have no bullet stack, and clears the stack on any currently active proof so REPL phrases are not checked - against it. Idempotent. Does not advance the undo level. *) -let disable_repl_bullets () = + against it. Idempotent. Does not advance the undo level. Returns + the stack that was in place (if any) at the moment the active + proof's bullets were first cleared; returns [None] on idempotent + calls (where the stack is already gone). Callers use the returned + stack to drive bullet-character selection in [COMMIT]. *) +let disable_repl_bullets () : EcBullets.stack option = pragma_strict_bullets false; match !context with - | None -> () + | None -> None | Some ctxt -> match EcScope.xgoal ctxt.ct_current with - | None -> () + | None -> None | Some puc -> match puc.EcScope.puc_active with - | None -> () - | Some (pac, _) when pac.EcScope.puc_bullets = None -> () + | None -> None | Some (pac, pct) -> - let pac = { pac with EcScope.puc_bullets = None } in - let puc = { puc with EcScope.puc_active = Some (pac, pct) } in - let scope = EcScope.set_xgoal ctxt.ct_current puc in - context := Some { ctxt with ct_current = scope } + match pac.EcScope.puc_bullets with + | None -> None + | Some _ as prior -> + let pac = { pac with EcScope.puc_bullets = None } in + let puc = { puc with EcScope.puc_active = Some (pac, pct) } in + let scope = EcScope.set_xgoal ctxt.ct_current puc in + context := Some { ctxt with ct_current = scope }; + prior (* -------------------------------------------------------------------- *) let initialize ~restart ~undo ~boot ~checkmode ~checkproof = @@ -1195,6 +1202,34 @@ let pp_current_goal ?(all = false) stream = let in_proof () = Option.is_some (S.xgoal (current ())) +(* Return the list of open-goal handles at the top level of the active + proof, focused-first, or [] if no proof is active. *) +let open_handles () : EcCoreGoal.handle list = + match S.xgoal (current ()) with + | Some { S.puc_active = + Some ({ S.puc_jdg = S.PSCheck pf }, _) } -> + EcCoreGoal.all_hd_opened pf + | _ -> [] + +(* Direct DAG children of [h] in the active proof. [] if no proof. *) +let children_of (h : EcCoreGoal.handle) : EcCoreGoal.handle list = + match S.xgoal (current ()) with + | Some { S.puc_active = + Some ({ S.puc_jdg = S.PSCheck pf }, _) } -> + EcCoreGoal.children_of_handle + (EcCoreGoal.proofenv_of_proof pf) h + | _ -> [] + +(* Parent of [h] in the active proof's DAG, or [None] if [h] is the + root or no proof is active. *) +let parent_of (h : EcCoreGoal.handle) : EcCoreGoal.handle option = + match S.xgoal (current ()) with + | Some { S.puc_active = + Some ({ S.puc_jdg = S.PSCheck pf }, _) } -> + EcCoreGoal.parent_of_handle + (EcCoreGoal.proofenv_of_proof pf) h + | _ -> None + let pp_current_goal_or_noproof ?(all = false) stream = if in_proof () then pp_current_goal ~all stream diff --git a/src/ecCommands.mli b/src/ecCommands.mli index 5248ef8b9..320aed1d6 100644 --- a/src/ecCommands.mli +++ b/src/ecCommands.mli @@ -65,9 +65,12 @@ val pp_current_goal_or_noproof : ?all:bool -> Format.formatter -> unit val pp_maybe_current_goal : Format.formatter -> unit val pp_all_goals : unit -> string list val in_proof : unit -> bool -val disable_repl_bullets : unit -> unit +val disable_repl_bullets : unit -> EcBullets.stack option val pp_tree : ?all:bool -> unit -> (int * bool * string) list val focus_goal : int -> (int, string) result +val open_handles : unit -> EcCoreGoal.handle list +val children_of : EcCoreGoal.handle -> EcCoreGoal.handle list +val parent_of : EcCoreGoal.handle -> EcCoreGoal.handle option (* -------------------------------------------------------------------- *) val pragma_verbose : bool -> unit diff --git a/src/ecCoreGoal.ml b/src/ecCoreGoal.ml index 1df78f2c7..fc79359c9 100644 --- a/src/ecCoreGoal.ml +++ b/src/ecCoreGoal.ml @@ -132,9 +132,13 @@ type proof = { } and proofenv = { - pr_uid : ID.id; (* unique ID for this proof *) - pr_main : ID.id; (* top goal, contains the final result *) - pr_goals : goal ID.Map.t; (* set of all goals, closed and opened *) + pr_uid : ID.id; (* unique ID for this proof *) + pr_main : ID.id; (* top goal, contains the final result *) + pr_goals : goal ID.Map.t; (* set of all goals, closed and opened *) + pr_parent : handle ID.Map.t; + (* For each non-root handle, the parent in the proof DAG: i.e. + the handle that was being worked on when this one was created + via [FApi.newgoal]. The root [pr_main] is absent. *) } and pregoal = { @@ -463,17 +467,24 @@ module FApi = struct tcenv (* ------------------------------------------------------------------ *) - let pf_newgoal (pe : proofenv) ?vx hyps concl = + let pf_newgoal (pe : proofenv) ?parent ?vx hyps concl = let hid = ID.gen () in let pregoal = { g_uid = hid; g_hyps = hyps; g_concl = concl; g_simpl = EcEnv.SimplifyContext.empty; } in let goal = { g_goal = pregoal; g_validation = vx; } in - let pe = { pe with pr_goals = ID.Map.add pregoal.g_uid goal pe.pr_goals; } in + let pr_goals = ID.Map.add pregoal.g_uid goal pe.pr_goals in + let pr_parent = + match parent with + | None -> pe.pr_parent + | Some p -> ID.Map.add pregoal.g_uid p pe.pr_parent + in + let pe = { pe with pr_goals; pr_parent } in (pe, pregoal) (* ------------------------------------------------------------------ *) let newgoal (tc : tcenv) ?(hyps : LDecl.hyps option) (concl : form) = let hyps = ofdfl (fun () -> tc_hyps tc) hyps in - let (pe, pg) = pf_newgoal (tc_penv tc) hyps concl in + let parent = tc.tce_tcenv.tce_goal |> Option.map (fun g -> g.g_uid) in + let (pe, pg) = pf_newgoal (tc_penv tc) ?parent hyps concl in let pg = { pg with g_simpl = tc1_simplify_context tc.tce_tcenv } in let pe = update_goal_map (fun g -> { g with g_goal = pg }) pg.g_uid pe in @@ -1006,9 +1017,10 @@ let start (hyps : LDecl.hyps) (goal : form) = let goal = { g_uid = hid; g_hyps = hyps; g_concl = goal; g_simpl = EcEnv.SimplifyContext.empty; } in let goal = { g_goal = goal; g_validation = None; } in - let env = { pr_uid = uid; - pr_main = hid; - pr_goals = ID.Map.singleton hid goal; } in + let env = { pr_uid = uid; + pr_main = hid; + pr_goals = ID.Map.singleton hid goal; + pr_parent = ID.Map.empty; } in { pr_env = env; pr_opened = [hid]; } @@ -1030,6 +1042,23 @@ let all_opened (pf : proof) = (* -------------------------------------------------------------------- *) let closed (pf : proof) = List.is_empty pf.pr_opened +(* -------------------------------------------------------------------- *) +(* Direct children of [h] in the proof DAG, in creation order. This is + driven by [pr_parent], the explicit parent edge recorded by + [FApi.newgoal] at the moment each child handle is allocated. The + iteration order matches creation order because handles are + generated by a monotonic counter and [ID.Map] iterates by key. *) +let children_of_handle (pe : proofenv) (h : handle) : handle list = + ID.Map.fold + (fun child parent acc -> + if eq_handle parent h then child :: acc else acc) + pe.pr_parent [] + |> List.rev + +(* Parent of [h] in the proof DAG, or [None] if [h] is the root. *) +let parent_of_handle (pe : proofenv) (h : handle) : handle option = + ID.Map.find_opt h pe.pr_parent + (* -------------------------------------------------------------------- *) let rotate_focus (k : int) (pf : proof) = let n = List.length pf.pr_opened in diff --git a/src/ecCoreGoal.mli b/src/ecCoreGoal.mli index 2b6e59074..19d0feb09 100644 --- a/src/ecCoreGoal.mli +++ b/src/ecCoreGoal.mli @@ -207,6 +207,12 @@ val all_opened : proof -> pregoal list (* Check if a proof is done *) val closed : proof -> bool +(* Direct children of [h] in the proof DAG, in creation order. *) +val children_of_handle : proofenv -> handle -> handle list + +(* Parent of [h] in the proof DAG, or [None] if [h] is the root. *) +val parent_of_handle : proofenv -> handle -> handle option + (* Rotate the list of opened goals at the top level. [rotate_focus k pf] makes the goal currently at 1-based index [k] the new focused goal, preserving the cyclic order of the others. Raises [Invalid_argument] From 17c4a0ec29ac74c64fef2cf97d92a85e7583b74a Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Sat, 30 May 2026 10:54:50 +0200 Subject: [PATCH 06/51] [llm] extract REPL into a dedicated EcLlm module The LLM REPL accumulated ~870 lines of closures and refs inside [main]'s body, intermixed with the unrelated compile/runtest/docgen plumbing. Move it out into [src/ecLlm.ml] (with a one-line .mli exposing just [val run]). The implementation organises its closed-over state (notices buffer, transcript, prior-bullets snapshot, checkpoints, quiet flag, initialized flag) at the top of [run], then groups the helpers into nested submodules so each concern is named: - Goals goal/error formatting, focus tag, tree rendering - Wire OK/ERROR/ envelope and replies - Transcript transcript trimming and clearing - Commit bullet-token generator and DAG walk for COMMIT - Load LOAD parser, prefix processor, and -trace block [ec.ml] keeps the small [Llm] dispatch arm that calls [EcLlm.run ~relocdir ~boot llmopts]. Pure move, no behavioural change; smoke-tested with COMMIT, TREE, FOCUS, and -trace. --- src/ec.ml | 870 +------------------------------------------------ src/ecLlm.ml | 873 ++++++++++++++++++++++++++++++++++++++++++++++++++ src/ecLlm.mli | 11 + 3 files changed, 885 insertions(+), 869 deletions(-) create mode 100644 src/ecLlm.ml create mode 100644 src/ecLlm.mli diff --git a/src/ec.ml b/src/ec.ml index ba874f49c..5d7244a72 100644 --- a/src/ec.ml +++ b/src/ec.ml @@ -438,874 +438,6 @@ let main () = (* Register user messages printers *) begin let open EcUserMessages in register () end; - (* -------------------------------------------------------------------- *) - (* LLM interactive mode *) - (* -------------------------------------------------------------------- *) - - let llm_guide_path () = - let (module Sites) = EcRelocate.sites in - match EcRelocate.sourceroot with - | Some root -> - Filename.concat (Filename.concat root "doc/llm") "CLAUDE.md" - | None -> - Filename.concat Sites.doc "llm-guide.md" - in - - let print_llm_guide () = - let path = llm_guide_path () in - try - let ic = open_in path in - begin try while true do - print_char (input_char ic) - done with End_of_file -> () end; - close_in ic - with Sys_error e -> - Printf.eprintf "cannot read LLM guide: %s\n%!" e - in - - let run_llm_repl (llmopts : llm_option) = - if llmopts.llmo_help then begin - print_llm_guide (); - exit 0 - end; - - let prvopts = llmopts.llmo_provers in - (* Initialize PRNG *) - Random.self_init (); - - (* Connect to external Why3 server if requested *) - prvopts.prvo_why3server |> oiter (fun server -> - try - Why3.Prove_client.connect_external server - with Why3.Prove_client.ConnectionError e -> - Format.eprintf - "cannot connect to Why3 server `%s': %s" server e; - exit 1); - - (* Add current directory to load path *) - (match relocdir with - | None -> EcCommands.addidir Filename.current_dir_name - | Some pwd -> EcCommands.addidir pwd); - - (* Proof engine configuration *) - let checkmode = { - EcCommands.cm_checkall = prvopts.prvo_checkall; - EcCommands.cm_timeout = odfl 3 prvopts.prvo_timeout; - EcCommands.cm_cpufactor = odfl 1 prvopts.prvo_cpufactor; - EcCommands.cm_nprovers = odfl 4 prvopts.prvo_maxjobs; - EcCommands.cm_provers = prvopts.prvo_provers; - EcCommands.cm_quorum = prvopts.prvo_quorum; - EcCommands.cm_profile = prvopts.prvo_profile; - } in - - (* Notice buffer: collects messages during command processing *) - let notices = Buffer.create 256 in - - let notifier (_ : EcGState.loglevel) (lazy msg) = - Buffer.add_string notices msg; - Buffer.add_char notices '\n' - in - - let initialized = ref false in - - let do_initialize () = - EcCommands.initialize - ~restart:!initialized ~undo:true - ~boot:ldropts.ldro_boot ~checkmode ~checkproof:true; - initialized := true; - (try - List.iter EcCommands.apply_pragma prvopts.prvo_pragmas - with EcCommands.InvalidPragma x -> - EcScope.hierror "invalid pragma: `%s'\n%!" x); - EcCommands.addnotifier notifier; - oiter (fun ppwidth -> - let gs = EcEnv.gstate (EcScope.env (EcCommands.current ())) in - EcGState.setvalue "PP:width" (`Int ppwidth) gs) - prvopts.prvo_ppwidth - in - - (* Error formatting *) - let format_error ?(src="") e = - let base = match e with - | EcScope.TopError (loc, e) -> - let msg = String.strip (EcPException.tostring e) in - if loc = EcLocation._dummy then msg - else Format.asprintf "%s: %s" (EcLocation.tostring loc) msg - | e -> - String.strip (EcPException.tostring e) - in - if src = "" then base - else Printf.sprintf "%s\nsource: %s" base src - in - - (* Output helpers *) - let goals_to_string ?(all=false) () = - let buf = Buffer.create 256 in - let fmt = Format.formatter_of_buffer buf in - EcCommands.pp_current_goal_or_noproof ~all fmt; - Format.pp_print_flush fmt (); - Buffer.contents buf - in - - (* Render the focus-tree of open subgoals. [all=false] gives a - one-line digest per goal (conclusion truncated); [all=true] - gives the full goal body for each. *) - let tree_to_string ?(all=false) () = - let entries = EcCommands.pp_tree ~all () in - match entries with - | [] -> "No active proof.\n" - | _ -> - let buf = Buffer.create 256 in - let one_line s = - let s = - match String.index_opt s '\n' with - | None -> s - | Some k -> String.sub s 0 k - in - let limit = 80 in - if String.length s > limit - then String.sub s 0 (limit - 1) ^ "…" - else s - in - List.iter (fun (i, focused, text) -> - let marker = if focused then " <- focused" else "" in - if all then - Buffer.add_string buf - (Printf.sprintf "[%d]%s\n%s\n" i marker text) - else - Buffer.add_string buf - (Printf.sprintf "[%d] %s%s\n" i (one_line text) marker) - ) entries; - Buffer.contents buf - in - - let quiet = ref false in - - let checkpoints : (string, int) Hashtbl.t = Hashtbl.create 16 in - - (* Transcript of REPL-typed phrases that succeeded. Each entry is - (uuid_before, src, parent, opens_at_entry), where: - - [parent] is the handle that was focused right before the - phrase ran ([None] iff outside a proof); - - [opens_at_entry] is the full open-handle list at entry time - (focused-first), used by COMMIT to seed the sibling map for - continuations whose first phrase already starts inside a - frame opened by the LOAD prefix. - Trimmed by UNDO/REVERT; cleared on LOAD/Restart. *) - let transcript : - (int * string * EcCoreGoal.handle option - * EcCoreGoal.handle list) list ref = ref [] in - - (* The bullet stack of the active proof at the moment REPL input - took over. Captured the first time [disable_repl_bullets] - actually clears a non-empty stack; subsequent (idempotent) - calls return [None] and leave this snapshot unchanged. Used by - [COMMIT] to pick bullet characters that don't collide with - tokens already in scope from the LOAD prefix. Cleared on - LOAD/Restart along with the transcript. *) - let prior_bullets : EcBullets.stack option ref = ref None in - - let transcript_trim target = - transcript := - List.filter - (fun (uuid_before, _, _, _) -> uuid_before < target) - !transcript - in - - (* Render the recorded transcript as a +strict_bullets-friendly - proof body. The algorithm reads the proof DAG via - [EcCommands.children_of]/[parent_of] (backed by [pr_parent], - which records the parent handle of every child at - [FApi.newgoal] time). Strategy: - - For each phrase, walk the subtree rooted at its parent - handle, finding every multi-child split, and register each - such child in [sibling_depth] at the corresponding depth. - Single-child links are continuations and don't bump depth. - - To decide whether a phrase needs a bullet, walk upward via - [parent_of] from its recorded parent until hitting a - registered sibling ancestor; if found, emit the bullet for - that depth and consume the registration. - Bullet tokens are chosen per depth, skipping any token - already in the LOAD prefix's [puc_bullets] stack (snapshotted - at the moment REPL input took over) so we never collide. *) - (* Token order matches PR 1017's lexer: -, +, *, --, ++, **, - ---, +++, *** ... *) - let token_at_index i = - let chars = [| "-"; "+"; "*" |] in - let rep = i / 3 + 1 in - let chr = chars.(i mod 3) in - String.concat "" (List.init rep (fun _ -> chr)) - in - - let commit_proof_text () = - let entries = List.rev !transcript in - let buf = Buffer.create 1024 in - let emit_indent depth = - for _ = 1 to depth do Buffer.add_string buf " " done - in - let module Hmap = - Map.Make (struct - type t = EcCoreGoal.handle - let compare = compare - end) - in - let sibling_depth : int Hmap.t ref = ref Hmap.empty in - let current_depth = ref 0 in - (* Pick a bullet token for each depth, skipping tokens already - in scope from the LOAD prefix's bullet stack (so we never - collide with frames the prefix opened). State is per-COMMIT - so each invocation starts from a clean slate. *) - let in_use_tokens = - match !prior_bullets with - | None -> [] - | Some stack -> - List.map (fun (f : EcBullets.frame) -> f.bf_token) stack - in - let depth_cache : (int, string) Hashtbl.t = Hashtbl.create 8 in - let next_tok_idx = ref 0 in - let assigned_tokens = ref [] in - let bullet_for_depth d = - match Hashtbl.find_opt depth_cache d with - | Some t -> t - | None -> - let rec pick () = - let t = token_at_index !next_tok_idx in - incr next_tok_idx; - if List.mem t in_use_tokens || List.mem t !assigned_tokens - then pick () - else t - in - let t = pick () in - assigned_tokens := t :: !assigned_tokens; - Hashtbl.add depth_cache d t; - t - in - (* Seed: if the first recorded phrase entered a state with - multiple open goals, the LOAD prefix opened a frame whose - siblings are still pending. Register all of them as depth-1 - pending siblings so the first phrase's parent gets a bullet. *) - (match entries with - | (_, _, Some _, (_ :: _ :: _ as opens)) :: _ -> - List.iter - (fun h -> sibling_depth := Hmap.add h 1 !sibling_depth) - opens - | _ -> ()); - List.iter (fun (_uuid, src, parent_opt, _opens) -> - match parent_opt with - | None -> - Buffer.add_string buf src; - Buffer.add_char buf '\n' - | Some parent -> - (* Walk upward through pr_parent until we find a registered - sibling ancestor, or run out. If found, emit a bullet at - that depth and consume the registration. *) - let rec find_ancestor h = - match Hmap.find_opt h !sibling_depth with - | Some d -> Some (h, d) - | None -> - match EcCommands.parent_of h with - | Some p -> find_ancestor p - | None -> None - in - (match find_ancestor parent with - | Some (h, d) -> - emit_indent (d - 1); - Buffer.add_string buf (bullet_for_depth d); - Buffer.add_char buf ' '; - current_depth := d; - sibling_depth := Hmap.remove h !sibling_depth - | None -> - emit_indent !current_depth); - Buffer.add_string buf src; - Buffer.add_char buf '\n'; - (* A phrase can chain multiple sub-validations internally - (e.g. [move=> hp hq; split.] is VIntros -> VApply(split) - on a single recorded phrase). Walk single-child - validations until we hit a real split (>=2 children) or - an open leaf. *) - (* Walk the entire subtree rooted at [parent], finding every - multi-child node, and register its children at the - corresponding nesting depth. A compound phrase like - [split; split.] can produce nested splits within a single - phrase; both levels of children need to be registered. - Single-child links don't bump the depth (continuations); - multi-child links do. *) - let rec walk h d = - match EcCommands.children_of h with - | [c] -> walk c d - | (_ :: _ :: _) as cs -> - List.iter - (fun c -> - sibling_depth := - Hmap.add c d !sibling_depth; - walk c (d + 1)) - cs - | [] -> () - in - walk parent (!current_depth + 1) - ) entries; - Buffer.contents buf - in - - let reply_ok ?(tag="") body = - let n = Buffer.contents notices in - Printf.printf "OK [uuid:%d]%s\n" (EcCommands.uuid ()) tag; - if n <> "" then print_string n; - if body <> "" then begin - print_string body; - let len = String.length body in - if len > 0 && body.[len - 1] <> '\n' then - print_char '\n' - end; - Printf.printf "\n%!"; - Buffer.clear notices - in - - let focus_tag () = - match EcCommands.pp_tree () with - | _ :: _ :: _ as entries -> - Printf.sprintf " [focus: 1/%d]" (List.length entries) - | _ -> "" - in - - let reply_ok_goals ?(all=false) () = - let tag = focus_tag () in - if !quiet then reply_ok ~tag "" - else reply_ok ~tag (goals_to_string ~all ()) - in - - let reply_error msg = - let goals = goals_to_string () in - Printf.printf "ERROR [uuid:%d]\n%s\n" (EcCommands.uuid ()) msg; - if goals <> "" then begin - print_string goals; - let len = String.length goals in - if len > 0 && goals.[len - 1] <> '\n' then - print_char '\n' - end; - Printf.printf "\n%!"; - Buffer.clear notices - in - - (* Process a single EasyCrypt command, respecting gl_fail. When - [~record:true], capture the parent handle (the focused goal - before the phrase ran) and append a transcript entry on - success. COMMIT will use [EcCommands.children_of] on each - parent to walk the proof DAG and recover bullet structure. *) - let process_action ?(record=false) ~src (p : EP.global) = - let loc = p.EP.gl_action.EcLocation.pl_loc in - let pre_uuid = EcCommands.uuid () in - let opens_pre = - if record then EcCommands.open_handles () else [] - in - let parent = - match opens_pre with h :: _ -> Some h | [] -> None - in - let succeeded = ref false in - begin try - ignore (EcCommands.process ~src p.EP.gl_action : float option); - succeeded := true - with - | EcCommands.Restart -> raise EcCommands.Restart - | _ when p.EP.gl_fail -> () - | e -> raise (EcScope.toperror_of_exn ~gloc:loc e) - end; - if !succeeded && p.EP.gl_fail then - raise (EcScope.toperror_of_exn ~gloc:loc - (EcScope.HiScopeError (None, - "this command is expected to fail"))); - if record && !succeeded && not p.EP.gl_fail then - transcript := (pre_uuid, src, parent, opens_pre) :: !transcript - in - - (* Process EasyCrypt input from a string (one parsed program) *) - let process_ec_input input = - Buffer.clear notices; - (* On the first REPL phrase of each proof, capture the bullet - stack that the LOAD prefix left so COMMIT can avoid token - collisions with it. Subsequent calls return [None] and don't - clobber the snapshot. *) - (match EcCommands.disable_repl_bullets () with - | None -> () - | Some _ as snapshot -> prior_bullets := snapshot); - let reader = EcIo.from_string input in - let last_src = ref "" in - begin try - let (src, prog) = EcIo.xparse reader in - let src = String.strip src in - last_src := src; - begin match EcLocation.unloc prog with - | EP.P_Prog (commands, _) -> - List.iter (process_action ~record:true ~src) commands; - reply_ok_goals () - | EP.P_Undo i -> - EcCommands.undo i; - transcript_trim i; - reply_ok_goals () - | EP.P_Exit -> - EcIo.finalize reader; exit 0 - | EP.P_DocComment doc -> - EcCommands.doc_comment doc; - reply_ok "" - end - with - | EcCommands.Restart -> - do_initialize (); - transcript := []; - prior_bullets := None; - reply_ok "Session restarted" - | e -> - reply_error (format_error ~src:!last_src e) - end; - EcIo.finalize reader - in - - (* Handle LOAD "file.ec" [LINE[:COL]] *) - let handle_load args = - Buffer.clear notices; - let args = String.strip args in - let last_src = ref "" in - let trace_prefix = ref "" in - let exception Trace_failed of exn in - - try - (* Parse quoted or unquoted filename *) - let filename, rest = - if String.length args > 0 && args.[0] = '"' then - let close = - try String.index_from args 1 '"' - with Not_found -> - failwith "LOAD: unterminated filename" - in - let fn = String.sub args 1 (close - 1) in - let rest = String.strip ( - String.sub args (close + 1) - (String.length args - close - 1)) in - (fn, rest) - else - match String.split_on_char ' ' args with - | [] -> failwith "LOAD: missing filename" - | [f] -> (f, "") - | f :: rest -> (f, String.concat " " rest) - in - - (* Parse optional LINE[:COL] and flags (-nosmt, -trace) *) - let upto, nosmt, trace = - let words = - String.split_on_char ' ' rest - |> List.filter (fun s -> s <> "") - in - let nosmt = List.mem "-nosmt" words in - let trace = List.mem "-trace" words in - let words = - List.filter - (fun s -> s <> "-nosmt" && s <> "-trace") - words - in - let upto = match words with - | [] -> None - | [w] -> - begin match String.split_on_char ':' w with - | [line] -> - Some (int_of_string line, None) - | [line; col] -> - Some (int_of_string line, Some (int_of_string col)) - | _ -> failwith "LOAD: invalid LINE[:COL] format" - end - | _ -> failwith "LOAD: unexpected arguments" - in - (upto, nosmt, trace) - in - - (* Validate file extension *) - begin try - ignore (EcLoader.getkind - (Filename.extension filename) : EcLoader.kind) - with EcLoader.BadExtension ext -> - failwith (Format.sprintf - "unknown file extension: %s" ext) - end; - - (* Reset proof engine and process file *) - do_initialize (); - Hashtbl.clear checkpoints; - transcript := []; - prior_bullets := None; - EcCommands.addidir (Filename.dirname filename); - - let reader = EcIo.from_file filename in - - let past_upto (loc : EcLocation.t) = - match upto with - | None -> false - | Some (line, col) -> - let (el, ec) = loc.loc_end in - el > line || (el = line && match col with - | None -> false - | Some c -> ec > c) - in - - let last_loc = ref None in - - (* For -trace: lazy whole-file bytes, used to slice the exact - source text of a sentence by byte offsets. *) - let input_bytes = lazy ( - let ic = open_in_bin filename in - let n = in_channel_length ic in - let b = Bytes.create n in - really_input ic b 0 n; - close_in ic; - Bytes.unsafe_to_string b) - in - let sentence_source (loc : EcLocation.t) = - let s = Lazy.force input_bytes in - let lo = max 0 loc.EcLocation.loc_bchar in - let hi = min (String.length s) loc.EcLocation.loc_echar in - if hi <= lo then "" else String.sub s lo (hi - lo) - in - - (* For -trace: defer execution of the last sentence within the - prefix so we can capture goals before and after it. *) - let pending : (string * EP.global) option ref = ref None in - let flush_pending () = - match !pending with - | None -> () - | Some (src, p) -> - last_src := src; - process_action ~src p; - last_loc := Some p.EP.gl_action.EcLocation.pl_loc; - pending := None - in - let step src p = - let loc = p.EP.gl_action.EcLocation.pl_loc in - if past_upto loc then raise Exit; - if trace then begin - flush_pending (); - pending := Some (src, p) - end else begin - last_src := src; - process_action ~src p; - last_loc := Some loc - end - in - - (* In -nosmt mode, admit all SMT calls during prefix loading *) - if nosmt then EcCommands.pragma_check `WeakCheck; - - begin try while true do - let (src, prog) = EcIo.xparse reader in - let src = String.strip src in - match EcLocation.unloc prog with - | EP.P_Prog (commands, locterm) -> - List.iter (step src) commands; - if locterm then raise Exit - | EP.P_Undo i -> - last_src := src; - EcCommands.undo i - | EP.P_Exit -> - raise Exit - | EP.P_DocComment doc -> - last_src := src; - EcCommands.doc_comment doc - done with - | Exit | End_of_file -> () - | e -> - EcIo.finalize reader; - if nosmt then EcCommands.pragma_check `Check; - raise e - end; - - EcIo.finalize reader; - - (* Restore full SMT checking for interactive tactics *) - if nosmt then EcCommands.pragma_check `Check; - - (* If -trace is set, the last in-prefix sentence is still - pending. Run it with goal capture before and after, and - build the BEFORE/TACTIC/AFTER/SUMMARY response body. *) - let body = - if not trace then - goals_to_string () - else - let pre_state = - match !pending with - | None -> `Nothing - | Some _ when not (EcCommands.in_proof ()) -> `NotInProof - | Some (src, p) -> `Ready (src, p) - in - match pre_state with - | `Nothing -> failwith "trace: nothing to trace" - | `NotInProof -> - failwith - "trace: target sentence is not in a proof context" - | `Ready (src, p) -> - let loc = p.EP.gl_action.EcLocation.pl_loc in - let (sl, sc) = loc.EcLocation.loc_start in - let (el, ec) = loc.EcLocation.loc_end in - let before_goals = EcCommands.pp_all_goals () in - let n1 = List.length before_goals in - let buf = Buffer.create 1024 in - let fmt = Format.formatter_of_buffer buf in - Format.fprintf fmt - "=== BEFORE: line %d (col %d) ===@\n" sl sc; - EcCommands.pp_current_goal_or_noproof ~all:false fmt; - Format.fprintf fmt - "@\n=== TACTIC (lines %d:%d - %d:%d) ===@\n%s@\n@\n" - sl sc el ec (sentence_source loc); - last_src := src; - begin - try - process_action ~src p; - last_loc := Some loc; - pending := None; - let after_goals = EcCommands.pp_all_goals () in - let n2 = List.length after_goals in - Format.fprintf fmt - "=== AFTER: line %d (col %d) ===@\n" sl sc; - let before_set = - List.fold_left - (fun s g -> EcMaps.Sstr.add g s) - EcMaps.Sstr.empty before_goals - in - (* The new focused goal always counts as "modified" - (its focus status changed even if its text matches - an old sibling); the rest are printed only if - they didn't appear in BEFORE. *) - let to_print = - match after_goals with - | [] -> [] - | head :: tl -> - head :: - List.filter - (fun g -> not (EcMaps.Sstr.mem g before_set)) - tl - in - begin match to_print with - | [] -> Format.fprintf fmt "(no open goals)@\n" - | _ -> - List.iteri (fun i g -> - if i > 0 then Format.fprintf fmt "@\n"; - Format.fprintf fmt "%s@\n" g) - to_print - end; - Format.fprintf fmt - "@\n=== SUMMARY ===@\nopen goals: %d -> %d@\n" n1 n2; - Format.pp_print_flush fmt (); - Buffer.contents buf - with e -> - Format.fprintf fmt - "=== AFTER: line %d (col %d) ===@\n@\n" - sl sc; - Format.pp_print_flush fmt (); - trace_prefix := Buffer.contents buf; - raise (Trace_failed e) - end - in - - let tag = - let loaded = - match !last_loc with - | None -> "" - | Some loc -> - let (el, _) = loc.EcLocation.loc_end in - Printf.sprintf " [loaded:%s:%d]" filename el - in - loaded ^ focus_tag () - in - reply_ok ~tag body - - with - | EcCommands.Restart -> - do_initialize (); - Hashtbl.clear checkpoints; - transcript := []; - prior_bullets := None; - reply_ok "Session restarted" - | Trace_failed e -> - let msg = format_error ~src:!last_src e in - reply_error (!trace_prefix ^ msg) - | Failure s -> - reply_error s - | e -> - reply_error (format_error ~src:!last_src e) - in - - (* Initialize proof engine *) - do_initialize (); - - (* Signal ready *) - Printf.printf "READY [uuid:%d]\n\n%!" - (EcCommands.uuid ()); - - (* Main REPL loop *) - let multi_buf = Buffer.create 256 in - let in_multi = ref false in - - begin try while true do - let line = input_line stdin in - let line = String.strip line in - - (* Multi-line input: starts, flushes *) - if line = "" then begin - Buffer.clear multi_buf; - in_multi := true - end - else if line = "" && !in_multi then begin - let input = Buffer.contents multi_buf in - Buffer.clear multi_buf; - in_multi := false; - if input <> "" then process_ec_input input - end - else if !in_multi then begin - if Buffer.length multi_buf > 0 then - Buffer.add_char multi_buf ' '; - Buffer.add_string multi_buf line - end - - else if line = "" then - () - else if line = "QUIT" then - exit 0 - else if line = "HELP" then begin - Buffer.clear notices; - let buf = Buffer.create 4096 in - let path = llm_guide_path () in - begin try - let ic = open_in path in - begin try while true do - Buffer.add_char buf (input_char ic) - done with End_of_file -> () end; - close_in ic; - reply_ok (Buffer.contents buf) - with Sys_error e -> - reply_error (Printf.sprintf "cannot read guide: %s" e) - end - end - else if line = "UNDO" then begin - Buffer.clear notices; - let uuid = EcCommands.uuid () in - if uuid > 0 then begin - EcCommands.undo (uuid - 1); - transcript_trim (uuid - 1); - reply_ok_goals () - end else - reply_error "nothing to undo" - end - else if line = "GOALS ALL" then begin - Buffer.clear notices; - reply_ok (goals_to_string ~all:true ()) - end - else if line = "GOALS" then begin - Buffer.clear notices; - reply_ok (goals_to_string ()) - end - else if line = "TREE ALL" then begin - Buffer.clear notices; - reply_ok (tree_to_string ~all:true ()) - end - else if line = "TREE" then begin - Buffer.clear notices; - reply_ok (tree_to_string ()) - end - else if line = "COMMIT" then begin - Buffer.clear notices; - reply_ok (commit_proof_text ()) - end - else if String.starts_with line "FOCUS " || line = "NEXT" then begin - Buffer.clear notices; - let request = - if line = "NEXT" then `Next - else - let arg = String.strip ( - String.sub line 6 (String.length line - 6)) in - try `At (int_of_string arg) - with Failure _ -> `Bad arg - in - match request with - | `Bad arg -> - reply_error (Printf.sprintf "FOCUS: not an integer: %s" arg) - | _ -> - let entries = EcCommands.pp_tree () in - let n = List.length entries in - let target = - match request with - | `Next -> if n <= 1 then 1 else 2 - | `At k -> k - | `Bad _ -> 1 - in - begin match EcCommands.focus_goal target with - | Ok _ -> reply_ok_goals () - | Error msg -> reply_error msg - end - end - else if String.starts_with line "CHECKPOINT " then begin - Buffer.clear notices; - let name = String.strip ( - String.sub line 11 (String.length line - 11)) in - if name = "" then - reply_error "CHECKPOINT: missing name" - else begin - Hashtbl.replace checkpoints name (EcCommands.uuid ()); - reply_ok (Printf.sprintf - "checkpoint '%s' set at uuid %d" name (EcCommands.uuid ())) - end - end - else if String.starts_with line "REVERT " then begin - Buffer.clear notices; - let n = String.strip ( - String.sub line 7 (String.length line - 7)) in - let target = - try Some (int_of_string n) - with Failure _ -> Hashtbl.find_opt checkpoints n - in - begin match target with - | None -> - reply_error (Printf.sprintf - "REVERT: '%s' is not a valid uuid or checkpoint name" n) - | Some target -> - let uuid = EcCommands.uuid () in - if target < 0 || target > uuid then - reply_error (Printf.sprintf - "REVERT: uuid %d out of range [0, %d]" target uuid) - else begin - EcCommands.undo target; - transcript_trim target; - reply_ok_goals () - end - end - end - else if line = "QUIET ON" then begin - Buffer.clear notices; - quiet := true; - reply_ok "" - end - else if line = "QUIET OFF" then begin - Buffer.clear notices; - quiet := false; - reply_ok "" - end - else if String.starts_with line "SEARCH " then begin - let query = String.strip ( - String.sub line 7 (String.length line - 7)) in - let query = - if String.ends_with query "." - then String.sub query 0 (String.length query - 1) - else query - in - process_ec_input (Printf.sprintf "search %s." query) - end - else if String.starts_with line "LOAD " then - handle_load (String.sub line 5 (String.length line - 5)) - else - (* Treat as EasyCrypt input *) - process_ec_input line - done with - | End_of_file -> () - end; - - exit 0 - in - (* Initialize I/O + interaction module *) let module State = struct type t = { @@ -1438,7 +570,7 @@ let main () = end | `Llm llmopts -> - run_llm_repl llmopts + EcLlm.run ~relocdir ~boot:ldropts.ldro_boot llmopts | `Runtest _ -> (* Eagerly executed *) diff --git a/src/ecLlm.ml b/src/ecLlm.ml new file mode 100644 index 000000000..7d2cbdde6 --- /dev/null +++ b/src/ecLlm.ml @@ -0,0 +1,873 @@ +(* -------------------------------------------------------------------- *) +(* The LLM coding-agent REPL. See [ecLlm.mli] for the entry point. + + Implementation note: the REPL holds a large amount of mutable state + (notice buffer, transcript, checkpoints, ...). To keep that state + sharable across the various helpers without resorting to a big + record, [run] is a single closure that opens nested [module] blocks + for grouping. The submodules are read-only views over the closed- + over refs. *) + +open EcUtils + +module EP = EcParsetree + +(* -------------------------------------------------------------------- *) +(* Path to the bundled LLM-agent guide. *) +let llm_guide_path () = + let (module Sites) = EcRelocate.sites in + match EcRelocate.sourceroot with + | Some root -> + Filename.concat (Filename.concat root "doc/llm") "CLAUDE.md" + | None -> + Filename.concat Sites.doc "llm-guide.md" + +(* Print the bundled guide to stdout. Used by [-help]. *) +let print_llm_guide () = + let path = llm_guide_path () in + try + let ic = open_in path in + begin try while true do + print_char (input_char ic) + done with End_of_file -> () end; + close_in ic + with Sys_error e -> + Printf.eprintf "cannot read LLM guide: %s\n%!" e + +(* -------------------------------------------------------------------- *) +let run ~relocdir ~boot (llmopts : EcOptions.llm_option) = + if llmopts.llmo_help then begin + print_llm_guide (); + exit 0 + end; + + let prvopts = llmopts.llmo_provers in + Random.self_init (); + + prvopts.prvo_why3server |> oiter (fun server -> + try + Why3.Prove_client.connect_external server + with Why3.Prove_client.ConnectionError e -> + Format.eprintf + "cannot connect to Why3 server `%s': %s" server e; + exit 1); + + (match relocdir with + | None -> EcCommands.addidir Filename.current_dir_name + | Some pwd -> EcCommands.addidir pwd); + + let checkmode = { + EcCommands.cm_checkall = prvopts.prvo_checkall; + EcCommands.cm_timeout = odfl 3 prvopts.prvo_timeout; + EcCommands.cm_cpufactor = odfl 1 prvopts.prvo_cpufactor; + EcCommands.cm_nprovers = odfl 4 prvopts.prvo_maxjobs; + EcCommands.cm_provers = prvopts.prvo_provers; + EcCommands.cm_quorum = prvopts.prvo_quorum; + EcCommands.cm_profile = prvopts.prvo_profile; + } in + + (* ------------------------------------------------------------------ *) + (* State. *) + + (* Messages emitted by the engine during a phrase; flushed into the + next OK/ERROR reply. *) + let notices = Buffer.create 256 in + + (* Has [EcCommands.initialize] been called? Subsequent calls pass + [~restart:true]. *) + let initialized = ref false in + + (* True iff replies should suppress goal bodies. Toggled by QUIET. *) + let quiet = ref false in + + (* CHECKPOINT name -> uuid. *) + let checkpoints : (string, int) Hashtbl.t = Hashtbl.create 16 in + + (* Transcript of REPL-typed phrases that succeeded. Each entry is + [(uuid_before, src, parent, opens_at_entry)]: + - [parent]: focused handle right before the phrase ([None] iff + outside a proof); + - [opens_at_entry]: full open-handle list (focused first), used + by [Commit] to seed the sibling map when the first recorded + phrase already sits inside a frame opened by the LOAD prefix. + Trimmed by UNDO/REVERT; cleared on LOAD/Restart. *) + let transcript : + (int * string * EcCoreGoal.handle option + * EcCoreGoal.handle list) list ref = ref [] in + + (* The bullet stack of the active proof at the moment REPL input + took over. Captured the first time [disable_repl_bullets] clears + a non-empty stack. Used by [Commit] to pick bullet characters + that don't collide with frames opened by the LOAD prefix. + Cleared with the transcript on LOAD/Restart. *) + let prior_bullets : EcBullets.stack option ref = ref None in + + let notifier (_ : EcGState.loglevel) (lazy msg) = + Buffer.add_string notices msg; + Buffer.add_char notices '\n' + in + + let do_initialize () = + EcCommands.initialize + ~restart:!initialized ~undo:true + ~boot ~checkmode ~checkproof:true; + initialized := true; + (try + List.iter EcCommands.apply_pragma prvopts.prvo_pragmas + with EcCommands.InvalidPragma x -> + EcScope.hierror "invalid pragma: `%s'\n%!" x); + EcCommands.addnotifier notifier; + oiter (fun ppwidth -> + let gs = EcEnv.gstate (EcScope.env (EcCommands.current ())) in + EcGState.setvalue "PP:width" (`Int ppwidth) gs) + prvopts.prvo_ppwidth + in + + (* ------------------------------------------------------------------ *) + (* Goal/error formatting: shared between the wire layer and the + -trace block. *) + let module Goals = struct + let format_error ?(src="") e = + let base = match e with + | EcScope.TopError (loc, e) -> + let msg = String.strip (EcPException.tostring e) in + if loc = EcLocation._dummy then msg + else Format.asprintf "%s: %s" (EcLocation.tostring loc) msg + | e -> + String.strip (EcPException.tostring e) + in + if src = "" then base + else Printf.sprintf "%s\nsource: %s" base src + + let goals_to_string ?(all=false) () = + let buf = Buffer.create 256 in + let fmt = Format.formatter_of_buffer buf in + EcCommands.pp_current_goal_or_noproof ~all fmt; + Format.pp_print_flush fmt (); + Buffer.contents buf + + (* Render the focus-tree of open subgoals. [all=false] gives a + one-line digest per goal; [all=true] gives the full goal body. *) + let tree_to_string ?(all=false) () = + let entries = EcCommands.pp_tree ~all () in + match entries with + | [] -> "No active proof.\n" + | _ -> + let buf = Buffer.create 256 in + let one_line s = + let s = + match String.index_opt s '\n' with + | None -> s + | Some k -> String.sub s 0 k + in + let limit = 80 in + if String.length s > limit + then String.sub s 0 (limit - 1) ^ "…" + else s + in + List.iter (fun (i, focused, text) -> + let marker = if focused then " <- focused" else "" in + if all then + Buffer.add_string buf + (Printf.sprintf "[%d]%s\n%s\n" i marker text) + else + Buffer.add_string buf + (Printf.sprintf "[%d] %s%s\n" i (one_line text) marker) + ) entries; + Buffer.contents buf + + (* Inline focus annotation ([focus: 1/N]) appended to reply tags + whenever the active proof has >=2 open subgoals. *) + let focus_tag () = + match EcCommands.pp_tree () with + | _ :: _ :: _ as entries -> + Printf.sprintf " [focus: 1/%d]" (List.length entries) + | _ -> "" + end in + + (* ------------------------------------------------------------------ *) + (* OK/ERROR/ wire envelope. *) + let module Wire = struct + let reply_ok ?(tag="") body = + let n = Buffer.contents notices in + Printf.printf "OK [uuid:%d]%s\n" (EcCommands.uuid ()) tag; + if n <> "" then print_string n; + if body <> "" then begin + print_string body; + let len = String.length body in + if len > 0 && body.[len - 1] <> '\n' then + print_char '\n' + end; + Printf.printf "\n%!"; + Buffer.clear notices + + let reply_ok_goals ?(all=false) () = + let tag = Goals.focus_tag () in + if !quiet then reply_ok ~tag "" + else reply_ok ~tag (Goals.goals_to_string ~all ()) + + let reply_error msg = + let goals = Goals.goals_to_string () in + Printf.printf "ERROR [uuid:%d]\n%s\n" (EcCommands.uuid ()) msg; + if goals <> "" then begin + print_string goals; + let len = String.length goals in + if len > 0 && goals.[len - 1] <> '\n' then + print_char '\n' + end; + Printf.printf "\n%!"; + Buffer.clear notices + end in + + (* ------------------------------------------------------------------ *) + (* Transcript manipulation. *) + let module Transcript = struct + let trim target = + transcript := + List.filter + (fun (uuid_before, _, _, _) -> uuid_before < target) + !transcript + + let clear () = + transcript := []; + prior_bullets := None + end in + + (* ------------------------------------------------------------------ *) + (* Process a single EasyCrypt command, respecting [gl_fail]. When + [~record:true], append a transcript entry on success: the parent + handle (focused goal before the phrase) and the open-handle list, + which together let [Commit] reconstruct bullet structure. *) + let process_action ?(record=false) ~src (p : EP.global) = + let loc = p.EP.gl_action.EcLocation.pl_loc in + let pre_uuid = EcCommands.uuid () in + let opens_pre = + if record then EcCommands.open_handles () else [] + in + let parent = + match opens_pre with h :: _ -> Some h | [] -> None + in + let succeeded = ref false in + begin try + ignore (EcCommands.process ~src p.EP.gl_action : float option); + succeeded := true + with + | EcCommands.Restart -> raise EcCommands.Restart + | _ when p.EP.gl_fail -> () + | e -> raise (EcScope.toperror_of_exn ~gloc:loc e) + end; + if !succeeded && p.EP.gl_fail then + raise (EcScope.toperror_of_exn ~gloc:loc + (EcScope.HiScopeError (None, + "this command is expected to fail"))); + if record && !succeeded && not p.EP.gl_fail then + transcript := (pre_uuid, src, parent, opens_pre) :: !transcript + in + + (* ------------------------------------------------------------------ *) + (* COMMIT: replay the transcript against the proof DAG (parent_of / + children_of, backed by [EcCoreGoal.pr_parent]), inserting bullets + at multi-child splits. Bullet tokens skip any character already on + the LOAD prefix's [puc_bullets] stack so emitted bullets cannot + collide with frames opened by the prefix. *) + let module Commit = struct + (* Token order matches PR 1017's lexer: -, +, *, --, ++, **, + ---, +++, *** ... *) + let token_at_index i = + let chars = [| "-"; "+"; "*" |] in + let rep = i / 3 + 1 in + let chr = chars.(i mod 3) in + String.concat "" (List.init rep (fun _ -> chr)) + + let proof_text () = + let entries = List.rev !transcript in + let buf = Buffer.create 1024 in + let emit_indent depth = + for _ = 1 to depth do Buffer.add_string buf " " done + in + let module Hmap = + Map.Make (struct + type t = EcCoreGoal.handle + let compare = compare + end) + in + let sibling_depth : int Hmap.t ref = ref Hmap.empty in + let current_depth = ref 0 in + (* Pick a bullet token for each depth, skipping tokens already + in scope from the LOAD prefix's bullet stack. *) + let bullet_to_string (b : EcParsetree.bullet) = + let ch = + match b.b_kind with + | `Minus -> "-" + | `Plus -> "+" + | `Star -> "*" + in + String.concat "" (List.init b.b_count (fun _ -> ch)) + in + let in_use_tokens = + match !prior_bullets with + | None -> [] + | Some stack -> + List.map + (fun (f : EcBullets.frame) -> bullet_to_string f.bf_bullet) + stack + in + let depth_cache : (int, string) Hashtbl.t = Hashtbl.create 8 in + let next_tok_idx = ref 0 in + let assigned_tokens = ref [] in + let bullet_for_depth d = + match Hashtbl.find_opt depth_cache d with + | Some t -> t + | None -> + let rec pick () = + let t = token_at_index !next_tok_idx in + incr next_tok_idx; + if List.mem t in_use_tokens || List.mem t !assigned_tokens + then pick () + else t + in + let t = pick () in + assigned_tokens := t :: !assigned_tokens; + Hashtbl.add depth_cache d t; + t + in + (* Seed: if the first recorded phrase entered a state with + multiple open goals, the LOAD prefix opened a frame whose + siblings are still pending. Register all of them at depth 1 + so the first phrase's parent gets a bullet. *) + (match entries with + | (_, _, Some _, (_ :: _ :: _ as opens)) :: _ -> + List.iter + (fun h -> sibling_depth := Hmap.add h 1 !sibling_depth) + opens + | _ -> ()); + List.iter (fun (_uuid, src, parent_opt, _opens) -> + match parent_opt with + | None -> + Buffer.add_string buf src; + Buffer.add_char buf '\n' + | Some parent -> + (* Walk upward via pr_parent until we hit a registered + sibling ancestor. If found, emit its bullet and consume + the registration. *) + let rec find_ancestor h = + match Hmap.find_opt h !sibling_depth with + | Some d -> Some (h, d) + | None -> + match EcCommands.parent_of h with + | Some p -> find_ancestor p + | None -> None + in + (match find_ancestor parent with + | Some (h, d) -> + emit_indent (d - 1); + Buffer.add_string buf (bullet_for_depth d); + Buffer.add_char buf ' '; + current_depth := d; + sibling_depth := Hmap.remove h !sibling_depth + | None -> + emit_indent !current_depth); + Buffer.add_string buf src; + Buffer.add_char buf '\n'; + (* Register fresh siblings: walk the subtree rooted at + [parent], finding every multi-child split, and register + each such child at the right depth. Single-child links + are continuations and don't bump depth; multi-child + links do. A compound phrase like [split; split.] can + produce nested splits within one phrase. *) + let rec walk h d = + match EcCommands.children_of h with + | [c] -> walk c d + | (_ :: _ :: _) as cs -> + List.iter + (fun c -> + sibling_depth := + Hmap.add c d !sibling_depth; + walk c (d + 1)) + cs + | [] -> () + in + walk parent (!current_depth + 1) + ) entries; + Buffer.contents buf + end in + + (* ------------------------------------------------------------------ *) + (* Process EasyCrypt input typed at the REPL prompt (single phrase + or a line ending with a "."). *) + let process_ec_input input = + Buffer.clear notices; + (* On the first REPL phrase of each proof, capture the bullet stack + the LOAD prefix left so COMMIT can avoid token collisions with + it. Subsequent calls return [None] and don't clobber the snapshot. *) + (match EcCommands.disable_repl_bullets () with + | None -> () + | Some _ as snapshot -> prior_bullets := snapshot); + let reader = EcIo.from_string input in + let last_src = ref "" in + begin try + let (src, prog) = EcIo.xparse reader in + let src = String.strip src in + last_src := src; + begin match EcLocation.unloc prog with + | EP.P_Prog (commands, _) -> + List.iter (process_action ~record:true ~src) commands; + Wire.reply_ok_goals () + | EP.P_Undo i -> + EcCommands.undo i; + Transcript.trim i; + Wire.reply_ok_goals () + | EP.P_Exit -> + EcIo.finalize reader; exit 0 + | EP.P_DocComment doc -> + EcCommands.doc_comment doc; + Wire.reply_ok "" + end + with + | EcCommands.Restart -> + do_initialize (); + Transcript.clear (); + Wire.reply_ok "Session restarted" + | e -> + Wire.reply_error (Goals.format_error ~src:!last_src e) + end; + EcIo.finalize reader + in + + (* ------------------------------------------------------------------ *) + (* LOAD "file.ec" [LINE[:COL]] [-nosmt] [-trace]. *) + let module Load = struct + let handle args = + Buffer.clear notices; + let args = String.strip args in + let last_src = ref "" in + let trace_prefix = ref "" in + let exception Trace_failed of exn in + + try + (* Parse quoted or unquoted filename. *) + let filename, rest = + if String.length args > 0 && args.[0] = '"' then + let close = + try String.index_from args 1 '"' + with Not_found -> + failwith "LOAD: unterminated filename" + in + let fn = String.sub args 1 (close - 1) in + let rest = String.strip ( + String.sub args (close + 1) + (String.length args - close - 1)) in + (fn, rest) + else + match String.split_on_char ' ' args with + | [] -> failwith "LOAD: missing filename" + | [f] -> (f, "") + | f :: rest -> (f, String.concat " " rest) + in + + (* Parse optional LINE[:COL] and flags (-nosmt, -trace). *) + let upto, nosmt, trace = + let words = + String.split_on_char ' ' rest + |> List.filter (fun s -> s <> "") + in + let nosmt = List.mem "-nosmt" words in + let trace = List.mem "-trace" words in + let words = + List.filter + (fun s -> s <> "-nosmt" && s <> "-trace") + words + in + let upto = match words with + | [] -> None + | [w] -> + begin match String.split_on_char ':' w with + | [line] -> + Some (int_of_string line, None) + | [line; col] -> + Some (int_of_string line, Some (int_of_string col)) + | _ -> failwith "LOAD: invalid LINE[:COL] format" + end + | _ -> failwith "LOAD: unexpected arguments" + in + (upto, nosmt, trace) + in + + begin try + ignore (EcLoader.getkind + (Filename.extension filename) : EcLoader.kind) + with EcLoader.BadExtension ext -> + failwith (Format.sprintf + "unknown file extension: %s" ext) + end; + + do_initialize (); + Hashtbl.clear checkpoints; + Transcript.clear (); + EcCommands.addidir (Filename.dirname filename); + + let reader = EcIo.from_file filename in + + let past_upto (loc : EcLocation.t) = + match upto with + | None -> false + | Some (line, col) -> + let (el, ec) = loc.loc_end in + el > line || (el = line && match col with + | None -> false + | Some c -> ec > c) + in + + let last_loc = ref None in + + (* For -trace: lazy whole-file bytes, used to slice the exact + source text of a sentence by byte offsets. *) + let input_bytes = lazy ( + let ic = open_in_bin filename in + let n = in_channel_length ic in + let b = Bytes.create n in + really_input ic b 0 n; + close_in ic; + Bytes.unsafe_to_string b) + in + let sentence_source (loc : EcLocation.t) = + let s = Lazy.force input_bytes in + let lo = max 0 loc.EcLocation.loc_bchar in + let hi = min (String.length s) loc.EcLocation.loc_echar in + if hi <= lo then "" else String.sub s lo (hi - lo) + in + + (* For -trace: defer execution of the last sentence within the + prefix so we can capture goals before and after it. *) + let pending : (string * EP.global) option ref = ref None in + let flush_pending () = + match !pending with + | None -> () + | Some (src, p) -> + last_src := src; + process_action ~src p; + last_loc := Some p.EP.gl_action.EcLocation.pl_loc; + pending := None + in + let step src p = + let loc = p.EP.gl_action.EcLocation.pl_loc in + if past_upto loc then raise Exit; + if trace then begin + flush_pending (); + pending := Some (src, p) + end else begin + last_src := src; + process_action ~src p; + last_loc := Some loc + end + in + + if nosmt then EcCommands.pragma_check `WeakCheck; + + begin try while true do + let (src, prog) = EcIo.xparse reader in + let src = String.strip src in + match EcLocation.unloc prog with + | EP.P_Prog (commands, locterm) -> + List.iter (step src) commands; + if locterm then raise Exit + | EP.P_Undo i -> + last_src := src; + EcCommands.undo i + | EP.P_Exit -> + raise Exit + | EP.P_DocComment doc -> + last_src := src; + EcCommands.doc_comment doc + done with + | Exit | End_of_file -> () + | e -> + EcIo.finalize reader; + if nosmt then EcCommands.pragma_check `Check; + raise e + end; + + EcIo.finalize reader; + + if nosmt then EcCommands.pragma_check `Check; + + (* If -trace is set, the last in-prefix sentence is still + pending. Run it under goal capture and build the + BEFORE/TACTIC/AFTER/SUMMARY response body. *) + let body = + if not trace then + Goals.goals_to_string () + else + let pre_state = + match !pending with + | None -> `Nothing + | Some _ when not (EcCommands.in_proof ()) -> `NotInProof + | Some (src, p) -> `Ready (src, p) + in + match pre_state with + | `Nothing -> failwith "trace: nothing to trace" + | `NotInProof -> + failwith + "trace: target sentence is not in a proof context" + | `Ready (src, p) -> + let loc = p.EP.gl_action.EcLocation.pl_loc in + let (sl, sc) = loc.EcLocation.loc_start in + let (el, ec) = loc.EcLocation.loc_end in + let before_goals = EcCommands.pp_all_goals () in + let n1 = List.length before_goals in + let buf = Buffer.create 1024 in + let fmt = Format.formatter_of_buffer buf in + Format.fprintf fmt + "=== BEFORE: line %d (col %d) ===@\n" sl sc; + EcCommands.pp_current_goal_or_noproof ~all:false fmt; + Format.fprintf fmt + "@\n=== TACTIC (lines %d:%d - %d:%d) ===@\n%s@\n@\n" + sl sc el ec (sentence_source loc); + last_src := src; + begin + try + process_action ~src p; + last_loc := Some loc; + pending := None; + let after_goals = EcCommands.pp_all_goals () in + let n2 = List.length after_goals in + Format.fprintf fmt + "=== AFTER: line %d (col %d) ===@\n" sl sc; + let before_set = + List.fold_left + (fun s g -> EcMaps.Sstr.add g s) + EcMaps.Sstr.empty before_goals + in + (* The new focused goal always counts as "modified" + (its focus status changed even if its text matches + an old sibling); the rest are printed only if they + didn't appear in BEFORE. *) + let to_print = + match after_goals with + | [] -> [] + | head :: tl -> + head :: + List.filter + (fun g -> not (EcMaps.Sstr.mem g before_set)) + tl + in + begin match to_print with + | [] -> Format.fprintf fmt "(no open goals)@\n" + | _ -> + List.iteri (fun i g -> + if i > 0 then Format.fprintf fmt "@\n"; + Format.fprintf fmt "%s@\n" g) + to_print + end; + Format.fprintf fmt + "@\n=== SUMMARY ===@\nopen goals: %d -> %d@\n" n1 n2; + Format.pp_print_flush fmt (); + Buffer.contents buf + with e -> + Format.fprintf fmt + "=== AFTER: line %d (col %d) ===@\n@\n" + sl sc; + Format.pp_print_flush fmt (); + trace_prefix := Buffer.contents buf; + raise (Trace_failed e) + end + in + + let tag = + let loaded = + match !last_loc with + | None -> "" + | Some loc -> + let (el, _) = loc.EcLocation.loc_end in + Printf.sprintf " [loaded:%s:%d]" filename el + in + loaded ^ Goals.focus_tag () + in + Wire.reply_ok ~tag body + + with + | EcCommands.Restart -> + do_initialize (); + Hashtbl.clear checkpoints; + Transcript.clear (); + Wire.reply_ok "Session restarted" + | Trace_failed e -> + let msg = Goals.format_error ~src:!last_src e in + Wire.reply_error (!trace_prefix ^ msg) + | Failure s -> + Wire.reply_error s + | e -> + Wire.reply_error (Goals.format_error ~src:!last_src e) + end in + + (* ------------------------------------------------------------------ *) + (* Main loop: line-by-line dispatcher. *) + + do_initialize (); + + Printf.printf "READY [uuid:%d]\n\n%!" (EcCommands.uuid ()); + + let multi_buf = Buffer.create 256 in + let in_multi = ref false in + + begin try while true do + let line = input_line stdin in + let line = String.strip line in + + if line = "" then begin + Buffer.clear multi_buf; + in_multi := true + end + else if line = "" && !in_multi then begin + let input = Buffer.contents multi_buf in + Buffer.clear multi_buf; + in_multi := false; + if input <> "" then process_ec_input input + end + else if !in_multi then begin + if Buffer.length multi_buf > 0 then + Buffer.add_char multi_buf ' '; + Buffer.add_string multi_buf line + end + + else if line = "" then + () + else if line = "QUIT" then + exit 0 + else if line = "HELP" then begin + Buffer.clear notices; + let buf = Buffer.create 4096 in + let path = llm_guide_path () in + begin try + let ic = open_in path in + begin try while true do + Buffer.add_char buf (input_char ic) + done with End_of_file -> () end; + close_in ic; + Wire.reply_ok (Buffer.contents buf) + with Sys_error e -> + Wire.reply_error (Printf.sprintf "cannot read guide: %s" e) + end + end + else if line = "UNDO" then begin + Buffer.clear notices; + let uuid = EcCommands.uuid () in + if uuid > 0 then begin + EcCommands.undo (uuid - 1); + Transcript.trim (uuid - 1); + Wire.reply_ok_goals () + end else + Wire.reply_error "nothing to undo" + end + else if line = "GOALS ALL" then begin + Buffer.clear notices; + Wire.reply_ok (Goals.goals_to_string ~all:true ()) + end + else if line = "GOALS" then begin + Buffer.clear notices; + Wire.reply_ok (Goals.goals_to_string ()) + end + else if line = "TREE ALL" then begin + Buffer.clear notices; + Wire.reply_ok (Goals.tree_to_string ~all:true ()) + end + else if line = "TREE" then begin + Buffer.clear notices; + Wire.reply_ok (Goals.tree_to_string ()) + end + else if line = "COMMIT" then begin + Buffer.clear notices; + Wire.reply_ok (Commit.proof_text ()) + end + else if String.starts_with line "FOCUS " || line = "NEXT" then begin + Buffer.clear notices; + let request = + if line = "NEXT" then `Next + else + let arg = String.strip ( + String.sub line 6 (String.length line - 6)) in + try `At (int_of_string arg) + with Failure _ -> `Bad arg + in + match request with + | `Bad arg -> + Wire.reply_error (Printf.sprintf "FOCUS: not an integer: %s" arg) + | _ -> + let entries = EcCommands.pp_tree () in + let n = List.length entries in + let target = + match request with + | `Next -> if n <= 1 then 1 else 2 + | `At k -> k + | `Bad _ -> 1 + in + begin match EcCommands.focus_goal target with + | Ok _ -> Wire.reply_ok_goals () + | Error msg -> Wire.reply_error msg + end + end + else if String.starts_with line "CHECKPOINT " then begin + Buffer.clear notices; + let name = String.strip ( + String.sub line 11 (String.length line - 11)) in + if name = "" then + Wire.reply_error "CHECKPOINT: missing name" + else begin + Hashtbl.replace checkpoints name (EcCommands.uuid ()); + Wire.reply_ok (Printf.sprintf + "checkpoint '%s' set at uuid %d" name (EcCommands.uuid ())) + end + end + else if String.starts_with line "REVERT " then begin + Buffer.clear notices; + let n = String.strip ( + String.sub line 7 (String.length line - 7)) in + let target = + try Some (int_of_string n) + with Failure _ -> Hashtbl.find_opt checkpoints n + in + begin match target with + | None -> + Wire.reply_error (Printf.sprintf + "REVERT: '%s' is not a valid uuid or checkpoint name" n) + | Some target -> + let uuid = EcCommands.uuid () in + if target < 0 || target > uuid then + Wire.reply_error (Printf.sprintf + "REVERT: uuid %d out of range [0, %d]" target uuid) + else begin + EcCommands.undo target; + Transcript.trim target; + Wire.reply_ok_goals () + end + end + end + else if line = "QUIET ON" then begin + Buffer.clear notices; + quiet := true; + Wire.reply_ok "" + end + else if line = "QUIET OFF" then begin + Buffer.clear notices; + quiet := false; + Wire.reply_ok "" + end + else if String.starts_with line "SEARCH " then begin + let query = String.strip ( + String.sub line 7 (String.length line - 7)) in + let query = + if String.ends_with query "." + then String.sub query 0 (String.length query - 1) + else query + in + process_ec_input (Printf.sprintf "search %s." query) + end + else if String.starts_with line "LOAD " then + Load.handle (String.sub line 5 (String.length line - 5)) + else + process_ec_input line + done with + | End_of_file -> () + end; + + exit 0 diff --git a/src/ecLlm.mli b/src/ecLlm.mli new file mode 100644 index 000000000..b12dde1ef --- /dev/null +++ b/src/ecLlm.mli @@ -0,0 +1,11 @@ +(* -------------------------------------------------------------------- *) +(* The LLM coding-agent REPL: an interactive proof-development protocol + over stdin/stdout. Driven via the [easycrypt llm] command. *) + +(* Run the REPL until [QUIT] or EOF, then exit the process. Never + returns. *) +val run : + relocdir:string option + -> boot:bool + -> EcOptions.llm_option + -> 'a From 633c26784ce93a7f84abbbfd749e878581c0182c Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Sat, 30 May 2026 11:01:59 +0200 Subject: [PATCH 07/51] [llm] split REPL dispatcher into Parse and Dispatch submodules The main loop was a flat ~150-line if/else chain that mixed line parsing (substrings, int_of_string, String.starts_with checks) with the actions to take. Split into: module Parse: a [command] variant covering every accepted line shape (Quit, Help, Undo, Goals of [`One|`All], Tree of [`One|`All], Commit, Focus of int, Next, Checkpoint of string, Revert of string, Quiet of bool, Search of string, Load of string, Ec of string, Begin_multi, Done_multi, Multi_line of string, Blank), plus [of_line ~multi_active] which is a stateless string -> command, and [Parse_error] for argument-shape mistakes (e.g. "FOCUS foo"). module Dispatch: a flat pattern match on the parsed command, delegating to small handlers (do_help, do_undo, do_focus_request, do_checkpoint, do_revert, do_quiet, do_search) and to the existing Load/Commit/Goals submodules. Holds the multi-line buffer state (multi_buf/in_multi) since Parse is pure. The main loop becomes 9 lines: read a line, parse to a command, dispatch, catch Parse_error and reply ERROR. Behaviour is unchanged; manual smoke tests cover COMMIT, TREE, FOCUS (good and bad arg), CHECKPOINT/REVERT, SEARCH, multi-line input, and the prior-bullets collision case. --- src/ecLlm.ml | 301 +++++++++++++++++++++++++++++++-------------------- 1 file changed, 184 insertions(+), 117 deletions(-) diff --git a/src/ecLlm.ml b/src/ecLlm.ml index 7d2cbdde6..dc89c2a12 100644 --- a/src/ecLlm.ml +++ b/src/ecLlm.ml @@ -703,38 +703,99 @@ let run ~relocdir ~boot (llmopts : EcOptions.llm_option) = (* ------------------------------------------------------------------ *) (* Main loop: line-by-line dispatcher. *) - do_initialize (); + (* ------------------------------------------------------------------ *) + (* Surface command vocabulary. Parsing turns each stdin line into one + of these, and dispatch is a flat pattern-match. Argument + parsing/validation lives in [Parse]; commands that interact with + mutable state (checkpoints table, multi-line buffer) carry only + the raw user-supplied data and let [Dispatch] do the lookup. *) + let module Parse = struct + type command = + | Quit + | Help + | Undo + | Goals of [`One | `All] + | Tree of [`One | `All] + | Commit + | Focus of int + | Next + | Checkpoint of string + | Revert of string (* uuid-or-name; Dispatch resolves *) + | Quiet of bool + | Search of string (* trailing "." already stripped *) + | Load of string (* raw arg tail; Load.handle parses *) + | Ec of string (* fall-through: raw EasyCrypt input *) + | Begin_multi + | Done_multi + | Multi_line of string + | Blank + + exception Parse_error of string + + let rest n line = + String.strip (String.sub line n (String.length line - n)) + + let parse_focus arg = + try Focus (int_of_string arg) + with Failure _ -> + raise (Parse_error + (Printf.sprintf "FOCUS: not an integer: %s" arg)) + + let parse_checkpoint name = + if name = "" then + raise (Parse_error "CHECKPOINT: missing name"); + Checkpoint name - Printf.printf "READY [uuid:%d]\n\n%!" (EcCommands.uuid ()); + let parse_search query = + let query = + if String.ends_with query "." + then String.sub query 0 (String.length query - 1) + else query + in + Search query + + let of_line ~multi_active (raw : string) : command = + let line = String.strip raw in + if multi_active then + if line = "" then Done_multi + else Multi_line line + else + match line with + | "" -> Begin_multi + | "" -> Blank + | "QUIT" -> Quit + | "HELP" -> Help + | "UNDO" -> Undo + | "GOALS" -> Goals `One + | "GOALS ALL" -> Goals `All + | "TREE" -> Tree `One + | "TREE ALL" -> Tree `All + | "COMMIT" -> Commit + | "NEXT" -> Next + | "QUIET ON" -> Quiet true + | "QUIET OFF" -> Quiet false + | _ when String.starts_with line "FOCUS " -> + parse_focus (rest 6 line) + | _ when String.starts_with line "CHECKPOINT " -> + parse_checkpoint (rest 11 line) + | _ when String.starts_with line "REVERT " -> + Revert (rest 7 line) + | _ when String.starts_with line "SEARCH " -> + parse_search (rest 7 line) + | _ when String.starts_with line "LOAD " -> + Load (rest 5 line) + | _ -> Ec line + end in + (* ------------------------------------------------------------------ *) + (* Command handlers. Each takes (already-parsed) data and produces a + wire reply via [Wire] (or exits the process). Multi-line state is + held here so [Parse] can stay pure. *) let multi_buf = Buffer.create 256 in - let in_multi = ref false in - - begin try while true do - let line = input_line stdin in - let line = String.strip line in + let in_multi = ref false in - if line = "" then begin - Buffer.clear multi_buf; - in_multi := true - end - else if line = "" && !in_multi then begin - let input = Buffer.contents multi_buf in - Buffer.clear multi_buf; - in_multi := false; - if input <> "" then process_ec_input input - end - else if !in_multi then begin - if Buffer.length multi_buf > 0 then - Buffer.add_char multi_buf ' '; - Buffer.add_string multi_buf line - end - - else if line = "" then - () - else if line = "QUIT" then - exit 0 - else if line = "HELP" then begin + let module Dispatch = struct + let do_help () = Buffer.clear notices; let buf = Buffer.create 4096 in let path = llm_guide_path () in @@ -748,8 +809,8 @@ let run ~relocdir ~boot (llmopts : EcOptions.llm_option) = with Sys_error e -> Wire.reply_error (Printf.sprintf "cannot read guide: %s" e) end - end - else if line = "UNDO" then begin + + let do_undo () = Buffer.clear notices; let uuid = EcCommands.uuid () in if uuid > 0 then begin @@ -758,78 +819,38 @@ let run ~relocdir ~boot (llmopts : EcOptions.llm_option) = Wire.reply_ok_goals () end else Wire.reply_error "nothing to undo" - end - else if line = "GOALS ALL" then begin - Buffer.clear notices; - Wire.reply_ok (Goals.goals_to_string ~all:true ()) - end - else if line = "GOALS" then begin - Buffer.clear notices; - Wire.reply_ok (Goals.goals_to_string ()) - end - else if line = "TREE ALL" then begin - Buffer.clear notices; - Wire.reply_ok (Goals.tree_to_string ~all:true ()) - end - else if line = "TREE" then begin - Buffer.clear notices; - Wire.reply_ok (Goals.tree_to_string ()) - end - else if line = "COMMIT" then begin - Buffer.clear notices; - Wire.reply_ok (Commit.proof_text ()) - end - else if String.starts_with line "FOCUS " || line = "NEXT" then begin + + let do_focus_request request = + (* [request] is the user's intent normalized; [`Next] is "second + sibling unless only one open". *) Buffer.clear notices; - let request = - if line = "NEXT" then `Next - else - let arg = String.strip ( - String.sub line 6 (String.length line - 6)) in - try `At (int_of_string arg) - with Failure _ -> `Bad arg + let entries = EcCommands.pp_tree () in + let n = List.length entries in + let target = + match request with + | `Next -> if n <= 1 then 1 else 2 + | `At k -> k in - match request with - | `Bad arg -> - Wire.reply_error (Printf.sprintf "FOCUS: not an integer: %s" arg) - | _ -> - let entries = EcCommands.pp_tree () in - let n = List.length entries in - let target = - match request with - | `Next -> if n <= 1 then 1 else 2 - | `At k -> k - | `Bad _ -> 1 - in - begin match EcCommands.focus_goal target with - | Ok _ -> Wire.reply_ok_goals () - | Error msg -> Wire.reply_error msg - end - end - else if String.starts_with line "CHECKPOINT " then begin + match EcCommands.focus_goal target with + | Ok _ -> Wire.reply_ok_goals () + | Error msg -> Wire.reply_error msg + + let do_checkpoint name = Buffer.clear notices; - let name = String.strip ( - String.sub line 11 (String.length line - 11)) in - if name = "" then - Wire.reply_error "CHECKPOINT: missing name" - else begin - Hashtbl.replace checkpoints name (EcCommands.uuid ()); - Wire.reply_ok (Printf.sprintf - "checkpoint '%s' set at uuid %d" name (EcCommands.uuid ())) - end - end - else if String.starts_with line "REVERT " then begin + Hashtbl.replace checkpoints name (EcCommands.uuid ()); + Wire.reply_ok (Printf.sprintf + "checkpoint '%s' set at uuid %d" name (EcCommands.uuid ())) + + let do_revert spec = Buffer.clear notices; - let n = String.strip ( - String.sub line 7 (String.length line - 7)) in let target = - try Some (int_of_string n) - with Failure _ -> Hashtbl.find_opt checkpoints n + try Some (int_of_string spec) + with Failure _ -> Hashtbl.find_opt checkpoints spec in - begin match target with + match target with | None -> Wire.reply_error (Printf.sprintf - "REVERT: '%s' is not a valid uuid or checkpoint name" n) + "REVERT: '%s' is not a valid uuid or checkpoint name" spec) | Some target -> let uuid = EcCommands.uuid () in if target < 0 || target > uuid then @@ -840,32 +861,78 @@ let run ~relocdir ~boot (llmopts : EcOptions.llm_option) = Transcript.trim target; Wire.reply_ok_goals () end - end - end - else if line = "QUIET ON" then begin - Buffer.clear notices; - quiet := true; - Wire.reply_ok "" - end - else if line = "QUIET OFF" then begin + + let do_quiet on = Buffer.clear notices; - quiet := false; + quiet := on; Wire.reply_ok "" - end - else if String.starts_with line "SEARCH " then begin - let query = String.strip ( - String.sub line 7 (String.length line - 7)) in - let query = - if String.ends_with query "." - then String.sub query 0 (String.length query - 1) - else query - in + + let do_search query = process_ec_input (Printf.sprintf "search %s." query) - end - else if String.starts_with line "LOAD " then - Load.handle (String.sub line 5 (String.length line - 5)) - else - process_ec_input line + + let do_begin_multi () = + Buffer.clear multi_buf; + in_multi := true + + let do_done_multi () = + let input = Buffer.contents multi_buf in + Buffer.clear multi_buf; + in_multi := false; + if input <> "" then process_ec_input input + + let do_multi_line s = + if Buffer.length multi_buf > 0 then + Buffer.add_char multi_buf ' '; + Buffer.add_string multi_buf s + + let run (cmd : Parse.command) = + match cmd with + | Blank -> () + | Quit -> exit 0 + | Help -> do_help () + | Undo -> do_undo () + | Goals `One -> + Buffer.clear notices; + Wire.reply_ok (Goals.goals_to_string ()) + | Goals `All -> + Buffer.clear notices; + Wire.reply_ok (Goals.goals_to_string ~all:true ()) + | Tree `One -> + Buffer.clear notices; + Wire.reply_ok (Goals.tree_to_string ()) + | Tree `All -> + Buffer.clear notices; + Wire.reply_ok (Goals.tree_to_string ~all:true ()) + | Commit -> + Buffer.clear notices; + Wire.reply_ok (Commit.proof_text ()) + | Focus k -> do_focus_request (`At k) + | Next -> do_focus_request `Next + | Checkpoint n -> do_checkpoint n + | Revert s -> do_revert s + | Quiet on -> do_quiet on + | Search q -> do_search q + | Load args -> Load.handle args + | Ec input -> process_ec_input input + | Begin_multi -> do_begin_multi () + | Done_multi -> do_done_multi () + | Multi_line s -> do_multi_line s + end in + + (* ------------------------------------------------------------------ *) + (* Main loop. *) + + do_initialize (); + + Printf.printf "READY [uuid:%d]\n\n%!" (EcCommands.uuid ()); + + begin try while true do + let line = input_line stdin in + (try + let cmd = Parse.of_line ~multi_active:!in_multi line in + Dispatch.run cmd + with Parse.Parse_error msg -> + Wire.reply_error msg) done with | End_of_file -> () end; From 6c657ddcea5601eb88794e4f05eddc3d46cff84a Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Sun, 31 May 2026 07:47:08 +0200 Subject: [PATCH 08/51] [llm] diagnose missing-argument REPL commands at parse time Before, lines like "CHECKPOINT" (no trailing argument) silently fell through to EC input because the dispatcher used [String.starts_with "CHECKPOINT "] -- requiring the trailing space that [String.strip] on the input line had just removed. The user saw EC's generic "parse error" instead of a command-specific message. Same shape applied to LOAD, FOCUS, REVERT, and SEARCH. Introduce a small [keyword_arg kw line] helper that accepts both [line = kw] and [line = kw ^ " " ^ ...], returning the stripped argument tail. Each prefix command now routes to its own parser even when the argument is empty, and each parser produces a specific [Parse_error] message: FOCUS -> "FOCUS: missing argument" CHECKPOINT -> "CHECKPOINT: missing name" REVERT -> "REVERT: missing uuid or checkpoint name" SEARCH -> "SEARCH: missing query" LOAD -> "LOAD: missing filename" (Load.handle, which already had this branch but it was unreachable) --- src/ecLlm.ml | 52 ++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 38 insertions(+), 14 deletions(-) diff --git a/src/ecLlm.ml b/src/ecLlm.ml index dc89c2a12..174734b4e 100644 --- a/src/ecLlm.ml +++ b/src/ecLlm.ml @@ -445,9 +445,10 @@ let run ~relocdir ~boot (llmopts : EcOptions.llm_option) = let exception Trace_failed of exn in try + if args = "" then failwith "LOAD: missing filename"; (* Parse quoted or unquoted filename. *) let filename, rest = - if String.length args > 0 && args.[0] = '"' then + if args.[0] = '"' then let close = try String.index_from args 1 '"' with Not_found -> @@ -464,6 +465,7 @@ let run ~relocdir ~boot (llmopts : EcOptions.llm_option) = | [f] -> (f, "") | f :: rest -> (f, String.concat " " rest) in + if filename = "" then failwith "LOAD: missing filename"; (* Parse optional LINE[:COL] and flags (-nosmt, -trace). *) let upto, nosmt, trace = @@ -732,10 +734,23 @@ let run ~relocdir ~boot (llmopts : EcOptions.llm_option) = exception Parse_error of string - let rest n line = - String.strip (String.sub line n (String.length line - n)) + (* Match [kw] as a prefix: succeeds on exactly [kw] (no argument) + or [kw ^ " " ^ ...] (with argument), returning the stripped + argument tail. Returns [None] otherwise. This recognises both + "CHECKPOINT" and "CHECKPOINT foo" the same way, so we can + diagnose the missing-name case ourselves instead of falling + through to EC's parser. *) + let keyword_arg kw line = + if line = kw then Some "" + else if String.starts_with line (kw ^ " ") then + let n = String.length kw + 1 in + Some (String.strip + (String.sub line n (String.length line - n))) + else None let parse_focus arg = + if arg = "" then + raise (Parse_error "FOCUS: missing argument"); try Focus (int_of_string arg) with Failure _ -> raise (Parse_error @@ -746,7 +761,15 @@ let run ~relocdir ~boot (llmopts : EcOptions.llm_option) = raise (Parse_error "CHECKPOINT: missing name"); Checkpoint name + let parse_revert spec = + if spec = "" then + raise (Parse_error + "REVERT: missing uuid or checkpoint name"); + Revert spec + let parse_search query = + if query = "" then + raise (Parse_error "SEARCH: missing query"); let query = if String.ends_with query "." then String.sub query 0 (String.length query - 1) @@ -754,6 +777,11 @@ let run ~relocdir ~boot (llmopts : EcOptions.llm_option) = in Search query + let parse_load args = + (* [Load.handle] accepts an empty argument and reports a + specific error; keep that responsibility there. *) + Load args + let of_line ~multi_active (raw : string) : command = let line = String.strip raw in if multi_active then @@ -774,17 +802,13 @@ let run ~relocdir ~boot (llmopts : EcOptions.llm_option) = | "NEXT" -> Next | "QUIET ON" -> Quiet true | "QUIET OFF" -> Quiet false - | _ when String.starts_with line "FOCUS " -> - parse_focus (rest 6 line) - | _ when String.starts_with line "CHECKPOINT " -> - parse_checkpoint (rest 11 line) - | _ when String.starts_with line "REVERT " -> - Revert (rest 7 line) - | _ when String.starts_with line "SEARCH " -> - parse_search (rest 7 line) - | _ when String.starts_with line "LOAD " -> - Load (rest 5 line) - | _ -> Ec line + | _ -> + match keyword_arg "FOCUS" line with Some a -> parse_focus a | None -> + match keyword_arg "CHECKPOINT" line with Some a -> parse_checkpoint a | None -> + match keyword_arg "REVERT" line with Some a -> parse_revert a | None -> + match keyword_arg "SEARCH" line with Some a -> parse_search a | None -> + match keyword_arg "LOAD" line with Some a -> parse_load a | None -> + Ec line end in (* ------------------------------------------------------------------ *) From c21749d1b9baefad8e1f266c251f7c63dca49a28 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Sun, 31 May 2026 08:00:48 +0200 Subject: [PATCH 09/51] [llm] nested TREE and dotted-path FOCUS Introduce a frame tree derived from pr_opened + parent_of: each open leaf's chain of multi-child ancestors (skipping single-child continuations) becomes its path through the tree. The same data structure backs both TREE rendering and FOCUS path lookup. TREE now shows depth-indented entries labelled with dotted paths matching what FOCUS accepts. Leading singleton frames are unwrapped: when all opens share an outermost split, the rendering starts at that split's branches, not at a redundant [1.] wrapper. FOCUS N1.N2.N3 walks the tree following each component and focuses the resolved leaf. A single integer (FOCUS k) still works (degree-1 path). The path must resolve to a leaf; selecting an internal frame yields "FOCUS: path must select a leaf goal, not a frame" and overshooting a leaf yields "FOCUS: path overshoots a leaf goal". After [split. split. split.] on [((a /\ b) /\ c) /\ d], TREE prints: [1.1.1] a = a <- focused [1.1.2] b = b [1.2] c = c [2] d = d and FOCUS 1.2 selects c, FOCUS 2 selects d, FOCUS 1.1.2 selects b. NEXT semantics unchanged. Flat proofs render unindented as before. --- doc/llm/CLAUDE.md | 4 +- src/ecLlm.ml | 221 +++++++++++++++++++++++++++++++++++++--------- 2 files changed, 183 insertions(+), 42 deletions(-) diff --git a/doc/llm/CLAUDE.md b/doc/llm/CLAUDE.md index a06725467..1c215cd82 100644 --- a/doc/llm/CLAUDE.md +++ b/doc/llm/CLAUDE.md @@ -61,9 +61,9 @@ These are protocol-level commands, not EasyCrypt syntax: | `REVERT ` | Revert to a specific state (by uuid or checkpoint name) | | `GOALS` | Print the current goal (first subgoal only, with remaining count) | | `GOALS ALL` | Print all subgoals | -| `TREE` | List open subgoals as `[N] `, marking the focused one | +| `TREE` | List open subgoals with dotted-path labels showing nesting, marking the focused one | | `TREE ALL` | Same as `TREE`, but with full goal bodies | -| `FOCUS N` | Rotate focus so subgoal `[N]` (from `TREE`) becomes the focused goal | +| `FOCUS P` | Rotate focus to the leaf addressed by path `P` (`N` or `N1.N2.N3...`) | | `NEXT` | Rotate focus to the next subgoal (equivalent to `FOCUS 2`) | | `COMMIT` | Emit recorded REPL phrases as a bulleted proof body (works under `+strict_bullets`) | | `CHECKPOINT ` | Save current uuid under a name for later `REVERT` | diff --git a/src/ecLlm.ml b/src/ecLlm.ml index 174734b4e..9a7ce317c 100644 --- a/src/ecLlm.ml +++ b/src/ecLlm.ml @@ -146,14 +146,108 @@ let run ~relocdir ~boot (llmopts : EcOptions.llm_option) = Format.pp_print_flush fmt (); Buffer.contents buf - (* Render the focus-tree of open subgoals. [all=false] gives a - one-line digest per goal; [all=true] gives the full goal body. *) - let tree_to_string ?(all=false) () = - let entries = EcCommands.pp_tree ~all () in - match entries with - | [] -> "No active proof.\n" - | _ -> - let buf = Buffer.create 256 in + (* Inline focus annotation ([focus: 1/N]) appended to reply tags + whenever the active proof has >=2 open subgoals. *) + let focus_tag () = + match EcCommands.pp_tree () with + | _ :: _ :: _ as entries -> + Printf.sprintf " [focus: 1/%d]" (List.length entries) + | _ -> "" + end in + + (* ------------------------------------------------------------------ *) + (* Frame tree: group currently-open goals by their shared multi-child + ancestors. Used by [Tree] (rendering) and [Focus] (path lookup). + The tree is a *derivation*: it depends only on [pr_opened] and + [parent_of], no recorded transcript. *) + let module FrameTree = struct + (* Internal nodes are split-point frames; leaves carry a handle + (the open goal), its index in [pr_opened] (1-based, used by + [EcCoreGoal.rotate_focus]), and its rendered text. *) + type node = + | Frame of node list (* >=2 child branches *) + | Leaf of + { idx : int (* 1-based in pr_opened *) + ; focused : bool (* idx = 1 *) + ; text : string } (* one-line conclusion *) + + (* Multi-child ancestors of [h], outermost first (= root-most + split first, deepest split last). This ordering means leaves + sharing the same OUTER frame will agree on the chain's first + element, which is what [group] partitions on. *) + let split_chain h = + let rec walk h acc = + match EcCommands.parent_of h with + | None -> acc + | Some p -> + match EcCommands.children_of p with + | [_] -> walk p acc + | _ -> walk p (p :: acc) + in + (* [walk] prepends each ancestor as we go up; the result has + outermost at the FRONT (we add it last). No reverse needed. *) + walk h [] + + (* Build the tree by grouping leaves with a common ancestor prefix. + [leaves] is a list of (chain, leaf) in [pr_opened] order. The + grouping is done recursively on the head of each chain. *) + let rec group (leaves : (EcCoreGoal.handle list * node) list) : node list = + let rec runs acc = function + | [] -> List.rev acc + | (chain, leaf) :: rest -> + match chain with + | [] -> runs (`Bare leaf :: acc) rest + | hd :: tl -> + let same_head, others = + List.partition_map (fun (c, l) -> + match c with + | h :: tail when EcCoreGoal.eq_handle h hd -> + Left (tail, l) + | _ -> Right (c, l)) + rest + in + runs (`Group ((tl, leaf) :: same_head) :: acc) others + in + List.map + (function + | `Bare leaf -> leaf + | `Group children -> Frame (group children)) + (runs [] leaves) + + (* Strip leading singleton frames so the top-level forest's + indices match what the user thinks of as "top-level subgoals + of the current frame." When all open leaves descend from a + single outermost split, the top-level forest has one Frame + containing the actual user-visible siblings; unwrap it. *) + let rec unwrap forest = + match forest with + | [Frame children] -> unwrap children + | _ -> forest + + let build () = + let handles = EcCommands.open_handles () in + let texts = EcCommands.pp_tree () in + if handles = [] then [] + else + let leaves = + List.mapi (fun i (h, (_, focused, text)) -> + let leaf = Leaf { idx = i + 1; focused; text } in + (split_chain h, leaf)) + (List.combine handles texts) + in + unwrap (group leaves) + + (* Render the tree with dotted-path labels matching what FOCUS + accepts. [all] requests full goal bodies (we re-query via + [pp_tree ~all:true] keyed by leaf index). *) + let render ?(all=false) () = + let forest = build () in + if forest = [] then "No active proof.\n" + else + let texts_all = + if all then Some (EcCommands.pp_tree ~all:true ()) + else None + in let one_line s = let s = match String.index_opt s '\n' with @@ -165,26 +259,61 @@ let run ~relocdir ~boot (llmopts : EcOptions.llm_option) = then String.sub s 0 (limit - 1) ^ "…" else s in - List.iter (fun (i, focused, text) -> - let marker = if focused then " <- focused" else "" in - if all then - Buffer.add_string buf - (Printf.sprintf "[%d]%s\n%s\n" i marker text) - else - Buffer.add_string buf - (Printf.sprintf "[%d] %s%s\n" i (one_line text) marker) - ) entries; + let buf = Buffer.create 256 in + let rec emit ~depth ~path = function + | Leaf { idx; focused; text } -> + let label = String.concat "." (List.rev_map string_of_int path) in + let marker = if focused then " <- focused" else "" in + for _ = 1 to depth do Buffer.add_string buf " " done; + (match texts_all with + | None -> + Buffer.add_string buf + (Printf.sprintf "[%s] %s%s\n" + label (one_line text) marker) + | Some entries -> + let (_, _, full) = + List.nth entries (idx - 1) + in + Buffer.add_string buf + (Printf.sprintf "[%s]%s\n%s\n" label marker full)) + | Frame children -> + List.iteri (fun i child -> + emit ~depth:(depth + 1) ~path:((i + 1) :: path) child) + children + in + List.iteri (fun i node -> + emit ~depth:0 ~path:[i + 1] node) + forest; Buffer.contents buf - (* Inline focus annotation ([focus: 1/N]) appended to reply tags - whenever the active proof has >=2 open subgoals. *) - let focus_tag () = - match EcCommands.pp_tree () with - | _ :: _ :: _ as entries -> - Printf.sprintf " [focus: 1/%d]" (List.length entries) - | _ -> "" + (* Resolve a dotted path against the tree. Returns [Ok idx] where + [idx] is the 1-based position in [pr_opened] of the selected + leaf, or [Error msg]. *) + let resolve_path (path : int list) : (int, string) result = + let forest = build () in + let rec walk ~components nodes = + match components with + | [] -> Error "FOCUS: path must select a leaf goal" + | k :: rest -> + if k < 1 || k > List.length nodes then + Error (Printf.sprintf + "FOCUS: index %d out of range (1..%d)" + k (List.length nodes)) + else + match List.nth nodes (k - 1), rest with + | Leaf { idx; _ }, [] -> Ok idx + | Leaf _, _ -> + Error "FOCUS: path overshoots a leaf goal" + | Frame _, [] -> + Error "FOCUS: path must select a leaf goal, \ + not a frame" + | Frame kids, _ -> walk ~components:rest kids + in + if forest = [] then Error "FOCUS: no active proof" + else walk ~components:path forest end in + (* ------------------------------------------------------------------ *) (* OK/ERROR/ wire envelope. *) let module Wire = struct @@ -719,7 +848,7 @@ let run ~relocdir ~boot (llmopts : EcOptions.llm_option) = | Goals of [`One | `All] | Tree of [`One | `All] | Commit - | Focus of int + | Focus of int list (* dotted path; [k] = "FOCUS k" *) | Next | Checkpoint of string | Revert of string (* uuid-or-name; Dispatch resolves *) @@ -751,10 +880,17 @@ let run ~relocdir ~boot (llmopts : EcOptions.llm_option) = let parse_focus arg = if arg = "" then raise (Parse_error "FOCUS: missing argument"); - try Focus (int_of_string arg) - with Failure _ -> + let parts = String.split_on_char '.' arg in + let path = + try List.map int_of_string parts + with Failure _ -> + raise (Parse_error + (Printf.sprintf "FOCUS: not a path of integers: %s" arg)) + in + if List.exists (fun k -> k < 1) path then raise (Parse_error - (Printf.sprintf "FOCUS: not an integer: %s" arg)) + (Printf.sprintf "FOCUS: path indices must be >= 1: %s" arg)); + Focus path let parse_checkpoint name = if name = "" then @@ -845,19 +981,24 @@ let run ~relocdir ~boot (llmopts : EcOptions.llm_option) = Wire.reply_error "nothing to undo" let do_focus_request request = - (* [request] is the user's intent normalized; [`Next] is "second - sibling unless only one open". *) + (* [request] is the user's intent normalized: + - [`Next] = rotate to the second open goal (or stay if <=1) + - [`Path p] = resolve dotted path [p] against the frame tree + and focus the matching leaf. *) Buffer.clear notices; - let entries = EcCommands.pp_tree () in - let n = List.length entries in - let target = + let resolved = match request with - | `Next -> if n <= 1 then 1 else 2 - | `At k -> k + | `Next -> + let n = List.length (EcCommands.open_handles ()) in + Ok (if n <= 1 then 1 else 2) + | `Path path -> FrameTree.resolve_path path in - match EcCommands.focus_goal target with - | Ok _ -> Wire.reply_ok_goals () + match resolved with | Error msg -> Wire.reply_error msg + | Ok target -> + match EcCommands.focus_goal target with + | Ok _ -> Wire.reply_ok_goals () + | Error msg -> Wire.reply_error msg let do_checkpoint name = Buffer.clear notices; @@ -923,14 +1064,14 @@ let run ~relocdir ~boot (llmopts : EcOptions.llm_option) = Wire.reply_ok (Goals.goals_to_string ~all:true ()) | Tree `One -> Buffer.clear notices; - Wire.reply_ok (Goals.tree_to_string ()) + Wire.reply_ok (FrameTree.render ()) | Tree `All -> Buffer.clear notices; - Wire.reply_ok (Goals.tree_to_string ~all:true ()) + Wire.reply_ok (FrameTree.render ~all:true ()) | Commit -> Buffer.clear notices; Wire.reply_ok (Commit.proof_text ()) - | Focus k -> do_focus_request (`At k) + | Focus path -> do_focus_request (`Path path) | Next -> do_focus_request `Next | Checkpoint n -> do_checkpoint n | Revert s -> do_revert s From e9472eccab977fc95f82e433dfbd1236c0f50d11 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Sun, 31 May 2026 19:51:21 +0200 Subject: [PATCH 10/51] [llm] update CLAUDE.md with current workflow status Add two workflow sections that were missing or stale: 4. Inspect and navigate nested subgoals with TREE and FOCUS Documents the dotted-path syntax (FOCUS 1.2.3), the [focus: k/N] reply tag, and the fact that TREE labels are dynamic (focus-first, not stable across focus changes). 5. Build a +strict_bullets-friendly proof with COMMIT Documents how COMMIT replays the recorded transcript and inserts bullets, plus the cycle (-, +, *, --, ++, **, ...) and the prior- bullet collision avoidance. Re-number the existing QUIET and SEARCH sections from 4/5 to 6/7. Fix one outdated pitfall ("subgoals must be closed in order") -- FOCUS path now lets the agent address them in any order. Reflects what's live in EcLlm; the protocol/meta-command table was already up to date. --- doc/llm/CLAUDE.md | 72 ++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 68 insertions(+), 4 deletions(-) diff --git a/doc/llm/CLAUDE.md b/doc/llm/CLAUDE.md index 1c215cd82..f18137631 100644 --- a/doc/llm/CLAUDE.md +++ b/doc/llm/CLAUDE.md @@ -177,7 +177,69 @@ REVERT before_split apply H. ← try a different approach ``` -**4. Use QUIET mode to save tokens during bulk tactic application:** +**4. Inspect and navigate nested subgoals with `TREE` and `FOCUS`:** + +When a tactic opens multiple subgoals, the engine focuses the first +one. By default subsequent tactics act on it; siblings wait their +turn. Use `TREE` to see the structure, including nested splits: + +``` +TREE +→ OK [uuid:N] + [1.1.1] x = 0 <- focused + [1.1.2] y = 1 + [1.2] z = 2 +[2] w = 3 + +``` + +The labels are dotted paths. `FOCUS P` rotates focus to the leaf at +path `P`: + +``` +FOCUS 1.2 ← work on `z = 2` +FOCUS 2 ← work on `w = 3` +FOCUS 1.1.1 ← back to `x = 0` +``` + +`FOCUS k` (a single integer) targets the k-th open goal in the flat +listing. `NEXT` is shorthand for `FOCUS 2`. Selecting an internal +frame errors (`FOCUS: path must select a leaf goal, not a frame`). + +Replies carry a `[focus: k/N]` tag when more than one goal is open +(e.g. `OK [uuid:42] [focus: 1/3]`) so you always know which goal the +next tactic will hit. **TREE labels are not stable across focus +changes** — `FOCUS 1.2` from one state may name a different goal in +another, because the tree always shows the focused goal first. + +**5. Build a `+strict_bullets`-friendly proof with `COMMIT`:** + +The REPL records every successful interactive phrase. `COMMIT` walks +the proof DAG and emits the recorded tactics with bullets inserted +at every multi-child split. The output is a proof body that compiles +under `pragma +strict_bullets`: + +``` +LOAD "myfile.ec" 42 +split. +- rewrite H. trivial. ← REPL accepts the unbulleted form +- exact hq. +COMMIT +→ OK [uuid:N] +split. +- rewrite H. trivial. +- exact hq. + +``` + +Bullet characters cycle through `-`, `+`, `*`, `--`, `++`, `**`, ... +and are chosen to avoid colliding with any frames the LOAD prefix +already opened. Use `COMMIT` once the proof is complete (or at any +checkpoint) and paste the result back into the source file. + +`UNDO` / `REVERT` trim the COMMIT transcript automatically. + +**6. Use QUIET mode to save tokens during bulk tactic application:** ``` QUIET ON @@ -188,7 +250,7 @@ QUIET OFF GOALS ``` -**5. Search for lemmas using patterns:** +**7. Search for lemmas using patterns:** EasyCrypt `search` uses pattern syntax, not keywords. Use `_` as wildcard: @@ -267,8 +329,10 @@ SEARCH (_ %/ _) - `by` closes **all** remaining subgoals. If it fails, the error refers to the first unclosed goal, which may not be the intended one. -- When a tactic generates multiple subgoals, each subgoal must be - closed in order. Use `GOALS ALL` or `TREE` to see them all. +- When a tactic generates multiple subgoals, the engine focuses the + first one. Address them in any order via `FOCUS path`, or in the + default order by closing each in turn. Use `TREE` or `GOALS ALL` + to see what's open. - When more than one subgoal is open, replies carry a `[focus: k/N]` tag (e.g. `OK [uuid:42] [focus: 1/3]`) so you know which one the next tactic will hit. From ea32ba5883b947069483668724facec775258140 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Wed, 15 Jul 2026 13:38:24 +0200 Subject: [PATCH 11/51] [llm] add -eval STR for scripted one-shot invocations Add llmo_eval : string option to llm_option, wired to the -eval CLI flag. When set, EcLlm.run splits the argument on newlines and feeds each line through the same Parse/Dispatch pipeline as stdin, then exits at end of script (no QUIT required, though QUIT still works). Enables cheap scripted use without piping: easycrypt llm -eval 'LOAD "myfile.ec" 42 GOALS COMMIT' The stdin path is unchanged when -eval is not given. Docs updated. --- doc/llm/CLAUDE.md | 10 ++++++++++ src/ecLlm.ml | 17 ++++++++++++++++- src/ecOptions.ml | 7 +++++-- src/ecOptions.mli | 1 + 4 files changed, 32 insertions(+), 3 deletions(-) diff --git a/doc/llm/CLAUDE.md b/doc/llm/CLAUDE.md index f18137631..df832874d 100644 --- a/doc/llm/CLAUDE.md +++ b/doc/llm/CLAUDE.md @@ -23,6 +23,16 @@ available. Use `-help` to print this guide and exit: easycrypt llm -help ``` +Use `-eval STR` to feed a newline-separated script instead of reading +stdin. Useful for scripted callers and CI: the REPL runs the given +commands and exits (implicit end-of-input, no `QUIT` required): + +``` +easycrypt llm -eval 'LOAD "myfile.ec" 42 +GOALS +COMMIT' +``` + ### Protocol **Startup.** EasyCrypt prints a `READY` message and waits for input: diff --git a/src/ecLlm.ml b/src/ecLlm.ml index 9a7ce317c..5f7c2565d 100644 --- a/src/ecLlm.ml +++ b/src/ecLlm.ml @@ -1091,8 +1091,23 @@ let run ~relocdir ~boot (llmopts : EcOptions.llm_option) = Printf.printf "READY [uuid:%d]\n\n%!" (EcCommands.uuid ()); + (* Input source: stdin by default, or the -eval string when given. + For -eval, we split on newlines up front (no lazy channel), which + keeps the driver simple and avoids ever touching stdin. *) + let read_line : unit -> string = + match llmopts.llmo_eval with + | None -> + fun () -> input_line stdin + | Some script -> + let lines = ref (String.split_on_char '\n' script) in + fun () -> + match !lines with + | [] -> raise End_of_file + | l :: tl -> lines := tl; l + in + begin try while true do - let line = input_line stdin in + let line = read_line () in (try let cmd = Parse.of_line ~multi_active:!in_multi line in Dispatch.run cmd diff --git a/src/ecOptions.ml b/src/ecOptions.ml index ba47648ae..0c3721a47 100644 --- a/src/ecOptions.ml +++ b/src/ecOptions.ml @@ -51,6 +51,7 @@ and doc_option = { and llm_option = { llmo_provers : prv_options; llmo_help : bool; + llmo_eval : string option; } and prv_options = { @@ -382,7 +383,8 @@ let specs = { ("llm", "LLM-friendly interactive mode", [ `Group "loader"; `Group "provers"; - `Spec ("help", `Flag, "Print the LLM agent guide and exit")]); + `Spec ("help", `Flag , "Print the LLM agent guide and exit"); + `Spec ("eval", `String, "Run the given commands (newline-separated) and exit, in lieu of reading stdin")]); ("cli", "Run EasyCrypt top-level", [ `Group "loader"; @@ -573,7 +575,8 @@ let doc_options_of_values values input = let llm_options_of_values ini values = { llmo_provers = prv_options_of_values ini values; - llmo_help = get_flag "help" values; } + llmo_help = get_flag "help" values; + llmo_eval = get_string "eval" values; } (* -------------------------------------------------------------------- *) let parse getini argv = diff --git a/src/ecOptions.mli b/src/ecOptions.mli index e5c4b5f04..80e747589 100644 --- a/src/ecOptions.mli +++ b/src/ecOptions.mli @@ -47,6 +47,7 @@ and doc_option = { and llm_option = { llmo_provers : prv_options; llmo_help : bool; + llmo_eval : string option; } and prv_options = { From dcef713e77d09d03cc9aebb7bbf62ab75aed4fed Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Wed, 15 Jul 2026 13:55:16 +0200 Subject: [PATCH 12/51] [llm] use apply_pragma_option for loader pragmas (align with ec.ml compile path) --- src/ecLlm.ml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/ecLlm.ml b/src/ecLlm.ml index 5f7c2565d..4a011d154 100644 --- a/src/ecLlm.ml +++ b/src/ecLlm.ml @@ -113,7 +113,7 @@ let run ~relocdir ~boot (llmopts : EcOptions.llm_option) = ~boot ~checkmode ~checkproof:true; initialized := true; (try - List.iter EcCommands.apply_pragma prvopts.prvo_pragmas + List.iter EcCommands.apply_pragma_option prvopts.prvo_pragmas with EcCommands.InvalidPragma x -> EcScope.hierror "invalid pragma: `%s'\n%!" x); EcCommands.addnotifier notifier; From f4846364430eb3c8e1ede3bc27166ab4c697ae26 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Fri, 17 Jul 2026 20:39:29 +0200 Subject: [PATCH 13/51] Add -stdlib DIR flag to override the built-in standard library Add ldro_stdlib : string list to ldr_options, wired to a repeatable -stdlib CLI flag. When non-empty, its entries replace Sites.theories as the roots of the System-namespace loading loop -- both the prelude add and (unless -boot) the recursive add. When empty (the default), the built-in Sites.theories list is used, preserving today's behaviour. This is stronger than -boot alone: -boot only suppresses the recursive-System add and still injects /prelude, so a caller wanting to fully sidestep the shipped stdlib currently has no clean way to do it. With -stdlib, the built-in Sites.theories is never touched. The two flags remain independent and compose: -stdlib DIR : DIR/prelude (System) + DIR (recursive System) -boot : /prelude (System) only -boot -stdlib DIR : DIR/prelude (System) only -stdlib is also propagated to subprocess invocations of runtest so child ec calls see the same effective load path. --- src/ec.ml | 18 ++++++++++++++++-- src/ecOptions.ml | 20 ++++++++++++-------- src/ecOptions.mli | 8 ++++++-- 3 files changed, 34 insertions(+), 12 deletions(-) diff --git a/src/ec.ml b/src/ec.ml index 5d7244a72..954cc433c 100644 --- a/src/ec.ml +++ b/src/ec.ml @@ -328,6 +328,11 @@ let main () = ["-boot"] else [] in + let stdlib = + options.o_options.o_loader.ldro_stdlib + |> List.map (fun d -> ["-stdlib"; d]) + |> List.flatten in + let idirs = options.o_options.o_loader.ldro_idirs |> List.map (fun (pfx, name, rec_) -> @@ -341,7 +346,7 @@ let main () = maxjobs; timeout; cpufactor; ppwidth; provers; quorum ; pragmas ; checkall; profile; why3srv ; why3 ; - reloc ; noevict; boot ; idirs ; + reloc ; noevict; boot ; stdlib ; idirs ; ] in @@ -420,11 +425,20 @@ let main () = let ldropts = options.o_options.o_loader in begin + (* [-stdlib DIR] (repeatable) fully replaces the built-in + [Sites.theories] roots. This is stronger than [-boot], which + only skips the recursive-System add but still injects + [/prelude]. *) + let theories = + match ldropts.ldro_stdlib with + | [] -> Sites.theories + | ds -> ds + in List.iter (fun theory -> EcCommands.addidir ~namespace:`System (Filename.concat theory "prelude"); if not ldropts.ldro_boot then EcCommands.addidir ~namespace:`System ~recursive:true theory - ) Sites.theories; + ) theories; List.iter (fun (onm, name, isrec) -> EcCommands.addidir ?namespace:(omap (fun nm -> `Named nm) onm) diff --git a/src/ecOptions.ml b/src/ecOptions.ml index 0c3721a47..e4746cc18 100644 --- a/src/ecOptions.ml +++ b/src/ecOptions.ml @@ -68,8 +68,9 @@ and prv_options = { } and ldr_options = { - ldro_idirs : (string option * string * bool) list; - ldro_boot : bool; + ldro_idirs : (string option * string * bool) list; + ldro_boot : bool; + ldro_stdlib : string list; } and glb_options = { @@ -423,9 +424,10 @@ let specs = { ]); ("loader", "Options related to loader", [ - `Spec ("I" , `String, "Add to the list of include directories"); - `Spec ("R" , `String, "Recursively add to the list of include directories"); - `Spec ("boot", `Flag , "Don't load prelude")]) + `Spec ("I" , `String, "Add to the list of include directories"); + `Spec ("R" , `String, "Recursively add to the list of include directories"); + `Spec ("stdlib", `String, "Use as a standard-library root (System namespace, prelude + recursive), replacing the built-in one; repeatable"); + `Spec ("boot" , `Flag , "Don't load prelude")]) ] } @@ -485,8 +487,9 @@ let dirs_of_env = (* -------------------------------------------------------------------- *) let ldr_options_of_values ~env ?(ini = []) values = + let stdlib = get_strings "stdlib" values in if get_flag "boot" values then - { ldro_idirs = []; ldro_boot = true; } + { ldro_idirs = []; ldro_boot = true; ldro_stdlib = stdlib; } else let add_rec (fl : bool) ((nm, x) : string option * string) = (nm, x, fl) in @@ -500,8 +503,9 @@ let ldr_options_of_values ~env ?(ini = []) values = let rdirs = List.map (add_rec true) rdirs in let idirs_R = List.map (add_rec true) (List.map parse_idir (get_strings "R" values)) in - { ldro_idirs = idirs @ idirs_I @ rdirs @ idirs_R; - ldro_boot = false; } + { ldro_idirs = idirs @ idirs_I @ rdirs @ idirs_R; + ldro_boot = false; + ldro_stdlib = stdlib; } let glb_options_of_values ~env ini values = let why3 = diff --git a/src/ecOptions.mli b/src/ecOptions.mli index 80e747589..def341dfd 100644 --- a/src/ecOptions.mli +++ b/src/ecOptions.mli @@ -64,8 +64,12 @@ and prv_options = { } and ldr_options = { - ldro_idirs : (string option * string * bool) list; - ldro_boot : bool; + ldro_idirs : (string option * string * bool) list; + ldro_boot : bool; + ldro_stdlib : string list; + (* When non-empty, these directories replace the built-in + [Sites.theories] for prelude and recursive-System namespace + loading. Empty means "use the built-in stdlib". *) } and glb_options = { From 5c221fcfcb443ec77fca56c0942a72ade10f3bee Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Thu, 23 Jul 2026 09:56:00 +0200 Subject: [PATCH 14/51] [llm] scripted -eval runs exit nonzero when any command errors --- src/ecLlm.ml | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/src/ecLlm.ml b/src/ecLlm.ml index 4a011d154..bf6a9328d 100644 --- a/src/ecLlm.ml +++ b/src/ecLlm.ml @@ -316,6 +316,8 @@ let run ~relocdir ~boot (llmopts : EcOptions.llm_option) = (* ------------------------------------------------------------------ *) (* OK/ERROR/ wire envelope. *) + let had_error = ref false in + let module Wire = struct let reply_ok ?(tag="") body = let n = Buffer.contents notices in @@ -336,6 +338,7 @@ let run ~relocdir ~boot (llmopts : EcOptions.llm_option) = else reply_ok ~tag (Goals.goals_to_string ~all ()) let reply_error msg = + had_error := true; let goals = Goals.goals_to_string () in Printf.printf "ERROR [uuid:%d]\n%s\n" (EcCommands.uuid ()) msg; if goals <> "" then begin @@ -1117,4 +1120,7 @@ let run ~relocdir ~boot (llmopts : EcOptions.llm_option) = | End_of_file -> () end; - exit 0 + (* Scripted runs (-eval) report in-band errors through the exit + status, so that automation does not mistake an ERROR reply for + success. Interactive sessions keep exiting 0. *) + exit (if llmopts.llmo_eval <> None && !had_error then 1 else 0) From 68e70575c32e96aa1a6853d18437bce8e53a7ed6 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Thu, 23 Jul 2026 10:11:11 +0200 Subject: [PATCH 15/51] [llm] LOAD applies the easycrypt.project of the loaded file The REPL discovers files at run time, after option parsing, so the project-file context that the batch compiler resolves from its command-line input was never applied: LOADing a file whose project supplies include dirs, provers, timeout or pragmas failed to locate theories (and checked with default prover settings). LOAD now resolves the easycrypt.project attached to the loaded file (same parent-directory walk as the compiler), extends the load path with its idirs/rdirs, re-initializes with the project's prover options overlaid on the command-line ones, and sets the current path to the file's directory, mirroring the compile path. Note: a project scalar (e.g. timeout) now takes precedence over the same setting given on the llm command line, since the two cannot be told apart after option parsing. --- src/ec.ml | 32 +++++++++++++++++--------------- src/ecLlm.ml | 37 ++++++++++++++++++++++++++++++++----- src/ecLlm.mli | 5 ++++- src/ecOptions.ml | 41 +++++++++++++++++++++++++++++++++++++++++ src/ecOptions.mli | 10 ++++++++++ 5 files changed, 104 insertions(+), 21 deletions(-) diff --git a/src/ec.ml b/src/ec.ml index 954cc433c..90630a31e 100644 --- a/src/ec.ml +++ b/src/ec.ml @@ -158,7 +158,7 @@ let main () = let (module Sites) = EcRelocate.sites in (* Parse command line arguments *) - let conffiles, options = + let conffiles, projini, options = let sysfile = let xdgini = XDG.Config.file @@ -220,6 +220,19 @@ let main () = exit 1 in + (* The [easycrypt.project] context of a file (walking up from the + file's directory; from the cwd when no file is given). Also used + by the LLM REPL to reconfigure per loaded file. *) + let projini (path : string option) = + Option.bind (projfile path) (fun conffile -> + Option.map + (fun ini -> { + inic_ini = ini; + inic_root = Some (Filename.dirname conffile); + }) + (read_ini_file conffile) + ) in + let getini (path : string option) = let inisys = List.filter_map @@ -230,20 +243,9 @@ let main () = conffiles in - let iniproj = - Option.bind (projfile path) (fun conffile -> - Option.map - (fun ini -> { - inic_ini = ini; - inic_root = Some (Filename.dirname conffile); - }) - (read_ini_file conffile) - ) - in - - List.ocons iniproj inisys in + List.ocons (projini path) inisys in - (conffiles, EcOptions.parse_cmdline ~ini:getini Sys.argv) in + (conffiles, projini, EcOptions.parse_cmdline ~ini:getini Sys.argv) in (* Execution of eager commands *) begin @@ -584,7 +586,7 @@ let main () = end | `Llm llmopts -> - EcLlm.run ~relocdir ~boot:ldropts.ldro_boot llmopts + EcLlm.run ~relocdir ~boot:ldropts.ldro_boot ~projini llmopts | `Runtest _ -> (* Eagerly executed *) diff --git a/src/ecLlm.ml b/src/ecLlm.ml index bf6a9328d..ca00b91c7 100644 --- a/src/ecLlm.ml +++ b/src/ecLlm.ml @@ -35,7 +35,7 @@ let print_llm_guide () = Printf.eprintf "cannot read LLM guide: %s\n%!" e (* -------------------------------------------------------------------- *) -let run ~relocdir ~boot (llmopts : EcOptions.llm_option) = +let run ~relocdir ~boot ~projini (llmopts : EcOptions.llm_option) = if llmopts.llmo_help then begin print_llm_guide (); exit 0 @@ -56,7 +56,12 @@ let run ~relocdir ~boot (llmopts : EcOptions.llm_option) = | None -> EcCommands.addidir Filename.current_dir_name | Some pwd -> EcCommands.addidir pwd); - let checkmode = { + (* Prover options in effect: refreshed by [LOAD] with the loaded + file's [easycrypt.project] settings overlaid on the command-line + options, as the batch compiler does at option-parsing time. *) + let cur_prvopts = ref prvopts in + + let checkmode_of (prvopts : EcOptions.prv_options) = { EcCommands.cm_checkall = prvopts.prvo_checkall; EcCommands.cm_timeout = odfl 3 prvopts.prvo_timeout; EcCommands.cm_cpufactor = odfl 1 prvopts.prvo_cpufactor; @@ -77,6 +82,10 @@ let run ~relocdir ~boot (llmopts : EcOptions.llm_option) = [~restart:true]. *) let initialized = ref false in + (* Project-file load-path entries already added to the (global) + loader, so repeated [LOAD]s do not pile up duplicates. *) + let projdirs : (string option * string * bool) list ref = ref [] in + (* True iff replies should suppress goal bodies. Toggled by QUIET. *) let quiet = ref false in @@ -110,17 +119,17 @@ let run ~relocdir ~boot (llmopts : EcOptions.llm_option) = let do_initialize () = EcCommands.initialize ~restart:!initialized ~undo:true - ~boot ~checkmode ~checkproof:true; + ~boot ~checkmode:(checkmode_of !cur_prvopts) ~checkproof:true; initialized := true; (try - List.iter EcCommands.apply_pragma_option prvopts.prvo_pragmas + List.iter EcCommands.apply_pragma_option !cur_prvopts.prvo_pragmas with EcCommands.InvalidPragma x -> EcScope.hierror "invalid pragma: `%s'\n%!" x); EcCommands.addnotifier notifier; oiter (fun ppwidth -> let gs = EcEnv.gstate (EcScope.env (EcCommands.current ())) in EcGState.setvalue "PP:width" (`Int ppwidth) gs) - prvopts.prvo_ppwidth + !cur_prvopts.prvo_ppwidth in (* ------------------------------------------------------------------ *) @@ -635,10 +644,28 @@ let run ~relocdir ~boot (llmopts : EcOptions.llm_option) = "unknown file extension: %s" ext) end; + (* Apply the configuration attached to the loaded file's + [easycrypt.project], as the batch compiler does when the + file is given on the command line: refresh the prover + options (timeout, provers, pragmas, ...) and extend the + load path with the project's include dirs. *) + let ini = Option.to_list (projini (Some filename)) in + cur_prvopts := + EcOptions.prv_options_with_ini ini llmopts.llmo_provers; + List.iter (fun ((nm, dir, isrec) as entry) -> + if not (List.mem entry !projdirs) then begin + projdirs := entry :: !projdirs; + EcCommands.addidir + ?namespace:(omap (fun nm -> `Named nm) nm) + ~recursive:isrec dir + end) + (EcOptions.ini_loadpath ini); + do_initialize (); Hashtbl.clear checkpoints; Transcript.clear (); EcCommands.addidir (Filename.dirname filename); + EcCommands.set_current_path (Filename.dirname filename); let reader = EcIo.from_file filename in diff --git a/src/ecLlm.mli b/src/ecLlm.mli index b12dde1ef..d426c4c6e 100644 --- a/src/ecLlm.mli +++ b/src/ecLlm.mli @@ -3,9 +3,12 @@ over stdin/stdout. Driven via the [easycrypt llm] command. *) (* Run the REPL until [QUIT] or EOF, then exit the process. Never - returns. *) + returns. [projini] resolves the [easycrypt.project] context of a + file path, so [LOAD] can apply the project's load path and prover + options the way the batch compiler does. *) val run : relocdir:string option -> boot:bool + -> projini:(string option -> EcOptions.ini_context option) -> EcOptions.llm_option -> 'a diff --git a/src/ecOptions.ml b/src/ecOptions.ml index e4746cc18..dccc5858c 100644 --- a/src/ecOptions.ml +++ b/src/ecOptions.ml @@ -551,6 +551,47 @@ let prv_options_of_values ini values = prvo_why3server = get_string "why3server" values; } +(* -------------------------------------------------------------------- *) +(* Overlay project INI settings (an [easycrypt.project] discovered when + a file is loaded at run time, e.g. by the LLM REPL's [LOAD]) on top + of already-parsed prover options. Mirrors the precedence used by + [prv_options_of_values] when the project file is known at + option-parsing time: project provers/pragmas extend the parsed + lists, project scalars take over the parsed values. *) +let prv_options_with_ini (ini : ini_context list) (prv : prv_options) = + let provers = + match Ini.get_all_provers ini with + | [] -> prv.prvo_provers + | ps -> + let old = odfl [] prv.prvo_provers in + Some (ps @ List.filter (fun p -> not (List.mem p ps)) old) + in + { prv with + prvo_provers = provers; + prvo_timeout = begin + match Ini.get_all_timeout ini with + | None -> prv.prvo_timeout + | Some _ as i -> i + end; + prvo_quorum = begin + match Ini.get_all_quorum ini with + | None -> prv.prvo_quorum + | Some _ as i -> i + end; + prvo_ppwidth = begin + match Ini.get_all_ppwidth ini with + | None -> prv.prvo_ppwidth + | Some _ as i -> i + end; + prvo_pragmas = Ini.get_all_pragmas ini @ prv.prvo_pragmas; } + +(* The load path contributed by INI contexts, in the shape and order of + [ldro_idirs]: plain include dirs first, then recursive ones. *) +let ini_loadpath (ini : ini_context list) = + List.map (fun (nm, dir) -> (nm, dir, false)) (Ini.get_all_idirs ini) + @ List.map (fun (nm, dir) -> (nm, dir, true)) (Ini.get_all_rdirs ini) + +(* -------------------------------------------------------------------- *) let cli_options_of_values ini values = { clio_emacs = get_flag "emacs" values; clio_provers = prv_options_of_values ini values; } diff --git a/src/ecOptions.mli b/src/ecOptions.mli index def341dfd..d409a10bb 100644 --- a/src/ecOptions.mli +++ b/src/ecOptions.mli @@ -102,6 +102,16 @@ exception InvalidIniFile of (int * string) val read_ini_file : string -> ini_options +(* -------------------------------------------------------------------- *) +(* Overlay project INI settings discovered at run time (e.g. by the LLM + REPL's [LOAD]) on top of already-parsed prover options, mirroring + the precedence of option parsing with a known project file. *) +val prv_options_with_ini : ini_context list -> prv_options -> prv_options + +(* The load path contributed by INI contexts, in [ldro_idirs] shape: + (namespace, dir, recursive). *) +val ini_loadpath : ini_context list -> (string option * string * bool) list + val parse_cmdline : ?ini:(string option -> ini_context list) -> string array From 87423689fb88f15b748bbe158a1b62e03f5334a5 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Fri, 21 Aug 2026 09:44:19 +0200 Subject: [PATCH 16/51] [llm] add golden-output regression harness for the REPL 17 scenarios under tests/llm (scripts + recorded goldens + fixtures), driven by scripts/testing/llm-golden and a `make test-llm` target. Each script declares its expected exit status on a `# exit: N` first line; the runner strips #-comment lines, feeds the rest to `ec llm -eval` with tests/llm as the working directory (relative fixture paths keep the [loaded:...] tags machine-independent), and diffs stdout against the golden. `--record` re-records, still checking the declared exit code. Coverage: LOAD (+ -nosmt, -trace happy/error, argument errors), GOALS/GOALS ALL, nested TREE/TREE ALL, FOCUS dotted paths and error paths, NEXT, UNDO/REVERT/CHECKPOINT (incl. errors), COMMIT (simple, nested, prior-bullets, mid-proof-LOAD continuation, after qed), SEARCH, QUIET, /, and -eval exit-code semantics. Fixtures use AllCore only and close goals without SMT so goldens are deterministic (verified over repeated runs). Goldens record CURRENT behaviour, bugs included, as the byte-identity gate for the upcoming EcLlmCore extraction. Known frozen defects, to fix separately (with deliberate golden updates): COMMIT emits a flat, bullet-less body once the proof is closed by qed (the DAG walk finds no ancestors in a discarded proof); COMMIT under a +strict_bullets prefix flattens outer-frame siblings to the inner depth; LOAD of a missing file leaks a raw Sys_error anomaly. --- Makefile | 6 + scripts/testing/llm-golden | 132 ++++++++++++++++++ tests/llm/README.md | 88 ++++++++++++ tests/llm/expected/commit-after-qed.out | 29 ++++ .../llm/expected/commit-load-continuation.out | 25 ++++ tests/llm/expected/commit-nested.out | 37 +++++ tests/llm/expected/commit-simple.out | 25 ++++ tests/llm/expected/commit-strict-bullets.out | 25 ++++ tests/llm/expected/error-exit.out | 6 + tests/llm/expected/focus-nav.out | 84 +++++++++++ tests/llm/expected/load-errors.out | 18 +++ tests/llm/expected/load-goals.out | 48 +++++++ tests/llm/expected/load-nosmt.out | 18 +++ tests/llm/expected/load-trace-notinproof.out | 6 + tests/llm/expected/load-trace.out | 30 ++++ tests/llm/expected/multiline.out | 37 +++++ tests/llm/expected/quiet.out | 26 ++++ tests/llm/expected/search.out | 43 ++++++ tests/llm/expected/tree-nested.out | 52 +++++++ tests/llm/expected/undo-revert.out | 105 ++++++++++++++ tests/llm/fixtures/midproof.ec | 8 ++ tests/llm/fixtures/nested.ec | 14 ++ tests/llm/fixtures/simple.ec | 10 ++ tests/llm/fixtures/strict.ec | 11 ++ tests/llm/scripts/commit-after-qed.script | 12 ++ .../scripts/commit-load-continuation.script | 9 ++ tests/llm/scripts/commit-nested.script | 13 ++ tests/llm/scripts/commit-simple.script | 9 ++ .../llm/scripts/commit-strict-bullets.script | 11 ++ tests/llm/scripts/error-exit.script | 3 + tests/llm/scripts/focus-nav.script | 17 +++ tests/llm/scripts/load-errors.script | 6 + tests/llm/scripts/load-goals.script | 7 + tests/llm/scripts/load-nosmt.script | 4 + .../llm/scripts/load-trace-notinproof.script | 3 + tests/llm/scripts/load-trace.script | 3 + tests/llm/scripts/multiline.script | 16 +++ tests/llm/scripts/quiet.script | 8 ++ tests/llm/scripts/search.script | 5 + tests/llm/scripts/tree-nested.script | 10 ++ tests/llm/scripts/undo-revert.script | 16 +++ 41 files changed, 1035 insertions(+) create mode 100755 scripts/testing/llm-golden create mode 100644 tests/llm/README.md create mode 100644 tests/llm/expected/commit-after-qed.out create mode 100644 tests/llm/expected/commit-load-continuation.out create mode 100644 tests/llm/expected/commit-nested.out create mode 100644 tests/llm/expected/commit-simple.out create mode 100644 tests/llm/expected/commit-strict-bullets.out create mode 100644 tests/llm/expected/error-exit.out create mode 100644 tests/llm/expected/focus-nav.out create mode 100644 tests/llm/expected/load-errors.out create mode 100644 tests/llm/expected/load-goals.out create mode 100644 tests/llm/expected/load-nosmt.out create mode 100644 tests/llm/expected/load-trace-notinproof.out create mode 100644 tests/llm/expected/load-trace.out create mode 100644 tests/llm/expected/multiline.out create mode 100644 tests/llm/expected/quiet.out create mode 100644 tests/llm/expected/search.out create mode 100644 tests/llm/expected/tree-nested.out create mode 100644 tests/llm/expected/undo-revert.out create mode 100644 tests/llm/fixtures/midproof.ec create mode 100644 tests/llm/fixtures/nested.ec create mode 100644 tests/llm/fixtures/simple.ec create mode 100644 tests/llm/fixtures/strict.ec create mode 100644 tests/llm/scripts/commit-after-qed.script create mode 100644 tests/llm/scripts/commit-load-continuation.script create mode 100644 tests/llm/scripts/commit-nested.script create mode 100644 tests/llm/scripts/commit-simple.script create mode 100644 tests/llm/scripts/commit-strict-bullets.script create mode 100644 tests/llm/scripts/error-exit.script create mode 100644 tests/llm/scripts/focus-nav.script create mode 100644 tests/llm/scripts/load-errors.script create mode 100644 tests/llm/scripts/load-goals.script create mode 100644 tests/llm/scripts/load-nosmt.script create mode 100644 tests/llm/scripts/load-trace-notinproof.script create mode 100644 tests/llm/scripts/load-trace.script create mode 100644 tests/llm/scripts/multiline.script create mode 100644 tests/llm/scripts/quiet.script create mode 100644 tests/llm/scripts/search.script create mode 100644 tests/llm/scripts/tree-nested.script create mode 100644 tests/llm/scripts/undo-revert.script diff --git a/Makefile b/Makefile index 62ace7ba2..05a1dcf8b 100644 --- a/Makefile +++ b/Makefile @@ -13,6 +13,8 @@ CHECK += --jobs="$(ECJOBS)" CHECK += --bin-args=-timeout --bin-args="$(ECTOUT)" CHECK += $(foreach arg,$(ECARGS),--bin-args="$(arg)") CHECK += $(ECEXTRA) config/tests.config +LLMCHECK := scripts/testing/llm-golden +LLMCHECK += --bin=./ec.native NIX ?= nix --extra-experimental-features "nix-command flakes" PROFILE ?= dev @@ -22,6 +24,7 @@ UNAME_S = $(shell uname -s) # -------------------------------------------------------------------- .PHONY: default build byte native tests check examples +.PHONY: test-llm .PHONY: nix-build nix-build-with-provers nix-develop .PHONY: clean install uninstall @@ -51,6 +54,9 @@ stdlib: build examples: build $(CHECK) examples mee-cbc +test-llm: build + $(LLMCHECK) + check: unit stdlib examples @true diff --git a/scripts/testing/llm-golden b/scripts/testing/llm-golden new file mode 100755 index 000000000..98361f38a --- /dev/null +++ b/scripts/testing/llm-golden @@ -0,0 +1,132 @@ +#! /bin/sh + +# -------------------------------------------------------------------- +# Golden-output regression harness for the `easycrypt llm` REPL. +# +# llm-golden [--bin PATH] [--record] [NAME...] +# +# Each tests/llm/scripts/NAME.script holds the newline-separated +# commands fed to `ec.exe llm -eval`. Lines starting with `#` are +# stripped before the script is passed to -eval; the first such line +# must be `# exit: N`, the expected process exit status. Stdout is +# compared against tests/llm/expected/NAME.out. +# +# Scripts run with tests/llm as the working directory, so fixture paths +# stay relative and the [loaded:...] reply tags remain machine +# independent. +# -------------------------------------------------------------------- + +set -u + +root=$(CDPATH= cd -- "$(dirname -- "$0")/../.." && pwd) +bin="$root/_build/default/src/ec.exe" +record=0 +names="" + +while [ $# -gt 0 ]; do + case "$1" in + --bin) + [ $# -ge 2 ] || { echo "llm-golden: --bin needs an argument" >&2; exit 2; } + bin=$2; shift 2 ;; + --bin=*) + bin=${1#--bin=}; shift ;; + --record) + record=1; shift ;; + -h|--help) + echo "usage: llm-golden [--bin PATH] [--record] [NAME...]"; exit 0 ;; + -*) + echo "llm-golden: unknown option: $1" >&2; exit 2 ;; + *) + names="$names $1"; shift ;; + esac +done + +case "$bin" in + /*) ;; + *) bin=$(CDPATH= cd -- "$(dirname -- "$bin")" && pwd)/$(basename -- "$bin") ;; +esac + +if [ ! -x "$bin" ]; then + echo "llm-golden: no such executable: $bin" >&2 + exit 2 +fi + +tests="$root/tests/llm" +scripts="$tests/scripts" +expected="$tests/expected" + +if [ -z "$names" ]; then + names=$(cd "$scripts" && ls *.script 2>/dev/null | sed 's/\.script$//') +fi + +mkdir -p "$expected" + +tmp=$(mktemp -d "${TMPDIR:-/tmp}/llm-golden.XXXXXX") || exit 2 +trap 'rm -rf "$tmp"' EXIT INT TERM + +nfail=0 +npass=0 + +for name in $names; do + script="$scripts/$name.script" + gold="$expected/$name.out" + + if [ ! -f "$script" ]; then + echo "FAIL $name (no such script: $script)" + nfail=$((nfail + 1)) + continue + fi + + want_exit=$(sed -n 's/^# *exit: *\([0-9][0-9]*\).*$/\1/p' "$script" | head -n 1) + if [ -z "$want_exit" ]; then + echo "FAIL $name (script has no '# exit: N' line)" + nfail=$((nfail + 1)) + continue + fi + + grep -v '^#' "$script" > "$tmp/eval.in" + + (cd "$tests" && "$bin" llm -eval "$(cat "$tmp/eval.in")") \ + > "$tmp/out" 2> "$tmp/err" + got_exit=$? + + if [ "$record" = 1 ]; then + cp "$tmp/out" "$gold" + if [ "$got_exit" != "$want_exit" ]; then + echo "RECORD $name (exit $got_exit, script declares $want_exit)" + nfail=$((nfail + 1)) + else + echo "RECORD $name" + npass=$((npass + 1)) + fi + continue + fi + + ok=1 + + if [ ! -f "$gold" ]; then + echo "FAIL $name (no golden: $gold; re-run with --record)" + ok=0 + elif ! diff -u "$gold" "$tmp/out" > "$tmp/diff"; then + echo "FAIL $name (stdout differs)" + sed 's/^/ /' "$tmp/diff" + ok=0 + fi + + if [ "$got_exit" != "$want_exit" ]; then + echo "FAIL $name (exit $got_exit, expected $want_exit)" + ok=0 + fi + + if [ "$ok" = 1 ]; then + echo "PASS $name" + npass=$((npass + 1)) + else + nfail=$((nfail + 1)) + fi +done + +echo "----" +echo "$npass passed, $nfail failed" + +[ "$nfail" = 0 ] diff --git a/tests/llm/README.md b/tests/llm/README.md new file mode 100644 index 000000000..c4fbf2672 --- /dev/null +++ b/tests/llm/README.md @@ -0,0 +1,88 @@ +# `easycrypt llm` golden-output tests + +Byte-identity regression harness for the LLM REPL (`src/ecLlm.ml`). +Each scenario is a small script of REPL commands fed to +`ec.exe llm -eval`; its raw stdout and its process exit status are +compared against recorded goldens. + +## Layout + +| Path | Contents | +|------|----------| +| `fixtures/*.ec` | tiny EasyCrypt files the scripts `LOAD` | +| `scripts/*.script` | the newline-separated commands passed to `-eval` | +| `expected/*.out` | recorded stdout, one file per script | +| `../../scripts/testing/llm-golden` | the runner | + +## Running + +From the repository root: + +``` +make test-llm # build + run every scenario +scripts/testing/llm-golden # run every scenario +scripts/testing/llm-golden tree-nested commit-nested +scripts/testing/llm-golden --bin /path/to/ec.exe +``` + +The runner defaults to `_build/default/src/ec.exe`, resolved relative +to the repository root. It prints `PASS`/`FAIL` per scenario, a unified +diff for each mismatch, and exits nonzero if anything failed. That is +the CI invocation. + +## Re-recording + +``` +scripts/testing/llm-golden --record # all scenarios +scripts/testing/llm-golden --record load-goals # one scenario +``` + +`--record` overwrites `expected/*.out` with the current binary's +output instead of diffing. It still checks the declared exit status +and reports a mismatch, so a stale `# exit:` line cannot go unnoticed. + +Re-record only deliberately: these goldens are the gate for refactors +of `src/ecLlm.ml`, and every diff must be reviewed by hand. + +## Expected exit status + +Each `.script` declares its expected process exit status on its first +line: + +``` +# exit: 1 +``` + +Lines starting with `#` are comment lines: the runner strips **all** of +them before handing the script to `-eval`, so they can also be used for +prose. The first `# exit: N` line wins; a script without one fails. + +`ec.exe llm -eval` exits 1 if any command produced an `ERROR` reply and +0 otherwise, so scenarios that deliberately exercise error paths +declare `# exit: 1`. + +## Determinism rules + +The goldens are compared byte for byte, so scenarios must not leak +anything machine- or environment-dependent: + +* **Relative paths only.** The runner `cd`s into `tests/llm` before + invoking the binary, and scripts must refer to fixtures as + `LOAD "fixtures/foo.ec"`. `LOAD` echoes the filename verbatim in its + `[loaded:...]` reply tag, so an absolute path would bake the + developer's home directory into the golden. +* **No SMT.** Fixtures and scripts must never use `smt()`, `smt(...)` + or `/#`. Proofs close with `trivial`, `done`, `reflexivity` or + `split`. SMT would make the goldens depend on which provers are + installed, and on their timing. +* **stdout only.** stderr is discarded; only stdout is compared. +* **No `HELP`.** `HELP` echoes `doc/llm/CLAUDE.md`, which would make + every documentation edit a test failure. +* Fixtures require `AllCore` only. + +## Adding a scenario + +1. Add `scripts/NAME.script` starting with `# exit: N`. +2. Add any new fixture under `fixtures/`. +3. `scripts/testing/llm-golden --record NAME`. +4. Read `expected/NAME.out` and check it is what you meant to freeze. diff --git a/tests/llm/expected/commit-after-qed.out b/tests/llm/expected/commit-after-qed.out new file mode 100644 index 000000000..d979ee6b9 --- /dev/null +++ b/tests/llm/expected/commit-after-qed.out @@ -0,0 +1,29 @@ +READY [uuid:0] + +OK [uuid:3] [loaded:fixtures/simple.ec:6] +Current goal + +Type variables: + +------------------------------------------------------------------------ +1 = 1 /\ 2 = 2 + +OK [uuid:3] + +OK [uuid:4] [focus: 1/2] + +OK [uuid:5] + +OK [uuid:6] + +OK [uuid:7] +added lemma: `simple_and' + +OK [uuid:7] + +OK [uuid:7] +split. +trivial. +trivial. +qed. + diff --git a/tests/llm/expected/commit-load-continuation.out b/tests/llm/expected/commit-load-continuation.out new file mode 100644 index 000000000..d90f6bebd --- /dev/null +++ b/tests/llm/expected/commit-load-continuation.out @@ -0,0 +1,25 @@ +READY [uuid:0] + +OK [uuid:4] [loaded:fixtures/midproof.ec:8] [focus: 1/2] +Current goal (remaining: 2) + +Type variables: + +------------------------------------------------------------------------ +1 = 1 + +OK [uuid:4] + +OK [uuid:5] + +OK [uuid:6] + +OK [uuid:6] + +OK [uuid:6] +No more goals + +OK [uuid:6] +- trivial. +- trivial. + diff --git a/tests/llm/expected/commit-nested.out b/tests/llm/expected/commit-nested.out new file mode 100644 index 000000000..0b8a2fa93 --- /dev/null +++ b/tests/llm/expected/commit-nested.out @@ -0,0 +1,37 @@ +READY [uuid:0] + +OK [uuid:3] [loaded:fixtures/nested.ec:6] +Current goal + +Type variables: + +------------------------------------------------------------------------ +((1 = 1 /\ 2 = 2) /\ 3 = 3) /\ 4 = 4 + +OK [uuid:3] + +OK [uuid:4] [focus: 1/2] + +OK [uuid:5] [focus: 1/3] + +OK [uuid:6] [focus: 1/4] + +OK [uuid:7] [focus: 1/3] + +OK [uuid:8] [focus: 1/2] + +OK [uuid:9] + +OK [uuid:10] + +OK [uuid:10] + +OK [uuid:10] +split. +- split. + + split. + * trivial. + * trivial. + + trivial. +- trivial. + diff --git a/tests/llm/expected/commit-simple.out b/tests/llm/expected/commit-simple.out new file mode 100644 index 000000000..3a1e13b34 --- /dev/null +++ b/tests/llm/expected/commit-simple.out @@ -0,0 +1,25 @@ +READY [uuid:0] + +OK [uuid:3] [loaded:fixtures/simple.ec:6] +Current goal + +Type variables: + +------------------------------------------------------------------------ +1 = 1 /\ 2 = 2 + +OK [uuid:3] + +OK [uuid:4] [focus: 1/2] + +OK [uuid:5] + +OK [uuid:6] + +OK [uuid:6] + +OK [uuid:6] +split. +- trivial. +- trivial. + diff --git a/tests/llm/expected/commit-strict-bullets.out b/tests/llm/expected/commit-strict-bullets.out new file mode 100644 index 000000000..8076cd54c --- /dev/null +++ b/tests/llm/expected/commit-strict-bullets.out @@ -0,0 +1,25 @@ +READY [uuid:0] + +OK [uuid:6] [loaded:fixtures/strict.ec:11] [focus: 1/3] +Current goal (remaining: 3) + +Type variables: + +------------------------------------------------------------------------ +1 = 1 + +OK [uuid:6] + +OK [uuid:7] [focus: 1/2] + +OK [uuid:8] + +OK [uuid:9] + +OK [uuid:9] + +OK [uuid:9] ++ trivial. ++ trivial. ++ trivial. + diff --git a/tests/llm/expected/error-exit.out b/tests/llm/expected/error-exit.out new file mode 100644 index 000000000..8346241ef --- /dev/null +++ b/tests/llm/expected/error-exit.out @@ -0,0 +1,6 @@ +READY [uuid:0] + +ERROR [uuid:0] +nothing to undo +No active proof. + diff --git a/tests/llm/expected/focus-nav.out b/tests/llm/expected/focus-nav.out new file mode 100644 index 000000000..a98f637a3 --- /dev/null +++ b/tests/llm/expected/focus-nav.out @@ -0,0 +1,84 @@ +READY [uuid:0] + +OK [uuid:3] [loaded:fixtures/nested.ec:6] +Current goal + +Type variables: + +------------------------------------------------------------------------ +((1 = 1 /\ 2 = 2) /\ 3 = 3) /\ 4 = 4 + +OK [uuid:3] + +OK [uuid:4] [focus: 1/2] + +OK [uuid:5] [focus: 1/3] + +OK [uuid:6] [focus: 1/4] + +OK [uuid:6] + +OK [uuid:6] + [1.1.1] 1 = 1 <- focused + [1.1.2] 2 = 2 + [1.2] 3 = 3 +[2] 4 = 4 + +OK [uuid:7] [focus: 1/4] +Current goal (remaining: 4) + +Type variables: + +------------------------------------------------------------------------ +3 = 3 + +ERROR [uuid:7] +FOCUS: path must select a leaf goal, not a frame +Current goal (remaining: 4) + +Type variables: + +------------------------------------------------------------------------ +3 = 3 + +ERROR [uuid:7] +FOCUS: index 9 out of range (1..2) +Current goal (remaining: 4) + +Type variables: + +------------------------------------------------------------------------ +3 = 3 + +ERROR [uuid:7] +FOCUS: not a path of integers: foo +Current goal (remaining: 4) + +Type variables: + +------------------------------------------------------------------------ +3 = 3 + +ERROR [uuid:7] +FOCUS: missing argument +Current goal (remaining: 4) + +Type variables: + +------------------------------------------------------------------------ +3 = 3 + +OK [uuid:8] [focus: 1/4] +Current goal (remaining: 4) + +Type variables: + +------------------------------------------------------------------------ +4 = 4 + +OK [uuid:8] +[1] 4 = 4 <- focused + [2.1.1] 1 = 1 + [2.1.2] 2 = 2 + [2.2] 3 = 3 + diff --git a/tests/llm/expected/load-errors.out b/tests/llm/expected/load-errors.out new file mode 100644 index 000000000..b8e9f0fc2 --- /dev/null +++ b/tests/llm/expected/load-errors.out @@ -0,0 +1,18 @@ +READY [uuid:0] + +ERROR [uuid:0] +LOAD: missing filename +No active proof. + +ERROR [uuid:0] +unknown file extension: .txt +No active proof. + +ERROR [uuid:0] +anomaly: Sys_error("fixtures/nosuch.ec: No such file or directory") +No active proof. + +ERROR [uuid:0] +LOAD: unexpected arguments +No active proof. + diff --git a/tests/llm/expected/load-goals.out b/tests/llm/expected/load-goals.out new file mode 100644 index 000000000..acd523c00 --- /dev/null +++ b/tests/llm/expected/load-goals.out @@ -0,0 +1,48 @@ +READY [uuid:0] + +OK [uuid:3] [loaded:fixtures/simple.ec:6] +Current goal + +Type variables: + +------------------------------------------------------------------------ +1 = 1 /\ 2 = 2 + +OK [uuid:3] +Current goal + +Type variables: + +------------------------------------------------------------------------ +1 = 1 /\ 2 = 2 + +OK [uuid:4] [focus: 1/2] +Current goal (remaining: 2) + +Type variables: + +------------------------------------------------------------------------ +1 = 1 + +OK [uuid:4] +Current goal (remaining: 2) + +Type variables: + +------------------------------------------------------------------------ +1 = 1 + +OK [uuid:4] +Current goal (remaining: 2) + +Type variables: + +------------------------------------------------------------------------ +1 = 1 + + + + Goal #2 + ------------------------------------------------------------------------ + 2 = 2 + diff --git a/tests/llm/expected/load-nosmt.out b/tests/llm/expected/load-nosmt.out new file mode 100644 index 000000000..e08378e82 --- /dev/null +++ b/tests/llm/expected/load-nosmt.out @@ -0,0 +1,18 @@ +READY [uuid:0] + +OK [uuid:3] [loaded:fixtures/simple.ec:6] +Current goal + +Type variables: + +------------------------------------------------------------------------ +1 = 1 /\ 2 = 2 + +OK [uuid:3] +Current goal + +Type variables: + +------------------------------------------------------------------------ +1 = 1 /\ 2 = 2 + diff --git a/tests/llm/expected/load-trace-notinproof.out b/tests/llm/expected/load-trace-notinproof.out new file mode 100644 index 000000000..a0ed496fd --- /dev/null +++ b/tests/llm/expected/load-trace-notinproof.out @@ -0,0 +1,6 @@ +READY [uuid:0] + +ERROR [uuid:0] +trace: target sentence is not in a proof context +No active proof. + diff --git a/tests/llm/expected/load-trace.out b/tests/llm/expected/load-trace.out new file mode 100644 index 000000000..b424004d7 --- /dev/null +++ b/tests/llm/expected/load-trace.out @@ -0,0 +1,30 @@ +READY [uuid:0] + +OK [uuid:4] [loaded:fixtures/midproof.ec:8] [focus: 1/2] +=== BEFORE: line 8 (col 0) === +Current goal + +Type variables: + +------------------------------------------------------------------------ +1 = 1 /\ 2 = 2 + +=== TACTIC (lines 8:0 - 8:6) === +split. + +=== AFTER: line 8 (col 0) === +Type variables: + +------------------------------------------------------------------------ +1 = 1 + + +Type variables: + +------------------------------------------------------------------------ +2 = 2 + + +=== SUMMARY === +open goals: 1 -> 2 + diff --git a/tests/llm/expected/multiline.out b/tests/llm/expected/multiline.out new file mode 100644 index 000000000..3fb160924 --- /dev/null +++ b/tests/llm/expected/multiline.out @@ -0,0 +1,37 @@ +READY [uuid:0] + +OK [uuid:1] [loaded:fixtures/simple.ec:3] +No active proof. + +OK [uuid:2] +Current goal + +Type variables: + +------------------------------------------------------------------------ +1 = 1 /\ 2 = 2 + +OK [uuid:2] + +OK [uuid:3] [focus: 1/2] + +OK [uuid:4] + +OK [uuid:5] + +OK [uuid:5] + +OK [uuid:5] +No more goals + +OK [uuid:6] +added lemma: `multi' +No active proof. + +OK [uuid:6] +lemma multi : 1 = 1 /\ 2 = 2. +split. +trivial. +trivial. +qed. + diff --git a/tests/llm/expected/quiet.out b/tests/llm/expected/quiet.out new file mode 100644 index 000000000..6d7186947 --- /dev/null +++ b/tests/llm/expected/quiet.out @@ -0,0 +1,26 @@ +READY [uuid:0] + +OK [uuid:3] [loaded:fixtures/simple.ec:6] +Current goal + +Type variables: + +------------------------------------------------------------------------ +1 = 1 /\ 2 = 2 + +OK [uuid:3] + +OK [uuid:4] [focus: 1/2] + +OK [uuid:5] + +OK [uuid:5] + +OK [uuid:5] +Current goal + +Type variables: + +------------------------------------------------------------------------ +2 = 2 + diff --git a/tests/llm/expected/search.out b/tests/llm/expected/search.out new file mode 100644 index 000000000..7a5c27fbe --- /dev/null +++ b/tests/llm/expected/search.out @@ -0,0 +1,43 @@ +READY [uuid:0] + +OK [uuid:3] [loaded:fixtures/simple.ec:6] +Current goal + +Type variables: + +------------------------------------------------------------------------ +1 = 1 /\ 2 = 2 + +OK [uuid:4] +(* RField.signr_odd *) +lemma signr_odd: + forall (n : int), 0 <= n => (- 1%r) ^ b2i (odd n) = (- 1%r) ^ n. +lemma b2rE: forall (b : bool), b2r b = (b2i b)%r. +lemma le_b2i: forall (b1 b2 : bool), (b1 => b2) <=> b2i b1 <= b2i b2. +lemma b2i_or: + forall (b1 b2 : bool), b2i (b1 \/ b2) = b2i b1 + b2i b2 - b2i b1 * b2i b2. +lemma b2i_le1: forall (b : bool), b2i b <= 1. +lemma b2i_ge0: forall (b : bool), 0 <= b2i b. +lemma b2i_eq1: forall (b : bool), b2i b = 1 <=> b. +lemma b2i_eq0: forall (b : bool), b2i b = 0 <=> !b. +lemma b2i_and: forall (b1 b2 : bool), b2i (b1 /\ b2) = b2i b1 * b2i b2. +lemma b2i1: b2i true = 1. +lemma b2i0: b2i false = 0. +lemma signr_odd: forall (n : int), 0 <= n => (-1) ^ b2i (odd n) = (-1) ^ n. + +Current goal + +Type variables: + +------------------------------------------------------------------------ +1 = 1 /\ 2 = 2 + +ERROR [uuid:4] +SEARCH: missing query +Current goal + +Type variables: + +------------------------------------------------------------------------ +1 = 1 /\ 2 = 2 + diff --git a/tests/llm/expected/tree-nested.out b/tests/llm/expected/tree-nested.out new file mode 100644 index 000000000..8246e232b --- /dev/null +++ b/tests/llm/expected/tree-nested.out @@ -0,0 +1,52 @@ +READY [uuid:0] + +OK [uuid:3] [loaded:fixtures/nested.ec:6] +Current goal + +Type variables: + +------------------------------------------------------------------------ +((1 = 1 /\ 2 = 2) /\ 3 = 3) /\ 4 = 4 + +OK [uuid:3] + +OK [uuid:4] [focus: 1/2] + +OK [uuid:5] [focus: 1/3] + +OK [uuid:6] [focus: 1/4] + +OK [uuid:6] + +OK [uuid:6] + [1.1.1] 1 = 1 <- focused + [1.1.2] 2 = 2 + [1.2] 3 = 3 +[2] 4 = 4 + +OK [uuid:6] + [1.1.1] <- focused +Type variables: + +------------------------------------------------------------------------ +1 = 1 + + [1.1.2] +Type variables: + +------------------------------------------------------------------------ +2 = 2 + + [1.2] +Type variables: + +------------------------------------------------------------------------ +3 = 3 + +[2] +Type variables: + +------------------------------------------------------------------------ +4 = 4 + + diff --git a/tests/llm/expected/undo-revert.out b/tests/llm/expected/undo-revert.out new file mode 100644 index 000000000..e00e426a8 --- /dev/null +++ b/tests/llm/expected/undo-revert.out @@ -0,0 +1,105 @@ +READY [uuid:0] + +OK [uuid:3] [loaded:fixtures/simple.ec:6] +Current goal + +Type variables: + +------------------------------------------------------------------------ +1 = 1 /\ 2 = 2 + +OK [uuid:3] +checkpoint 'start' set at uuid 3 + +OK [uuid:4] [focus: 1/2] +Current goal (remaining: 2) + +Type variables: + +------------------------------------------------------------------------ +1 = 1 + +OK [uuid:3] +Current goal + +Type variables: + +------------------------------------------------------------------------ +1 = 1 /\ 2 = 2 + +OK [uuid:4] [focus: 1/2] +Current goal (remaining: 2) + +Type variables: + +------------------------------------------------------------------------ +1 = 1 + +OK [uuid:5] +Current goal + +Type variables: + +------------------------------------------------------------------------ +2 = 2 + +OK [uuid:4] [focus: 1/2] +Current goal (remaining: 2) + +Type variables: + +------------------------------------------------------------------------ +1 = 1 + +OK [uuid:3] +Current goal + +Type variables: + +------------------------------------------------------------------------ +1 = 1 /\ 2 = 2 + +ERROR [uuid:3] +CHECKPOINT: missing name +Current goal + +Type variables: + +------------------------------------------------------------------------ +1 = 1 /\ 2 = 2 + +ERROR [uuid:3] +REVERT: missing uuid or checkpoint name +Current goal + +Type variables: + +------------------------------------------------------------------------ +1 = 1 /\ 2 = 2 + +OK [uuid:3] +Current goal + +Type variables: + +------------------------------------------------------------------------ +1 = 1 /\ 2 = 2 + +ERROR [uuid:3] +REVERT: 'nosuch' is not a valid uuid or checkpoint name +Current goal + +Type variables: + +------------------------------------------------------------------------ +1 = 1 /\ 2 = 2 + +ERROR [uuid:3] +REVERT: uuid 999 out of range [0, 3] +Current goal + +Type variables: + +------------------------------------------------------------------------ +1 = 1 /\ 2 = 2 + diff --git a/tests/llm/fixtures/midproof.ec b/tests/llm/fixtures/midproof.ec new file mode 100644 index 000000000..6b2bad195 --- /dev/null +++ b/tests/llm/fixtures/midproof.ec @@ -0,0 +1,8 @@ +(* Deliberately truncated: the file ends inside the proof, so a bare + `LOAD "fixtures/midproof.ec"` lands mid-proof with two open goals. + Used for the LOAD-continuation and -trace scenarios. *) +require import AllCore. + +lemma cont_and : 1 = 1 /\ 2 = 2. +proof. +split. diff --git a/tests/llm/fixtures/nested.ec b/tests/llm/fixtures/nested.ec new file mode 100644 index 000000000..b30ba0234 --- /dev/null +++ b/tests/llm/fixtures/nested.ec @@ -0,0 +1,14 @@ +(* Nested conjunction: `split. split. split.` from the state at line 6 + opens four goals nested as [1.1.1] [1.1.2] [1.2] [2]. *) +require import AllCore. + +lemma nested_and : ((1 = 1 /\ 2 = 2) /\ 3 = 3) /\ 4 = 4. +proof. +split. +split. +split. +trivial. +trivial. +trivial. +trivial. +qed. diff --git a/tests/llm/fixtures/simple.ec b/tests/llm/fixtures/simple.ec new file mode 100644 index 000000000..ceb2d9c37 --- /dev/null +++ b/tests/llm/fixtures/simple.ec @@ -0,0 +1,10 @@ +(* Simple conjunction: LOAD stops on line 5 (the `proof.`), leaving one + open goal `1 = 1 /\ 2 = 2`. *) +require import AllCore. + +lemma simple_and : 1 = 1 /\ 2 = 2. +proof. +split. +trivial. +trivial. +qed. diff --git a/tests/llm/fixtures/strict.ec b/tests/llm/fixtures/strict.ec new file mode 100644 index 000000000..6d0948646 --- /dev/null +++ b/tests/llm/fixtures/strict.ec @@ -0,0 +1,11 @@ +(* Deliberately truncated, under +strict_bullets: the LOAD prefix leaves + the bullet stack holding `-`, so COMMIT must pick a different token + for the bullets it emits. *) +pragma +strict_bullets. + +require import AllCore. + +lemma strict_and : (1 = 1 /\ 2 = 2) /\ 3 = 3. +proof. +split. +- split. diff --git a/tests/llm/scripts/commit-after-qed.script b/tests/llm/scripts/commit-after-qed.script new file mode 100644 index 000000000..f0b45e483 --- /dev/null +++ b/tests/llm/scripts/commit-after-qed.script @@ -0,0 +1,12 @@ +# exit: 0 +# COMMIT run after `qed.`: the proof DAG is gone, so the current +# implementation emits the transcript with no bullets at all. Frozen +# here so a refactor cannot change it silently. +LOAD "fixtures/simple.ec" 6 +QUIET ON +split. +trivial. +trivial. +qed. +QUIET OFF +COMMIT diff --git a/tests/llm/scripts/commit-load-continuation.script b/tests/llm/scripts/commit-load-continuation.script new file mode 100644 index 000000000..fcd9a3c1b --- /dev/null +++ b/tests/llm/scripts/commit-load-continuation.script @@ -0,0 +1,9 @@ +# exit: 0 +# LOAD a file that ends mid-proof, continue it at the REPL, COMMIT. +LOAD "fixtures/midproof.ec" +QUIET ON +trivial. +trivial. +QUIET OFF +GOALS +COMMIT diff --git a/tests/llm/scripts/commit-nested.script b/tests/llm/scripts/commit-nested.script new file mode 100644 index 000000000..175da21f8 --- /dev/null +++ b/tests/llm/scripts/commit-nested.script @@ -0,0 +1,13 @@ +# exit: 0 +# COMMIT after nested splits: bullets nest as - / + / *. +LOAD "fixtures/nested.ec" 6 +QUIET ON +split. +split. +split. +trivial. +trivial. +trivial. +trivial. +QUIET OFF +COMMIT diff --git a/tests/llm/scripts/commit-simple.script b/tests/llm/scripts/commit-simple.script new file mode 100644 index 000000000..7dd1fead9 --- /dev/null +++ b/tests/llm/scripts/commit-simple.script @@ -0,0 +1,9 @@ +# exit: 0 +# COMMIT after a plain split + two trivials. +LOAD "fixtures/simple.ec" 6 +QUIET ON +split. +trivial. +trivial. +QUIET OFF +COMMIT diff --git a/tests/llm/scripts/commit-strict-bullets.script b/tests/llm/scripts/commit-strict-bullets.script new file mode 100644 index 000000000..8912983aa --- /dev/null +++ b/tests/llm/scripts/commit-strict-bullets.script @@ -0,0 +1,11 @@ +# exit: 0 +# The LOAD prefix of fixtures/strict.ec ends under `pragma +# +strict_bullets` with `-` on the bullet stack, so COMMIT must pick a +# token other than `-`. +LOAD "fixtures/strict.ec" +QUIET ON +trivial. +trivial. +trivial. +QUIET OFF +COMMIT diff --git a/tests/llm/scripts/error-exit.script b/tests/llm/scripts/error-exit.script new file mode 100644 index 000000000..01bb443d2 --- /dev/null +++ b/tests/llm/scripts/error-exit.script @@ -0,0 +1,3 @@ +# exit: 1 +# A script whose only command errors: the process must exit 1. +UNDO diff --git a/tests/llm/scripts/focus-nav.script b/tests/llm/scripts/focus-nav.script new file mode 100644 index 000000000..54963c73e --- /dev/null +++ b/tests/llm/scripts/focus-nav.script @@ -0,0 +1,17 @@ +# exit: 1 +# FOCUS with a dotted path, on a frame (error), out of range (error), +# with a non-integer path (parse error), then NEXT. +LOAD "fixtures/nested.ec" 6 +QUIET ON +split. +split. +split. +QUIET OFF +TREE +FOCUS 1.2 +FOCUS 1 +FOCUS 9 +FOCUS foo +FOCUS +NEXT +TREE diff --git a/tests/llm/scripts/load-errors.script b/tests/llm/scripts/load-errors.script new file mode 100644 index 000000000..5461f2d11 --- /dev/null +++ b/tests/llm/scripts/load-errors.script @@ -0,0 +1,6 @@ +# exit: 1 +# LOAD argument errors: missing filename, unknown extension, missing file. +LOAD +LOAD "fixtures/simple.txt" +LOAD "fixtures/nosuch.ec" +LOAD "fixtures/simple.ec" 6 7 diff --git a/tests/llm/scripts/load-goals.script b/tests/llm/scripts/load-goals.script new file mode 100644 index 000000000..83d64c2b8 --- /dev/null +++ b/tests/llm/scripts/load-goals.script @@ -0,0 +1,7 @@ +# exit: 0 +# LOAD a file up to a proof point, then inspect with GOALS / GOALS ALL. +LOAD "fixtures/simple.ec" 6 +GOALS +split. +GOALS +GOALS ALL diff --git a/tests/llm/scripts/load-nosmt.script b/tests/llm/scripts/load-nosmt.script new file mode 100644 index 000000000..deb495e0b --- /dev/null +++ b/tests/llm/scripts/load-nosmt.script @@ -0,0 +1,4 @@ +# exit: 0 +# LOAD -nosmt just has to load. +LOAD "fixtures/simple.ec" 6 -nosmt +GOALS diff --git a/tests/llm/scripts/load-trace-notinproof.script b/tests/llm/scripts/load-trace-notinproof.script new file mode 100644 index 000000000..bce7fef81 --- /dev/null +++ b/tests/llm/scripts/load-trace-notinproof.script @@ -0,0 +1,3 @@ +# exit: 1 +# LOAD -trace whose target sentence is outside any proof. +LOAD "fixtures/simple.ec" 3 -trace diff --git a/tests/llm/scripts/load-trace.script b/tests/llm/scripts/load-trace.script new file mode 100644 index 000000000..b02e820bc --- /dev/null +++ b/tests/llm/scripts/load-trace.script @@ -0,0 +1,3 @@ +# exit: 0 +# LOAD -trace on a file ending mid-proof: BEFORE/TACTIC/AFTER/SUMMARY. +LOAD "fixtures/midproof.ec" -trace diff --git a/tests/llm/scripts/multiline.script b/tests/llm/scripts/multiline.script new file mode 100644 index 000000000..32299a6b0 --- /dev/null +++ b/tests/llm/scripts/multiline.script @@ -0,0 +1,16 @@ +# exit: 0 +# / multi-line EasyCrypt input. +LOAD "fixtures/simple.ec" 3 + +lemma multi : + 1 = 1 /\ + 2 = 2. + +QUIET ON +split. +trivial. +trivial. +QUIET OFF +GOALS +qed. +COMMIT diff --git a/tests/llm/scripts/quiet.script b/tests/llm/scripts/quiet.script new file mode 100644 index 000000000..aa2917b6b --- /dev/null +++ b/tests/llm/scripts/quiet.script @@ -0,0 +1,8 @@ +# exit: 0 +# QUIET ON suppresses goal bodies; QUIET OFF restores them. +LOAD "fixtures/simple.ec" 6 +QUIET ON +split. +trivial. +QUIET OFF +GOALS diff --git a/tests/llm/scripts/search.script b/tests/llm/scripts/search.script new file mode 100644 index 000000000..80a5404ac --- /dev/null +++ b/tests/llm/scripts/search.script @@ -0,0 +1,5 @@ +# exit: 1 +# SEARCH with a pattern, then SEARCH with no argument (error). +LOAD "fixtures/simple.ec" 6 +SEARCH (b2i _) +SEARCH diff --git a/tests/llm/scripts/tree-nested.script b/tests/llm/scripts/tree-nested.script new file mode 100644 index 000000000..ce9dd92a7 --- /dev/null +++ b/tests/llm/scripts/tree-nested.script @@ -0,0 +1,10 @@ +# exit: 0 +# Nested splits: TREE and TREE ALL show dotted labels [1.1.1] ... [2]. +LOAD "fixtures/nested.ec" 6 +QUIET ON +split. +split. +split. +QUIET OFF +TREE +TREE ALL diff --git a/tests/llm/scripts/undo-revert.script b/tests/llm/scripts/undo-revert.script new file mode 100644 index 000000000..34e6800f2 --- /dev/null +++ b/tests/llm/scripts/undo-revert.script @@ -0,0 +1,16 @@ +# exit: 1 +# UNDO, REVERT by numeric uuid, CHECKPOINT + REVERT by name, and the +# CHECKPOINT/REVERT argument errors. +LOAD "fixtures/simple.ec" 6 +CHECKPOINT start +split. +UNDO +split. +trivial. +REVERT 4 +REVERT 3 +CHECKPOINT +REVERT +REVERT start +REVERT nosuch +REVERT 999 From 55599f42cd86ccc5ce5a06cf687428bb168aa7a4 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Fri, 21 Aug 2026 09:51:47 +0200 Subject: [PATCH 17/51] [llm] COMMIT keeps bullets when run after qed COMMIT reconstructed bullet structure by walking the proof DAG through EcCommands.parent_of/children_of, which read the ACTIVE proof from the scope. Once `qed.` has run there is no active proof, so both accessors answered None/[]: COMMIT registered no siblings and emitted a flat, bullet-less body -- exactly the case where the body is most useful (the proof is finished and ready to be pasted back). A [proofenv] is immutable and cumulative, so it can outlive the proof it belongs to. Add `EcCommands.current_proofenv` (Some the proofenv of the active PSCheck proof, else None), snapshot it in EcLlm's new `commit_env` ref at every recorded phrase -- keeping the last non-empty snapshot, so the phrase that closes the proof does not clear it -- and have Commit.proof_text query EcCoreGoal.parent_of_handle / children_of_handle against that snapshot, falling back to the live proof when nothing was recorded. Transcript.clear resets it along with the transcript. The snapshot is re-taken at every recorded phrase, so after UNDO plus a new phrase it is the rewound environment plus the new children; handles from undone phrases are either gone or inert (no transcript entry references them). Goldens changed: * tests/llm/expected/commit-after-qed.out -- the two `trivial.` lines now carry `- ` bullets; `qed.` stays flat, since no goal was open right before it and it therefore has no parent handle. Intended. * tests/llm/expected/multiline.out -- same, for the proof built from the multi-line lemma statement. Intended. * tests/llm/scripts/commit-after-qed.script -- comment updated: it described the old flat output as the frozen behaviour. commit-simple, commit-nested, commit-load-continuation and commit-strict-bullets are byte-identical (verified). --- doc/llm/CLAUDE.md | 4 ++- src/ecCommands.ml | 11 +++++++ src/ecCommands.mli | 1 + src/ecLlm.ml | 38 ++++++++++++++++++++--- tests/llm/expected/commit-after-qed.out | 4 +-- tests/llm/expected/multiline.out | 4 +-- tests/llm/scripts/commit-after-qed.script | 7 +++-- 7 files changed, 56 insertions(+), 13 deletions(-) diff --git a/doc/llm/CLAUDE.md b/doc/llm/CLAUDE.md index df832874d..706a55ef2 100644 --- a/doc/llm/CLAUDE.md +++ b/doc/llm/CLAUDE.md @@ -245,7 +245,9 @@ split. Bullet characters cycle through `-`, `+`, `*`, `--`, `++`, `**`, ... and are chosen to avoid colliding with any frames the LOAD prefix already opened. Use `COMMIT` once the proof is complete (or at any -checkpoint) and paste the result back into the source file. +checkpoint) and paste the result back into the source file. Running +`COMMIT` after `qed.` still emits a bulleted body: the proof structure +is read from a snapshot taken while the proof was open. `UNDO` / `REVERT` trim the COMMIT transcript automatically. diff --git a/src/ecCommands.ml b/src/ecCommands.ml index 012a77110..1c6a8bb7e 100644 --- a/src/ecCommands.ml +++ b/src/ecCommands.ml @@ -1211,6 +1211,17 @@ let open_handles () : EcCoreGoal.handle list = EcCoreGoal.all_hd_opened pf | _ -> [] +(* The proof environment of the active proof, or [None] if no proof is + active. A [proofenv] is immutable and cumulative, so a snapshot taken + while the proof was open keeps answering DAG queries after [qed] has + discarded the active proof. *) +let current_proofenv () : EcCoreGoal.proofenv option = + match S.xgoal (current ()) with + | Some { S.puc_active = + Some ({ S.puc_jdg = S.PSCheck pf }, _) } -> + Some (EcCoreGoal.proofenv_of_proof pf) + | _ -> None + (* Direct DAG children of [h] in the active proof. [] if no proof. *) let children_of (h : EcCoreGoal.handle) : EcCoreGoal.handle list = match S.xgoal (current ()) with diff --git a/src/ecCommands.mli b/src/ecCommands.mli index 320aed1d6..b44d30a75 100644 --- a/src/ecCommands.mli +++ b/src/ecCommands.mli @@ -69,6 +69,7 @@ val disable_repl_bullets : unit -> EcBullets.stack option val pp_tree : ?all:bool -> unit -> (int * bool * string) list val focus_goal : int -> (int, string) result val open_handles : unit -> EcCoreGoal.handle list +val current_proofenv : unit -> EcCoreGoal.proofenv option val children_of : EcCoreGoal.handle -> EcCoreGoal.handle list val parent_of : EcCoreGoal.handle -> EcCoreGoal.handle option diff --git a/src/ecLlm.ml b/src/ecLlm.ml index ca00b91c7..2e69d3f4c 100644 --- a/src/ecLlm.ml +++ b/src/ecLlm.ml @@ -104,6 +104,13 @@ let run ~relocdir ~boot ~projini (llmopts : EcOptions.llm_option) = (int * string * EcCoreGoal.handle option * EcCoreGoal.handle list) list ref = ref [] in + (* Proof environment snapshot, refreshed at every recorded phrase. + [COMMIT] queries the proof DAG through it rather than through the + active proof, which is gone once [qed] has run. A [proofenv] is + immutable and cumulative, so the last snapshot knows about every + handle any transcript entry can mention. *) + let commit_env : EcCoreGoal.proofenv option ref = ref None in + (* The bullet stack of the active proof at the moment REPL input took over. Captured the first time [disable_repl_bullets] clears a non-empty stack. Used by [Commit] to pick bullet characters @@ -371,7 +378,8 @@ let run ~relocdir ~boot ~projini (llmopts : EcOptions.llm_option) = let clear () = transcript := []; - prior_bullets := None + prior_bullets := None; + commit_env := None end in (* ------------------------------------------------------------------ *) @@ -401,8 +409,15 @@ let run ~relocdir ~boot ~projini (llmopts : EcOptions.llm_option) = raise (EcScope.toperror_of_exn ~gloc:loc (EcScope.HiScopeError (None, "this command is expected to fail"))); - if record && !succeeded && not p.EP.gl_fail then - transcript := (pre_uuid, src, parent, opens_pre) :: !transcript + if record && !succeeded && not p.EP.gl_fail then begin + transcript := (pre_uuid, src, parent, opens_pre) :: !transcript; + (* Keep the newest non-empty snapshot: a phrase that closes the + proof ([qed]) leaves no active proof, and precisely then we + still need the environment the previous phrases built. *) + match EcCommands.current_proofenv () with + | None -> () + | Some _ as penv -> commit_env := penv + end in (* ------------------------------------------------------------------ *) @@ -420,6 +435,19 @@ let run ~relocdir ~boot ~projini (llmopts : EcOptions.llm_option) = let chr = chars.(i mod 3) in String.concat "" (List.init rep (fun _ -> chr)) + (* DAG queries go through the snapshot recorded at the last phrase, + so COMMIT still sees the structure after [qed]. Fall back to the + live proof when no phrase was recorded under a proof. *) + let parent_of h = + match !commit_env with + | Some penv -> EcCoreGoal.parent_of_handle penv h + | None -> EcCommands.parent_of h + + let children_of h = + match !commit_env with + | Some penv -> EcCoreGoal.children_of_handle penv h + | None -> EcCommands.children_of h + let proof_text () = let entries = List.rev !transcript in let buf = Buffer.create 1024 in @@ -495,7 +523,7 @@ let run ~relocdir ~boot ~projini (llmopts : EcOptions.llm_option) = match Hmap.find_opt h !sibling_depth with | Some d -> Some (h, d) | None -> - match EcCommands.parent_of h with + match parent_of h with | Some p -> find_ancestor p | None -> None in @@ -517,7 +545,7 @@ let run ~relocdir ~boot ~projini (llmopts : EcOptions.llm_option) = links do. A compound phrase like [split; split.] can produce nested splits within one phrase. *) let rec walk h d = - match EcCommands.children_of h with + match children_of h with | [c] -> walk c d | (_ :: _ :: _) as cs -> List.iter diff --git a/tests/llm/expected/commit-after-qed.out b/tests/llm/expected/commit-after-qed.out index d979ee6b9..c55cefa12 100644 --- a/tests/llm/expected/commit-after-qed.out +++ b/tests/llm/expected/commit-after-qed.out @@ -23,7 +23,7 @@ OK [uuid:7] OK [uuid:7] split. -trivial. -trivial. +- trivial. +- trivial. qed. diff --git a/tests/llm/expected/multiline.out b/tests/llm/expected/multiline.out index 3fb160924..73772ea35 100644 --- a/tests/llm/expected/multiline.out +++ b/tests/llm/expected/multiline.out @@ -31,7 +31,7 @@ No active proof. OK [uuid:6] lemma multi : 1 = 1 /\ 2 = 2. split. -trivial. -trivial. +- trivial. +- trivial. qed. diff --git a/tests/llm/scripts/commit-after-qed.script b/tests/llm/scripts/commit-after-qed.script index f0b45e483..48e5c1493 100644 --- a/tests/llm/scripts/commit-after-qed.script +++ b/tests/llm/scripts/commit-after-qed.script @@ -1,7 +1,8 @@ # exit: 0 -# COMMIT run after `qed.`: the proof DAG is gone, so the current -# implementation emits the transcript with no bullets at all. Frozen -# here so a refactor cannot change it silently. +# COMMIT run after `qed.`: the active proof is gone, but COMMIT queries +# the proofenv snapshot taken at the last recorded phrase, so the body +# still carries bullets. `qed.` itself stays flat (no goal was open +# right before it, hence no parent handle). LOAD "fixtures/simple.ec" 6 QUIET ON split. From 1e608b3734f8fa23c8d86115a253727a6b19acf0 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Fri, 21 Aug 2026 09:55:48 +0200 Subject: [PATCH 18/51] [llm] COMMIT respects the LOAD prefix's open bullet frames Under a `+strict_bullets` LOAD prefix, COMMIT seeded *every* goal that was open when the first REPL phrase ran at depth 1 and then handed the whole run a single freshly-picked token. Two things went wrong: * goals still owned by different prefix frames were flattened onto one level, so the emitted body no longer described the proof's shape; * the prefix's own tokens were never reused, even though addressing a frame's next sibling is spelled with exactly that frame's token -- the emitted token instead opened a brand-new nested level under the frame that was meant to be left. With fixtures/strict.ec (stack `[-]` with floor 1, three goals open) the old body was three flat `+ trivial.` lines: the third goal, a sibling of the prefix's `-`, was emitted as if it lived inside it. New rule. Let the prefix frames be t_1..t_k, outermost first (the stack stores the innermost frame at its head), and n = the number of goals open at the first recorded phrase. A frame with floor f is discharged once f goals remain, so it still owns the first (n - f) goals of the focused-first list. A goal covered by c frames is seeded at depth c+1. For depth d <= k the token IS t_d's own token (the depth-to-token cache is pre-populated with those before any fresh pick happens); deeper levels keep cycling -, +, *, --, ... skipping every token on the stack and every token already assigned. When the prefix left no frame, coverage is empty and every goal lands at depth 1 exactly as before; the case of a single open goal and no frame stays unseeded, so the REPL simply continues on the prefix's own focus. Empty-stack scenarios are byte-identical (verified). Manual recompilation checks (deliberately not in the harness, which only diffs REPL stdout): for both strict fixtures I concatenated the fixture prefix, the body COMMIT emitted and `qed.`, and compiled the result with `ec.exe compile -no-eco`. Both succeed, and both now read as the proof a human would have written: split. split. - split. - split. + trivial. + split. + trivial. * trivial. - trivial. * trivial. + trivial. - trivial. Goldens changed: * tests/llm/expected/commit-strict-bullets.out -- was three flat `+ trivial.` lines, now ` + trivial.` / ` + trivial.` / `- trivial.`: the first two goals are inside the prefix's `-` frame (fresh `+`, indented), the third is that frame's next sibling and reuses `-`. Intended. Goldens added: * tests/llm/{fixtures/strictnested.ec,scripts/commit-strict-nested.script, expected/commit-strict-nested.out} -- new scenario with two prefix frames (`-` then `+`) and four open goals, exercising token reuse at two depths plus one fresh level (`*`). --- src/ecLlm.ml | 55 ++++++++++++++----- tests/llm/expected/commit-strict-bullets.out | 6 +- tests/llm/expected/commit-strict-nested.out | 28 ++++++++++ tests/llm/fixtures/strictnested.ec | 13 +++++ tests/llm/scripts/commit-strict-nested.script | 13 +++++ 5 files changed, 97 insertions(+), 18 deletions(-) create mode 100644 tests/llm/expected/commit-strict-nested.out create mode 100644 tests/llm/fixtures/strictnested.ec create mode 100644 tests/llm/scripts/commit-strict-nested.script diff --git a/src/ecLlm.ml b/src/ecLlm.ml index 2e69d3f4c..498469af2 100644 --- a/src/ecLlm.ml +++ b/src/ecLlm.ml @@ -462,8 +462,6 @@ let run ~relocdir ~boot ~projini (llmopts : EcOptions.llm_option) = in let sibling_depth : int Hmap.t ref = ref Hmap.empty in let current_depth = ref 0 in - (* Pick a bullet token for each depth, skipping tokens already - in scope from the LOAD prefix's bullet stack. *) let bullet_to_string (b : EcParsetree.bullet) = let ch = match b.b_kind with @@ -473,17 +471,31 @@ let run ~relocdir ~boot ~projini (llmopts : EcOptions.llm_option) = in String.concat "" (List.init b.b_count (fun _ -> ch)) in - let in_use_tokens = + (* Bullet frames the LOAD prefix left open, OUTERMOST first (the + stack stores the innermost frame at its head). Frame [t_d] is + the one whose siblings live at emitted depth [d]. *) + let frames : EcBullets.frame list = match !prior_bullets with - | None -> [] - | Some stack -> - List.map - (fun (f : EcBullets.frame) -> bullet_to_string f.bf_bullet) - stack + | None -> [] + | Some stack -> List.rev stack + in + let in_use_tokens = + List.map + (fun (f : EcBullets.frame) -> bullet_to_string f.bf_bullet) + frames in let depth_cache : (int, string) Hashtbl.t = Hashtbl.create 8 in let next_tok_idx = ref 0 in let assigned_tokens = ref [] in + (* Depths 1..k address the next sibling of a frame the prefix + already opened, and strict bullets accepts nothing but that + frame's own token there. Deeper levels get fresh tokens, so + pre-populate the cache before any fresh pick happens. *) + List.iteri (fun i (f : EcBullets.frame) -> + let t = bullet_to_string f.bf_bullet in + Hashtbl.replace depth_cache (i + 1) t; + assigned_tokens := t :: !assigned_tokens) + frames; let bullet_for_depth d = match Hashtbl.find_opt depth_cache d with | Some t -> t @@ -500,14 +512,27 @@ let run ~relocdir ~boot ~projini (llmopts : EcOptions.llm_option) = Hashtbl.add depth_cache d t; t in - (* Seed: if the first recorded phrase entered a state with - multiple open goals, the LOAD prefix opened a frame whose - siblings are still pending. Register all of them at depth 1 - so the first phrase's parent gets a bullet. *) + (* Seed: the goals already open when the first recorded phrase ran + were left there by the LOAD prefix, so COMMIT must place each of + them at the depth the prefix's own bullets put it at. A frame + with floor [f] is discharged once [f] goals remain, hence it + still owns the first [n - f] goals of the focused-first list; + a goal covered by [c] frames sits at depth [c + 1]. + Nothing to seed when the prefix left no frame and a single goal + (the REPL just continues on the prefix's own focus). *) (match entries with - | (_, _, Some _, (_ :: _ :: _ as opens)) :: _ -> - List.iter - (fun h -> sibling_depth := Hmap.add h 1 !sibling_depth) + | (_, _, Some _, (_ :: _ as opens)) :: _ + when frames <> [] || List.length opens >= 2 -> + let n = List.length opens in + List.iteri (fun i h -> + let pos = i + 1 in + let covering = + List.length + (List.filter + (fun (f : EcBullets.frame) -> pos <= n - f.bf_floor) + frames) + in + sibling_depth := Hmap.add h (covering + 1) !sibling_depth) opens | _ -> ()); List.iter (fun (_uuid, src, parent_opt, _opens) -> diff --git a/tests/llm/expected/commit-strict-bullets.out b/tests/llm/expected/commit-strict-bullets.out index 8076cd54c..f844bb224 100644 --- a/tests/llm/expected/commit-strict-bullets.out +++ b/tests/llm/expected/commit-strict-bullets.out @@ -19,7 +19,7 @@ OK [uuid:9] OK [uuid:9] OK [uuid:9] -+ trivial. -+ trivial. -+ trivial. + + trivial. + + trivial. +- trivial. diff --git a/tests/llm/expected/commit-strict-nested.out b/tests/llm/expected/commit-strict-nested.out new file mode 100644 index 000000000..2a5a8c31f --- /dev/null +++ b/tests/llm/expected/commit-strict-nested.out @@ -0,0 +1,28 @@ +READY [uuid:0] + +OK [uuid:7] [loaded:fixtures/strictnested.ec:13] [focus: 1/4] +Current goal (remaining: 4) + +Type variables: + +------------------------------------------------------------------------ +1 = 1 + +OK [uuid:7] + +OK [uuid:8] [focus: 1/3] + +OK [uuid:9] [focus: 1/2] + +OK [uuid:10] + +OK [uuid:11] + +OK [uuid:11] + +OK [uuid:11] + * trivial. + * trivial. + + trivial. +- trivial. + diff --git a/tests/llm/fixtures/strictnested.ec b/tests/llm/fixtures/strictnested.ec new file mode 100644 index 000000000..3dccf4d83 --- /dev/null +++ b/tests/llm/fixtures/strictnested.ec @@ -0,0 +1,13 @@ +(* Deliberately truncated, under +strict_bullets: the LOAD prefix leaves + two frames on the bullet stack (`-` outermost, `+` inside it) and + four open goals. COMMIT must address the goals still owned by those + frames with the frames' own tokens, and open one fresh level. *) +pragma +strict_bullets. + +require import AllCore. + +lemma strict_nested : ((1 = 1 /\ 2 = 2) /\ 3 = 3) /\ 4 = 4. +proof. +split. +- split. + + split. diff --git a/tests/llm/scripts/commit-strict-nested.script b/tests/llm/scripts/commit-strict-nested.script new file mode 100644 index 000000000..c1fd92099 --- /dev/null +++ b/tests/llm/scripts/commit-strict-nested.script @@ -0,0 +1,13 @@ +# exit: 0 +# The LOAD prefix of fixtures/strictnested.ec stops under two open +# bullet frames (`-` then `+`) with four goals open. COMMIT must reuse +# `-` for the outer frame's next sibling, `+` for the inner frame's, +# and pick `*` fresh for the level the prefix never opened. +LOAD "fixtures/strictnested.ec" +QUIET ON +trivial. +trivial. +trivial. +trivial. +QUIET OFF +COMMIT From b62259c1d1badbb23bd271bcf366c18bdbdf5ce8 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Fri, 21 Aug 2026 09:56:44 +0200 Subject: [PATCH 19/51] [llm] LOAD reports a missing file instead of an anomaly `LOAD "nosuch.ec"` ran the whole LOAD preamble -- project-file lookup, scope re-initialisation, transcript reset -- and only failed when EcIo.from_file opened the file, surfacing as `anomaly: Sys_error("nosuch.ec: No such file or directory")`. Besides being an unhelpful message, the session had already been reset by the time it was reported. Check `Sys.file_exists` in Load.handle right after filename parsing, before anything touches the session, and fail with `LOAD: no such file: `. The unknown-extension error is unchanged; it now only fires for files that exist. Goldens changed: * tests/llm/expected/load-errors.out -- the anomaly line becomes `LOAD: no such file: fixtures/nosuch.ec`. Intended. * tests/llm/scripts/load-errors.script -- the unknown-extension case used `fixtures/simple.txt`, which does not exist and would now be caught by the existence check first; it points at a new file that does exist, and the two cases are ordered missing-then-extension. Goldens added: * tests/llm/fixtures/notec.txt -- an existing non-EasyCrypt file, so the unknown-extension path stays covered. * tests/llm/README.md -- layout table no longer claims fixtures are all `.ec`. --- src/ecLlm.ml | 7 +++++++ tests/llm/README.md | 2 +- tests/llm/expected/load-errors.out | 4 ++-- tests/llm/fixtures/notec.txt | 2 ++ tests/llm/scripts/load-errors.script | 5 +++-- 5 files changed, 15 insertions(+), 5 deletions(-) create mode 100644 tests/llm/fixtures/notec.txt diff --git a/src/ecLlm.ml b/src/ecLlm.ml index 498469af2..0483944e3 100644 --- a/src/ecLlm.ml +++ b/src/ecLlm.ml @@ -660,6 +660,13 @@ let run ~relocdir ~boot ~projini (llmopts : EcOptions.llm_option) = | f :: rest -> (f, String.concat " " rest) in if filename = "" then failwith "LOAD: missing filename"; + (* Checked here, before anything else touches the session: the + reader would otherwise raise [Sys_error] far downstream, and + the REPL would report it as an anomaly after having already + reset the scope. *) + if not (Sys.file_exists filename) then + failwith + (Printf.sprintf "LOAD: no such file: %s" filename); (* Parse optional LINE[:COL] and flags (-nosmt, -trace). *) let upto, nosmt, trace = diff --git a/tests/llm/README.md b/tests/llm/README.md index c4fbf2672..a21a91641 100644 --- a/tests/llm/README.md +++ b/tests/llm/README.md @@ -9,7 +9,7 @@ compared against recorded goldens. | Path | Contents | |------|----------| -| `fixtures/*.ec` | tiny EasyCrypt files the scripts `LOAD` | +| `fixtures/*` | tiny EasyCrypt files the scripts `LOAD` (plus one non-`.ec` file, for the unknown-extension error) | | `scripts/*.script` | the newline-separated commands passed to `-eval` | | `expected/*.out` | recorded stdout, one file per script | | `../../scripts/testing/llm-golden` | the runner | diff --git a/tests/llm/expected/load-errors.out b/tests/llm/expected/load-errors.out index b8e9f0fc2..7628f4c9e 100644 --- a/tests/llm/expected/load-errors.out +++ b/tests/llm/expected/load-errors.out @@ -5,11 +5,11 @@ LOAD: missing filename No active proof. ERROR [uuid:0] -unknown file extension: .txt +LOAD: no such file: fixtures/nosuch.ec No active proof. ERROR [uuid:0] -anomaly: Sys_error("fixtures/nosuch.ec: No such file or directory") +unknown file extension: .txt No active proof. ERROR [uuid:0] diff --git a/tests/llm/fixtures/notec.txt b/tests/llm/fixtures/notec.txt new file mode 100644 index 000000000..a3de7e3c6 --- /dev/null +++ b/tests/llm/fixtures/notec.txt @@ -0,0 +1,2 @@ +This file exists but is not an EasyCrypt source: LOAD must reject it +with the unknown-extension error, not with the missing-file error. diff --git a/tests/llm/scripts/load-errors.script b/tests/llm/scripts/load-errors.script index 5461f2d11..66338b489 100644 --- a/tests/llm/scripts/load-errors.script +++ b/tests/llm/scripts/load-errors.script @@ -1,6 +1,7 @@ # exit: 1 -# LOAD argument errors: missing filename, unknown extension, missing file. +# LOAD argument errors: missing filename, missing file, an existing +# file with an unknown extension, trailing junk. LOAD -LOAD "fixtures/simple.txt" LOAD "fixtures/nosuch.ec" +LOAD "fixtures/notec.txt" LOAD "fixtures/simple.ec" 6 7 From 345a5d4abe1ce785fc916d6808d3cd7759f77503 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Fri, 21 Aug 2026 09:57:41 +0200 Subject: [PATCH 20/51] [llm] queries no longer pollute the COMMIT transcript Every successful REPL phrase was appended to the COMMIT transcript, queries included. Looking a lemma up mid-proof with `SEARCH`, `search`, `print` or `locate` therefore inserted the query into the proof body COMMIT emits -- a body that no longer describes the proof, and that a reader would have to strip by hand. process_action now skips the transcript append (and the proofenv snapshot) for the query actions Gprint / Gsearch / Glocate. Everything else -- declarations, `proof.`, tactics, `qed.` -- is recorded as before, and the engine still runs the query, so uuid behaviour and the printed results are unchanged. The SEARCH meta-command reaches the same code path through process_ec_input, so it is covered too. Goldens added: * tests/llm/{scripts/search-in-proof.script,expected/search-in-proof.out} -- mid-proof SEARCH between two tactics; the recorded body is the two `- trivial.` lines only. Golden search.out is unchanged (it never ran COMMIT); verified, as is the rest of the suite. doc/llm/CLAUDE.md: the COMMIT section now states the exemption. --- doc/llm/CLAUDE.md | 4 ++- src/ecLlm.ml | 9 +++++- tests/llm/expected/search-in-proof.out | 40 ++++++++++++++++++++++++ tests/llm/scripts/search-in-proof.script | 10 ++++++ 4 files changed, 61 insertions(+), 2 deletions(-) create mode 100644 tests/llm/expected/search-in-proof.out create mode 100644 tests/llm/scripts/search-in-proof.script diff --git a/doc/llm/CLAUDE.md b/doc/llm/CLAUDE.md index 706a55ef2..b61f9730f 100644 --- a/doc/llm/CLAUDE.md +++ b/doc/llm/CLAUDE.md @@ -224,7 +224,9 @@ another, because the tree always shows the focused goal first. **5. Build a `+strict_bullets`-friendly proof with `COMMIT`:** -The REPL records every successful interactive phrase. `COMMIT` walks +The REPL records every successful interactive phrase except queries +(`search`, `print`, `locate`, and the `SEARCH` command), so you can +look things up mid-proof without polluting the body. `COMMIT` walks the proof DAG and emits the recorded tactics with bullets inserted at every multi-child split. The output is a proof body that compiles under `pragma +strict_bullets`: diff --git a/src/ecLlm.ml b/src/ecLlm.ml index 0483944e3..cd17cefd8 100644 --- a/src/ecLlm.ml +++ b/src/ecLlm.ml @@ -409,7 +409,14 @@ let run ~relocdir ~boot ~projini (llmopts : EcOptions.llm_option) = raise (EcScope.toperror_of_exn ~gloc:loc (EcScope.HiScopeError (None, "this command is expected to fail"))); - if record && !succeeded && not p.EP.gl_fail then begin + (* Queries only inspect the environment: they neither advance the + proof nor belong in the body COMMIT emits. *) + let is_query = + match EcLocation.unloc p.EP.gl_action with + | EP.Gprint _ | EP.Gsearch _ | EP.Glocate _ -> true + | _ -> false + in + if record && !succeeded && not p.EP.gl_fail && not is_query then begin transcript := (pre_uuid, src, parent, opens_pre) :: !transcript; (* Keep the newest non-empty snapshot: a phrase that closes the proof ([qed]) leaves no active proof, and precisely then we diff --git a/tests/llm/expected/search-in-proof.out b/tests/llm/expected/search-in-proof.out new file mode 100644 index 000000000..4c588e2e7 --- /dev/null +++ b/tests/llm/expected/search-in-proof.out @@ -0,0 +1,40 @@ +READY [uuid:0] + +OK [uuid:4] [loaded:fixtures/midproof.ec:8] [focus: 1/2] +Current goal (remaining: 2) + +Type variables: + +------------------------------------------------------------------------ +1 = 1 + +OK [uuid:4] + +OK [uuid:5] + +OK [uuid:6] +(* RField.signr_odd *) +lemma signr_odd: + forall (n : int), 0 <= n => (- 1%r) ^ b2i (odd n) = (- 1%r) ^ n. +lemma b2rE: forall (b : bool), b2r b = (b2i b)%r. +lemma le_b2i: forall (b1 b2 : bool), (b1 => b2) <=> b2i b1 <= b2i b2. +lemma b2i_or: + forall (b1 b2 : bool), b2i (b1 \/ b2) = b2i b1 + b2i b2 - b2i b1 * b2i b2. +lemma b2i_le1: forall (b : bool), b2i b <= 1. +lemma b2i_ge0: forall (b : bool), 0 <= b2i b. +lemma b2i_eq1: forall (b : bool), b2i b = 1 <=> b. +lemma b2i_eq0: forall (b : bool), b2i b = 0 <=> !b. +lemma b2i_and: forall (b1 b2 : bool), b2i (b1 /\ b2) = b2i b1 * b2i b2. +lemma b2i1: b2i true = 1. +lemma b2i0: b2i false = 0. +lemma signr_odd: forall (n : int), 0 <= n => (-1) ^ b2i (odd n) = (-1) ^ n. + + +OK [uuid:7] + +OK [uuid:7] + +OK [uuid:7] +- trivial. +- trivial. + diff --git a/tests/llm/scripts/search-in-proof.script b/tests/llm/scripts/search-in-proof.script new file mode 100644 index 000000000..f462e9cc3 --- /dev/null +++ b/tests/llm/scripts/search-in-proof.script @@ -0,0 +1,10 @@ +# exit: 0 +# A query issued in the middle of a proof must not end up in the body +# COMMIT emits: the two `trivial.` lines are the whole proof. +LOAD "fixtures/midproof.ec" +QUIET ON +trivial. +SEARCH (b2i _) +trivial. +QUIET OFF +COMMIT From 86bc1f7044056aafc7f785fed3222fe8f0a5cbd1 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Fri, 21 Aug 2026 09:59:15 +0200 Subject: [PATCH 21/51] [llm] tag GOALS / TREE / COMMIT replies with the focus indicator Only the replies produced by Wire.reply_ok_goals -- tactic phrases, FOCUS, NEXT, UNDO, REVERT -- and LOAD carried the `[focus: k/N]` tag. The inspection commands did not, so `GOALS`, `GOALS ALL`, `TREE`, `TREE ALL` and `COMMIT` answered with a bare `OK [uuid:N]` even with several goals open: exactly the commands an agent uses to find out how many goals there are. Dispatch now passes ~tag:(Goals.focus_tag ()) on those five arms. HELP, QUIET and CHECKPOINT do not report proof state and stay untagged. focus_tag is unchanged (empty below two open goals), so single-goal replies are unaffected. Goldens changed (each gains a tag on an otherwise identical line): * tests/llm/expected/load-goals.out -- the `GOALS` and `GOALS ALL` replies after `split.` (2 open goals) become `[focus: 1/2]`. * tests/llm/expected/tree-nested.out -- the `TREE` and `TREE ALL` replies (4 open goals) become `[focus: 1/4]`. * tests/llm/expected/focus-nav.out -- the two `TREE` replies (4 open goals) become `[focus: 1/4]`. quiet, load-nosmt and the commit-* scenarios are unchanged: they have at most one goal open when they run an inspection command. The tagged COMMIT reply is not exercised by any scenario; checked by hand (`COMMIT` after two splits replies `OK [uuid:5] [focus: 1/3]`). Also in this commit: the Commit module header still described the old "skip every stack token" policy replaced two commits ago; corrected. doc/llm/CLAUDE.md: the pitfall bullet listed no commands, which read as if only tactic replies were tagged; it now names the tagged commands and the untagged ones, and the TREE example shows its tag. --- doc/llm/CLAUDE.md | 10 ++++++---- src/ecLlm.ml | 22 ++++++++++++++-------- tests/llm/expected/focus-nav.out | 4 ++-- tests/llm/expected/load-goals.out | 4 ++-- tests/llm/expected/tree-nested.out | 4 ++-- 5 files changed, 26 insertions(+), 18 deletions(-) diff --git a/doc/llm/CLAUDE.md b/doc/llm/CLAUDE.md index b61f9730f..2f6bdf4c9 100644 --- a/doc/llm/CLAUDE.md +++ b/doc/llm/CLAUDE.md @@ -195,7 +195,7 @@ turn. Use `TREE` to see the structure, including nested splits: ``` TREE -→ OK [uuid:N] +→ OK [uuid:N] [focus: 1/4] [1.1.1] x = 0 <- focused [1.1.2] y = 1 [1.2] z = 2 @@ -347,9 +347,11 @@ SEARCH (_ %/ _) first one. Address them in any order via `FOCUS path`, or in the default order by closing each in turn. Use `TREE` or `GOALS ALL` to see what's open. -- When more than one subgoal is open, replies carry a - `[focus: k/N]` tag (e.g. `OK [uuid:42] [focus: 1/3]`) so you know - which one the next tactic will hit. +- When more than one subgoal is open, every `OK` reply that reflects + proof state -- tactics, `GOALS`, `GOALS ALL`, `TREE`, `TREE ALL`, + `FOCUS`, `NEXT`, `COMMIT`, `LOAD` -- carries a `[focus: k/N]` tag + (e.g. `OK [uuid:42] [focus: 1/3]`) so you know which one the next + tactic will hit. `HELP`, `QUIET` and `CHECKPOINT` are untagged. - `pragma +strict_bullets` does **not** apply to REPL input. Files loaded via `LOAD` still respect their own pragmas, but tactics typed at the REPL prompt are never rejected for missing bullets — the diff --git a/src/ecLlm.ml b/src/ecLlm.ml index cd17cefd8..9bd6d1694 100644 --- a/src/ecLlm.ml +++ b/src/ecLlm.ml @@ -430,9 +430,10 @@ let run ~relocdir ~boot ~projini (llmopts : EcOptions.llm_option) = (* ------------------------------------------------------------------ *) (* COMMIT: replay the transcript against the proof DAG (parent_of / children_of, backed by [EcCoreGoal.pr_parent]), inserting bullets - at multi-child splits. Bullet tokens skip any character already on - the LOAD prefix's [puc_bullets] stack so emitted bullets cannot - collide with frames opened by the prefix. *) + at multi-child splits. Levels the LOAD prefix's [puc_bullets] stack + already opened are addressed with that frame's own token; deeper + levels get fresh tokens, chosen so they collide with neither the + stack nor each other. *) let module Commit = struct (* Token order matches PR 1017's lexer: -, +, *, --, ++, **, ---, +++, *** ... *) @@ -1155,19 +1156,24 @@ let run ~relocdir ~boot ~projini (llmopts : EcOptions.llm_option) = | Undo -> do_undo () | Goals `One -> Buffer.clear notices; - Wire.reply_ok (Goals.goals_to_string ()) + Wire.reply_ok ~tag:(Goals.focus_tag ()) + (Goals.goals_to_string ()) | Goals `All -> Buffer.clear notices; - Wire.reply_ok (Goals.goals_to_string ~all:true ()) + Wire.reply_ok ~tag:(Goals.focus_tag ()) + (Goals.goals_to_string ~all:true ()) | Tree `One -> Buffer.clear notices; - Wire.reply_ok (FrameTree.render ()) + Wire.reply_ok ~tag:(Goals.focus_tag ()) + (FrameTree.render ()) | Tree `All -> Buffer.clear notices; - Wire.reply_ok (FrameTree.render ~all:true ()) + Wire.reply_ok ~tag:(Goals.focus_tag ()) + (FrameTree.render ~all:true ()) | Commit -> Buffer.clear notices; - Wire.reply_ok (Commit.proof_text ()) + Wire.reply_ok ~tag:(Goals.focus_tag ()) + (Commit.proof_text ()) | Focus path -> do_focus_request (`Path path) | Next -> do_focus_request `Next | Checkpoint n -> do_checkpoint n diff --git a/tests/llm/expected/focus-nav.out b/tests/llm/expected/focus-nav.out index a98f637a3..c7829c4bd 100644 --- a/tests/llm/expected/focus-nav.out +++ b/tests/llm/expected/focus-nav.out @@ -18,7 +18,7 @@ OK [uuid:6] [focus: 1/4] OK [uuid:6] -OK [uuid:6] +OK [uuid:6] [focus: 1/4] [1.1.1] 1 = 1 <- focused [1.1.2] 2 = 2 [1.2] 3 = 3 @@ -76,7 +76,7 @@ Type variables: ------------------------------------------------------------------------ 4 = 4 -OK [uuid:8] +OK [uuid:8] [focus: 1/4] [1] 4 = 4 <- focused [2.1.1] 1 = 1 [2.1.2] 2 = 2 diff --git a/tests/llm/expected/load-goals.out b/tests/llm/expected/load-goals.out index acd523c00..b48f2bebd 100644 --- a/tests/llm/expected/load-goals.out +++ b/tests/llm/expected/load-goals.out @@ -24,7 +24,7 @@ Type variables: ------------------------------------------------------------------------ 1 = 1 -OK [uuid:4] +OK [uuid:4] [focus: 1/2] Current goal (remaining: 2) Type variables: @@ -32,7 +32,7 @@ Type variables: ------------------------------------------------------------------------ 1 = 1 -OK [uuid:4] +OK [uuid:4] [focus: 1/2] Current goal (remaining: 2) Type variables: diff --git a/tests/llm/expected/tree-nested.out b/tests/llm/expected/tree-nested.out index 8246e232b..d8936191a 100644 --- a/tests/llm/expected/tree-nested.out +++ b/tests/llm/expected/tree-nested.out @@ -18,13 +18,13 @@ OK [uuid:6] [focus: 1/4] OK [uuid:6] -OK [uuid:6] +OK [uuid:6] [focus: 1/4] [1.1.1] 1 = 1 <- focused [1.1.2] 2 = 2 [1.2] 3 = 3 [2] 4 = 4 -OK [uuid:6] +OK [uuid:6] [focus: 1/4] [1.1.1] <- focused Type variables: From dbe5acb9d87fc70664ec0d6c4677369a981cdeee Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Fri, 21 Aug 2026 10:00:30 +0200 Subject: [PATCH 22/51] [llm] a -trace LOAD that cannot trace keeps the prefix state `-trace` defers the last in-prefix sentence so it can be run under goal capture. When the epilogue decided it could not trace -- no sentence pending (`trace: nothing to trace`), or the sentence sits outside any proof (`trace: target sentence is not in a proof context`) -- it failed with the deferred sentence still pending, so it was never run. The session was therefore one sentence short of a plain LOAD of the same prefix, and everything that sentence brought in was missing: `LOAD "f.ec" 3 -trace` stopping on `require import AllCore.` left a session where `b2i` did not resolve. Recovering meant reloading. Both branches now flush the deferred sentence (plainly, without trace capture) before failing, so the state matches a plain LOAD. The ERROR text is unchanged; the reply uuid now reflects the flushed sentence. A failure inside the flush propagates to the enclosing handler and is reported like any other prefix failure. `trace: nothing to trace` has nothing pending by construction, so flushing there is a no-op, kept for uniformity. Goldens changed: * tests/llm/expected/load-trace-notinproof.out -- the ERROR reply moves from `[uuid:0]` to `[uuid:1]` (the `require` ran). Intended. * tests/llm/scripts/load-trace-notinproof.script -- extended with `GOALS`, a lemma whose statement needs AllCore, and a second `GOALS`, so the golden actually demonstrates the preserved state. Before this fix that lemma failed with "no matching operator, named `b2i'". doc/llm/CLAUDE.md: the `-trace` section now states that a trace that cannot run still leaves a usable session. --- doc/llm/CLAUDE.md | 5 +++++ src/ecLlm.ml | 10 ++++++++- tests/llm/expected/load-trace-notinproof.out | 21 ++++++++++++++++++- .../llm/scripts/load-trace-notinproof.script | 8 ++++++- 4 files changed, 41 insertions(+), 3 deletions(-) diff --git a/doc/llm/CLAUDE.md b/doc/llm/CLAUDE.md index 2f6bdf4c9..a4fb29087 100644 --- a/doc/llm/CLAUDE.md +++ b/doc/llm/CLAUDE.md @@ -153,6 +153,11 @@ trace the file's last sentence. On tactic failure the reply uses the `ERROR` envelope and still includes the BEFORE/TACTIC blocks plus an `` marker in the AFTER block. +A `-trace` LOAD that cannot trace at all (the target sentence is not +inside a proof, or there is no sentence to trace) reports the error but +still leaves the session where the same LOAD without `-trace` would: +you can carry on from there instead of reloading. + **2. Try tactics, using REVERT to restart:** The uuid returned by LOAD is a revertible state. Use `REVERT` to diff --git a/src/ecLlm.ml b/src/ecLlm.ml index 9bd6d1694..b2748a306 100644 --- a/src/ecLlm.ml +++ b/src/ecLlm.ml @@ -834,8 +834,16 @@ let run ~relocdir ~boot ~projini (llmopts : EcOptions.llm_option) = | Some (src, p) -> `Ready (src, p) in match pre_state with - | `Nothing -> failwith "trace: nothing to trace" + (* Tracing is off the table, but the prefix is not: run the + sentence we deferred so that the session ends up exactly + where a plain LOAD of the same prefix would leave it. A + failure inside the flush is reported by the enclosing + handler, as any prefix failure is. *) + | `Nothing -> + flush_pending (); + failwith "trace: nothing to trace" | `NotInProof -> + flush_pending (); failwith "trace: target sentence is not in a proof context" | `Ready (src, p) -> diff --git a/tests/llm/expected/load-trace-notinproof.out b/tests/llm/expected/load-trace-notinproof.out index a0ed496fd..345e3ab82 100644 --- a/tests/llm/expected/load-trace-notinproof.out +++ b/tests/llm/expected/load-trace-notinproof.out @@ -1,6 +1,25 @@ READY [uuid:0] -ERROR [uuid:0] +ERROR [uuid:1] trace: target sentence is not in a proof context No active proof. +OK [uuid:1] +No active proof. + +OK [uuid:2] +Current goal + +Type variables: + +------------------------------------------------------------------------ +b2i true = 1 + +OK [uuid:2] +Current goal + +Type variables: + +------------------------------------------------------------------------ +b2i true = 1 + diff --git a/tests/llm/scripts/load-trace-notinproof.script b/tests/llm/scripts/load-trace-notinproof.script index bce7fef81..4022dc223 100644 --- a/tests/llm/scripts/load-trace-notinproof.script +++ b/tests/llm/scripts/load-trace-notinproof.script @@ -1,3 +1,9 @@ # exit: 1 -# LOAD -trace whose target sentence is outside any proof. +# LOAD -trace whose target sentence is outside any proof. Tracing +# fails, but the prefix must be in effect exactly as after a plain +# LOAD: the deferred `require import AllCore.` has run, so `b2i' below +# resolves and GOALS shows its goal. LOAD "fixtures/simple.ec" 3 -trace +GOALS +lemma preserved : b2i true = 1. +GOALS From 2355e0781265920a29b1b5dc2587da642c620752 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Fri, 21 Aug 2026 10:34:04 +0200 Subject: [PATCH 23/51] [llm] split ecLlm into an engine core and a text front-end MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 0b of the `easycrypt mcp` plan: `easycrypt llm` is about to grow a second front-end (JSON-RPC over the same engine), so the part of the REPL that talks to EasyCrypt is extracted verbatim into a module that neither prints nor exits. What moved to src/ecLlmCore.ml(i): - the session state, previously the closure refs of [run], now a [state] record (cur_prvopts, notice buffer, initialized, projdirs, checkpoints, transcript, commit_env, prior_bullets); [create] performs the why3 connect / Random.self_init / relocdir addidir / first [do_initialize] in the same order as before. The engine is a global singleton, hence at most one [state] per process; - the [Goals], [FrameTree], [Transcript] and [Commit] submodules and [process_action], unchanged except for taking [state]; - one function per meta-command ([load], [step], [goals], [tree], [focus], [undo], [revert], [checkpoint], [commit], [search]), lifted from the [Dispatch] handlers. Presentation state stays in src/ecLlm.ml: [quiet], the / buffer, [had_error], HELP (a file read), the READY banner, the -help early exit, the -eval driver, the line parser (now including the LOAD argument-string parser) and the OK/ERROR/ envelope. Reply types: an operation returns [reply] {uuid; tag; notices; body; changed} or [failure] {uuid; message; goals; notices} rather than printing. The front-end reconstructs today's exact bytes from those fields; MCP will build JSON from the same records. [body] is [Goals] or [Text] because QUIET suppression applies to precisely the replies that used to go through [reply_ok_goals] — that choice is presentation, so the core only says "this reply ends on the goals" and the REPL renders them via [current_goals]. Notices are captured-and-cleared at the exact points [Wire] used to read the buffer, so interleaving is preserved; [failure.notices] is captured but unused by the REPL, which never printed notices on errors. [changed] (uuid advanced?) is unused today and exists for MCP. De-exiting, with no observable change: - `exit.` typed at the prompt used to call [exit 0] from inside [process_ec_input]. [step] now returns [Quit] after finalizing the reader, and the front-end exits — same order, same status. - LOAD's -trace failure used a local [Trace_failed] exception plus a [trace_prefix] ref to prepend the BEFORE/TACTIC block to the error message; the prefix is now concatenated into [failure.message] exactly where [reply_error] received it. - [EcCommands.Restart] still reinitializes (and clears checkpoints in LOAD) inside the core, and returns an ordinary [Text "Session restarted"] reply. - a why3 connection failure raises [Init_error] instead of exiting; the front-end prints the same message and exits 1. Byte-identity compromises worth knowing about for the MCP phase: - LOAD argument errors are raised as [Parse_error] from the front-end parser, including the bare "int_of_string" that a malformed LINE[:COL] produces: the old code let those [Failure]s reach [reply_error] unchanged. The parser therefore maps [Failure msg] to [Parse_error msg] wholesale. - the file-existence check stays in the front-end parser (it must run before the flags parse, as it did), so [load] trusts its [file] argument to exist. The extension check stays in the core, after the flags parse, to keep the error ordering of `LOAD f.txt 6 7`. - GOALS/GOALS ALL return [Text], not [Goals]: QUIET never suppressed them. Gate: `dune build` clean, `make test-llm` 19/19 PASS, `git diff tests/` empty. Ten further scripts covering paths the goldens do not reach (`exit.`, QUIT, HELP, `pragma restart.`, a failing -trace sentence, LOAD argument errors, doc comments, checkpoint/revert/focus errors, the multi-line block, QUIET) produce byte-identical stdout, stderr and exit status before and after. --- src/ecLlm.ml | 1329 +++++++++------------------------------------ src/ecLlmCore.ml | 1003 ++++++++++++++++++++++++++++++++++ src/ecLlmCore.mli | 98 ++++ 3 files changed, 1344 insertions(+), 1086 deletions(-) create mode 100644 src/ecLlmCore.ml create mode 100644 src/ecLlmCore.mli diff --git a/src/ecLlm.ml b/src/ecLlm.ml index b2748a306..8a2c1ead6 100644 --- a/src/ecLlm.ml +++ b/src/ecLlm.ml @@ -1,17 +1,13 @@ (* -------------------------------------------------------------------- *) (* The LLM coding-agent REPL. See [ecLlm.mli] for the entry point. - Implementation note: the REPL holds a large amount of mutable state - (notice buffer, transcript, checkpoints, ...). To keep that state - sharable across the various helpers without resorting to a big - record, [run] is a single closure that opens nested [module] blocks - for grouping. The submodules are read-only views over the closed- - over refs. *) + This is the text front-end only: line parsing, the OK/ERROR/ + envelope, the multi-line block buffer, QUIET, HELP and the -eval + driver. Everything engine-facing lives in [EcLlmCore], which the + MCP front-end shares. *) open EcUtils -module EP = EcParsetree - (* -------------------------------------------------------------------- *) (* Path to the bundled LLM-agent guide. *) let llm_guide_path () = @@ -34,6 +30,188 @@ let print_llm_guide () = with Sys_error e -> Printf.eprintf "cannot read LLM guide: %s\n%!" e +(* -------------------------------------------------------------------- *) +(* Surface command vocabulary. Parsing turns each stdin line into one + of these, and dispatch is a flat pattern-match. Argument + parsing/validation lives here; commands that interact with mutable + session state (checkpoints table) carry only the raw user-supplied + data and let [EcLlmCore] do the lookup. *) +module Parse = struct + type command = + | Quit + | Help + | Undo + | Goals of [`One | `All] + | Tree of [`One | `All] + | Commit + | Focus of int list (* dotted path; [k] = "FOCUS k" *) + | Next + | Checkpoint of string + | Revert of string (* uuid-or-name; the core resolves *) + | Quiet of bool + | Search of string (* trailing "." already stripped *) + | Load of load (* parsed LOAD arguments *) + | Ec of string (* fall-through: raw EasyCrypt input *) + | Begin_multi + | Done_multi + | Multi_line of string + | Blank + + and load = { + ld_file : string; + ld_upto : (int * int option) option; + ld_nosmt : bool; + ld_trace : bool; + } + + exception Parse_error of string + + (* Match [kw] as a prefix: succeeds on exactly [kw] (no argument) + or [kw ^ " " ^ ...] (with argument), returning the stripped + argument tail. Returns [None] otherwise. This recognises both + "CHECKPOINT" and "CHECKPOINT foo" the same way, so we can + diagnose the missing-name case ourselves instead of falling + through to EC's parser. *) + let keyword_arg kw line = + if line = kw then Some "" + else if String.starts_with line (kw ^ " ") then + let n = String.length kw + 1 in + Some (String.strip + (String.sub line n (String.length line - n))) + else None + + let parse_focus arg = + if arg = "" then + raise (Parse_error "FOCUS: missing argument"); + let parts = String.split_on_char '.' arg in + let path = + try List.map int_of_string parts + with Failure _ -> + raise (Parse_error + (Printf.sprintf "FOCUS: not a path of integers: %s" arg)) + in + if List.exists (fun k -> k < 1) path then + raise (Parse_error + (Printf.sprintf "FOCUS: path indices must be >= 1: %s" arg)); + Focus path + + let parse_checkpoint name = + if name = "" then + raise (Parse_error "CHECKPOINT: missing name"); + Checkpoint name + + let parse_revert spec = + if spec = "" then + raise (Parse_error + "REVERT: missing uuid or checkpoint name"); + Revert spec + + let parse_search query = + if query = "" then + raise (Parse_error "SEARCH: missing query"); + let query = + if String.ends_with query "." + then String.sub query 0 (String.length query - 1) + else query + in + Search query + + (* LOAD "file.ec" [LINE[:COL]] [-nosmt] [-trace]. Argument errors are + signalled with [failwith] and turned into [Parse_error] below, so + they reach the wire exactly as any other line-parse error does + (including the bare "int_of_string" of a malformed LINE:COL). *) + let parse_load args = + try + let args = String.strip args in + if args = "" then failwith "LOAD: missing filename"; + (* Parse quoted or unquoted filename. *) + let filename, rest = + if args.[0] = '"' then + let close = + try String.index_from args 1 '"' + with Not_found -> + failwith "LOAD: unterminated filename" + in + let fn = String.sub args 1 (close - 1) in + let rest = String.strip ( + String.sub args (close + 1) + (String.length args - close - 1)) in + (fn, rest) + else + match String.split_on_char ' ' args with + | [] -> failwith "LOAD: missing filename" + | [f] -> (f, "") + | f :: rest -> (f, String.concat " " rest) + in + if filename = "" then failwith "LOAD: missing filename"; + (* Checked here, before anything else touches the session: the + reader would otherwise raise [Sys_error] far downstream, and + the REPL would report it as an anomaly after having already + reset the scope. *) + if not (Sys.file_exists filename) then + failwith + (Printf.sprintf "LOAD: no such file: %s" filename); + + (* Parse optional LINE[:COL] and flags (-nosmt, -trace). *) + let upto, nosmt, trace = + let words = + String.split_on_char ' ' rest + |> List.filter (fun s -> s <> "") + in + let nosmt = List.mem "-nosmt" words in + let trace = List.mem "-trace" words in + let words = + List.filter + (fun s -> s <> "-nosmt" && s <> "-trace") + words + in + let upto = match words with + | [] -> None + | [w] -> + begin match String.split_on_char ':' w with + | [line] -> + Some (int_of_string line, None) + | [line; col] -> + Some (int_of_string line, Some (int_of_string col)) + | _ -> failwith "LOAD: invalid LINE[:COL] format" + end + | _ -> failwith "LOAD: unexpected arguments" + in + (upto, nosmt, trace) + in + Load { ld_file = filename; ld_upto = upto; + ld_nosmt = nosmt; ld_trace = trace; } + with Failure msg -> raise (Parse_error msg) + + let of_line ~multi_active (raw : string) : command = + let line = String.strip raw in + if multi_active then + if line = "" then Done_multi + else Multi_line line + else + match line with + | "" -> Begin_multi + | "" -> Blank + | "QUIT" -> Quit + | "HELP" -> Help + | "UNDO" -> Undo + | "GOALS" -> Goals `One + | "GOALS ALL" -> Goals `All + | "TREE" -> Tree `One + | "TREE ALL" -> Tree `All + | "COMMIT" -> Commit + | "NEXT" -> Next + | "QUIET ON" -> Quiet true + | "QUIET OFF" -> Quiet false + | _ -> + match keyword_arg "FOCUS" line with Some a -> parse_focus a | None -> + match keyword_arg "CHECKPOINT" line with Some a -> parse_checkpoint a | None -> + match keyword_arg "REVERT" line with Some a -> parse_revert a | None -> + match keyword_arg "SEARCH" line with Some a -> parse_search a | None -> + match keyword_arg "LOAD" line with Some a -> parse_load a | None -> + Ec line +end + (* -------------------------------------------------------------------- *) let run ~relocdir ~boot ~projini (llmopts : EcOptions.llm_option) = if llmopts.llmo_help then begin @@ -42,302 +220,31 @@ let run ~relocdir ~boot ~projini (llmopts : EcOptions.llm_option) = end; let prvopts = llmopts.llmo_provers in - Random.self_init (); - - prvopts.prvo_why3server |> oiter (fun server -> - try - Why3.Prove_client.connect_external server - with Why3.Prove_client.ConnectionError e -> - Format.eprintf - "cannot connect to Why3 server `%s': %s" server e; - exit 1); - - (match relocdir with - | None -> EcCommands.addidir Filename.current_dir_name - | Some pwd -> EcCommands.addidir pwd); - - (* Prover options in effect: refreshed by [LOAD] with the loaded - file's [easycrypt.project] settings overlaid on the command-line - options, as the batch compiler does at option-parsing time. *) - let cur_prvopts = ref prvopts in - let checkmode_of (prvopts : EcOptions.prv_options) = { - EcCommands.cm_checkall = prvopts.prvo_checkall; - EcCommands.cm_timeout = odfl 3 prvopts.prvo_timeout; - EcCommands.cm_cpufactor = odfl 1 prvopts.prvo_cpufactor; - EcCommands.cm_nprovers = odfl 4 prvopts.prvo_maxjobs; - EcCommands.cm_provers = prvopts.prvo_provers; - EcCommands.cm_quorum = prvopts.prvo_quorum; - EcCommands.cm_profile = prvopts.prvo_profile; - } in - - (* ------------------------------------------------------------------ *) - (* State. *) - - (* Messages emitted by the engine during a phrase; flushed into the - next OK/ERROR reply. *) - let notices = Buffer.create 256 in - - (* Has [EcCommands.initialize] been called? Subsequent calls pass - [~restart:true]. *) - let initialized = ref false in - - (* Project-file load-path entries already added to the (global) - loader, so repeated [LOAD]s do not pile up duplicates. *) - let projdirs : (string option * string * bool) list ref = ref [] in + let st = + try EcLlmCore.create ~relocdir ~boot ~projini ~prvopts + with EcLlmCore.Init_error msg -> + Format.eprintf "%s" msg; + exit 1 + in (* True iff replies should suppress goal bodies. Toggled by QUIET. *) let quiet = ref false in - (* CHECKPOINT name -> uuid. *) - let checkpoints : (string, int) Hashtbl.t = Hashtbl.create 16 in - - (* Transcript of REPL-typed phrases that succeeded. Each entry is - [(uuid_before, src, parent, opens_at_entry)]: - - [parent]: focused handle right before the phrase ([None] iff - outside a proof); - - [opens_at_entry]: full open-handle list (focused first), used - by [Commit] to seed the sibling map when the first recorded - phrase already sits inside a frame opened by the LOAD prefix. - Trimmed by UNDO/REVERT; cleared on LOAD/Restart. *) - let transcript : - (int * string * EcCoreGoal.handle option - * EcCoreGoal.handle list) list ref = ref [] in - - (* Proof environment snapshot, refreshed at every recorded phrase. - [COMMIT] queries the proof DAG through it rather than through the - active proof, which is gone once [qed] has run. A [proofenv] is - immutable and cumulative, so the last snapshot knows about every - handle any transcript entry can mention. *) - let commit_env : EcCoreGoal.proofenv option ref = ref None in - - (* The bullet stack of the active proof at the moment REPL input - took over. Captured the first time [disable_repl_bullets] clears - a non-empty stack. Used by [Commit] to pick bullet characters - that don't collide with frames opened by the LOAD prefix. - Cleared with the transcript on LOAD/Restart. *) - let prior_bullets : EcBullets.stack option ref = ref None in - - let notifier (_ : EcGState.loglevel) (lazy msg) = - Buffer.add_string notices msg; - Buffer.add_char notices '\n' - in - - let do_initialize () = - EcCommands.initialize - ~restart:!initialized ~undo:true - ~boot ~checkmode:(checkmode_of !cur_prvopts) ~checkproof:true; - initialized := true; - (try - List.iter EcCommands.apply_pragma_option !cur_prvopts.prvo_pragmas - with EcCommands.InvalidPragma x -> - EcScope.hierror "invalid pragma: `%s'\n%!" x); - EcCommands.addnotifier notifier; - oiter (fun ppwidth -> - let gs = EcEnv.gstate (EcScope.env (EcCommands.current ())) in - EcGState.setvalue "PP:width" (`Int ppwidth) gs) - !cur_prvopts.prvo_ppwidth - in - (* ------------------------------------------------------------------ *) - (* Goal/error formatting: shared between the wire layer and the - -trace block. *) - let module Goals = struct - let format_error ?(src="") e = - let base = match e with - | EcScope.TopError (loc, e) -> - let msg = String.strip (EcPException.tostring e) in - if loc = EcLocation._dummy then msg - else Format.asprintf "%s: %s" (EcLocation.tostring loc) msg - | e -> - String.strip (EcPException.tostring e) - in - if src = "" then base - else Printf.sprintf "%s\nsource: %s" base src - - let goals_to_string ?(all=false) () = - let buf = Buffer.create 256 in - let fmt = Format.formatter_of_buffer buf in - EcCommands.pp_current_goal_or_noproof ~all fmt; - Format.pp_print_flush fmt (); - Buffer.contents buf - - (* Inline focus annotation ([focus: 1/N]) appended to reply tags - whenever the active proof has >=2 open subgoals. *) - let focus_tag () = - match EcCommands.pp_tree () with - | _ :: _ :: _ as entries -> - Printf.sprintf " [focus: 1/%d]" (List.length entries) - | _ -> "" - end in - - (* ------------------------------------------------------------------ *) - (* Frame tree: group currently-open goals by their shared multi-child - ancestors. Used by [Tree] (rendering) and [Focus] (path lookup). - The tree is a *derivation*: it depends only on [pr_opened] and - [parent_of], no recorded transcript. *) - let module FrameTree = struct - (* Internal nodes are split-point frames; leaves carry a handle - (the open goal), its index in [pr_opened] (1-based, used by - [EcCoreGoal.rotate_focus]), and its rendered text. *) - type node = - | Frame of node list (* >=2 child branches *) - | Leaf of - { idx : int (* 1-based in pr_opened *) - ; focused : bool (* idx = 1 *) - ; text : string } (* one-line conclusion *) - - (* Multi-child ancestors of [h], outermost first (= root-most - split first, deepest split last). This ordering means leaves - sharing the same OUTER frame will agree on the chain's first - element, which is what [group] partitions on. *) - let split_chain h = - let rec walk h acc = - match EcCommands.parent_of h with - | None -> acc - | Some p -> - match EcCommands.children_of p with - | [_] -> walk p acc - | _ -> walk p (p :: acc) - in - (* [walk] prepends each ancestor as we go up; the result has - outermost at the FRONT (we add it last). No reverse needed. *) - walk h [] - - (* Build the tree by grouping leaves with a common ancestor prefix. - [leaves] is a list of (chain, leaf) in [pr_opened] order. The - grouping is done recursively on the head of each chain. *) - let rec group (leaves : (EcCoreGoal.handle list * node) list) : node list = - let rec runs acc = function - | [] -> List.rev acc - | (chain, leaf) :: rest -> - match chain with - | [] -> runs (`Bare leaf :: acc) rest - | hd :: tl -> - let same_head, others = - List.partition_map (fun (c, l) -> - match c with - | h :: tail when EcCoreGoal.eq_handle h hd -> - Left (tail, l) - | _ -> Right (c, l)) - rest - in - runs (`Group ((tl, leaf) :: same_head) :: acc) others - in - List.map - (function - | `Bare leaf -> leaf - | `Group children -> Frame (group children)) - (runs [] leaves) - - (* Strip leading singleton frames so the top-level forest's - indices match what the user thinks of as "top-level subgoals - of the current frame." When all open leaves descend from a - single outermost split, the top-level forest has one Frame - containing the actual user-visible siblings; unwrap it. *) - let rec unwrap forest = - match forest with - | [Frame children] -> unwrap children - | _ -> forest - - let build () = - let handles = EcCommands.open_handles () in - let texts = EcCommands.pp_tree () in - if handles = [] then [] - else - let leaves = - List.mapi (fun i (h, (_, focused, text)) -> - let leaf = Leaf { idx = i + 1; focused; text } in - (split_chain h, leaf)) - (List.combine handles texts) - in - unwrap (group leaves) - - (* Render the tree with dotted-path labels matching what FOCUS - accepts. [all] requests full goal bodies (we re-query via - [pp_tree ~all:true] keyed by leaf index). *) - let render ?(all=false) () = - let forest = build () in - if forest = [] then "No active proof.\n" - else - let texts_all = - if all then Some (EcCommands.pp_tree ~all:true ()) - else None - in - let one_line s = - let s = - match String.index_opt s '\n' with - | None -> s - | Some k -> String.sub s 0 k - in - let limit = 80 in - if String.length s > limit - then String.sub s 0 (limit - 1) ^ "…" - else s - in - let buf = Buffer.create 256 in - let rec emit ~depth ~path = function - | Leaf { idx; focused; text } -> - let label = String.concat "." (List.rev_map string_of_int path) in - let marker = if focused then " <- focused" else "" in - for _ = 1 to depth do Buffer.add_string buf " " done; - (match texts_all with - | None -> - Buffer.add_string buf - (Printf.sprintf "[%s] %s%s\n" - label (one_line text) marker) - | Some entries -> - let (_, _, full) = - List.nth entries (idx - 1) - in - Buffer.add_string buf - (Printf.sprintf "[%s]%s\n%s\n" label marker full)) - | Frame children -> - List.iteri (fun i child -> - emit ~depth:(depth + 1) ~path:((i + 1) :: path) child) - children - in - List.iteri (fun i node -> - emit ~depth:0 ~path:[i + 1] node) - forest; - Buffer.contents buf - - (* Resolve a dotted path against the tree. Returns [Ok idx] where - [idx] is the 1-based position in [pr_opened] of the selected - leaf, or [Error msg]. *) - let resolve_path (path : int list) : (int, string) result = - let forest = build () in - let rec walk ~components nodes = - match components with - | [] -> Error "FOCUS: path must select a leaf goal" - | k :: rest -> - if k < 1 || k > List.length nodes then - Error (Printf.sprintf - "FOCUS: index %d out of range (1..%d)" - k (List.length nodes)) - else - match List.nth nodes (k - 1), rest with - | Leaf { idx; _ }, [] -> Ok idx - | Leaf _, _ -> - Error "FOCUS: path overshoots a leaf goal" - | Frame _, [] -> - Error "FOCUS: path must select a leaf goal, \ - not a frame" - | Frame kids, _ -> walk ~components:rest kids - in - if forest = [] then Error "FOCUS: no active proof" - else walk ~components:path forest - end in - - - (* ------------------------------------------------------------------ *) - (* OK/ERROR/ wire envelope. *) + (* OK/ERROR/ wire envelope: the only printers. *) let had_error = ref false in let module Wire = struct - let reply_ok ?(tag="") body = - let n = Buffer.contents notices in - Printf.printf "OK [uuid:%d]%s\n" (EcCommands.uuid ()) tag; + let reply_ok (r : EcLlmCore.reply) = + let body = + match r.EcLlmCore.body with + | EcLlmCore.Text body -> body + | EcLlmCore.Goals -> + if !quiet then "" else EcLlmCore.current_goals st + in + Printf.printf "OK [uuid:%d]%s\n" r.EcLlmCore.uuid r.EcLlmCore.tag; + let n = r.EcLlmCore.notices in if n <> "" then print_string n; if body <> "" then begin print_string body; @@ -345,712 +252,33 @@ let run ~relocdir ~boot ~projini (llmopts : EcOptions.llm_option) = if len > 0 && body.[len - 1] <> '\n' then print_char '\n' end; - Printf.printf "\n%!"; - Buffer.clear notices - - let reply_ok_goals ?(all=false) () = - let tag = Goals.focus_tag () in - if !quiet then reply_ok ~tag "" - else reply_ok ~tag (Goals.goals_to_string ~all ()) + Printf.printf "\n%!" - let reply_error msg = + let reply_failure (f : EcLlmCore.failure) = had_error := true; - let goals = Goals.goals_to_string () in - Printf.printf "ERROR [uuid:%d]\n%s\n" (EcCommands.uuid ()) msg; + let goals = f.EcLlmCore.goals in + Printf.printf "ERROR [uuid:%d]\n%s\n" + f.EcLlmCore.uuid f.EcLlmCore.message; if goals <> "" then begin print_string goals; let len = String.length goals in if len > 0 && goals.[len - 1] <> '\n' then print_char '\n' end; - Printf.printf "\n%!"; - Buffer.clear notices - end in - - (* ------------------------------------------------------------------ *) - (* Transcript manipulation. *) - let module Transcript = struct - let trim target = - transcript := - List.filter - (fun (uuid_before, _, _, _) -> uuid_before < target) - !transcript - - let clear () = - transcript := []; - prior_bullets := None; - commit_env := None - end in + Printf.printf "\n%!" - (* ------------------------------------------------------------------ *) - (* Process a single EasyCrypt command, respecting [gl_fail]. When - [~record:true], append a transcript entry on success: the parent - handle (focused goal before the phrase) and the open-handle list, - which together let [Commit] reconstruct bullet structure. *) - let process_action ?(record=false) ~src (p : EP.global) = - let loc = p.EP.gl_action.EcLocation.pl_loc in - let pre_uuid = EcCommands.uuid () in - let opens_pre = - if record then EcCommands.open_handles () else [] - in - let parent = - match opens_pre with h :: _ -> Some h | [] -> None - in - let succeeded = ref false in - begin try - ignore (EcCommands.process ~src p.EP.gl_action : float option); - succeeded := true - with - | EcCommands.Restart -> raise EcCommands.Restart - | _ when p.EP.gl_fail -> () - | e -> raise (EcScope.toperror_of_exn ~gloc:loc e) - end; - if !succeeded && p.EP.gl_fail then - raise (EcScope.toperror_of_exn ~gloc:loc - (EcScope.HiScopeError (None, - "this command is expected to fail"))); - (* Queries only inspect the environment: they neither advance the - proof nor belong in the body COMMIT emits. *) - let is_query = - match EcLocation.unloc p.EP.gl_action with - | EP.Gprint _ | EP.Gsearch _ | EP.Glocate _ -> true - | _ -> false - in - if record && !succeeded && not p.EP.gl_fail && not is_query then begin - transcript := (pre_uuid, src, parent, opens_pre) :: !transcript; - (* Keep the newest non-empty snapshot: a phrase that closes the - proof ([qed]) leaves no active proof, and precisely then we - still need the environment the previous phrases built. *) - match EcCommands.current_proofenv () with - | None -> () - | Some _ as penv -> commit_env := penv - end - in + (* Render an operation's outcome. *) + let reply = function + | Ok reply -> reply_ok reply + | Error failed -> reply_failure failed - (* ------------------------------------------------------------------ *) - (* COMMIT: replay the transcript against the proof DAG (parent_of / - children_of, backed by [EcCoreGoal.pr_parent]), inserting bullets - at multi-child splits. Levels the LOAD prefix's [puc_bullets] stack - already opened are addressed with that frame's own token; deeper - levels get fresh tokens, chosen so they collide with neither the - stack nor each other. *) - let module Commit = struct - (* Token order matches PR 1017's lexer: -, +, *, --, ++, **, - ---, +++, *** ... *) - let token_at_index i = - let chars = [| "-"; "+"; "*" |] in - let rep = i / 3 + 1 in - let chr = chars.(i mod 3) in - String.concat "" (List.init rep (fun _ -> chr)) - - (* DAG queries go through the snapshot recorded at the last phrase, - so COMMIT still sees the structure after [qed]. Fall back to the - live proof when no phrase was recorded under a proof. *) - let parent_of h = - match !commit_env with - | Some penv -> EcCoreGoal.parent_of_handle penv h - | None -> EcCommands.parent_of h + (* Same, for operations that may end the session. *) + let answer = function + | EcLlmCore.Quit -> exit 0 + | EcLlmCore.Done outcome -> reply outcome - let children_of h = - match !commit_env with - | Some penv -> EcCoreGoal.children_of_handle penv h - | None -> EcCommands.children_of h - - let proof_text () = - let entries = List.rev !transcript in - let buf = Buffer.create 1024 in - let emit_indent depth = - for _ = 1 to depth do Buffer.add_string buf " " done - in - let module Hmap = - Map.Make (struct - type t = EcCoreGoal.handle - let compare = compare - end) - in - let sibling_depth : int Hmap.t ref = ref Hmap.empty in - let current_depth = ref 0 in - let bullet_to_string (b : EcParsetree.bullet) = - let ch = - match b.b_kind with - | `Minus -> "-" - | `Plus -> "+" - | `Star -> "*" - in - String.concat "" (List.init b.b_count (fun _ -> ch)) - in - (* Bullet frames the LOAD prefix left open, OUTERMOST first (the - stack stores the innermost frame at its head). Frame [t_d] is - the one whose siblings live at emitted depth [d]. *) - let frames : EcBullets.frame list = - match !prior_bullets with - | None -> [] - | Some stack -> List.rev stack - in - let in_use_tokens = - List.map - (fun (f : EcBullets.frame) -> bullet_to_string f.bf_bullet) - frames - in - let depth_cache : (int, string) Hashtbl.t = Hashtbl.create 8 in - let next_tok_idx = ref 0 in - let assigned_tokens = ref [] in - (* Depths 1..k address the next sibling of a frame the prefix - already opened, and strict bullets accepts nothing but that - frame's own token there. Deeper levels get fresh tokens, so - pre-populate the cache before any fresh pick happens. *) - List.iteri (fun i (f : EcBullets.frame) -> - let t = bullet_to_string f.bf_bullet in - Hashtbl.replace depth_cache (i + 1) t; - assigned_tokens := t :: !assigned_tokens) - frames; - let bullet_for_depth d = - match Hashtbl.find_opt depth_cache d with - | Some t -> t - | None -> - let rec pick () = - let t = token_at_index !next_tok_idx in - incr next_tok_idx; - if List.mem t in_use_tokens || List.mem t !assigned_tokens - then pick () - else t - in - let t = pick () in - assigned_tokens := t :: !assigned_tokens; - Hashtbl.add depth_cache d t; - t - in - (* Seed: the goals already open when the first recorded phrase ran - were left there by the LOAD prefix, so COMMIT must place each of - them at the depth the prefix's own bullets put it at. A frame - with floor [f] is discharged once [f] goals remain, hence it - still owns the first [n - f] goals of the focused-first list; - a goal covered by [c] frames sits at depth [c + 1]. - Nothing to seed when the prefix left no frame and a single goal - (the REPL just continues on the prefix's own focus). *) - (match entries with - | (_, _, Some _, (_ :: _ as opens)) :: _ - when frames <> [] || List.length opens >= 2 -> - let n = List.length opens in - List.iteri (fun i h -> - let pos = i + 1 in - let covering = - List.length - (List.filter - (fun (f : EcBullets.frame) -> pos <= n - f.bf_floor) - frames) - in - sibling_depth := Hmap.add h (covering + 1) !sibling_depth) - opens - | _ -> ()); - List.iter (fun (_uuid, src, parent_opt, _opens) -> - match parent_opt with - | None -> - Buffer.add_string buf src; - Buffer.add_char buf '\n' - | Some parent -> - (* Walk upward via pr_parent until we hit a registered - sibling ancestor. If found, emit its bullet and consume - the registration. *) - let rec find_ancestor h = - match Hmap.find_opt h !sibling_depth with - | Some d -> Some (h, d) - | None -> - match parent_of h with - | Some p -> find_ancestor p - | None -> None - in - (match find_ancestor parent with - | Some (h, d) -> - emit_indent (d - 1); - Buffer.add_string buf (bullet_for_depth d); - Buffer.add_char buf ' '; - current_depth := d; - sibling_depth := Hmap.remove h !sibling_depth - | None -> - emit_indent !current_depth); - Buffer.add_string buf src; - Buffer.add_char buf '\n'; - (* Register fresh siblings: walk the subtree rooted at - [parent], finding every multi-child split, and register - each such child at the right depth. Single-child links - are continuations and don't bump depth; multi-child - links do. A compound phrase like [split; split.] can - produce nested splits within one phrase. *) - let rec walk h d = - match children_of h with - | [c] -> walk c d - | (_ :: _ :: _) as cs -> - List.iter - (fun c -> - sibling_depth := - Hmap.add c d !sibling_depth; - walk c (d + 1)) - cs - | [] -> () - in - walk parent (!current_depth + 1) - ) entries; - Buffer.contents buf - end in - - (* ------------------------------------------------------------------ *) - (* Process EasyCrypt input typed at the REPL prompt (single phrase - or a line ending with a "."). *) - let process_ec_input input = - Buffer.clear notices; - (* On the first REPL phrase of each proof, capture the bullet stack - the LOAD prefix left so COMMIT can avoid token collisions with - it. Subsequent calls return [None] and don't clobber the snapshot. *) - (match EcCommands.disable_repl_bullets () with - | None -> () - | Some _ as snapshot -> prior_bullets := snapshot); - let reader = EcIo.from_string input in - let last_src = ref "" in - begin try - let (src, prog) = EcIo.xparse reader in - let src = String.strip src in - last_src := src; - begin match EcLocation.unloc prog with - | EP.P_Prog (commands, _) -> - List.iter (process_action ~record:true ~src) commands; - Wire.reply_ok_goals () - | EP.P_Undo i -> - EcCommands.undo i; - Transcript.trim i; - Wire.reply_ok_goals () - | EP.P_Exit -> - EcIo.finalize reader; exit 0 - | EP.P_DocComment doc -> - EcCommands.doc_comment doc; - Wire.reply_ok "" - end - with - | EcCommands.Restart -> - do_initialize (); - Transcript.clear (); - Wire.reply_ok "Session restarted" - | e -> - Wire.reply_error (Goals.format_error ~src:!last_src e) - end; - EcIo.finalize reader - in - - (* ------------------------------------------------------------------ *) - (* LOAD "file.ec" [LINE[:COL]] [-nosmt] [-trace]. *) - let module Load = struct - let handle args = - Buffer.clear notices; - let args = String.strip args in - let last_src = ref "" in - let trace_prefix = ref "" in - let exception Trace_failed of exn in - - try - if args = "" then failwith "LOAD: missing filename"; - (* Parse quoted or unquoted filename. *) - let filename, rest = - if args.[0] = '"' then - let close = - try String.index_from args 1 '"' - with Not_found -> - failwith "LOAD: unterminated filename" - in - let fn = String.sub args 1 (close - 1) in - let rest = String.strip ( - String.sub args (close + 1) - (String.length args - close - 1)) in - (fn, rest) - else - match String.split_on_char ' ' args with - | [] -> failwith "LOAD: missing filename" - | [f] -> (f, "") - | f :: rest -> (f, String.concat " " rest) - in - if filename = "" then failwith "LOAD: missing filename"; - (* Checked here, before anything else touches the session: the - reader would otherwise raise [Sys_error] far downstream, and - the REPL would report it as an anomaly after having already - reset the scope. *) - if not (Sys.file_exists filename) then - failwith - (Printf.sprintf "LOAD: no such file: %s" filename); - - (* Parse optional LINE[:COL] and flags (-nosmt, -trace). *) - let upto, nosmt, trace = - let words = - String.split_on_char ' ' rest - |> List.filter (fun s -> s <> "") - in - let nosmt = List.mem "-nosmt" words in - let trace = List.mem "-trace" words in - let words = - List.filter - (fun s -> s <> "-nosmt" && s <> "-trace") - words - in - let upto = match words with - | [] -> None - | [w] -> - begin match String.split_on_char ':' w with - | [line] -> - Some (int_of_string line, None) - | [line; col] -> - Some (int_of_string line, Some (int_of_string col)) - | _ -> failwith "LOAD: invalid LINE[:COL] format" - end - | _ -> failwith "LOAD: unexpected arguments" - in - (upto, nosmt, trace) - in - - begin try - ignore (EcLoader.getkind - (Filename.extension filename) : EcLoader.kind) - with EcLoader.BadExtension ext -> - failwith (Format.sprintf - "unknown file extension: %s" ext) - end; - - (* Apply the configuration attached to the loaded file's - [easycrypt.project], as the batch compiler does when the - file is given on the command line: refresh the prover - options (timeout, provers, pragmas, ...) and extend the - load path with the project's include dirs. *) - let ini = Option.to_list (projini (Some filename)) in - cur_prvopts := - EcOptions.prv_options_with_ini ini llmopts.llmo_provers; - List.iter (fun ((nm, dir, isrec) as entry) -> - if not (List.mem entry !projdirs) then begin - projdirs := entry :: !projdirs; - EcCommands.addidir - ?namespace:(omap (fun nm -> `Named nm) nm) - ~recursive:isrec dir - end) - (EcOptions.ini_loadpath ini); - - do_initialize (); - Hashtbl.clear checkpoints; - Transcript.clear (); - EcCommands.addidir (Filename.dirname filename); - EcCommands.set_current_path (Filename.dirname filename); - - let reader = EcIo.from_file filename in - - let past_upto (loc : EcLocation.t) = - match upto with - | None -> false - | Some (line, col) -> - let (el, ec) = loc.loc_end in - el > line || (el = line && match col with - | None -> false - | Some c -> ec > c) - in - - let last_loc = ref None in - - (* For -trace: lazy whole-file bytes, used to slice the exact - source text of a sentence by byte offsets. *) - let input_bytes = lazy ( - let ic = open_in_bin filename in - let n = in_channel_length ic in - let b = Bytes.create n in - really_input ic b 0 n; - close_in ic; - Bytes.unsafe_to_string b) - in - let sentence_source (loc : EcLocation.t) = - let s = Lazy.force input_bytes in - let lo = max 0 loc.EcLocation.loc_bchar in - let hi = min (String.length s) loc.EcLocation.loc_echar in - if hi <= lo then "" else String.sub s lo (hi - lo) - in - - (* For -trace: defer execution of the last sentence within the - prefix so we can capture goals before and after it. *) - let pending : (string * EP.global) option ref = ref None in - let flush_pending () = - match !pending with - | None -> () - | Some (src, p) -> - last_src := src; - process_action ~src p; - last_loc := Some p.EP.gl_action.EcLocation.pl_loc; - pending := None - in - let step src p = - let loc = p.EP.gl_action.EcLocation.pl_loc in - if past_upto loc then raise Exit; - if trace then begin - flush_pending (); - pending := Some (src, p) - end else begin - last_src := src; - process_action ~src p; - last_loc := Some loc - end - in - - if nosmt then EcCommands.pragma_check `WeakCheck; - - begin try while true do - let (src, prog) = EcIo.xparse reader in - let src = String.strip src in - match EcLocation.unloc prog with - | EP.P_Prog (commands, locterm) -> - List.iter (step src) commands; - if locterm then raise Exit - | EP.P_Undo i -> - last_src := src; - EcCommands.undo i - | EP.P_Exit -> - raise Exit - | EP.P_DocComment doc -> - last_src := src; - EcCommands.doc_comment doc - done with - | Exit | End_of_file -> () - | e -> - EcIo.finalize reader; - if nosmt then EcCommands.pragma_check `Check; - raise e - end; - - EcIo.finalize reader; - - if nosmt then EcCommands.pragma_check `Check; - - (* If -trace is set, the last in-prefix sentence is still - pending. Run it under goal capture and build the - BEFORE/TACTIC/AFTER/SUMMARY response body. *) - let body = - if not trace then - Goals.goals_to_string () - else - let pre_state = - match !pending with - | None -> `Nothing - | Some _ when not (EcCommands.in_proof ()) -> `NotInProof - | Some (src, p) -> `Ready (src, p) - in - match pre_state with - (* Tracing is off the table, but the prefix is not: run the - sentence we deferred so that the session ends up exactly - where a plain LOAD of the same prefix would leave it. A - failure inside the flush is reported by the enclosing - handler, as any prefix failure is. *) - | `Nothing -> - flush_pending (); - failwith "trace: nothing to trace" - | `NotInProof -> - flush_pending (); - failwith - "trace: target sentence is not in a proof context" - | `Ready (src, p) -> - let loc = p.EP.gl_action.EcLocation.pl_loc in - let (sl, sc) = loc.EcLocation.loc_start in - let (el, ec) = loc.EcLocation.loc_end in - let before_goals = EcCommands.pp_all_goals () in - let n1 = List.length before_goals in - let buf = Buffer.create 1024 in - let fmt = Format.formatter_of_buffer buf in - Format.fprintf fmt - "=== BEFORE: line %d (col %d) ===@\n" sl sc; - EcCommands.pp_current_goal_or_noproof ~all:false fmt; - Format.fprintf fmt - "@\n=== TACTIC (lines %d:%d - %d:%d) ===@\n%s@\n@\n" - sl sc el ec (sentence_source loc); - last_src := src; - begin - try - process_action ~src p; - last_loc := Some loc; - pending := None; - let after_goals = EcCommands.pp_all_goals () in - let n2 = List.length after_goals in - Format.fprintf fmt - "=== AFTER: line %d (col %d) ===@\n" sl sc; - let before_set = - List.fold_left - (fun s g -> EcMaps.Sstr.add g s) - EcMaps.Sstr.empty before_goals - in - (* The new focused goal always counts as "modified" - (its focus status changed even if its text matches - an old sibling); the rest are printed only if they - didn't appear in BEFORE. *) - let to_print = - match after_goals with - | [] -> [] - | head :: tl -> - head :: - List.filter - (fun g -> not (EcMaps.Sstr.mem g before_set)) - tl - in - begin match to_print with - | [] -> Format.fprintf fmt "(no open goals)@\n" - | _ -> - List.iteri (fun i g -> - if i > 0 then Format.fprintf fmt "@\n"; - Format.fprintf fmt "%s@\n" g) - to_print - end; - Format.fprintf fmt - "@\n=== SUMMARY ===@\nopen goals: %d -> %d@\n" n1 n2; - Format.pp_print_flush fmt (); - Buffer.contents buf - with e -> - Format.fprintf fmt - "=== AFTER: line %d (col %d) ===@\n@\n" - sl sc; - Format.pp_print_flush fmt (); - trace_prefix := Buffer.contents buf; - raise (Trace_failed e) - end - in - - let tag = - let loaded = - match !last_loc with - | None -> "" - | Some loc -> - let (el, _) = loc.EcLocation.loc_end in - Printf.sprintf " [loaded:%s:%d]" filename el - in - loaded ^ Goals.focus_tag () - in - Wire.reply_ok ~tag body - - with - | EcCommands.Restart -> - do_initialize (); - Hashtbl.clear checkpoints; - Transcript.clear (); - Wire.reply_ok "Session restarted" - | Trace_failed e -> - let msg = Goals.format_error ~src:!last_src e in - Wire.reply_error (!trace_prefix ^ msg) - | Failure s -> - Wire.reply_error s - | e -> - Wire.reply_error (Goals.format_error ~src:!last_src e) - end in - - (* ------------------------------------------------------------------ *) - (* Main loop: line-by-line dispatcher. *) - - (* ------------------------------------------------------------------ *) - (* Surface command vocabulary. Parsing turns each stdin line into one - of these, and dispatch is a flat pattern-match. Argument - parsing/validation lives in [Parse]; commands that interact with - mutable state (checkpoints table, multi-line buffer) carry only - the raw user-supplied data and let [Dispatch] do the lookup. *) - let module Parse = struct - type command = - | Quit - | Help - | Undo - | Goals of [`One | `All] - | Tree of [`One | `All] - | Commit - | Focus of int list (* dotted path; [k] = "FOCUS k" *) - | Next - | Checkpoint of string - | Revert of string (* uuid-or-name; Dispatch resolves *) - | Quiet of bool - | Search of string (* trailing "." already stripped *) - | Load of string (* raw arg tail; Load.handle parses *) - | Ec of string (* fall-through: raw EasyCrypt input *) - | Begin_multi - | Done_multi - | Multi_line of string - | Blank - - exception Parse_error of string - - (* Match [kw] as a prefix: succeeds on exactly [kw] (no argument) - or [kw ^ " " ^ ...] (with argument), returning the stripped - argument tail. Returns [None] otherwise. This recognises both - "CHECKPOINT" and "CHECKPOINT foo" the same way, so we can - diagnose the missing-name case ourselves instead of falling - through to EC's parser. *) - let keyword_arg kw line = - if line = kw then Some "" - else if String.starts_with line (kw ^ " ") then - let n = String.length kw + 1 in - Some (String.strip - (String.sub line n (String.length line - n))) - else None - - let parse_focus arg = - if arg = "" then - raise (Parse_error "FOCUS: missing argument"); - let parts = String.split_on_char '.' arg in - let path = - try List.map int_of_string parts - with Failure _ -> - raise (Parse_error - (Printf.sprintf "FOCUS: not a path of integers: %s" arg)) - in - if List.exists (fun k -> k < 1) path then - raise (Parse_error - (Printf.sprintf "FOCUS: path indices must be >= 1: %s" arg)); - Focus path - - let parse_checkpoint name = - if name = "" then - raise (Parse_error "CHECKPOINT: missing name"); - Checkpoint name - - let parse_revert spec = - if spec = "" then - raise (Parse_error - "REVERT: missing uuid or checkpoint name"); - Revert spec - - let parse_search query = - if query = "" then - raise (Parse_error "SEARCH: missing query"); - let query = - if String.ends_with query "." - then String.sub query 0 (String.length query - 1) - else query - in - Search query - - let parse_load args = - (* [Load.handle] accepts an empty argument and reports a - specific error; keep that responsibility there. *) - Load args - - let of_line ~multi_active (raw : string) : command = - let line = String.strip raw in - if multi_active then - if line = "" then Done_multi - else Multi_line line - else - match line with - | "" -> Begin_multi - | "" -> Blank - | "QUIT" -> Quit - | "HELP" -> Help - | "UNDO" -> Undo - | "GOALS" -> Goals `One - | "GOALS ALL" -> Goals `All - | "TREE" -> Tree `One - | "TREE ALL" -> Tree `All - | "COMMIT" -> Commit - | "NEXT" -> Next - | "QUIET ON" -> Quiet true - | "QUIET OFF" -> Quiet false - | _ -> - match keyword_arg "FOCUS" line with Some a -> parse_focus a | None -> - match keyword_arg "CHECKPOINT" line with Some a -> parse_checkpoint a | None -> - match keyword_arg "REVERT" line with Some a -> parse_revert a | None -> - match keyword_arg "SEARCH" line with Some a -> parse_search a | None -> - match keyword_arg "LOAD" line with Some a -> parse_load a | None -> - Ec line + let reply_error msg = + reply_failure (EcLlmCore.make_failure st msg) end in (* ------------------------------------------------------------------ *) @@ -1062,7 +290,7 @@ let run ~relocdir ~boot ~projini (llmopts : EcOptions.llm_option) = let module Dispatch = struct let do_help () = - Buffer.clear notices; + EcLlmCore.clear_notices st; let buf = Buffer.create 4096 in let path = llm_guide_path () in begin try @@ -1071,75 +299,16 @@ let run ~relocdir ~boot ~projini (llmopts : EcOptions.llm_option) = Buffer.add_char buf (input_char ic) done with End_of_file -> () end; close_in ic; - Wire.reply_ok (Buffer.contents buf) + Wire.reply_ok + (EcLlmCore.make_reply st (EcLlmCore.Text (Buffer.contents buf))) with Sys_error e -> Wire.reply_error (Printf.sprintf "cannot read guide: %s" e) end - let do_undo () = - Buffer.clear notices; - let uuid = EcCommands.uuid () in - if uuid > 0 then begin - EcCommands.undo (uuid - 1); - Transcript.trim (uuid - 1); - Wire.reply_ok_goals () - end else - Wire.reply_error "nothing to undo" - - let do_focus_request request = - (* [request] is the user's intent normalized: - - [`Next] = rotate to the second open goal (or stay if <=1) - - [`Path p] = resolve dotted path [p] against the frame tree - and focus the matching leaf. *) - Buffer.clear notices; - let resolved = - match request with - | `Next -> - let n = List.length (EcCommands.open_handles ()) in - Ok (if n <= 1 then 1 else 2) - | `Path path -> FrameTree.resolve_path path - in - match resolved with - | Error msg -> Wire.reply_error msg - | Ok target -> - match EcCommands.focus_goal target with - | Ok _ -> Wire.reply_ok_goals () - | Error msg -> Wire.reply_error msg - - let do_checkpoint name = - Buffer.clear notices; - Hashtbl.replace checkpoints name (EcCommands.uuid ()); - Wire.reply_ok (Printf.sprintf - "checkpoint '%s' set at uuid %d" name (EcCommands.uuid ())) - - let do_revert spec = - Buffer.clear notices; - let target = - try Some (int_of_string spec) - with Failure _ -> Hashtbl.find_opt checkpoints spec - in - match target with - | None -> - Wire.reply_error (Printf.sprintf - "REVERT: '%s' is not a valid uuid or checkpoint name" spec) - | Some target -> - let uuid = EcCommands.uuid () in - if target < 0 || target > uuid then - Wire.reply_error (Printf.sprintf - "REVERT: uuid %d out of range [0, %d]" target uuid) - else begin - EcCommands.undo target; - Transcript.trim target; - Wire.reply_ok_goals () - end - let do_quiet on = - Buffer.clear notices; + EcLlmCore.clear_notices st; quiet := on; - Wire.reply_ok "" - - let do_search query = - process_ec_input (Printf.sprintf "search %s." query) + Wire.reply_ok (EcLlmCore.make_reply st (EcLlmCore.Text "")) let do_begin_multi () = Buffer.clear multi_buf; @@ -1149,7 +318,7 @@ let run ~relocdir ~boot ~projini (llmopts : EcOptions.llm_option) = let input = Buffer.contents multi_buf in Buffer.clear multi_buf; in_multi := false; - if input <> "" then process_ec_input input + if input <> "" then Wire.answer (EcLlmCore.step st input) let do_multi_line s = if Buffer.length multi_buf > 0 then @@ -1161,35 +330,25 @@ let run ~relocdir ~boot ~projini (llmopts : EcOptions.llm_option) = | Blank -> () | Quit -> exit 0 | Help -> do_help () - | Undo -> do_undo () - | Goals `One -> - Buffer.clear notices; - Wire.reply_ok ~tag:(Goals.focus_tag ()) - (Goals.goals_to_string ()) - | Goals `All -> - Buffer.clear notices; - Wire.reply_ok ~tag:(Goals.focus_tag ()) - (Goals.goals_to_string ~all:true ()) - | Tree `One -> - Buffer.clear notices; - Wire.reply_ok ~tag:(Goals.focus_tag ()) - (FrameTree.render ()) - | Tree `All -> - Buffer.clear notices; - Wire.reply_ok ~tag:(Goals.focus_tag ()) - (FrameTree.render ~all:true ()) - | Commit -> - Buffer.clear notices; - Wire.reply_ok ~tag:(Goals.focus_tag ()) - (Commit.proof_text ()) - | Focus path -> do_focus_request (`Path path) - | Next -> do_focus_request `Next - | Checkpoint n -> do_checkpoint n - | Revert s -> do_revert s + | Undo -> Wire.reply (EcLlmCore.undo st) + | Goals `One -> Wire.reply (EcLlmCore.goals st ~all:false) + | Goals `All -> Wire.reply (EcLlmCore.goals st ~all:true) + | Tree `One -> Wire.reply (EcLlmCore.tree st ~all:false) + | Tree `All -> Wire.reply (EcLlmCore.tree st ~all:true) + | Commit -> Wire.reply (EcLlmCore.commit st) + | Focus path -> Wire.reply (EcLlmCore.focus st (`Path path)) + | Next -> Wire.reply (EcLlmCore.focus st `Next) + | Checkpoint n -> Wire.reply (EcLlmCore.checkpoint st ~name:n) + | Revert s -> Wire.reply (EcLlmCore.revert st s) | Quiet on -> do_quiet on - | Search q -> do_search q - | Load args -> Load.handle args - | Ec input -> process_ec_input input + | Search q -> Wire.answer (EcLlmCore.search st ~pattern:q) + | Load args -> + Wire.reply (EcLlmCore.load st + ~file:args.Parse.ld_file + ~upto:args.Parse.ld_upto + ~nosmt:args.Parse.ld_nosmt + ~trace:args.Parse.ld_trace) + | Ec input -> Wire.answer (EcLlmCore.step st input) | Begin_multi -> do_begin_multi () | Done_multi -> do_done_multi () | Multi_line s -> do_multi_line s @@ -1198,9 +357,7 @@ let run ~relocdir ~boot ~projini (llmopts : EcOptions.llm_option) = (* ------------------------------------------------------------------ *) (* Main loop. *) - do_initialize (); - - Printf.printf "READY [uuid:%d]\n\n%!" (EcCommands.uuid ()); + Printf.printf "READY [uuid:%d]\n\n%!" (EcLlmCore.uuid st); (* Input source: stdin by default, or the -eval string when given. For -eval, we split on newlines up front (no lazy channel), which diff --git a/src/ecLlmCore.ml b/src/ecLlmCore.ml new file mode 100644 index 000000000..672b06367 --- /dev/null +++ b/src/ecLlmCore.ml @@ -0,0 +1,1003 @@ +(* -------------------------------------------------------------------- *) +(* Engine-facing core of the LLM interaction protocol. See + [ecLlmCore.mli]. This module owns the session state and implements + one function per meta-command; it never prints and never exits. The + text envelope ([OK]/[ERROR]/[]) is the front-end's business. *) + +open EcUtils + +module EP = EcParsetree + +(* -------------------------------------------------------------------- *) +type body = + | Goals + | Text of string + +type reply = { + uuid : int; + tag : string; + notices : string; + body : body; + changed : bool; +} + +type failure = { + uuid : int; + message : string; + goals : string; + notices : string; +} + +type answer = + | Done of (reply, failure) result + | Quit + +exception Init_error of string + +(* -------------------------------------------------------------------- *) +(* Session state. The proof engine ([EcCommands]) is a global mutable + singleton, so at most one [state] may exist per process. *) +type state = { + (* Prover options as given on the command line: the base [LOAD] + overlays the loaded file's [easycrypt.project] settings onto. *) + base_prvopts : EcOptions.prv_options; + + (* Resolves the [easycrypt.project] context of a file path. *) + projini : string option -> EcOptions.ini_context option; + + boot : bool; + + (* Prover options in effect: refreshed by [LOAD] with the loaded + file's [easycrypt.project] settings overlaid on the command-line + options, as the batch compiler does at option-parsing time. *) + cur_prvopts : EcOptions.prv_options ref; + + (* Messages emitted by the engine during a phrase; flushed into the + next reply. *) + notices : Buffer.t; + + (* Has [EcCommands.initialize] been called? Subsequent calls pass + [~restart:true]. *) + initialized : bool ref; + + (* Project-file load-path entries already added to the (global) + loader, so repeated [LOAD]s do not pile up duplicates. *) + projdirs : (string option * string * bool) list ref; + + (* CHECKPOINT name -> uuid. *) + checkpoints : (string, int) Hashtbl.t; + + (* Transcript of REPL-typed phrases that succeeded. Each entry is + [(uuid_before, src, parent, opens_at_entry)]: + - [parent]: focused handle right before the phrase ([None] iff + outside a proof); + - [opens_at_entry]: full open-handle list (focused first), used + by [Commit] to seed the sibling map when the first recorded + phrase already sits inside a frame opened by the LOAD prefix. + Trimmed by UNDO/REVERT; cleared on LOAD/Restart. *) + transcript : + (int * string * EcCoreGoal.handle option + * EcCoreGoal.handle list) list ref; + + (* Proof environment snapshot, refreshed at every recorded phrase. + [COMMIT] queries the proof DAG through it rather than through the + active proof, which is gone once [qed] has run. A [proofenv] is + immutable and cumulative, so the last snapshot knows about every + handle any transcript entry can mention. *) + commit_env : EcCoreGoal.proofenv option ref; + + (* The bullet stack of the active proof at the moment REPL input + took over. Captured the first time [disable_repl_bullets] clears + a non-empty stack. Used by [Commit] to pick bullet characters + that don't collide with frames opened by the LOAD prefix. + Cleared with the transcript on LOAD/Restart. *) + prior_bullets : EcBullets.stack option ref; +} + +(* -------------------------------------------------------------------- *) +let checkmode_of (prvopts : EcOptions.prv_options) = { + EcCommands.cm_checkall = prvopts.prvo_checkall; + EcCommands.cm_timeout = odfl 3 prvopts.prvo_timeout; + EcCommands.cm_cpufactor = odfl 1 prvopts.prvo_cpufactor; + EcCommands.cm_nprovers = odfl 4 prvopts.prvo_maxjobs; + EcCommands.cm_provers = prvopts.prvo_provers; + EcCommands.cm_quorum = prvopts.prvo_quorum; + EcCommands.cm_profile = prvopts.prvo_profile; +} + +let notifier (st : state) = + fun (_ : EcGState.loglevel) (lazy msg) -> + Buffer.add_string st.notices msg; + Buffer.add_char st.notices '\n' + +let do_initialize (st : state) = + let initialized = st.initialized in + let cur_prvopts = st.cur_prvopts in + EcCommands.initialize + ~restart:!initialized ~undo:true + ~boot:st.boot ~checkmode:(checkmode_of !cur_prvopts) ~checkproof:true; + initialized := true; + (try + List.iter EcCommands.apply_pragma_option !cur_prvopts.prvo_pragmas + with EcCommands.InvalidPragma x -> + EcScope.hierror "invalid pragma: `%s'\n%!" x); + EcCommands.addnotifier (notifier st); + oiter (fun ppwidth -> + let gs = EcEnv.gstate (EcScope.env (EcCommands.current ())) in + EcGState.setvalue "PP:width" (`Int ppwidth) gs) + !cur_prvopts.prvo_ppwidth + +(* -------------------------------------------------------------------- *) +let create ~relocdir ~boot ~projini ~prvopts = + Random.self_init (); + + prvopts.EcOptions.prvo_why3server |> oiter (fun server -> + try + Why3.Prove_client.connect_external server + with Why3.Prove_client.ConnectionError e -> + raise (Init_error (Format.asprintf + "cannot connect to Why3 server `%s': %s" server e))); + + (match relocdir with + | None -> EcCommands.addidir Filename.current_dir_name + | Some pwd -> EcCommands.addidir pwd); + + let st = { + base_prvopts = prvopts; + projini; + boot; + cur_prvopts = ref prvopts; + notices = Buffer.create 256; + initialized = ref false; + projdirs = ref []; + checkpoints = Hashtbl.create 16; + transcript = ref []; + commit_env = ref None; + prior_bullets = ref None; + } in + + do_initialize st; st + +(* -------------------------------------------------------------------- *) +(* Goal/error formatting: shared between the reply layer and the + -trace block. *) +module Goals = struct + let format_error ?(src="") e = + let base = match e with + | EcScope.TopError (loc, e) -> + let msg = String.strip (EcPException.tostring e) in + if loc = EcLocation._dummy then msg + else Format.asprintf "%s: %s" (EcLocation.tostring loc) msg + | e -> + String.strip (EcPException.tostring e) + in + if src = "" then base + else Printf.sprintf "%s\nsource: %s" base src + + let goals_to_string ?(all=false) () = + let buf = Buffer.create 256 in + let fmt = Format.formatter_of_buffer buf in + EcCommands.pp_current_goal_or_noproof ~all fmt; + Format.pp_print_flush fmt (); + Buffer.contents buf + + (* Inline focus annotation ([focus: 1/N]) appended to reply tags + whenever the active proof has >=2 open subgoals. *) + let focus_tag () = + match EcCommands.pp_tree () with + | _ :: _ :: _ as entries -> + Printf.sprintf " [focus: 1/%d]" (List.length entries) + | _ -> "" +end + +(* -------------------------------------------------------------------- *) +(* Frame tree: group currently-open goals by their shared multi-child + ancestors. Used by [Tree] (rendering) and [Focus] (path lookup). + The tree is a *derivation*: it depends only on [pr_opened] and + [parent_of], no recorded transcript. *) +module FrameTree = struct + (* Internal nodes are split-point frames; leaves carry a handle + (the open goal), its index in [pr_opened] (1-based, used by + [EcCoreGoal.rotate_focus]), and its rendered text. *) + type node = + | Frame of node list (* >=2 child branches *) + | Leaf of + { idx : int (* 1-based in pr_opened *) + ; focused : bool (* idx = 1 *) + ; text : string } (* one-line conclusion *) + + (* Multi-child ancestors of [h], outermost first (= root-most + split first, deepest split last). This ordering means leaves + sharing the same OUTER frame will agree on the chain's first + element, which is what [group] partitions on. *) + let split_chain h = + let rec walk h acc = + match EcCommands.parent_of h with + | None -> acc + | Some p -> + match EcCommands.children_of p with + | [_] -> walk p acc + | _ -> walk p (p :: acc) + in + (* [walk] prepends each ancestor as we go up; the result has + outermost at the FRONT (we add it last). No reverse needed. *) + walk h [] + + (* Build the tree by grouping leaves with a common ancestor prefix. + [leaves] is a list of (chain, leaf) in [pr_opened] order. The + grouping is done recursively on the head of each chain. *) + let rec group (leaves : (EcCoreGoal.handle list * node) list) : node list = + let rec runs acc = function + | [] -> List.rev acc + | (chain, leaf) :: rest -> + match chain with + | [] -> runs (`Bare leaf :: acc) rest + | hd :: tl -> + let same_head, others = + List.partition_map (fun (c, l) -> + match c with + | h :: tail when EcCoreGoal.eq_handle h hd -> + Left (tail, l) + | _ -> Right (c, l)) + rest + in + runs (`Group ((tl, leaf) :: same_head) :: acc) others + in + List.map + (function + | `Bare leaf -> leaf + | `Group children -> Frame (group children)) + (runs [] leaves) + + (* Strip leading singleton frames so the top-level forest's + indices match what the user thinks of as "top-level subgoals + of the current frame." When all open leaves descend from a + single outermost split, the top-level forest has one Frame + containing the actual user-visible siblings; unwrap it. *) + let rec unwrap forest = + match forest with + | [Frame children] -> unwrap children + | _ -> forest + + let build () = + let handles = EcCommands.open_handles () in + let texts = EcCommands.pp_tree () in + if handles = [] then [] + else + let leaves = + List.mapi (fun i (h, (_, focused, text)) -> + let leaf = Leaf { idx = i + 1; focused; text } in + (split_chain h, leaf)) + (List.combine handles texts) + in + unwrap (group leaves) + + (* Render the tree with dotted-path labels matching what FOCUS + accepts. [all] requests full goal bodies (we re-query via + [pp_tree ~all:true] keyed by leaf index). *) + let render ?(all=false) () = + let forest = build () in + if forest = [] then "No active proof.\n" + else + let texts_all = + if all then Some (EcCommands.pp_tree ~all:true ()) + else None + in + let one_line s = + let s = + match String.index_opt s '\n' with + | None -> s + | Some k -> String.sub s 0 k + in + let limit = 80 in + if String.length s > limit + then String.sub s 0 (limit - 1) ^ "…" + else s + in + let buf = Buffer.create 256 in + let rec emit ~depth ~path = function + | Leaf { idx; focused; text } -> + let label = String.concat "." (List.rev_map string_of_int path) in + let marker = if focused then " <- focused" else "" in + for _ = 1 to depth do Buffer.add_string buf " " done; + (match texts_all with + | None -> + Buffer.add_string buf + (Printf.sprintf "[%s] %s%s\n" + label (one_line text) marker) + | Some entries -> + let (_, _, full) = + List.nth entries (idx - 1) + in + Buffer.add_string buf + (Printf.sprintf "[%s]%s\n%s\n" label marker full)) + | Frame children -> + List.iteri (fun i child -> + emit ~depth:(depth + 1) ~path:((i + 1) :: path) child) + children + in + List.iteri (fun i node -> + emit ~depth:0 ~path:[i + 1] node) + forest; + Buffer.contents buf + + (* Resolve a dotted path against the tree. Returns [Ok idx] where + [idx] is the 1-based position in [pr_opened] of the selected + leaf, or [Error msg]. *) + let resolve_path (path : int list) : (int, string) result = + let forest = build () in + let rec walk ~components nodes = + match components with + | [] -> Error "FOCUS: path must select a leaf goal" + | k :: rest -> + if k < 1 || k > List.length nodes then + Error (Printf.sprintf + "FOCUS: index %d out of range (1..%d)" + k (List.length nodes)) + else + match List.nth nodes (k - 1), rest with + | Leaf { idx; _ }, [] -> Ok idx + | Leaf _, _ -> + Error "FOCUS: path overshoots a leaf goal" + | Frame _, [] -> + Error "FOCUS: path must select a leaf goal, \ + not a frame" + | Frame kids, _ -> walk ~components:rest kids + in + if forest = [] then Error "FOCUS: no active proof" + else walk ~components:path forest +end + +(* -------------------------------------------------------------------- *) +(* Reply construction. The notice buffer is captured and cleared at + exactly the points the text front-end used to print it, so that + engine messages keep interleaving with replies as before. *) +let mk_reply (st : state) ~(pre : int) ?(tag = "") (body : body) = + let notices = Buffer.contents st.notices in + Buffer.clear st.notices; + let uuid = EcCommands.uuid () in + { uuid; tag; notices; body; changed = uuid <> pre; } + +(* The body of a reply that ends on the current goals. The front-end + decides whether to render them (QUIET is a presentation setting). *) +let mk_reply_goals (st : state) ~(pre : int) = + let tag = Goals.focus_tag () in + mk_reply st ~pre ~tag Goals + +let mk_failure (st : state) (message : string) = + let notices = Buffer.contents st.notices in + Buffer.clear st.notices; + { uuid = EcCommands.uuid (); message; + goals = Goals.goals_to_string (); notices; } + +(* -------------------------------------------------------------------- *) +(* Transcript manipulation. *) +module Transcript = struct + let trim (st : state) target = + let transcript = st.transcript in + transcript := + List.filter + (fun (uuid_before, _, _, _) -> uuid_before < target) + !transcript + + let clear (st : state) = + st.transcript := []; + st.prior_bullets := None; + st.commit_env := None +end + +(* -------------------------------------------------------------------- *) +(* Process a single EasyCrypt command, respecting [gl_fail]. When + [~record:true], append a transcript entry on success: the parent + handle (focused goal before the phrase) and the open-handle list, + which together let [Commit] reconstruct bullet structure. *) +let process_action (st : state) ?(record=false) ~src (p : EP.global) = + let transcript = st.transcript in + let commit_env = st.commit_env in + let loc = p.EP.gl_action.EcLocation.pl_loc in + let pre_uuid = EcCommands.uuid () in + let opens_pre = + if record then EcCommands.open_handles () else [] + in + let parent = + match opens_pre with h :: _ -> Some h | [] -> None + in + let succeeded = ref false in + begin try + ignore (EcCommands.process ~src p.EP.gl_action : float option); + succeeded := true + with + | EcCommands.Restart -> raise EcCommands.Restart + | _ when p.EP.gl_fail -> () + | e -> raise (EcScope.toperror_of_exn ~gloc:loc e) + end; + if !succeeded && p.EP.gl_fail then + raise (EcScope.toperror_of_exn ~gloc:loc + (EcScope.HiScopeError (None, + "this command is expected to fail"))); + (* Queries only inspect the environment: they neither advance the + proof nor belong in the body COMMIT emits. *) + let is_query = + match EcLocation.unloc p.EP.gl_action with + | EP.Gprint _ | EP.Gsearch _ | EP.Glocate _ -> true + | _ -> false + in + if record && !succeeded && not p.EP.gl_fail && not is_query then begin + transcript := (pre_uuid, src, parent, opens_pre) :: !transcript; + (* Keep the newest non-empty snapshot: a phrase that closes the + proof ([qed]) leaves no active proof, and precisely then we + still need the environment the previous phrases built. *) + match EcCommands.current_proofenv () with + | None -> () + | Some _ as penv -> commit_env := penv + end + +(* -------------------------------------------------------------------- *) +(* COMMIT: replay the transcript against the proof DAG (parent_of / + children_of, backed by [EcCoreGoal.pr_parent]), inserting bullets + at multi-child splits. Levels the LOAD prefix's [puc_bullets] stack + already opened are addressed with that frame's own token; deeper + levels get fresh tokens, chosen so they collide with neither the + stack nor each other. *) +module Commit = struct + (* Token order matches PR 1017's lexer: -, +, *, --, ++, **, + ---, +++, *** ... *) + let token_at_index i = + let chars = [| "-"; "+"; "*" |] in + let rep = i / 3 + 1 in + let chr = chars.(i mod 3) in + String.concat "" (List.init rep (fun _ -> chr)) + + (* DAG queries go through the snapshot recorded at the last phrase, + so COMMIT still sees the structure after [qed]. Fall back to the + live proof when no phrase was recorded under a proof. *) + let parent_of (st : state) h = + match !(st.commit_env) with + | Some penv -> EcCoreGoal.parent_of_handle penv h + | None -> EcCommands.parent_of h + + let children_of (st : state) h = + match !(st.commit_env) with + | Some penv -> EcCoreGoal.children_of_handle penv h + | None -> EcCommands.children_of h + + let proof_text (st : state) = + let parent_of = parent_of st in + let children_of = children_of st in + let transcript = st.transcript in + let prior_bullets = st.prior_bullets in + let entries = List.rev !transcript in + let buf = Buffer.create 1024 in + let emit_indent depth = + for _ = 1 to depth do Buffer.add_string buf " " done + in + let module Hmap = + Map.Make (struct + type t = EcCoreGoal.handle + let compare = compare + end) + in + let sibling_depth : int Hmap.t ref = ref Hmap.empty in + let current_depth = ref 0 in + let bullet_to_string (b : EcParsetree.bullet) = + let ch = + match b.b_kind with + | `Minus -> "-" + | `Plus -> "+" + | `Star -> "*" + in + String.concat "" (List.init b.b_count (fun _ -> ch)) + in + (* Bullet frames the LOAD prefix left open, OUTERMOST first (the + stack stores the innermost frame at its head). Frame [t_d] is + the one whose siblings live at emitted depth [d]. *) + let frames : EcBullets.frame list = + match !prior_bullets with + | None -> [] + | Some stack -> List.rev stack + in + let in_use_tokens = + List.map + (fun (f : EcBullets.frame) -> bullet_to_string f.bf_bullet) + frames + in + let depth_cache : (int, string) Hashtbl.t = Hashtbl.create 8 in + let next_tok_idx = ref 0 in + let assigned_tokens = ref [] in + (* Depths 1..k address the next sibling of a frame the prefix + already opened, and strict bullets accepts nothing but that + frame's own token there. Deeper levels get fresh tokens, so + pre-populate the cache before any fresh pick happens. *) + List.iteri (fun i (f : EcBullets.frame) -> + let t = bullet_to_string f.bf_bullet in + Hashtbl.replace depth_cache (i + 1) t; + assigned_tokens := t :: !assigned_tokens) + frames; + let bullet_for_depth d = + match Hashtbl.find_opt depth_cache d with + | Some t -> t + | None -> + let rec pick () = + let t = token_at_index !next_tok_idx in + incr next_tok_idx; + if List.mem t in_use_tokens || List.mem t !assigned_tokens + then pick () + else t + in + let t = pick () in + assigned_tokens := t :: !assigned_tokens; + Hashtbl.add depth_cache d t; + t + in + (* Seed: the goals already open when the first recorded phrase ran + were left there by the LOAD prefix, so COMMIT must place each of + them at the depth the prefix's own bullets put it at. A frame + with floor [f] is discharged once [f] goals remain, hence it + still owns the first [n - f] goals of the focused-first list; + a goal covered by [c] frames sits at depth [c + 1]. + Nothing to seed when the prefix left no frame and a single goal + (the REPL just continues on the prefix's own focus). *) + (match entries with + | (_, _, Some _, (_ :: _ as opens)) :: _ + when frames <> [] || List.length opens >= 2 -> + let n = List.length opens in + List.iteri (fun i h -> + let pos = i + 1 in + let covering = + List.length + (List.filter + (fun (f : EcBullets.frame) -> pos <= n - f.bf_floor) + frames) + in + sibling_depth := Hmap.add h (covering + 1) !sibling_depth) + opens + | _ -> ()); + List.iter (fun (_uuid, src, parent_opt, _opens) -> + match parent_opt with + | None -> + Buffer.add_string buf src; + Buffer.add_char buf '\n' + | Some parent -> + (* Walk upward via pr_parent until we hit a registered + sibling ancestor. If found, emit its bullet and consume + the registration. *) + let rec find_ancestor h = + match Hmap.find_opt h !sibling_depth with + | Some d -> Some (h, d) + | None -> + match parent_of h with + | Some p -> find_ancestor p + | None -> None + in + (match find_ancestor parent with + | Some (h, d) -> + emit_indent (d - 1); + Buffer.add_string buf (bullet_for_depth d); + Buffer.add_char buf ' '; + current_depth := d; + sibling_depth := Hmap.remove h !sibling_depth + | None -> + emit_indent !current_depth); + Buffer.add_string buf src; + Buffer.add_char buf '\n'; + (* Register fresh siblings: walk the subtree rooted at + [parent], finding every multi-child split, and register + each such child at the right depth. Single-child links + are continuations and don't bump depth; multi-child + links do. A compound phrase like [split; split.] can + produce nested splits within one phrase. *) + let rec walk h d = + match children_of h with + | [c] -> walk c d + | (_ :: _ :: _) as cs -> + List.iter + (fun c -> + sibling_depth := + Hmap.add c d !sibling_depth; + walk c (d + 1)) + cs + | [] -> () + in + walk parent (!current_depth + 1) + ) entries; + Buffer.contents buf +end + +(* -------------------------------------------------------------------- *) +(* Accessors used by front-ends to build their own replies (HELP, + QUIET, parse errors) and to render a [Goals] body. *) +let uuid (_ : state) = + EcCommands.uuid () + +let clear_notices (st : state) = + Buffer.clear st.notices + +let current_goals (_ : state) = + Goals.goals_to_string () + +let make_reply (st : state) ?tag (body : body) = + mk_reply st ~pre:(EcCommands.uuid ()) ?tag body + +let make_failure (st : state) (message : string) = + mk_failure st message + +(* -------------------------------------------------------------------- *) +(* Process EasyCrypt input typed at the prompt (single phrase or a + line ending with a "."). *) +let step (st : state) input = + let notices = st.notices in + let prior_bullets = st.prior_bullets in + let pre = EcCommands.uuid () in + Buffer.clear notices; + (* On the first REPL phrase of each proof, capture the bullet stack + the LOAD prefix left so COMMIT can avoid token collisions with + it. Subsequent calls return [None] and don't clobber the snapshot. *) + (match EcCommands.disable_repl_bullets () with + | None -> () + | Some _ as snapshot -> prior_bullets := snapshot); + let reader = EcIo.from_string input in + let last_src = ref "" in + let answer = + begin try + let (src, prog) = EcIo.xparse reader in + let src = String.strip src in + last_src := src; + begin match EcLocation.unloc prog with + | EP.P_Prog (commands, _) -> + List.iter (process_action st ~record:true ~src) commands; + Done (Ok (mk_reply_goals st ~pre)) + | EP.P_Undo i -> + EcCommands.undo i; + Transcript.trim st i; + Done (Ok (mk_reply_goals st ~pre)) + | EP.P_Exit -> + Quit + | EP.P_DocComment doc -> + EcCommands.doc_comment doc; + Done (Ok (mk_reply st ~pre (Text ""))) + end + with + | EcCommands.Restart -> + do_initialize st; + Transcript.clear st; + Done (Ok (mk_reply st ~pre (Text "Session restarted"))) + | e -> + Done (Error (mk_failure st (Goals.format_error ~src:!last_src e))) + end + in + EcIo.finalize reader; + answer + +(* -------------------------------------------------------------------- *) +(* LOAD: run [file] up to [upto], optionally with SMT calls weakened + ([nosmt]) or with the last sentence of the prefix traced. The + argument string is parsed by the front-end. *) +let load (st : state) ~file ~upto ~nosmt ~trace = + let notices = st.notices in + let cur_prvopts = st.cur_prvopts in + let projdirs = st.projdirs in + let checkpoints = st.checkpoints in + let pre = EcCommands.uuid () in + Buffer.clear notices; + let filename = file in + let last_src = ref "" in + let trace_prefix = ref "" in + let exception Trace_failed of exn in + + try + begin try + ignore (EcLoader.getkind + (Filename.extension filename) : EcLoader.kind) + with EcLoader.BadExtension ext -> + failwith (Format.sprintf + "unknown file extension: %s" ext) + end; + + (* Apply the configuration attached to the loaded file's + [easycrypt.project], as the batch compiler does when the + file is given on the command line: refresh the prover + options (timeout, provers, pragmas, ...) and extend the + load path with the project's include dirs. *) + let ini = Option.to_list (st.projini (Some filename)) in + cur_prvopts := + EcOptions.prv_options_with_ini ini st.base_prvopts; + List.iter (fun ((nm, dir, isrec) as entry) -> + if not (List.mem entry !projdirs) then begin + projdirs := entry :: !projdirs; + EcCommands.addidir + ?namespace:(omap (fun nm -> `Named nm) nm) + ~recursive:isrec dir + end) + (EcOptions.ini_loadpath ini); + + do_initialize st; + Hashtbl.clear checkpoints; + Transcript.clear st; + EcCommands.addidir (Filename.dirname filename); + EcCommands.set_current_path (Filename.dirname filename); + + let reader = EcIo.from_file filename in + + let past_upto (loc : EcLocation.t) = + match upto with + | None -> false + | Some (line, col) -> + let (el, ec) = loc.loc_end in + el > line || (el = line && match col with + | None -> false + | Some c -> ec > c) + in + + let last_loc = ref None in + + (* For -trace: lazy whole-file bytes, used to slice the exact + source text of a sentence by byte offsets. *) + let input_bytes = lazy ( + let ic = open_in_bin filename in + let n = in_channel_length ic in + let b = Bytes.create n in + really_input ic b 0 n; + close_in ic; + Bytes.unsafe_to_string b) + in + let sentence_source (loc : EcLocation.t) = + let s = Lazy.force input_bytes in + let lo = max 0 loc.EcLocation.loc_bchar in + let hi = min (String.length s) loc.EcLocation.loc_echar in + if hi <= lo then "" else String.sub s lo (hi - lo) + in + + (* For -trace: defer execution of the last sentence within the + prefix so we can capture goals before and after it. *) + let pending : (string * EP.global) option ref = ref None in + let flush_pending () = + match !pending with + | None -> () + | Some (src, p) -> + last_src := src; + process_action st ~src p; + last_loc := Some p.EP.gl_action.EcLocation.pl_loc; + pending := None + in + let step src p = + let loc = p.EP.gl_action.EcLocation.pl_loc in + if past_upto loc then raise Exit; + if trace then begin + flush_pending (); + pending := Some (src, p) + end else begin + last_src := src; + process_action st ~src p; + last_loc := Some loc + end + in + + if nosmt then EcCommands.pragma_check `WeakCheck; + + begin try while true do + let (src, prog) = EcIo.xparse reader in + let src = String.strip src in + match EcLocation.unloc prog with + | EP.P_Prog (commands, locterm) -> + List.iter (step src) commands; + if locterm then raise Exit + | EP.P_Undo i -> + last_src := src; + EcCommands.undo i + | EP.P_Exit -> + raise Exit + | EP.P_DocComment doc -> + last_src := src; + EcCommands.doc_comment doc + done with + | Exit | End_of_file -> () + | e -> + EcIo.finalize reader; + if nosmt then EcCommands.pragma_check `Check; + raise e + end; + + EcIo.finalize reader; + + if nosmt then EcCommands.pragma_check `Check; + + (* If -trace is set, the last in-prefix sentence is still + pending. Run it under goal capture and build the + BEFORE/TACTIC/AFTER/SUMMARY response body. *) + let body = + if not trace then + Goals.goals_to_string () + else + let pre_state = + match !pending with + | None -> `Nothing + | Some _ when not (EcCommands.in_proof ()) -> `NotInProof + | Some (src, p) -> `Ready (src, p) + in + match pre_state with + (* Tracing is off the table, but the prefix is not: run the + sentence we deferred so that the session ends up exactly + where a plain LOAD of the same prefix would leave it. A + failure inside the flush is reported by the enclosing + handler, as any prefix failure is. *) + | `Nothing -> + flush_pending (); + failwith "trace: nothing to trace" + | `NotInProof -> + flush_pending (); + failwith + "trace: target sentence is not in a proof context" + | `Ready (src, p) -> + let loc = p.EP.gl_action.EcLocation.pl_loc in + let (sl, sc) = loc.EcLocation.loc_start in + let (el, ec) = loc.EcLocation.loc_end in + let before_goals = EcCommands.pp_all_goals () in + let n1 = List.length before_goals in + let buf = Buffer.create 1024 in + let fmt = Format.formatter_of_buffer buf in + Format.fprintf fmt + "=== BEFORE: line %d (col %d) ===@\n" sl sc; + EcCommands.pp_current_goal_or_noproof ~all:false fmt; + Format.fprintf fmt + "@\n=== TACTIC (lines %d:%d - %d:%d) ===@\n%s@\n@\n" + sl sc el ec (sentence_source loc); + last_src := src; + begin + try + process_action st ~src p; + last_loc := Some loc; + pending := None; + let after_goals = EcCommands.pp_all_goals () in + let n2 = List.length after_goals in + Format.fprintf fmt + "=== AFTER: line %d (col %d) ===@\n" sl sc; + let before_set = + List.fold_left + (fun s g -> EcMaps.Sstr.add g s) + EcMaps.Sstr.empty before_goals + in + (* The new focused goal always counts as "modified" + (its focus status changed even if its text matches + an old sibling); the rest are printed only if they + didn't appear in BEFORE. *) + let to_print = + match after_goals with + | [] -> [] + | head :: tl -> + head :: + List.filter + (fun g -> not (EcMaps.Sstr.mem g before_set)) + tl + in + begin match to_print with + | [] -> Format.fprintf fmt "(no open goals)@\n" + | _ -> + List.iteri (fun i g -> + if i > 0 then Format.fprintf fmt "@\n"; + Format.fprintf fmt "%s@\n" g) + to_print + end; + Format.fprintf fmt + "@\n=== SUMMARY ===@\nopen goals: %d -> %d@\n" n1 n2; + Format.pp_print_flush fmt (); + Buffer.contents buf + with e -> + Format.fprintf fmt + "=== AFTER: line %d (col %d) ===@\n@\n" + sl sc; + Format.pp_print_flush fmt (); + trace_prefix := Buffer.contents buf; + raise (Trace_failed e) + end + in + + let tag = + let loaded = + match !last_loc with + | None -> "" + | Some loc -> + let (el, _) = loc.EcLocation.loc_end in + Printf.sprintf " [loaded:%s:%d]" filename el + in + loaded ^ Goals.focus_tag () + in + Ok (mk_reply st ~pre ~tag (Text body)) + + with + | EcCommands.Restart -> + do_initialize st; + Hashtbl.clear checkpoints; + Transcript.clear st; + Ok (mk_reply st ~pre (Text "Session restarted")) + | Trace_failed e -> + let msg = Goals.format_error ~src:!last_src e in + Error (mk_failure st (!trace_prefix ^ msg)) + | Failure s -> + Error (mk_failure st s) + | e -> + Error (mk_failure st (Goals.format_error ~src:!last_src e)) + +(* -------------------------------------------------------------------- *) +(* The remaining meta-commands. *) + +let goals (st : state) ~all = + let pre = EcCommands.uuid () in + Buffer.clear st.notices; + Ok (mk_reply st ~pre ~tag:(Goals.focus_tag ()) + (Text (Goals.goals_to_string ~all ()))) + +let tree (st : state) ~all = + let pre = EcCommands.uuid () in + Buffer.clear st.notices; + Ok (mk_reply st ~pre ~tag:(Goals.focus_tag ()) + (Text (FrameTree.render ~all ()))) + +let commit (st : state) = + let pre = EcCommands.uuid () in + Buffer.clear st.notices; + Ok (mk_reply st ~pre ~tag:(Goals.focus_tag ()) + (Text (Commit.proof_text st))) + +let undo (st : state) = + let pre = EcCommands.uuid () in + Buffer.clear st.notices; + let uuid = EcCommands.uuid () in + if uuid > 0 then begin + EcCommands.undo (uuid - 1); + Transcript.trim st (uuid - 1); + Ok (mk_reply_goals st ~pre) + end else + Error (mk_failure st "nothing to undo") + +let focus (st : state) request = + (* [request] is the user's intent normalized: + - [`Next] = rotate to the second open goal (or stay if <=1) + - [`Path p] = resolve dotted path [p] against the frame tree + and focus the matching leaf. *) + let pre = EcCommands.uuid () in + Buffer.clear st.notices; + let resolved = + match request with + | `Next -> + let n = List.length (EcCommands.open_handles ()) in + Ok (if n <= 1 then 1 else 2) + | `Path path -> FrameTree.resolve_path path + in + match resolved with + | Error msg -> Error (mk_failure st msg) + | Ok target -> + match EcCommands.focus_goal target with + | Ok _ -> Ok (mk_reply_goals st ~pre) + | Error msg -> Error (mk_failure st msg) + +let checkpoint (st : state) ~name = + let pre = EcCommands.uuid () in + Buffer.clear st.notices; + Hashtbl.replace st.checkpoints name (EcCommands.uuid ()); + Ok (mk_reply st ~pre (Text (Printf.sprintf + "checkpoint '%s' set at uuid %d" name (EcCommands.uuid ())))) + +let revert (st : state) spec = + let pre = EcCommands.uuid () in + Buffer.clear st.notices; + let target = + try Some (int_of_string spec) + with Failure _ -> Hashtbl.find_opt st.checkpoints spec + in + match target with + | None -> + Error (mk_failure st (Printf.sprintf + "REVERT: '%s' is not a valid uuid or checkpoint name" spec)) + | Some target -> + let uuid = EcCommands.uuid () in + if target < 0 || target > uuid then + Error (mk_failure st (Printf.sprintf + "REVERT: uuid %d out of range [0, %d]" target uuid)) + else begin + EcCommands.undo target; + Transcript.trim st target; + Ok (mk_reply_goals st ~pre) + end + +let search (st : state) ~pattern = + step st (Printf.sprintf "search %s." pattern) diff --git a/src/ecLlmCore.mli b/src/ecLlmCore.mli new file mode 100644 index 000000000..c5539e7e7 --- /dev/null +++ b/src/ecLlmCore.mli @@ -0,0 +1,98 @@ +(* -------------------------------------------------------------------- *) +(* Engine-facing core of the LLM interaction protocol: one operation per + meta-command of the [easycrypt llm] REPL, with the text protocol + factored out. Operations never print, never exit, and never format a + wire envelope; they return structured values a front-end renders + (the REPL in [ecLlm.ml], the MCP server next to it). + + One process = one session: the proof engine ([EcCommands]) is global + mutable state, so at most one [state] may exist per process. *) + +(* -------------------------------------------------------------------- *) +type state + +(* Reply body. [Goals] means "the current goals"; the front-end renders + them through [current_goals] and may suppress them (the REPL does, + under QUIET). [Text] is a literal body and is never suppressed. *) +type body = + | Goals + | Text of string + +(* [notices] are the engine messages emitted while the operation ran, + captured and cleared at the point the REPL used to print them. + [changed] tells whether the engine uuid advanced. *) +type reply = { + uuid : int; + tag : string; + notices : string; + body : body; + changed : bool; +} + +(* [goals] is the goal state at the point of failure. The REPL does not + render [notices] on failures (it never did); they are captured all + the same, so the buffer is left clean for the next operation. *) +type failure = { + uuid : int; + message : string; + goals : string; + notices : string; +} + +(* Operations that can be asked to end the session ([exit.]) return an + [answer]: the front-end owns the process, hence the exit. *) +type answer = + | Done of (reply, failure) result + | Quit + +(* Raised by [create] when the session cannot be set up. *) +exception Init_error of string + +(* -------------------------------------------------------------------- *) +(* Open a session: connect to the Why3 server, seed the loader with + [relocdir], and initialize the engine. [projini] resolves the + [easycrypt.project] context of a file path, so [load] can apply the + project's load path and prover options the way the batch compiler + does. *) +val create : + relocdir:string option + -> boot:bool + -> projini:(string option -> EcOptions.ini_context option) + -> prvopts:EcOptions.prv_options + -> state + +(* -------------------------------------------------------------------- *) +(* Operations. *) + +(* LOAD, on already-parsed arguments. *) +val load : + state + -> file:string + -> upto:(int * int option) option + -> nosmt:bool + -> trace:bool + -> (reply, failure) result + +(* One line of raw EasyCrypt input (or a multi-line block). *) +val step : state -> string -> answer + +val goals : state -> all:bool -> (reply, failure) result +val tree : state -> all:bool -> (reply, failure) result +val focus : state -> [`Next | `Path of int list] -> (reply, failure) result +val undo : state -> (reply, failure) result +val revert : state -> string -> (reply, failure) result +val checkpoint : state -> name:string -> (reply, failure) result +val commit : state -> (reply, failure) result +val search : state -> pattern:string -> answer + +(* -------------------------------------------------------------------- *) +(* Front-end helpers: for replies a front-end produces on its own (the + REPL's HELP and QUIET) and for errors it detects itself (line-parse + errors). Both capture-and-clear the notice buffer, as the operations + above do. *) + +val uuid : state -> int +val current_goals : state -> string +val clear_notices : state -> unit +val make_reply : state -> ?tag:string -> body -> reply +val make_failure : state -> string -> failure From 31dad7cda4eb8f213c1740bb035f3da935f36e34 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Fri, 21 Aug 2026 11:13:49 +0200 Subject: [PATCH 24/51] [llm] core: TRY semantics via EcLlmCore.try_step A failed phrase can still have advanced the engine: a compound sentence whose leading tactics went through leaves a new uuid and a transcript entry behind. [try_step] runs [step] and, on failure, restores the pre-entry uuid the way REVERT does (EcCommands.undo + Transcript.trim), then re-stamps the failure -- its uuid and goal text described a state that no longer exists. [failure] gains a [reverted] flag so a front-end can tell the caller that the rollback happened. It is false everywhere else, and the REPL front-end is untouched: it builds failures through [make_failure]. No REPL surface change; this is the primitive the MCP ec_try tool needs. --- src/ecLlmCore.ml | 32 +++++++++++++++++++++++++++----- src/ecLlmCore.mli | 21 ++++++++++++++++----- 2 files changed, 43 insertions(+), 10 deletions(-) diff --git a/src/ecLlmCore.ml b/src/ecLlmCore.ml index 672b06367..e958c7a88 100644 --- a/src/ecLlmCore.ml +++ b/src/ecLlmCore.ml @@ -22,10 +22,11 @@ type reply = { } type failure = { - uuid : int; - message : string; - goals : string; - notices : string; + uuid : int; + message : string; + goals : string; + notices : string; + reverted : bool; } type answer = @@ -368,7 +369,7 @@ let mk_failure (st : state) (message : string) = let notices = Buffer.contents st.notices in Buffer.clear st.notices; { uuid = EcCommands.uuid (); message; - goals = Goals.goals_to_string (); notices; } + goals = Goals.goals_to_string (); notices; reverted = false; } (* -------------------------------------------------------------------- *) (* Transcript manipulation. *) @@ -668,6 +669,27 @@ let step (st : state) input = EcIo.finalize reader; answer +(* -------------------------------------------------------------------- *) +(* [step] with an automatic rollback on failure. A phrase can fail + after having advanced the engine (a compound sentence whose first + tactics went through, say), so the pre-entry uuid is the only + faithful notion of "unchanged": restore it the way REVERT does. + The failure is then re-stamped, because its uuid and goal text + described the state at the point of failure, which no longer + exists. *) +let try_step (st : state) input = + let pre = EcCommands.uuid () in + match step st input with + | Quit -> Quit + | Done (Ok _) as answer -> answer + | Done (Error failure) -> + EcCommands.undo pre; + Transcript.trim st pre; + Done (Error { failure with + uuid = EcCommands.uuid (); + goals = Goals.goals_to_string (); + reverted = true; }) + (* -------------------------------------------------------------------- *) (* LOAD: run [file] up to [upto], optionally with SMT calls weakened ([nosmt]) or with the last sentence of the prefix traced. The diff --git a/src/ecLlmCore.mli b/src/ecLlmCore.mli index c5539e7e7..4f1906fd9 100644 --- a/src/ecLlmCore.mli +++ b/src/ecLlmCore.mli @@ -31,12 +31,16 @@ type reply = { (* [goals] is the goal state at the point of failure. The REPL does not render [notices] on failures (it never did); they are captured all - the same, so the buffer is left clean for the next operation. *) + the same, so the buffer is left clean for the next operation. + [reverted] is set by [try_step] only: it says the engine was rolled + back to the state it had before the operation ran, so [uuid] and + [goals] describe that restored state, not the point of failure. *) type failure = { - uuid : int; - message : string; - goals : string; - notices : string; + uuid : int; + message : string; + goals : string; + notices : string; + reverted : bool; } (* Operations that can be asked to end the session ([exit.]) return an @@ -76,6 +80,13 @@ val load : (* One line of raw EasyCrypt input (or a multi-line block). *) val step : state -> string -> answer +(* [step], but a failure leaves no trace: the engine is rolled back to + the uuid it had on entry (as REVERT does) and the failure comes back + with [reverted = true]. Successes and [Quit] behave exactly as in + [step]. A phrase that fails halfway through a compound sentence is + rolled back whole. *) +val try_step : state -> string -> answer + val goals : state -> all:bool -> (reply, failure) result val tree : state -> all:bool -> (reply, failure) result val focus : state -> [`Next | `Path of int list] -> (reply, failure) result From 31320d7c487695b01430df5636f4704c6ed07ef8 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Fri, 21 Aug 2026 11:24:15 +0200 Subject: [PATCH 25/51] [llm] add `easycrypt mcp`: an MCP server front-end over EcLlmCore Adds a second front-end over EcLlmCore speaking the Model Context Protocol on stdio (JSON-RPC 2.0, newline-delimited, hand-rolled over yojson, which ecLib already depends on). Wiring mirrors `llm`: an mcp_option record and command spec in ecOptions (so -I/-timeout/-p/ -stdlib come from the shared option groups) and a dispatch arm in ec.ml. Protocol: initialize with version negotiation, ping, tools/list, tools/call, tolerance for the lifecycle notifications, and the -32700/-32600/-32601/-32602 errors. Batch arrays are rejected: they were removed from the spec in 2025-06-18 and have not returned. The loop is synchronous and single-threaded on purpose: the engine is a global mutable singleton and uuid ordering is what makes ec_revert meaningful, so calls must run strictly in arrival order. Spec era: the server implements the initialize-handshake era, pinned to 2025-11-25/2025-06-18/2025-03-26. The 2026-07-28 revision removed the handshake in favor of stateless per-request _meta fields and server/discover; deployed clients still speak the handshake era, and per the spec's own compatibility matrix a dual-era client that receives -32601 for server/discover falls back to initialize. Supporting the stateless era is future work (noted at the top of ecMcp.ml). stdout is reserved for the protocol at the file-descriptor level: the wire keeps a private dup of fd 1 and the process's stdout is pointed at stderr, so a stray print anywhere under the engine lands in the client's log instead of corrupting the message stream. Eleven tools route to EcLlmCore: ec_load, ec_step, ec_try, ec_goals, ec_tree, ec_focus, ec_undo, ec_revert, ec_checkpoint, ec_commit, ec_search. The front-end validates arguments before the engine is touched, since the core trusts what it is handed: ec_load checks that the file exists rather than letting a Sys_error surface downstream, ec_focus parses the dotted path itself, and JSON typing covers the rest. That split keeps the two error channels honest -- an unknown tool or an argument violating the declared schema is a JSON-RPC -32602, while a prover error is a successful response with isError set and the error text (plus the goals at that point) as content, which is what an agent needs in order to react to it. Every result carries structuredContent {uuid, changed} so an agent can address the state later with ec_revert; ec_try adds reverted on the failure side. Each tool declares a matching outputSchema. `exit.' answers "session terminated" and stops the process. ec_step's description states that only the first sentence of a phrase runs -- pre-existing behavior of the shared core, inherited from the REPL. --- src/ec.ml | 3 + src/ecLlmCore.ml | 3 +- src/ecLlmCore.mli | 4 +- src/ecMcp.ml | 723 ++++++++++++++++++++++++++++++++++++++++++++++ src/ecMcp.mli | 14 + src/ecOptions.ml | 24 ++ src/ecOptions.mli | 6 + 7 files changed, 773 insertions(+), 4 deletions(-) create mode 100644 src/ecMcp.ml create mode 100644 src/ecMcp.mli diff --git a/src/ec.ml b/src/ec.ml index 90630a31e..f798081e9 100644 --- a/src/ec.ml +++ b/src/ec.ml @@ -588,6 +588,9 @@ let main () = | `Llm llmopts -> EcLlm.run ~relocdir ~boot:ldropts.ldro_boot ~projini llmopts + | `Mcp mcpopts -> + EcMcp.run ~relocdir ~boot:ldropts.ldro_boot ~projini mcpopts + | `Runtest _ -> (* Eagerly executed *) assert false diff --git a/src/ecLlmCore.ml b/src/ecLlmCore.ml index e958c7a88..cd75a6afc 100644 --- a/src/ecLlmCore.ml +++ b/src/ecLlmCore.ml @@ -671,8 +671,7 @@ let step (st : state) input = (* -------------------------------------------------------------------- *) (* [step] with an automatic rollback on failure. A phrase can fail - after having advanced the engine (a compound sentence whose first - tactics went through, say), so the pre-entry uuid is the only + after having advanced the engine, so the pre-entry uuid is the only faithful notion of "unchanged": restore it the way REVERT does. The failure is then re-stamped, because its uuid and goal text described the state at the point of failure, which no longer diff --git a/src/ecLlmCore.mli b/src/ecLlmCore.mli index 4f1906fd9..8580feb6b 100644 --- a/src/ecLlmCore.mli +++ b/src/ecLlmCore.mli @@ -83,8 +83,8 @@ val step : state -> string -> answer (* [step], but a failure leaves no trace: the engine is rolled back to the uuid it had on entry (as REVERT does) and the failure comes back with [reverted = true]. Successes and [Quit] behave exactly as in - [step]. A phrase that fails halfway through a compound sentence is - rolled back whole. *) + [step]. A phrase that fails after having already advanced the engine + is rolled back whole. *) val try_step : state -> string -> answer val goals : state -> all:bool -> (reply, failure) result diff --git a/src/ecMcp.ml b/src/ecMcp.ml new file mode 100644 index 000000000..49e3b2aef --- /dev/null +++ b/src/ecMcp.ml @@ -0,0 +1,723 @@ +(* -------------------------------------------------------------------- *) +(* The Model Context Protocol front-end. See [ecMcp.mli]. + + This module is to MCP what [EcLlm] is to the text protocol: a wire + layer only. Every engine-facing operation goes through [EcLlmCore], + which the two front-ends share. + + The loop is synchronous and single-threaded, which is not an + implementation shortcut but the correctness anchor: the proof engine + is a global mutable singleton and uuid ordering is what makes + [ec_revert] meaningful, so tool calls must run strictly in arrival + order even when a client pipelines them. *) + +module J = Yojson.Safe + +(* -------------------------------------------------------------------- *) +(* Protocol revisions. + + We speak the handshake-based ("legacy", in the vocabulary of the + 2026-07-28 spec) era: [initialize] / [notifications/initialized], + with the negotiated version fixed for the life of the process. Every + deployed client speaks it. + + Revision 2026-07-28 replaced the handshake with per-request [_meta] + and a mandatory [server/discover]; supporting it is a separate piece + of work. A dual-era client probes with [server/discover], gets our + [-32601] -- not a recognized modern error -- and falls back to + [initialize], which is exactly the intended detection path. *) +let protocol_latest = "2025-11-25" + +let protocol_supported = [ + "2025-11-25"; + "2025-06-18"; + "2025-03-26"; +] + +let server_name = "easycrypt" + +let server_version = + match EcVersion.hash with "n/a" -> "dev" | v -> v + +(* -------------------------------------------------------------------- *) +(* JSON-RPC 2.0 error codes. *) +let e_parse_error = -32700 +let e_invalid_request = -32600 +let e_method_not_found = -32601 +let e_invalid_params = -32602 + +(* Raised by argument validation: a malformed [tools/call] is a + *protocol* failure, and must not be dressed up as a prover error. *) +exception Invalid_params of string + +(* Raised by the checks a tool performs on its own behalf before + reaching the engine (a missing file, say). Those are EasyCrypt-level + failures and travel as successful responses with [isError]. *) +exception Tool_error of string + +(* -------------------------------------------------------------------- *) +(* TODO(phase 3): replace with the MCP section of doc/llm/CLAUDE.md, + the way [llm -help] prints the whole guide. *) +let usage = {|easycrypt mcp -- Model Context Protocol server (stdio, JSON-RPC 2.0) + +Exposes the EasyCrypt proof engine to an MCP client as a set of tools. +Reads newline-delimited JSON-RPC messages on stdin and writes responses +on stdout; diagnostics go to stderr. One client = one process = one +proof session. Standard loader and prover options (-I, -timeout, -p, +-stdlib, ...) are accepted, as for `easycrypt llm'. + +Tools: + ec_load compile a file up to a position and start a session + ec_step run one EasyCrypt sentence + ec_try run one sentence, rolling back if it fails + ec_goals print the current goal state + ec_tree list the open subgoals as a labelled tree + ec_focus focus the subgoal at a dotted path (or `next') + ec_undo undo the last step + ec_revert return to a uuid or to a named checkpoint + ec_checkpoint name the current state for a later ec_revert + ec_commit emit the recorded phrases as a bulleted proof body + ec_search search for lemmas matching a pattern + +Client configuration and the full protocol description live in +doc/llm/CLAUDE.md. +|} + +(* -------------------------------------------------------------------- *) +(* JSON schema fragments for the tool declarations. *) +module Schema = struct + let str ?description () = + `Assoc (("type", `String "string") + :: (match description with + | None -> [] + | Some d -> [("description", `String d)])) + + let int ~description () = + `Assoc [("type", `String "integer"); + ("description", `String description)] + + let bool ~description ~default () = + `Assoc [("type", `String "boolean"); + ("description", `String description); + ("default", `Bool default)] + + let obj ?(required = []) props = + `Assoc ([("type", `String "object"); + ("properties", `Assoc props)] + @ (match required with + | [] -> [] + | _ -> [("required", + `List (List.map (fun s -> `String s) required))]) + @ [("additionalProperties", `Bool false)]) + + (* Every tool answers with the same structured payload: the engine + state the call left behind, and whether it moved. *) + let output ?(reverted = false) () = + let base = [ + ("uuid", int ~description:"engine state identifier after the call; \ + pass it to ec_revert to come back here" ()); + ("changed", `Assoc [("type", `String "boolean"); + ("description", + `String "whether the engine state advanced")]); + ] in + let base = + if not reverted then base + else base @ [ + ("reverted", + `Assoc [("type", `String "boolean"); + ("description", + `String "set when the phrase failed and the engine was \ + rolled back to its pre-call state")]); + ] + in + `Assoc [("type", `String "object"); + ("properties", `Assoc base); + ("required", `List [`String "uuid"; `String "changed"])] +end + +(* -------------------------------------------------------------------- *) +(* The static tool table, in [tools/list] order. Descriptions are + agent-facing and track the wording of doc/llm/CLAUDE.md. *) +let tools : J.t list = + let tool ~name ~description ~input ?(annotations = []) ~output () = + `Assoc ([ + ("name", `String name); + ("description", `String description); + ("inputSchema", input); + ("outputSchema", output); + ] @ (match annotations with + | [] -> [] + | _ -> [("annotations", `Assoc annotations)])) + in [ + tool + ~name:"ec_load" + ~description: + "Reset the session and compile FILE from the top, stopping after \ + the last sentence that ends on or before LINE (and column COL \ + when given). This is the entry point: every other tool needs a \ + loaded file, and tactics need the position to land inside a \ + proof. Set nosmt to weaken SMT calls while replaying a prefix \ + that was already verified, which is much faster on large files. \ + Set trace to have the reply describe the last loaded sentence as \ + BEFORE / TACTIC / AFTER / SUMMARY blocks. The reply reports \ + where compilation stopped and the resulting goal state; note the \ + uuid it returns, reverting to it is the instant way back to the \ + start of the proof." + ~input:(Schema.obj ~required:["file"] [ + ("file", Schema.str ~description:"path to the .ec/.eca file" ()); + ("line", Schema.int + ~description:"stop after the last sentence ending on \ + or before this line; omit to compile the \ + whole file" ()); + ("col", Schema.int + ~description:"column bound within `line'; requires \ + `line'" ()); + ("nosmt", Schema.bool + ~description:"weaken SMT calls while compiling the \ + prefix" ~default:false ()); + ("trace", Schema.bool + ~description:"report the proof state around the last \ + loaded sentence" ~default:false ()); + ]) + ~annotations:[("destructiveHint", `Bool true)] + ~output:(Schema.output ()) + (); + + tool + ~name:"ec_step" + ~description: + "Run one EasyCrypt sentence -- a tactic, a declaration, require, \ + print, ... -- against the current session. Send exactly one \ + complete sentence, ending with a period: anything past the \ + first sentence of the phrase is ignored, so chain tactics with \ + `;' rather than with `.'. A sentence spanning several lines is \ + fine as one string. Requires a file loaded with ec_load, and, \ + for tactics, an open proof. On success the reply carries the \ + new goal state; on failure the prover's error text comes back \ + with isError set and the engine is left wherever the sentence \ + left it -- use ec_try when you want a guaranteed rollback. \ + Successful non-query phrases are recorded for ec_commit." + ~input:(Schema.obj ~required:["phrase"] [ + ("phrase", Schema.str + ~description:"one complete EasyCrypt sentence, \ + ending with `.'" ()); + ]) + ~annotations:[("destructiveHint", `Bool false); + ("idempotentHint", `Bool false)] + ~output:(Schema.output ()) + (); + + tool + ~name:"ec_try" + ~description: + "Like ec_step, but the engine is rolled back to the state it had \ + before the call whenever the phrase fails, including a phrase \ + that failed only after having already advanced the proof. The \ + failure reply sets structuredContent.reverted to true, and its \ + uuid and goal text describe the restored state, not the point \ + of failure. Use this to probe a tactic without having to \ + ec_revert afterwards; use ec_step when you mean to keep \ + whatever progress the phrase makes. A successful phrase behaves \ + exactly as under ec_step and is recorded for ec_commit." + ~input:(Schema.obj ~required:["phrase"] [ + ("phrase", Schema.str + ~description:"one complete EasyCrypt sentence, \ + ending with `.'" ()); + ]) + ~annotations:[("destructiveHint", `Bool false)] + ~output:(Schema.output ~reverted:true ()) + (); + + tool + ~name:"ec_goals" + ~description: + "Print the current proof state: the focused subgoal alone, or, \ + with all set, every open subgoal. Requires an open proof, and \ + does not advance the engine." + ~input:(Schema.obj [ + ("all", Schema.bool + ~description:"print every open subgoal instead of the \ + focused one" ~default:false ()); + ]) + ~annotations:[("readOnlyHint", `Bool true)] + ~output:(Schema.output ()) + (); + + tool + ~name:"ec_tree" + ~description: + "List the open subgoals as a tree of dotted-path labels -- [1], \ + [1.2], [2.1.1] -- showing how the splits nest, and marking the \ + focused one. Those labels are exactly what ec_focus accepts. \ + Set full for whole goal bodies rather than one-line \ + conclusions. The labels are not stable across focus changes: \ + the tree always shows the focused goal first, so re-read it \ + after every ec_focus. Does not advance the engine." + ~input:(Schema.obj [ + ("full", Schema.bool + ~description:"print full goal bodies instead of \ + one-line conclusions" ~default:false ()); + ]) + ~annotations:[("readOnlyHint", `Bool true)] + ~output:(Schema.output ()) + (); + + tool + ~name:"ec_focus" + ~description: + "Rotate the focus onto the subgoal at dotted path PATH, as \ + printed by ec_tree (\"2\", \"1.2\", \"1.1.1\"); a single \ + integer selects the k-th goal of the flat listing, and the \ + special value \"next\" moves to the next open subgoal. \ + Subsequent tactics act on the focused goal. Selecting an \ + internal frame instead of a leaf goal is an error." + ~input:(Schema.obj ~required:["path"] [ + ("path", Schema.str + ~description:"\"N\", a dotted path \"N1.N2...\", or \ + \"next\"" ()); + ]) + ~annotations:[("destructiveHint", `Bool false)] + ~output:(Schema.output ()) + (); + + tool + ~name:"ec_undo" + ~description: + "Undo the last engine step, returning to the immediately \ + preceding state. The ec_commit transcript is trimmed to match. \ + Fails when there is nothing left to undo." + ~input:(Schema.obj []) + ~annotations:[("destructiveHint", `Bool false)] + ~output:(Schema.output ()) + (); + + tool + ~name:"ec_revert" + ~description: + "Return the session to an earlier state, named either by a uuid \ + reported in some previous structuredContent or by a name given \ + to ec_checkpoint. Reverting is instant, unlike re-running \ + ec_load, so going back to the uuid ec_load returned is the cheap \ + way to restart a proof from scratch after a failed experiment. \ + The ec_commit transcript is trimmed to match." + ~input:(Schema.obj ~required:["target"] [ + ("target", Schema.str + ~description:"a uuid (as a decimal string) or a \ + checkpoint name" ()); + ]) + ~annotations:[("destructiveHint", `Bool true)] + ~output:(Schema.output ()) + (); + + tool + ~name:"ec_checkpoint" + ~description: + "Record the current uuid under NAME, so that ec_revert can \ + address it by name later. Worth doing before a branching \ + experiment, when carrying the bare uuid around is awkward. Does \ + not change the proof state." + ~input:(Schema.obj ~required:["name"] [ + ("name", Schema.str ~description:"checkpoint name" ()); + ]) + ~annotations:[("readOnlyHint", `Bool true)] + ~output:(Schema.output ()) + (); + + tool + ~name:"ec_commit" + ~description: + "Emit the phrases recorded since the last ec_load as a proof \ + body, with bullets inserted at every multi-child split: the \ + result compiles under `pragma +strict_bullets' and can be \ + pasted straight into the source file. Queries (search, print, \ + locate, ec_search) are never recorded, so looking things up \ + mid-proof does not pollute the body, and ec_undo / ec_revert \ + trim the transcript. Still works after `qed.'. Does not change \ + the proof state." + ~input:(Schema.obj []) + ~annotations:[("readOnlyHint", `Bool true)] + ~output:(Schema.output ()) + (); + + tool + ~name:"ec_search" + ~description: + "Search the environment for lemmas matching an EasyCrypt search \ + pattern. This is pattern syntax, not keyword search: use _ as \ + the wildcard, as in \"(fdom _)\", \"(_ %/ _)\" or \"(mu _ _) (_ \ + <= _)\". Requires a loaded file. The query neither advances the \ + proof nor enters the ec_commit transcript." + ~input:(Schema.obj ~required:["pattern"] [ + ("pattern", Schema.str + ~description:"an EasyCrypt search pattern" ()); + ]) + ~annotations:[("readOnlyHint", `Bool true)] + ~output:(Schema.output ()) + (); + ] + +(* -------------------------------------------------------------------- *) +(* Argument access. Everything here reports through [Invalid_params]: + these are failures to satisfy the declared input schema, which the + spec classifies as protocol errors, not tool-execution errors. *) +module Args = struct + let of_params (params : J.t option) = + match params with + | None | Some `Null -> [] + | Some (`Assoc fields) -> fields + | Some _ -> raise (Invalid_params "`params' must be an object") + + let arguments (params : J.t option) = + match List.assoc_opt "arguments" (of_params params) with + | None | Some `Null -> [] + | Some (`Assoc fields) -> fields + | Some _ -> raise (Invalid_params "`arguments' must be an object") + + let bad tool name expected = + raise (Invalid_params + (Printf.sprintf "%s: `%s' must be %s" tool name expected)) + + let string_req tool args name = + match List.assoc_opt name args with + | Some (`String s) -> s + | Some _ -> bad tool name "a string" + | None -> + raise (Invalid_params + (Printf.sprintf "%s: missing required argument `%s'" tool name)) + + let bool_opt tool args name ~default = + match List.assoc_opt name args with + | None | Some `Null -> default + | Some (`Bool b) -> b + | Some _ -> bad tool name "a boolean" + + let int_opt tool args name = + match List.assoc_opt name args with + | None | Some `Null -> None + | Some (`Int i) -> Some i + | Some _ -> bad tool name "an integer" +end + +(* The [ec_focus] path is a string in the schema, so its shape is ours + to check: "next", or a dotted sequence of positive integers. *) +let focus_target (arg : string) = + if String.lowercase_ascii arg = "next" then `Next + else begin + let path = + try List.map int_of_string (String.split_on_char '.' arg) + with Failure _ -> + raise (Invalid_params + (Printf.sprintf "ec_focus: not a path of integers: %s" arg)) + in + if List.exists (fun k -> k < 1) path then + raise (Invalid_params + (Printf.sprintf "ec_focus: path indices must be >= 1: %s" arg)); + `Path path + end + +(* -------------------------------------------------------------------- *) +let run ~relocdir ~boot ~projini (mcpopts : EcOptions.mcp_option) = + if mcpopts.mcpo_help then begin + print_string usage; + exit 0 + end; + + (* stdout carries the protocol and nothing else. Rather than trust + every code path under the engine to stay silent, keep a private + descriptor for the protocol and point the process's stdout at + stderr, so a stray [print_string] anywhere lands in the client's + log instead of corrupting the message stream. *) + let wire = + let fd = Unix.dup Unix.stdout in + Unix.dup2 Unix.stderr Unix.stdout; + Unix.out_channel_of_descr fd + in + + let prvopts = mcpopts.mcpo_provers in + + let st = + try EcLlmCore.create ~relocdir ~boot ~projini ~prvopts + with EcLlmCore.Init_error msg -> + Printf.eprintf "%s\n%!" msg; + exit 1 + in + + (* ------------------------------------------------------------------ *) + (* The wire: one JSON value per line, flushed at once. Yojson escapes + newlines inside strings, so a message never contains one, as the + stdio transport requires. *) + let module Wire = struct + let send (msg : J.t) = + output_string wire (J.to_string msg); + output_char wire '\n'; + flush wire + + let result id (result : J.t) = + send (`Assoc [ + ("jsonrpc", `String "2.0"); + ("id", id); + ("result", result); + ]) + + let error ?data id code message = + send (`Assoc [ + ("jsonrpc", `String "2.0"); + ("id", id); + ("error", `Assoc ([ + ("code", `Int code); + ("message", `String message); + ] @ (match data with None -> [] | Some d -> [("data", d)]))); + ]) + end in + + (* ------------------------------------------------------------------ *) + (* Rendering [EcLlmCore] outcomes as tool results. *) + let module Result_of = struct + let content text = + `List [`Assoc [("type", `String "text"); ("text", `String text)]] + + let make ~text ~uuid ~changed ~is_error ~extra = + `Assoc [ + ("content", content text); + ("structuredContent", + `Assoc ([("uuid", `Int uuid); ("changed", `Bool changed)] @ extra)); + ("isError", `Bool is_error); + ] + + (* The notice buffer holds whatever the engine said while the + operation ran; it precedes the body, as it does in the REPL. *) + let join notices body = + if notices = "" then body + else if body = "" then notices + else if String.length notices > 0 + && notices.[String.length notices - 1] = '\n' + then notices ^ body + else notices ^ "\n" ^ body + + let reply (r : EcLlmCore.reply) = + let body = + match r.EcLlmCore.body with + | EcLlmCore.Text body -> body + | EcLlmCore.Goals -> EcLlmCore.current_goals st + in + make + ~text:(join r.EcLlmCore.notices body) + ~uuid:r.EcLlmCore.uuid + ~changed:r.EcLlmCore.changed + ~is_error:false ~extra:[] + + (* A prover error is data, not a protocol failure: it comes back as + a successful response the agent can read and act on. [changed] is + computed against the uuid the call started from, since a phrase + can fail after having advanced the engine. *) + let failure ~pre ~extra (f : EcLlmCore.failure) = + let body = + if f.EcLlmCore.goals = "" then f.EcLlmCore.message + else f.EcLlmCore.message ^ "\n" ^ f.EcLlmCore.goals + in + make + ~text:(join f.EcLlmCore.notices body) + ~uuid:f.EcLlmCore.uuid + ~changed:(f.EcLlmCore.uuid <> pre) + ~is_error:true ~extra:(extra f) + + let outcome ?(extra = fun _ -> []) ~pre = function + | Ok r -> reply r + | Error f -> failure ~pre ~extra f + end in + + (* ------------------------------------------------------------------ *) + (* Tool dispatch. + + Argument checking happens here, before the engine is touched: the + core trusts what it is handed (it does not test that a LOAD path + exists, for one), and a raw [int_of_string] message has no business + reaching an agent. Schema violations raise [Invalid_params] and + become JSON-RPC errors; checks a tool makes on its own behalf raise + [Tool_error] and become [isError] results. *) + + (* Set by a phrase that ends the session ([exit.]): the response still + goes out, then the process stops. *) + let quitting = ref false in + + let answer ~pre ?(extra = fun _ -> []) = function + | EcLlmCore.Quit -> + quitting := true; + Result_of.make ~text:"session terminated" + ~uuid:(EcLlmCore.uuid st) ~changed:false ~is_error:false ~extra:[] + | EcLlmCore.Done outcome -> + Result_of.outcome ~extra ~pre outcome + in + + let call_tool (name : string) (params : J.t option) : J.t = + let args = Args.arguments params in + let pre = EcLlmCore.uuid st in + let outcome = Result_of.outcome ~pre in + + match name with + | "ec_load" -> + let file = Args.string_req name args "file" in + let line = Args.int_opt name args "line" in + let col = Args.int_opt name args "col" in + let nosmt = Args.bool_opt name args "nosmt" ~default:false in + let trace = Args.bool_opt name args "trace" ~default:false in + if line = None && col <> None then + raise (Invalid_params "ec_load: `col' requires `line'"); + if not (Sys.file_exists file) then + raise (Tool_error + (Printf.sprintf "LOAD: no such file: %s" file)); + let upto = Option.map (fun line -> (line, col)) line in + outcome (EcLlmCore.load st ~file ~upto ~nosmt ~trace) + + | "ec_step" -> + answer ~pre (EcLlmCore.step st (Args.string_req name args "phrase")) + + | "ec_try" -> + answer ~pre + ~extra:(fun (f : EcLlmCore.failure) -> + [("reverted", `Bool f.EcLlmCore.reverted)]) + (EcLlmCore.try_step st (Args.string_req name args "phrase")) + + | "ec_goals" -> + outcome (EcLlmCore.goals st + ~all:(Args.bool_opt name args "all" ~default:false)) + + | "ec_tree" -> + outcome (EcLlmCore.tree st + ~all:(Args.bool_opt name args "full" ~default:false)) + + | "ec_focus" -> + outcome (EcLlmCore.focus st + (focus_target (Args.string_req name args "path"))) + + | "ec_undo" -> + outcome (EcLlmCore.undo st) + + | "ec_revert" -> + outcome (EcLlmCore.revert st (Args.string_req name args "target")) + + | "ec_checkpoint" -> + outcome (EcLlmCore.checkpoint st + ~name:(Args.string_req name args "name")) + + | "ec_commit" -> + outcome (EcLlmCore.commit st) + + | "ec_search" -> + answer ~pre (EcLlmCore.search st + ~pattern:(Args.string_req name args "pattern")) + + | _ -> + raise (Invalid_params (Printf.sprintf "unknown tool: %s" name)) + in + + (* ------------------------------------------------------------------ *) + (* Requests. *) + let initialize (params : J.t option) = + let requested = + match List.assoc_opt "protocolVersion" (Args.of_params params) with + | Some (`String v) -> Some v + | _ -> None + in + (* Spec: answer with the requested version when we speak it, + otherwise with the latest one we do speak. *) + let negotiated = + match requested with + | Some v when List.mem v protocol_supported -> v + | _ -> protocol_latest + in + `Assoc [ + ("protocolVersion", `String negotiated); + ("capabilities", `Assoc [("tools", `Assoc [])]); + ("serverInfo", `Assoc [ + ("name", `String server_name); + ("version", `String server_version); + ]); + ] + in + + let request id (meth : string) (params : J.t option) = + try + match meth with + | "initialize" -> + Wire.result id (initialize params) + | "ping" -> + Wire.result id (`Assoc []) + | "tools/list" -> + (* The tool set is static and short: no pagination, and a + [cursor] argument is simply ignored. *) + Wire.result id (`Assoc [("tools", `List tools)]) + | "tools/call" -> + let name = + match List.assoc_opt "name" (Args.of_params params) with + | Some (`String s) -> s + | Some _ -> raise (Invalid_params "`name' must be a string") + | None -> raise (Invalid_params "missing tool `name'") + in + let result = + try call_tool name params with + | Tool_error msg -> + Result_of.make ~text:msg ~uuid:(EcLlmCore.uuid st) + ~changed:false ~is_error:true ~extra:[] + in + Wire.result id result; + if !quitting then exit 0 + | _ -> + Wire.error id e_method_not_found + (Printf.sprintf "method not found: %s" meth) + with + | Invalid_params msg -> Wire.error id e_invalid_params msg + in + + (* Notifications never get a reply, whatever they are. The ones the + spec has us tolerate ([initialized], [cancelled], + [roots/list_changed]) are no-ops here, and so is anything else: + cancellation cannot preempt a synchronous tool call. *) + let notification (_ : string) (_ : J.t option) = () in + + (* ------------------------------------------------------------------ *) + let dispatch (msg : J.t) = + match msg with + | `List _ -> + (* Batching was removed from the protocol in revision 2025-06-18 + and has not come back. *) + Wire.error `Null e_invalid_request + "JSON-RPC batches are not supported by this protocol revision" + | `Assoc fields -> + let params = List.assoc_opt "params" fields in + let id = + (* A message is a request exactly when it carries a usable id; + MCP forbids a null id, so we read one as "no id" and stay + silent rather than answer a malformed request. *) + match List.assoc_opt "id" fields with + | None | Some `Null -> None + | Some id -> Some id + in + begin match List.assoc_opt "method" fields, id with + | Some (`String meth), Some id -> request id meth params + | Some (`String meth), None -> notification meth params + | Some _, Some id -> + Wire.error id e_invalid_request "`method' must be a string" + | Some _, None -> () + | None, Some id -> + Wire.error id e_invalid_request "missing `method'" + | None, None -> () + end + | _ -> + Wire.error `Null e_invalid_request + "a JSON-RPC message must be an object" + in + + (* ------------------------------------------------------------------ *) + (* Main loop. A blank line is not a message; skipping it keeps a + client's trailing newline from drawing a parse error. *) + begin try while true do + let line = input_line stdin in + if String.trim line <> "" then + match J.from_string line with + | exception _ -> + Wire.error `Null e_parse_error "invalid JSON" + | msg -> dispatch msg + done with End_of_file -> () end; + + exit 0 diff --git a/src/ecMcp.mli b/src/ecMcp.mli new file mode 100644 index 000000000..b8eeb3ac0 --- /dev/null +++ b/src/ecMcp.mli @@ -0,0 +1,14 @@ +(* -------------------------------------------------------------------- *) +(* Model Context Protocol server over stdio: a second front-end, next to + the [easycrypt llm] REPL, over the shared engine core in + [EcLlmCore]. Driven via the [easycrypt mcp] command. *) + +(* Serve JSON-RPC 2.0 messages on stdin/stdout until end of input, then + exit the process. Never returns. [projini] resolves the + [easycrypt.project] context of a file path, as for the REPL. *) +val run : + relocdir:string option + -> boot:bool + -> projini:(string option -> EcOptions.ini_context option) + -> EcOptions.mcp_option + -> 'a diff --git a/src/ecOptions.ml b/src/ecOptions.ml index dccc5858c..82e4d2dd9 100644 --- a/src/ecOptions.ml +++ b/src/ecOptions.ml @@ -11,6 +11,7 @@ type command = [ | `Why3Config | `DocGen of doc_option | `Llm of llm_option + | `Mcp of mcp_option ] and options = { @@ -54,6 +55,11 @@ and llm_option = { llmo_eval : string option; } +and mcp_option = { + mcpo_provers : prv_options; + mcpo_help : bool; +} + and prv_options = { prvo_maxjobs : int option; prvo_timeout : int option; @@ -387,6 +393,11 @@ let specs = { `Spec ("help", `Flag , "Print the LLM agent guide and exit"); `Spec ("eval", `String, "Run the given commands (newline-separated) and exit, in lieu of reading stdin")]); + ("mcp", "Model Context Protocol server (stdio)", [ + `Group "loader"; + `Group "provers"; + `Spec ("help", `Flag , "Print the MCP server usage and exit")]); + ("cli", "Run EasyCrypt top-level", [ `Group "loader"; `Group "provers"; @@ -623,6 +634,10 @@ let llm_options_of_values ini values = llmo_help = get_flag "help" values; llmo_eval = get_string "eval" values; } +let mcp_options_of_values ini values = + { mcpo_provers = prv_options_of_values ini values; + mcpo_help = get_flag "help" values; } + (* -------------------------------------------------------------------- *) let parse getini argv = let (command, values, anons) = parse specs argv in @@ -703,6 +718,15 @@ let parse getini argv = (cmd, ini, true) + | "mcp" -> + if not (List.is_empty anons) then + raise (Arg.Bad "this command does not take arguments"); + + let ini = getini None in + let cmd = `Mcp (mcp_options_of_values ini values) in + + (cmd, ini, true) + | _ -> assert false in { diff --git a/src/ecOptions.mli b/src/ecOptions.mli index d409a10bb..2c10f3a1d 100644 --- a/src/ecOptions.mli +++ b/src/ecOptions.mli @@ -7,6 +7,7 @@ type command = [ | `Why3Config | `DocGen of doc_option | `Llm of llm_option + | `Mcp of mcp_option ] and options = { @@ -50,6 +51,11 @@ and llm_option = { llmo_eval : string option; } +and mcp_option = { + mcpo_provers : prv_options; + mcpo_help : bool; +} + and prv_options = { prvo_maxjobs : int option; prvo_timeout : int option; From 8ab6678de09e750c198378410ba073b985856df2 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Fri, 21 Aug 2026 15:32:43 +0200 Subject: [PATCH 26/51] [llm] step runs every sentence of its input, not just the first `EcLlmCore.step` parsed one toplevel phrase and silently dropped the rest, so `split. trivial. trivial.` ran a single `split.`. It now loops over `EcIo.xparse` the way the LOAD prefix does, handling each item as before and answering with one reply at the end. The semantics are those of a file: sentences run in order, a failure stops the run there and comes back as the reply, and everything applied before it stays applied. `exit.` ends the session immediately, with the sentences that preceded it applied. The reply body is decided by the last item that did something, so a lone doc comment still answers with an empty body rather than with goals. Both front-ends inherit this through the core; ec_step's and ec_try's tool descriptions and the guide's "EasyCrypt commands" section are updated to the new truth. Goldens: no existing scenario changes bytes (all 19 pass untouched -- none of them packed two sentences onto one line). Two new scenarios: * multi-sentence -- `split. trivial. trivial.` closes the proof in one line (uuid 3 -> 6) and COMMIT shows all three sentences; * multi-sentence-error -- `split. apply nosuchlemma. trivial.` stops at the failure with `split.` applied (uuid 4, two open goals) and COMMIT holding `split.' alone. --- doc/llm/CLAUDE.md | 14 ++++- src/ecLlmCore.ml | 57 ++++++++++++------- src/ecLlmCore.mli | 13 ++++- src/ecMcp.ml | 31 +++++----- tests/llm/expected/multi-sentence-error.out | 27 +++++++++ tests/llm/expected/multi-sentence.out | 21 +++++++ tests/llm/scripts/multi-sentence-error.script | 10 ++++ tests/llm/scripts/multi-sentence.script | 7 +++ 8 files changed, 142 insertions(+), 38 deletions(-) create mode 100644 tests/llm/expected/multi-sentence-error.out create mode 100644 tests/llm/expected/multi-sentence.out create mode 100644 tests/llm/scripts/multi-sentence-error.script create mode 100644 tests/llm/scripts/multi-sentence.script diff --git a/doc/llm/CLAUDE.md b/doc/llm/CLAUDE.md index a4fb29087..b3d6cfe7f 100644 --- a/doc/llm/CLAUDE.md +++ b/doc/llm/CLAUDE.md @@ -87,7 +87,7 @@ These are protocol-level commands, not EasyCrypt syntax: Any line that is not a meta-command is parsed as EasyCrypt input. This covers tactics, declarations, `search`, `print`, `require`, -etc. The line must be a complete EasyCrypt statement ending with `.` +etc. Every statement on the line must be complete and end with `.` ``` smt(). @@ -96,6 +96,18 @@ search (%/). print mulzK. ``` +A line may hold several statements; all of them are executed, in +order, exactly as if the text had been appended to the source file, +and a single reply describes the state they leave behind: + +``` +split. trivial. trivial. +``` + +If one of them fails, the reply is that failure and the statements +before it stay applied — again as in a file. `exit.` ends the session +there, with the statements that preceded it applied. + For multi-line statements, wrap with `` and ``: ``` diff --git a/src/ecLlmCore.ml b/src/ecLlmCore.ml index cd75a6afc..830d5e346 100644 --- a/src/ecLlmCore.ml +++ b/src/ecLlmCore.ml @@ -623,8 +623,11 @@ let make_failure (st : state) (message : string) = mk_failure st message (* -------------------------------------------------------------------- *) -(* Process EasyCrypt input typed at the prompt (single phrase or a - line ending with a "."). *) +(* Process EasyCrypt input typed at the prompt. The input is a file + fragment, not a single phrase: every sentence it holds runs, in + order, and one reply describes the state they leave behind. A + failure stops the run there; the sentences before it stay applied, + exactly as they would in a compiled file. *) let step (st : state) input = let notices = st.notices in let prior_bullets = st.prior_bullets in @@ -638,25 +641,41 @@ let step (st : state) input = | Some _ as snapshot -> prior_bullets := snapshot); let reader = EcIo.from_string input in let last_src = ref "" in + (* Reply body, decided by the last item that did something: a run of + sentences ends on the goals, a doc comment on an empty body. The + end-of-input marker is an empty [P_Prog], and must not count. *) + let body = ref Goals in + let quit = ref false in let answer = begin try - let (src, prog) = EcIo.xparse reader in - let src = String.strip src in - last_src := src; - begin match EcLocation.unloc prog with - | EP.P_Prog (commands, _) -> - List.iter (process_action st ~record:true ~src) commands; - Done (Ok (mk_reply_goals st ~pre)) - | EP.P_Undo i -> - EcCommands.undo i; - Transcript.trim st i; - Done (Ok (mk_reply_goals st ~pre)) - | EP.P_Exit -> - Quit - | EP.P_DocComment doc -> - EcCommands.doc_comment doc; - Done (Ok (mk_reply st ~pre (Text ""))) - end + begin try while true do + last_src := ""; + let (src, prog) = EcIo.xparse reader in + let src = String.strip src in + last_src := src; + match EcLocation.unloc prog with + | EP.P_Prog (commands, locterm) -> + if commands <> [] then begin + body := Goals; + List.iter (process_action st ~record:true ~src) commands + end; + if locterm then raise Exit + | EP.P_Undo i -> + body := Goals; + EcCommands.undo i; + Transcript.trim st i + | EP.P_Exit -> + (* Everything before [exit.] stays applied; the front-end + owns what happens next. *) + quit := true; raise Exit + | EP.P_DocComment doc -> + body := Text ""; + EcCommands.doc_comment doc + done with Exit | End_of_file -> () end; + if !quit then Quit else + match !body with + | Goals -> Done (Ok (mk_reply_goals st ~pre)) + | Text _ as b -> Done (Ok (mk_reply st ~pre b)) with | EcCommands.Restart -> do_initialize st; diff --git a/src/ecLlmCore.mli b/src/ecLlmCore.mli index 8580feb6b..19e588497 100644 --- a/src/ecLlmCore.mli +++ b/src/ecLlmCore.mli @@ -77,14 +77,21 @@ val load : -> trace:bool -> (reply, failure) result -(* One line of raw EasyCrypt input (or a multi-line block). *) +(* Raw EasyCrypt input: one line, or a multi-line block. Every + sentence the input holds is executed, in order, and a single reply + describes the state they leave behind. A sentence that fails stops + the run at that point and its failure is returned; the sentences + before it stay applied, as they would in a compiled file. An + [exit.] ends the session immediately, with the sentences that + preceded it applied. *) val step : state -> string -> answer (* [step], but a failure leaves no trace: the engine is rolled back to the uuid it had on entry (as REVERT does) and the failure comes back with [reverted = true]. Successes and [Quit] behave exactly as in - [step]. A phrase that fails after having already advanced the engine - is rolled back whole. *) + [step]. Input that fails after having already advanced the engine + -- a phrase with a side effect, or an earlier sentence of a + multi-sentence input -- is rolled back whole. *) val try_step : state -> string -> answer val goals : state -> all:bool -> (reply, failure) result diff --git a/src/ecMcp.ml b/src/ecMcp.ml index 49e3b2aef..6ac8b04c0 100644 --- a/src/ecMcp.ml +++ b/src/ecMcp.ml @@ -68,8 +68,8 @@ proof session. Standard loader and prover options (-I, -timeout, -p, Tools: ec_load compile a file up to a position and start a session - ec_step run one EasyCrypt sentence - ec_try run one sentence, rolling back if it fails + ec_step run one or more EasyCrypt sentences + ec_try run sentences, rolling back if any of them fails ec_goals print the current goal state ec_tree list the open subgoals as a labelled tree ec_focus focus the subgoal at a dotted path (or `next') @@ -186,21 +186,22 @@ let tools : J.t list = tool ~name:"ec_step" ~description: - "Run one EasyCrypt sentence -- a tactic, a declaration, require, \ - print, ... -- against the current session. Send exactly one \ - complete sentence, ending with a period: anything past the \ - first sentence of the phrase is ignored, so chain tactics with \ - `;' rather than with `.'. A sentence spanning several lines is \ - fine as one string. Requires a file loaded with ec_load, and, \ + "Run EasyCrypt sentences -- tactics, declarations, require, \ + print, ... -- against the current session. Every complete \ + sentence in the argument is executed, in order, exactly as if \ + the text had been appended to the source file, and a single \ + reply describes the state they leave behind; sentences may \ + span several lines. Requires a file loaded with ec_load, and, \ for tactics, an open proof. On success the reply carries the \ new goal state; on failure the prover's error text comes back \ - with isError set and the engine is left wherever the sentence \ - left it -- use ec_try when you want a guaranteed rollback. \ - Successful non-query phrases are recorded for ec_commit." + with isError set, the sentences before the failing one stay \ + applied and the engine is left wherever that sentence left it \ + -- use ec_try when you want a guaranteed rollback. Successful \ + non-query phrases are recorded for ec_commit." ~input:(Schema.obj ~required:["phrase"] [ ("phrase", Schema.str - ~description:"one complete EasyCrypt sentence, \ - ending with `.'" ()); + ~description:"one or more complete EasyCrypt \ + sentences, each ending with `.'" ()); ]) ~annotations:[("destructiveHint", `Bool false); ("idempotentHint", `Bool false)] @@ -211,8 +212,8 @@ let tools : J.t list = ~name:"ec_try" ~description: "Like ec_step, but the engine is rolled back to the state it had \ - before the call whenever the phrase fails, including a phrase \ - that failed only after having already advanced the proof. The \ + before the call whenever a sentence fails, including input that \ + failed only after having already advanced the proof. The \ failure reply sets structuredContent.reverted to true, and its \ uuid and goal text describe the restored state, not the point \ of failure. Use this to probe a tactic without having to \ diff --git a/tests/llm/expected/multi-sentence-error.out b/tests/llm/expected/multi-sentence-error.out new file mode 100644 index 000000000..2bd1cf2bf --- /dev/null +++ b/tests/llm/expected/multi-sentence-error.out @@ -0,0 +1,27 @@ +READY [uuid:0] + +OK [uuid:3] [loaded:fixtures/simple.ec:6] +Current goal + +Type variables: + +------------------------------------------------------------------------ +1 = 1 /\ 2 = 2 + +ERROR [uuid:4] +: line 1 (7-25): unknown lemma `nosuchlemma' +source: apply nosuchlemma. +Current goal (remaining: 2) + +Type variables: + +------------------------------------------------------------------------ +1 = 1 + +OK [uuid:4] [focus: 1/2] +[1] 1 = 1 <- focused +[2] 2 = 2 + +OK [uuid:4] [focus: 1/2] +split. + diff --git a/tests/llm/expected/multi-sentence.out b/tests/llm/expected/multi-sentence.out new file mode 100644 index 000000000..46f132906 --- /dev/null +++ b/tests/llm/expected/multi-sentence.out @@ -0,0 +1,21 @@ +READY [uuid:0] + +OK [uuid:3] [loaded:fixtures/simple.ec:6] +Current goal + +Type variables: + +------------------------------------------------------------------------ +1 = 1 /\ 2 = 2 + +OK [uuid:6] +No more goals + +OK [uuid:6] +No more goals + +OK [uuid:6] +split. +- trivial. +- trivial. + diff --git a/tests/llm/scripts/multi-sentence-error.script b/tests/llm/scripts/multi-sentence-error.script new file mode 100644 index 000000000..3471c6909 --- /dev/null +++ b/tests/llm/scripts/multi-sentence-error.script @@ -0,0 +1,10 @@ +# exit: 1 +# File semantics for a multi-sentence line: the sentences before the +# failing one stay applied. `split.' succeeds, `apply nosuchlemma.' +# fails, and the trailing `trivial.' never runs -- so the session is +# left with the two goals `split.' opened, and COMMIT holds `split.' +# alone. +LOAD "fixtures/simple.ec" 6 +split. apply nosuchlemma. trivial. +TREE +COMMIT diff --git a/tests/llm/scripts/multi-sentence.script b/tests/llm/scripts/multi-sentence.script new file mode 100644 index 000000000..29efb00d1 --- /dev/null +++ b/tests/llm/scripts/multi-sentence.script @@ -0,0 +1,7 @@ +# exit: 0 +# Several sentences on one line: every one of them runs, and a single +# reply describes the state they leave behind. COMMIT records all three. +LOAD "fixtures/simple.ec" 6 +split. trivial. trivial. +GOALS +COMMIT From ccfad83484c1269093cd1896a262dcfa9562dfe6 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Fri, 21 Aug 2026 15:34:50 +0200 Subject: [PATCH 27/51] [llm] queries no longer spend a uuid `Gsearch`/`Gprint`/`Glocate` reached the engine through `EcCommands.process`, which pushes an undo context for whatever it runs. A query hands back the very scope it was given, so the pushed context was a duplicate of the current one -- but it still bumped `ct_level`, so SEARCH advanced the uuid while being advertised as read-only, and an UNDO right after a SEARCH only undid the query. `EcLlmCore.process_action` now pops that context back off for the three query constructors, restoring `ct_level` and the undo stack exactly as they were (a no-op when the query itself failed, since nothing was pushed then). The fix sits in the llm core rather than in `EcCommands.process` so the batch compiler and the ProofGeneral terminal keep counting phrases the way they always have. SEARCH / `search .` / `print .` / `locate .` now reply with the pre-call uuid, and ec_search reports changed:false, consistent with its readOnlyHint:true annotation. Manual UNDO/REVERT check (tests/llm, ec.native llm -eval): LOAD "fixtures/simple.ec" 6 -> uuid 3, goal `1 = 1 /\ 2 = 2' split. -> uuid 4, two goals SEARCH (b2i _) -> uuid 4 (was 5), goals unchanged UNDO -> uuid 3, back to `1 = 1 /\ 2 = 2' (before: landed on uuid 4, i.e. it undid nothing but the query) CHECKPOINT c0 / trivial. / SEARCH / REVERT c0 -> uuid 3, goal restored, COMMIT empty Goldens re-recorded, uuid shifts only: * search -- SEARCH reply and the following ERROR go 4 -> 3; * search-in-proof -- the SEARCH reply goes 6 -> 5 and the three replies after it go 7 -> 6. --- doc/llm/CLAUDE.md | 6 ++++-- src/ecLlmCore.ml | 20 +++++++++++++------- tests/llm/expected/search-in-proof.out | 8 ++++---- tests/llm/expected/search.out | 4 ++-- 4 files changed, 23 insertions(+), 15 deletions(-) diff --git a/doc/llm/CLAUDE.md b/doc/llm/CLAUDE.md index b3d6cfe7f..0bd38c7a4 100644 --- a/doc/llm/CLAUDE.md +++ b/doc/llm/CLAUDE.md @@ -58,7 +58,9 @@ ERROR [uuid:N] ``` The `uuid` is a monotonically increasing integer identifying the proof -engine state. It increments with each successful command. +engine state. It increments with each successful command that changes +that state. Queries do not change it: `SEARCH`, and the `search`, +`print` and `locate` statements, report the uuid they were called at. ### Meta-commands @@ -77,7 +79,7 @@ These are protocol-level commands, not EasyCrypt syntax: | `NEXT` | Rotate focus to the next subgoal (equivalent to `FOCUS 2`) | | `COMMIT` | Emit recorded REPL phrases as a bulleted proof body (works under `+strict_bullets`) | | `CHECKPOINT ` | Save current uuid under a name for later `REVERT` | -| `SEARCH ` | Search for lemmas matching a pattern | +| `SEARCH ` | Search for lemmas matching a pattern (read-only: the uuid does not move) | | `QUIET ON` / `QUIET OFF` | Suppress/enable automatic goal display after tactics | | `` / `` | Delimit multi-line EasyCrypt input | | `HELP` | Print this guide | diff --git a/src/ecLlmCore.ml b/src/ecLlmCore.ml index 830d5e346..aa161470e 100644 --- a/src/ecLlmCore.ml +++ b/src/ecLlmCore.ml @@ -403,6 +403,13 @@ let process_action (st : state) ?(record=false) ~src (p : EP.global) = let parent = match opens_pre with h :: _ -> Some h | [] -> None in + (* Queries only inspect the environment: they neither advance the + proof nor belong in the body COMMIT emits. *) + let is_query = + match EcLocation.unloc p.EP.gl_action with + | EP.Gprint _ | EP.Gsearch _ | EP.Glocate _ -> true + | _ -> false + in let succeeded = ref false in begin try ignore (EcCommands.process ~src p.EP.gl_action : float option); @@ -412,17 +419,16 @@ let process_action (st : state) ?(record=false) ~src (p : EP.global) = | _ when p.EP.gl_fail -> () | e -> raise (EcScope.toperror_of_exn ~gloc:loc e) end; + (* The engine pushes an undo context for every command it runs, a + query included -- with the *same* scope, since a query returns the + scope it was handed. Pop it back off: a read-only command must not + spend a uuid, or REVERT targets and the MCP [readOnlyHint] would + both be lying. A no-op when the query failed (nothing was pushed). *) + if is_query then EcCommands.undo pre_uuid; if !succeeded && p.EP.gl_fail then raise (EcScope.toperror_of_exn ~gloc:loc (EcScope.HiScopeError (None, "this command is expected to fail"))); - (* Queries only inspect the environment: they neither advance the - proof nor belong in the body COMMIT emits. *) - let is_query = - match EcLocation.unloc p.EP.gl_action with - | EP.Gprint _ | EP.Gsearch _ | EP.Glocate _ -> true - | _ -> false - in if record && !succeeded && not p.EP.gl_fail && not is_query then begin transcript := (pre_uuid, src, parent, opens_pre) :: !transcript; (* Keep the newest non-empty snapshot: a phrase that closes the diff --git a/tests/llm/expected/search-in-proof.out b/tests/llm/expected/search-in-proof.out index 4c588e2e7..e2d3a9984 100644 --- a/tests/llm/expected/search-in-proof.out +++ b/tests/llm/expected/search-in-proof.out @@ -12,7 +12,7 @@ OK [uuid:4] OK [uuid:5] -OK [uuid:6] +OK [uuid:5] (* RField.signr_odd *) lemma signr_odd: forall (n : int), 0 <= n => (- 1%r) ^ b2i (odd n) = (- 1%r) ^ n. @@ -30,11 +30,11 @@ lemma b2i0: b2i false = 0. lemma signr_odd: forall (n : int), 0 <= n => (-1) ^ b2i (odd n) = (-1) ^ n. -OK [uuid:7] +OK [uuid:6] -OK [uuid:7] +OK [uuid:6] -OK [uuid:7] +OK [uuid:6] - trivial. - trivial. diff --git a/tests/llm/expected/search.out b/tests/llm/expected/search.out index 7a5c27fbe..055e55810 100644 --- a/tests/llm/expected/search.out +++ b/tests/llm/expected/search.out @@ -8,7 +8,7 @@ Type variables: ------------------------------------------------------------------------ 1 = 1 /\ 2 = 2 -OK [uuid:4] +OK [uuid:3] (* RField.signr_odd *) lemma signr_odd: forall (n : int), 0 <= n => (- 1%r) ^ b2i (odd n) = (- 1%r) ^ n. @@ -32,7 +32,7 @@ Type variables: ------------------------------------------------------------------------ 1 = 1 /\ 2 = 2 -ERROR [uuid:4] +ERROR [uuid:3] SEARCH: missing query Current goal From 19e2922c20d1b3f903c4a3f27f8078b8894bcade Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Fri, 21 Aug 2026 15:36:43 +0200 Subject: [PATCH 28/51] [llm] core computes `changed' for failures too `EcLlmCore.failure` gained a `changed` field, computed like the one on `reply`: the uuid after the operation against the uuid it started from. A failing phrase can advance the engine before failing, so the flag is not derivable from the failure alone -- `ecMcp.ml` was recomputing it from a `pre` it had to carry through `call_tool`, `answer` and `Result_of`, which is exactly the engine knowledge a front-end should not hold. That plumbing is gone; the front-end reads the field. `try_step` re-stamps it after the rollback, so it reports the *net* effect of the call: false for a phrase that advanced, failed and was rolled back, since after the rollback there is nothing left to have changed. It stays true only when the rollback cannot reach the entry uuid -- a `pragma Reset` that dropped the engine below it. The choice is documented on the type in ecLlmCore.mli. The REPL ignores the field (its envelope has never reported `changed`). No golden changes, as expected: all 21 scenarios pass untouched. Checked over the wire (tests/llm, ec.native mcp): ec_load -> uuid 3, changed true ec_search "(b2i _)" -> uuid 3, changed false ec_step "apply nosuchlemma." -> uuid 3, changed false, isError ec_step "split. apply nosuchlemma."-> uuid 4, changed true, isError ec_try "split. apply nosuchlemma."-> uuid 3, changed false, reverted true, isError --- src/ecLlmCore.ml | 45 +++++++++++++++++++++++++++------------------ src/ecLlmCore.mli | 8 +++++++- src/ecMcp.ml | 25 +++++++++++-------------- 3 files changed, 45 insertions(+), 33 deletions(-) diff --git a/src/ecLlmCore.ml b/src/ecLlmCore.ml index aa161470e..dbc5d7b45 100644 --- a/src/ecLlmCore.ml +++ b/src/ecLlmCore.ml @@ -27,6 +27,7 @@ type failure = { goals : string; notices : string; reverted : bool; + changed : bool; } type answer = @@ -365,11 +366,12 @@ let mk_reply_goals (st : state) ~(pre : int) = let tag = Goals.focus_tag () in mk_reply st ~pre ~tag Goals -let mk_failure (st : state) (message : string) = +let mk_failure (st : state) ~(pre : int) (message : string) = let notices = Buffer.contents st.notices in Buffer.clear st.notices; - { uuid = EcCommands.uuid (); message; - goals = Goals.goals_to_string (); notices; reverted = false; } + let uuid = EcCommands.uuid () in + { uuid; message; goals = Goals.goals_to_string (); notices; + reverted = false; changed = uuid <> pre; } (* -------------------------------------------------------------------- *) (* Transcript manipulation. *) @@ -626,7 +628,7 @@ let make_reply (st : state) ?tag (body : body) = mk_reply st ~pre:(EcCommands.uuid ()) ?tag body let make_failure (st : state) (message : string) = - mk_failure st message + mk_failure st ~pre:(EcCommands.uuid ()) message (* -------------------------------------------------------------------- *) (* Process EasyCrypt input typed at the prompt. The input is a file @@ -688,7 +690,7 @@ let step (st : state) input = Transcript.clear st; Done (Ok (mk_reply st ~pre (Text "Session restarted"))) | e -> - Done (Error (mk_failure st (Goals.format_error ~src:!last_src e))) + Done (Error (mk_failure st ~pre (Goals.format_error ~src:!last_src e))) end in EcIo.finalize reader; @@ -698,9 +700,14 @@ let step (st : state) input = (* [step] with an automatic rollback on failure. A phrase can fail after having advanced the engine, so the pre-entry uuid is the only faithful notion of "unchanged": restore it the way REVERT does. - The failure is then re-stamped, because its uuid and goal text - described the state at the point of failure, which no longer - exists. *) + The failure is then re-stamped, because its uuid, goal text and + [changed] flag described the state at the point of failure, which no + longer exists. [changed] is recomputed against the same pre-entry + uuid, so it reports the *net* effect of the call: normally [false], + the rollback having undone whatever the input managed to do. It + stays [true] in the one case where the rollback cannot reach [pre], + namely a phrase that reset the engine ([pragma Reset]) and landed + below it -- then the state really did change. *) let try_step (st : state) input = let pre = EcCommands.uuid () in match step st input with @@ -709,10 +716,12 @@ let try_step (st : state) input = | Done (Error failure) -> EcCommands.undo pre; Transcript.trim st pre; + let uuid = EcCommands.uuid () in Done (Error { failure with - uuid = EcCommands.uuid (); + uuid; goals = Goals.goals_to_string (); - reverted = true; }) + reverted = true; + changed = uuid <> pre; }) (* -------------------------------------------------------------------- *) (* LOAD: run [file] up to [upto], optionally with SMT calls weakened @@ -957,11 +966,11 @@ let load (st : state) ~file ~upto ~nosmt ~trace = Ok (mk_reply st ~pre (Text "Session restarted")) | Trace_failed e -> let msg = Goals.format_error ~src:!last_src e in - Error (mk_failure st (!trace_prefix ^ msg)) + Error (mk_failure st ~pre (!trace_prefix ^ msg)) | Failure s -> - Error (mk_failure st s) + Error (mk_failure st ~pre s) | e -> - Error (mk_failure st (Goals.format_error ~src:!last_src e)) + Error (mk_failure st ~pre (Goals.format_error ~src:!last_src e)) (* -------------------------------------------------------------------- *) (* The remaining meta-commands. *) @@ -993,7 +1002,7 @@ let undo (st : state) = Transcript.trim st (uuid - 1); Ok (mk_reply_goals st ~pre) end else - Error (mk_failure st "nothing to undo") + Error (mk_failure st ~pre "nothing to undo") let focus (st : state) request = (* [request] is the user's intent normalized: @@ -1010,11 +1019,11 @@ let focus (st : state) request = | `Path path -> FrameTree.resolve_path path in match resolved with - | Error msg -> Error (mk_failure st msg) + | Error msg -> Error (mk_failure st ~pre msg) | Ok target -> match EcCommands.focus_goal target with | Ok _ -> Ok (mk_reply_goals st ~pre) - | Error msg -> Error (mk_failure st msg) + | Error msg -> Error (mk_failure st ~pre msg) let checkpoint (st : state) ~name = let pre = EcCommands.uuid () in @@ -1032,12 +1041,12 @@ let revert (st : state) spec = in match target with | None -> - Error (mk_failure st (Printf.sprintf + Error (mk_failure st ~pre (Printf.sprintf "REVERT: '%s' is not a valid uuid or checkpoint name" spec)) | Some target -> let uuid = EcCommands.uuid () in if target < 0 || target > uuid then - Error (mk_failure st (Printf.sprintf + Error (mk_failure st ~pre (Printf.sprintf "REVERT: uuid %d out of range [0, %d]" target uuid)) else begin EcCommands.undo target; diff --git a/src/ecLlmCore.mli b/src/ecLlmCore.mli index 19e588497..ba85d13b3 100644 --- a/src/ecLlmCore.mli +++ b/src/ecLlmCore.mli @@ -34,13 +34,19 @@ type reply = { the same, so the buffer is left clean for the next operation. [reverted] is set by [try_step] only: it says the engine was rolled back to the state it had before the operation ran, so [uuid] and - [goals] describe that restored state, not the point of failure. *) + [goals] describe that restored state, not the point of failure. + [changed] tells whether the engine uuid advanced -- a failing + operation may well have moved the engine before failing. It reports + the *net* effect of the call, so under [try_step] it is [false] for + a phrase that advanced, failed and was rolled back: after the + rollback there is nothing left to have changed. *) type failure = { uuid : int; message : string; goals : string; notices : string; reverted : bool; + changed : bool; } (* Operations that can be asked to end the session ([exit.]) return an diff --git a/src/ecMcp.ml b/src/ecMcp.ml index 6ac8b04c0..3d7d5c352 100644 --- a/src/ecMcp.ml +++ b/src/ecMcp.ml @@ -508,10 +508,8 @@ let run ~relocdir ~boot ~projini (mcpopts : EcOptions.mcp_option) = ~is_error:false ~extra:[] (* A prover error is data, not a protocol failure: it comes back as - a successful response the agent can read and act on. [changed] is - computed against the uuid the call started from, since a phrase - can fail after having advanced the engine. *) - let failure ~pre ~extra (f : EcLlmCore.failure) = + a successful response the agent can read and act on. *) + let failure ~extra (f : EcLlmCore.failure) = let body = if f.EcLlmCore.goals = "" then f.EcLlmCore.message else f.EcLlmCore.message ^ "\n" ^ f.EcLlmCore.goals @@ -519,12 +517,12 @@ let run ~relocdir ~boot ~projini (mcpopts : EcOptions.mcp_option) = make ~text:(join f.EcLlmCore.notices body) ~uuid:f.EcLlmCore.uuid - ~changed:(f.EcLlmCore.uuid <> pre) + ~changed:f.EcLlmCore.changed ~is_error:true ~extra:(extra f) - let outcome ?(extra = fun _ -> []) ~pre = function + let outcome ?(extra = fun _ -> []) = function | Ok r -> reply r - | Error f -> failure ~pre ~extra f + | Error f -> failure ~extra f end in (* ------------------------------------------------------------------ *) @@ -541,19 +539,18 @@ let run ~relocdir ~boot ~projini (mcpopts : EcOptions.mcp_option) = goes out, then the process stops. *) let quitting = ref false in - let answer ~pre ?(extra = fun _ -> []) = function + let answer ?(extra = fun _ -> []) = function | EcLlmCore.Quit -> quitting := true; Result_of.make ~text:"session terminated" ~uuid:(EcLlmCore.uuid st) ~changed:false ~is_error:false ~extra:[] | EcLlmCore.Done outcome -> - Result_of.outcome ~extra ~pre outcome + Result_of.outcome ~extra outcome in let call_tool (name : string) (params : J.t option) : J.t = let args = Args.arguments params in - let pre = EcLlmCore.uuid st in - let outcome = Result_of.outcome ~pre in + let outcome = Result_of.outcome in match name with | "ec_load" -> @@ -571,10 +568,10 @@ let run ~relocdir ~boot ~projini (mcpopts : EcOptions.mcp_option) = outcome (EcLlmCore.load st ~file ~upto ~nosmt ~trace) | "ec_step" -> - answer ~pre (EcLlmCore.step st (Args.string_req name args "phrase")) + answer (EcLlmCore.step st (Args.string_req name args "phrase")) | "ec_try" -> - answer ~pre + answer ~extra:(fun (f : EcLlmCore.failure) -> [("reverted", `Bool f.EcLlmCore.reverted)]) (EcLlmCore.try_step st (Args.string_req name args "phrase")) @@ -605,7 +602,7 @@ let run ~relocdir ~boot ~projini (mcpopts : EcOptions.mcp_option) = outcome (EcLlmCore.commit st) | "ec_search" -> - answer ~pre (EcLlmCore.search st + answer (EcLlmCore.search st ~pattern:(Args.string_req name args "pattern")) | _ -> From 172d30fb290c9f545a96eaafb067911e127ed1a6 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Fri, 21 Aug 2026 15:40:13 +0200 Subject: [PATCH 29/51] [llm] a golden harness for the MCP front-end `make test-mcp` mirrors `make test-llm`: each tests/mcp/scripts/*.script is a newline-delimited stream of JSON-RPC messages piped into `ec.exe mcp', and its raw stdout -- the protocol, one message per line -- is diffed against tests/mcp/expected/*.out. `#'-comment lines are stripped and the first must declare the expected exit status, exactly as in llm-golden; --record and --bin behave the same way. The runner cd's into tests/mcp so fixture paths stay relative, and the scripts load the REPL harness's fixtures through ../llm/fixtures rather than duplicating them. One field of the stream is not reproducible -- serverInfo.version is a git-describe string -- and is rewritten to "VERSION" with sed before diffing; nothing else is normalized. Twelve scenarios, covering the plan's list: initialize (handshake + notifications/initialized + ping), version-negotiation (an unsupported "2099-01-01" falls back to 2025-11-25, a supported older revision is echoed), tools-list (the whole tool table verbatim), happy-path (load, step, goals, tree, focus, multi-sentence step, commit), prover-error (isError results carrying the goal state), try-revert (a phrase that advanced before failing, rolled back, with the following ec_goals proving the restored state), protocol-errors (-32700, -32600 for both a batch array and a malformed envelope, -32601, and eight -32602 cases), revert (by uuid and by checkpoint name), load-missing (a missing file and an unknown extension are isError results, NOT -32602), notifications (known and unknown ones draw no reply), exit (`exit.' answers "session terminated" and the process stops without reading further), and eof (clean shutdown). Gate: make test-llm (21) and make test-mcp (12) both green, twice in a row. --- Makefile | 7 +- scripts/testing/mcp-golden | 140 +++++++++++++++++++ tests/mcp/README.md | 124 ++++++++++++++++ tests/mcp/expected/eof.out | 1 + tests/mcp/expected/exit.out | 2 + tests/mcp/expected/happy-path.out | 8 ++ tests/mcp/expected/initialize.out | 2 + tests/mcp/expected/load-missing.out | 3 + tests/mcp/expected/notifications.out | 2 + tests/mcp/expected/protocol-errors.out | 16 +++ tests/mcp/expected/prover-error.out | 6 + tests/mcp/expected/revert.out | 11 ++ tests/mcp/expected/tools-list.out | 1 + tests/mcp/expected/try-revert.out | 7 + tests/mcp/expected/version-negotiation.out | 3 + tests/mcp/scripts/eof.script | 3 + tests/mcp/scripts/exit.script | 7 + tests/mcp/scripts/happy-path.script | 13 ++ tests/mcp/scripts/initialize.script | 6 + tests/mcp/scripts/load-missing.script | 8 ++ tests/mcp/scripts/notifications.script | 13 ++ tests/mcp/scripts/protocol-errors.script | 22 +++ tests/mcp/scripts/prover-error.script | 11 ++ tests/mcp/scripts/revert.script | 15 ++ tests/mcp/scripts/tools-list.script | 5 + tests/mcp/scripts/try-revert.script | 12 ++ tests/mcp/scripts/version-negotiation.script | 7 + 27 files changed, 454 insertions(+), 1 deletion(-) create mode 100755 scripts/testing/mcp-golden create mode 100644 tests/mcp/README.md create mode 100644 tests/mcp/expected/eof.out create mode 100644 tests/mcp/expected/exit.out create mode 100644 tests/mcp/expected/happy-path.out create mode 100644 tests/mcp/expected/initialize.out create mode 100644 tests/mcp/expected/load-missing.out create mode 100644 tests/mcp/expected/notifications.out create mode 100644 tests/mcp/expected/protocol-errors.out create mode 100644 tests/mcp/expected/prover-error.out create mode 100644 tests/mcp/expected/revert.out create mode 100644 tests/mcp/expected/tools-list.out create mode 100644 tests/mcp/expected/try-revert.out create mode 100644 tests/mcp/expected/version-negotiation.out create mode 100644 tests/mcp/scripts/eof.script create mode 100644 tests/mcp/scripts/exit.script create mode 100644 tests/mcp/scripts/happy-path.script create mode 100644 tests/mcp/scripts/initialize.script create mode 100644 tests/mcp/scripts/load-missing.script create mode 100644 tests/mcp/scripts/notifications.script create mode 100644 tests/mcp/scripts/protocol-errors.script create mode 100644 tests/mcp/scripts/prover-error.script create mode 100644 tests/mcp/scripts/revert.script create mode 100644 tests/mcp/scripts/tools-list.script create mode 100644 tests/mcp/scripts/try-revert.script create mode 100644 tests/mcp/scripts/version-negotiation.script diff --git a/Makefile b/Makefile index 05a1dcf8b..0fdb6d262 100644 --- a/Makefile +++ b/Makefile @@ -15,6 +15,8 @@ CHECK += $(foreach arg,$(ECARGS),--bin-args="$(arg)") CHECK += $(ECEXTRA) config/tests.config LLMCHECK := scripts/testing/llm-golden LLMCHECK += --bin=./ec.native +MCPCHECK := scripts/testing/mcp-golden +MCPCHECK += --bin=./ec.native NIX ?= nix --extra-experimental-features "nix-command flakes" PROFILE ?= dev @@ -24,7 +26,7 @@ UNAME_S = $(shell uname -s) # -------------------------------------------------------------------- .PHONY: default build byte native tests check examples -.PHONY: test-llm +.PHONY: test-llm test-mcp .PHONY: nix-build nix-build-with-provers nix-develop .PHONY: clean install uninstall @@ -57,6 +59,9 @@ examples: build test-llm: build $(LLMCHECK) +test-mcp: build + $(MCPCHECK) + check: unit stdlib examples @true diff --git a/scripts/testing/mcp-golden b/scripts/testing/mcp-golden new file mode 100755 index 000000000..56ee903e8 --- /dev/null +++ b/scripts/testing/mcp-golden @@ -0,0 +1,140 @@ +#! /bin/sh + +# -------------------------------------------------------------------- +# Golden-output regression harness for the `easycrypt mcp' server. +# +# mcp-golden [--bin PATH] [--record] [NAME...] +# +# Each tests/mcp/scripts/NAME.script holds the newline-delimited +# JSON-RPC messages fed to `ec.exe mcp' on stdin. Lines starting with +# `#' are stripped before the script is handed to the server; the first +# such line must be `# exit: N', the expected process exit status. +# Stdout -- the protocol stream, one JSON message per line -- is +# compared against tests/mcp/expected/NAME.out. +# +# Scripts run with tests/mcp as the working directory, so fixture paths +# stay relative and no golden bakes in a developer's home directory. +# The fixtures are the ones the REPL harness uses, under +# ../llm/fixtures. +# +# One field of the stream is not reproducible: serverInfo.version is a +# git-describe string. It is rewritten to "VERSION" before diffing. +# -------------------------------------------------------------------- + +set -u + +root=$(CDPATH= cd -- "$(dirname -- "$0")/../.." && pwd) +bin="$root/_build/default/src/ec.exe" +record=0 +names="" + +while [ $# -gt 0 ]; do + case "$1" in + --bin) + [ $# -ge 2 ] || { echo "mcp-golden: --bin needs an argument" >&2; exit 2; } + bin=$2; shift 2 ;; + --bin=*) + bin=${1#--bin=}; shift ;; + --record) + record=1; shift ;; + -h|--help) + echo "usage: mcp-golden [--bin PATH] [--record] [NAME...]"; exit 0 ;; + -*) + echo "mcp-golden: unknown option: $1" >&2; exit 2 ;; + *) + names="$names $1"; shift ;; + esac +done + +case "$bin" in + /*) ;; + *) bin=$(CDPATH= cd -- "$(dirname -- "$bin")" && pwd)/$(basename -- "$bin") ;; +esac + +if [ ! -x "$bin" ]; then + echo "mcp-golden: no such executable: $bin" >&2 + exit 2 +fi + +tests="$root/tests/mcp" +scripts="$tests/scripts" +expected="$tests/expected" + +if [ -z "$names" ]; then + names=$(cd "$scripts" && ls *.script 2>/dev/null | sed 's/\.script$//') +fi + +mkdir -p "$expected" + +tmp=$(mktemp -d "${TMPDIR:-/tmp}/mcp-golden.XXXXXX") || exit 2 +trap 'rm -rf "$tmp"' EXIT INT TERM + +nfail=0 +npass=0 + +for name in $names; do + script="$scripts/$name.script" + gold="$expected/$name.out" + + if [ ! -f "$script" ]; then + echo "FAIL $name (no such script: $script)" + nfail=$((nfail + 1)) + continue + fi + + want_exit=$(sed -n 's/^# *exit: *\([0-9][0-9]*\).*$/\1/p' "$script" | head -n 1) + if [ -z "$want_exit" ]; then + echo "FAIL $name (script has no '# exit: N' line)" + nfail=$((nfail + 1)) + continue + fi + + grep -v '^#' "$script" > "$tmp/rpc.in" + + (cd "$tests" && "$bin" mcp < "$tmp/rpc.in") \ + > "$tmp/raw" 2> "$tmp/err" + got_exit=$? + + sed 's/\("serverInfo":{"name":"easycrypt","version":\)"[^"]*"/\1"VERSION"/' \ + "$tmp/raw" > "$tmp/out" + + if [ "$record" = 1 ]; then + cp "$tmp/out" "$gold" + if [ "$got_exit" != "$want_exit" ]; then + echo "RECORD $name (exit $got_exit, script declares $want_exit)" + nfail=$((nfail + 1)) + else + echo "RECORD $name" + npass=$((npass + 1)) + fi + continue + fi + + ok=1 + + if [ ! -f "$gold" ]; then + echo "FAIL $name (no golden: $gold; re-run with --record)" + ok=0 + elif ! diff -u "$gold" "$tmp/out" > "$tmp/diff"; then + echo "FAIL $name (stdout differs)" + sed 's/^/ /' "$tmp/diff" + ok=0 + fi + + if [ "$got_exit" != "$want_exit" ]; then + echo "FAIL $name (exit $got_exit, expected $want_exit)" + ok=0 + fi + + if [ "$ok" = 1 ]; then + echo "PASS $name" + npass=$((npass + 1)) + else + nfail=$((nfail + 1)) + fi +done + +echo "----" +echo "$npass passed, $nfail failed" + +[ "$nfail" = 0 ] diff --git a/tests/mcp/README.md b/tests/mcp/README.md new file mode 100644 index 000000000..8f2b1a0b2 --- /dev/null +++ b/tests/mcp/README.md @@ -0,0 +1,124 @@ +# `easycrypt mcp` golden-output tests + +Byte-identity regression harness for the MCP server (`src/ecMcp.ml`). +Each scenario is a newline-delimited script of JSON-RPC messages fed to +`ec.exe mcp` on stdin; the raw protocol stream it writes on stdout, and +its process exit status, are compared against recorded goldens. + +This is the sibling of `../llm`, which does the same for the REPL. The +two front-ends share `src/ecLlmCore.ml`, so most behaviour changes show +up in both sets of goldens — that is the point. + +## Layout + +| Path | Contents | +|------|----------| +| `scripts/*.script` | the JSON-RPC messages piped into the server | +| `expected/*.out` | recorded stdout, one file per script | +| `../llm/fixtures/*` | the EasyCrypt files the scripts load (shared with the REPL harness, never duplicated) | +| `../../scripts/testing/mcp-golden` | the runner | + +## Running + +From the repository root: + +``` +make test-mcp # build + run every scenario +scripts/testing/mcp-golden # run every scenario +scripts/testing/mcp-golden happy-path protocol-errors +scripts/testing/mcp-golden --bin /path/to/ec.exe +``` + +The runner defaults to `_build/default/src/ec.exe`, resolved relative +to the repository root. It prints `PASS`/`FAIL` per scenario, a unified +diff for each mismatch, and exits nonzero if anything failed. That is +the CI invocation. + +## Re-recording + +``` +scripts/testing/mcp-golden --record # all scenarios +scripts/testing/mcp-golden --record tools-list # one scenario +``` + +`--record` overwrites `expected/*.out` with the current binary's +output instead of diffing. It still checks the declared exit status. + +Re-record only deliberately, and read the diff: these goldens are the +gate for changes to the protocol layer. + +## Scenarios + +| Scenario | What it pins | +|----------|--------------| +| `initialize` | the lifecycle handshake, the `initialized` notification, `ping` | +| `version-negotiation` | an unsupported revision falls back to the latest we speak; a supported one is echoed | +| `tools-list` | the whole tool table: names, descriptions, input/output schemas, annotations | +| `happy-path` | a session end to end: load, step, goals, tree, focus, commit | +| `prover-error` | EasyCrypt-level failures as `isError` results carrying the goal state | +| `try-revert` | `ec_try` rolling back a phrase that had already advanced the proof | +| `protocol-errors` | `-32700`, `-32600`, `-32601` and the `-32602` family | +| `revert` | `ec_revert` by uuid and by checkpoint name | +| `load-missing` | a missing file and an unknown extension: `isError`, *not* `-32602` | +| `notifications` | notifications, known and unknown, draw no reply | +| `exit` | `exit.` answers "session terminated", then the process stops | +| `eof` | end of input is a clean shutdown, exit 0 | + +## Expected exit status + +Each `.script` declares its expected process exit status on its first +line: + +``` +# exit: 0 +``` + +Lines starting with `#` are comment lines: the runner strips **all** of +them before piping the script into the server, so they can also be used +for prose. The first `# exit: N` line wins; a script without one fails. + +`easycrypt mcp` exits 0 on end of input and 0 after an `exit.` phrase; +EasyCrypt-level failures are `isError` results, not exit statuses, so +every scenario here declares `# exit: 0`. The field is kept all the +same, so that a future exit path cannot change silently. + +## Determinism rules + +The goldens are compared byte for byte, so scenarios must not leak +anything machine- or environment-dependent: + +* **Relative paths only.** The runner `cd`s into `tests/mcp` before + invoking the binary, and scripts refer to fixtures as + `"../llm/fixtures/simple.ec"`. Error messages echo the path + verbatim, so an absolute one would bake the developer's home + directory into the golden. +* **One normalization, and only one.** `serverInfo.version` is a + git-describe string; the runner rewrites it to `VERSION` with `sed` + before diffing. Nothing else is touched — if a second unstable field + ever appears, that is a bug in the server, not a reason to normalize + more. +* **No SMT.** As in `../llm`: proofs close with `trivial`, `done` or + `split`, never with `smt`, whose availability and timing vary by + machine. +* **stdout only.** stderr carries the engine's diagnostics (the server + points the process's stdout at stderr and keeps a private descriptor + for the protocol); it is discarded. +* Fixtures require `AllCore` only. + +## Reading a golden + +The stream is the protocol: one JSON message per line, unindented, +exactly as a client sees it. `tools-list.out` is therefore a single +very long line, and `diff` will show it whole. To read one by hand: + +``` +python3 -m json.tool < <(head -n 1 tests/mcp/expected/tools-list.out) +``` + +## Adding a scenario + +1. Add `scripts/NAME.script` starting with `# exit: N`. +2. Reuse a fixture from `../llm/fixtures/`; add a new one there (not + here) if none fits. +3. `scripts/testing/mcp-golden --record NAME`. +4. Read `expected/NAME.out` and check it is what you meant to freeze. diff --git a/tests/mcp/expected/eof.out b/tests/mcp/expected/eof.out new file mode 100644 index 000000000..8ddc98e50 --- /dev/null +++ b/tests/mcp/expected/eof.out @@ -0,0 +1 @@ +{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2025-11-25","capabilities":{"tools":{}},"serverInfo":{"name":"easycrypt","version":"VERSION"}}} diff --git a/tests/mcp/expected/exit.out b/tests/mcp/expected/exit.out new file mode 100644 index 000000000..2c9c64214 --- /dev/null +++ b/tests/mcp/expected/exit.out @@ -0,0 +1,2 @@ +{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2025-11-25","capabilities":{"tools":{}},"serverInfo":{"name":"easycrypt","version":"VERSION"}}} +{"jsonrpc":"2.0","id":2,"result":{"content":[{"type":"text","text":"session terminated"}],"structuredContent":{"uuid":0,"changed":false},"isError":false}} diff --git a/tests/mcp/expected/happy-path.out b/tests/mcp/expected/happy-path.out new file mode 100644 index 000000000..4ea16c67e --- /dev/null +++ b/tests/mcp/expected/happy-path.out @@ -0,0 +1,8 @@ +{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2025-11-25","capabilities":{"tools":{}},"serverInfo":{"name":"easycrypt","version":"VERSION"}}} +{"jsonrpc":"2.0","id":2,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"uuid":3,"changed":true},"isError":false}} +{"jsonrpc":"2.0","id":3,"result":{"content":[{"type":"text","text":"Current goal (remaining: 2)\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1\n"}],"structuredContent":{"uuid":4,"changed":true},"isError":false}} +{"jsonrpc":"2.0","id":4,"result":{"content":[{"type":"text","text":"Current goal (remaining: 2)\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1\n\n\n\n Goal #2\n ------------------------------------------------------------------------\n 2 = 2\n"}],"structuredContent":{"uuid":4,"changed":false},"isError":false}} +{"jsonrpc":"2.0","id":5,"result":{"content":[{"type":"text","text":"[1] 1 = 1 <- focused\n[2] 2 = 2\n"}],"structuredContent":{"uuid":4,"changed":false},"isError":false}} +{"jsonrpc":"2.0","id":6,"result":{"content":[{"type":"text","text":"Current goal (remaining: 2)\n\nType variables: \n\n------------------------------------------------------------------------\n2 = 2\n"}],"structuredContent":{"uuid":5,"changed":true},"isError":false}} +{"jsonrpc":"2.0","id":7,"result":{"content":[{"type":"text","text":"No more goals\n"}],"structuredContent":{"uuid":7,"changed":true},"isError":false}} +{"jsonrpc":"2.0","id":8,"result":{"content":[{"type":"text","text":"split.\n- trivial.\n- trivial.\n"}],"structuredContent":{"uuid":7,"changed":false},"isError":false}} diff --git a/tests/mcp/expected/initialize.out b/tests/mcp/expected/initialize.out new file mode 100644 index 000000000..1f7c0553c --- /dev/null +++ b/tests/mcp/expected/initialize.out @@ -0,0 +1,2 @@ +{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2025-11-25","capabilities":{"tools":{}},"serverInfo":{"name":"easycrypt","version":"VERSION"}}} +{"jsonrpc":"2.0","id":2,"result":{}} diff --git a/tests/mcp/expected/load-missing.out b/tests/mcp/expected/load-missing.out new file mode 100644 index 000000000..021b5bf4a --- /dev/null +++ b/tests/mcp/expected/load-missing.out @@ -0,0 +1,3 @@ +{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2025-11-25","capabilities":{"tools":{}},"serverInfo":{"name":"easycrypt","version":"VERSION"}}} +{"jsonrpc":"2.0","id":2,"result":{"content":[{"type":"text","text":"LOAD: no such file: ../llm/fixtures/nosuchfile.ec"}],"structuredContent":{"uuid":0,"changed":false},"isError":true}} +{"jsonrpc":"2.0","id":3,"result":{"content":[{"type":"text","text":"unknown file extension: .txt\nNo active proof.\n"}],"structuredContent":{"uuid":0,"changed":false},"isError":true}} diff --git a/tests/mcp/expected/notifications.out b/tests/mcp/expected/notifications.out new file mode 100644 index 000000000..c71fd037e --- /dev/null +++ b/tests/mcp/expected/notifications.out @@ -0,0 +1,2 @@ +{"jsonrpc":"2.0","id":1,"result":{}} +{"jsonrpc":"2.0","id":2,"result":{}} diff --git a/tests/mcp/expected/protocol-errors.out b/tests/mcp/expected/protocol-errors.out new file mode 100644 index 000000000..d920f69df --- /dev/null +++ b/tests/mcp/expected/protocol-errors.out @@ -0,0 +1,16 @@ +{"jsonrpc":"2.0","id":null,"error":{"code":-32700,"message":"invalid JSON"}} +{"jsonrpc":"2.0","id":null,"error":{"code":-32600,"message":"JSON-RPC batches are not supported by this protocol revision"}} +{"jsonrpc":"2.0","id":2,"error":{"code":-32601,"message":"method not found: server/discover"}} +{"jsonrpc":"2.0","id":3,"error":{"code":-32600,"message":"missing `method'"}} +{"jsonrpc":"2.0","id":4,"error":{"code":-32600,"message":"`method' must be a string"}} +{"jsonrpc":"2.0","id":null,"error":{"code":-32600,"message":"a JSON-RPC message must be an object"}} +{"jsonrpc":"2.0","id":5,"error":{"code":-32602,"message":"unknown tool: ec_nosuchtool"}} +{"jsonrpc":"2.0","id":6,"error":{"code":-32602,"message":"missing tool `name'"}} +{"jsonrpc":"2.0","id":7,"error":{"code":-32602,"message":"`name' must be a string"}} +{"jsonrpc":"2.0","id":8,"error":{"code":-32602,"message":"ec_step: missing required argument `phrase'"}} +{"jsonrpc":"2.0","id":9,"error":{"code":-32602,"message":"ec_goals: `all' must be a boolean"}} +{"jsonrpc":"2.0","id":10,"error":{"code":-32602,"message":"ec_load: `col' requires `line'"}} +{"jsonrpc":"2.0","id":11,"error":{"code":-32602,"message":"ec_load: `line' must be an integer"}} +{"jsonrpc":"2.0","id":12,"error":{"code":-32602,"message":"ec_focus: not a path of integers: 1.oops"}} +{"jsonrpc":"2.0","id":13,"error":{"code":-32602,"message":"ec_focus: path indices must be >= 1: 0"}} +{"jsonrpc":"2.0","id":14,"error":{"code":-32602,"message":"`params' must be an object"}} diff --git a/tests/mcp/expected/prover-error.out b/tests/mcp/expected/prover-error.out new file mode 100644 index 000000000..337f01463 --- /dev/null +++ b/tests/mcp/expected/prover-error.out @@ -0,0 +1,6 @@ +{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2025-11-25","capabilities":{"tools":{}},"serverInfo":{"name":"easycrypt","version":"VERSION"}}} +{"jsonrpc":"2.0","id":2,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"uuid":3,"changed":true},"isError":false}} +{"jsonrpc":"2.0","id":3,"result":{"content":[{"type":"text","text":": line 1 (0-18): unknown lemma `nosuchlemma'\nsource: apply nosuchlemma.\nCurrent goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"uuid":3,"changed":false},"isError":true}} +{"jsonrpc":"2.0","id":4,"result":{"content":[{"type":"text","text":"FOCUS: index 7 out of range (1..1)\nCurrent goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"uuid":3,"changed":false},"isError":true}} +{"jsonrpc":"2.0","id":5,"result":{"content":[{"type":"text","text":"No active proof.\n"}],"structuredContent":{"uuid":0,"changed":true},"isError":false}} +{"jsonrpc":"2.0","id":6,"result":{"content":[{"type":"text","text":"nothing to undo\nNo active proof.\n"}],"structuredContent":{"uuid":0,"changed":false},"isError":true}} diff --git a/tests/mcp/expected/revert.out b/tests/mcp/expected/revert.out new file mode 100644 index 000000000..e62225ec1 --- /dev/null +++ b/tests/mcp/expected/revert.out @@ -0,0 +1,11 @@ +{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2025-11-25","capabilities":{"tools":{}},"serverInfo":{"name":"easycrypt","version":"VERSION"}}} +{"jsonrpc":"2.0","id":2,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"uuid":3,"changed":true},"isError":false}} +{"jsonrpc":"2.0","id":3,"result":{"content":[{"type":"text","text":"checkpoint 'start' set at uuid 3"}],"structuredContent":{"uuid":3,"changed":false},"isError":false}} +{"jsonrpc":"2.0","id":4,"result":{"content":[{"type":"text","text":"Current goal (remaining: 2)\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1\n"}],"structuredContent":{"uuid":4,"changed":true},"isError":false}} +{"jsonrpc":"2.0","id":5,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"uuid":3,"changed":true},"isError":false}} +{"jsonrpc":"2.0","id":6,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"uuid":3,"changed":false},"isError":false}} +{"jsonrpc":"2.0","id":7,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n2 = 2\n"}],"structuredContent":{"uuid":5,"changed":true},"isError":false}} +{"jsonrpc":"2.0","id":8,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"uuid":3,"changed":true},"isError":false}} +{"jsonrpc":"2.0","id":9,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"uuid":3,"changed":false},"isError":false}} +{"jsonrpc":"2.0","id":10,"result":{"content":[{"type":"text","text":""}],"structuredContent":{"uuid":3,"changed":false},"isError":false}} +{"jsonrpc":"2.0","id":11,"result":{"content":[{"type":"text","text":"REVERT: 'nosuchname' is not a valid uuid or checkpoint name\nCurrent goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"uuid":3,"changed":false},"isError":true}} diff --git a/tests/mcp/expected/tools-list.out b/tests/mcp/expected/tools-list.out new file mode 100644 index 000000000..9de3be4a9 --- /dev/null +++ b/tests/mcp/expected/tools-list.out @@ -0,0 +1 @@ +{"jsonrpc":"2.0","id":1,"result":{"tools":[{"name":"ec_load","description":"Reset the session and compile FILE from the top, stopping after the last sentence that ends on or before LINE (and column COL when given). This is the entry point: every other tool needs a loaded file, and tactics need the position to land inside a proof. Set nosmt to weaken SMT calls while replaying a prefix that was already verified, which is much faster on large files. Set trace to have the reply describe the last loaded sentence as BEFORE / TACTIC / AFTER / SUMMARY blocks. The reply reports where compilation stopped and the resulting goal state; note the uuid it returns, reverting to it is the instant way back to the start of the proof.","inputSchema":{"type":"object","properties":{"file":{"type":"string","description":"path to the .ec/.eca file"},"line":{"type":"integer","description":"stop after the last sentence ending on or before this line; omit to compile the whole file"},"col":{"type":"integer","description":"column bound within `line'; requires `line'"},"nosmt":{"type":"boolean","description":"weaken SMT calls while compiling the prefix","default":false},"trace":{"type":"boolean","description":"report the proof state around the last loaded sentence","default":false}},"required":["file"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["uuid","changed"]},"annotations":{"destructiveHint":true}},{"name":"ec_step","description":"Run EasyCrypt sentences -- tactics, declarations, require, print, ... -- against the current session. Every complete sentence in the argument is executed, in order, exactly as if the text had been appended to the source file, and a single reply describes the state they leave behind; sentences may span several lines. Requires a file loaded with ec_load, and, for tactics, an open proof. On success the reply carries the new goal state; on failure the prover's error text comes back with isError set, the sentences before the failing one stay applied and the engine is left wherever that sentence left it -- use ec_try when you want a guaranteed rollback. Successful non-query phrases are recorded for ec_commit.","inputSchema":{"type":"object","properties":{"phrase":{"type":"string","description":"one or more complete EasyCrypt sentences, each ending with `.'"}},"required":["phrase"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["uuid","changed"]},"annotations":{"destructiveHint":false,"idempotentHint":false}},{"name":"ec_try","description":"Like ec_step, but the engine is rolled back to the state it had before the call whenever a sentence fails, including input that failed only after having already advanced the proof. The failure reply sets structuredContent.reverted to true, and its uuid and goal text describe the restored state, not the point of failure. Use this to probe a tactic without having to ec_revert afterwards; use ec_step when you mean to keep whatever progress the phrase makes. A successful phrase behaves exactly as under ec_step and is recorded for ec_commit.","inputSchema":{"type":"object","properties":{"phrase":{"type":"string","description":"one complete EasyCrypt sentence, ending with `.'"}},"required":["phrase"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"},"reverted":{"type":"boolean","description":"set when the phrase failed and the engine was rolled back to its pre-call state"}},"required":["uuid","changed"]},"annotations":{"destructiveHint":false}},{"name":"ec_goals","description":"Print the current proof state: the focused subgoal alone, or, with all set, every open subgoal. Requires an open proof, and does not advance the engine.","inputSchema":{"type":"object","properties":{"all":{"type":"boolean","description":"print every open subgoal instead of the focused one","default":false}},"additionalProperties":false},"outputSchema":{"type":"object","properties":{"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["uuid","changed"]},"annotations":{"readOnlyHint":true}},{"name":"ec_tree","description":"List the open subgoals as a tree of dotted-path labels -- [1], [1.2], [2.1.1] -- showing how the splits nest, and marking the focused one. Those labels are exactly what ec_focus accepts. Set full for whole goal bodies rather than one-line conclusions. The labels are not stable across focus changes: the tree always shows the focused goal first, so re-read it after every ec_focus. Does not advance the engine.","inputSchema":{"type":"object","properties":{"full":{"type":"boolean","description":"print full goal bodies instead of one-line conclusions","default":false}},"additionalProperties":false},"outputSchema":{"type":"object","properties":{"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["uuid","changed"]},"annotations":{"readOnlyHint":true}},{"name":"ec_focus","description":"Rotate the focus onto the subgoal at dotted path PATH, as printed by ec_tree (\"2\", \"1.2\", \"1.1.1\"); a single integer selects the k-th goal of the flat listing, and the special value \"next\" moves to the next open subgoal. Subsequent tactics act on the focused goal. Selecting an internal frame instead of a leaf goal is an error.","inputSchema":{"type":"object","properties":{"path":{"type":"string","description":"\"N\", a dotted path \"N1.N2...\", or \"next\""}},"required":["path"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["uuid","changed"]},"annotations":{"destructiveHint":false}},{"name":"ec_undo","description":"Undo the last engine step, returning to the immediately preceding state. The ec_commit transcript is trimmed to match. Fails when there is nothing left to undo.","inputSchema":{"type":"object","properties":{},"additionalProperties":false},"outputSchema":{"type":"object","properties":{"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["uuid","changed"]},"annotations":{"destructiveHint":false}},{"name":"ec_revert","description":"Return the session to an earlier state, named either by a uuid reported in some previous structuredContent or by a name given to ec_checkpoint. Reverting is instant, unlike re-running ec_load, so going back to the uuid ec_load returned is the cheap way to restart a proof from scratch after a failed experiment. The ec_commit transcript is trimmed to match.","inputSchema":{"type":"object","properties":{"target":{"type":"string","description":"a uuid (as a decimal string) or a checkpoint name"}},"required":["target"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["uuid","changed"]},"annotations":{"destructiveHint":true}},{"name":"ec_checkpoint","description":"Record the current uuid under NAME, so that ec_revert can address it by name later. Worth doing before a branching experiment, when carrying the bare uuid around is awkward. Does not change the proof state.","inputSchema":{"type":"object","properties":{"name":{"type":"string","description":"checkpoint name"}},"required":["name"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["uuid","changed"]},"annotations":{"readOnlyHint":true}},{"name":"ec_commit","description":"Emit the phrases recorded since the last ec_load as a proof body, with bullets inserted at every multi-child split: the result compiles under `pragma +strict_bullets' and can be pasted straight into the source file. Queries (search, print, locate, ec_search) are never recorded, so looking things up mid-proof does not pollute the body, and ec_undo / ec_revert trim the transcript. Still works after `qed.'. Does not change the proof state.","inputSchema":{"type":"object","properties":{},"additionalProperties":false},"outputSchema":{"type":"object","properties":{"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["uuid","changed"]},"annotations":{"readOnlyHint":true}},{"name":"ec_search","description":"Search the environment for lemmas matching an EasyCrypt search pattern. This is pattern syntax, not keyword search: use _ as the wildcard, as in \"(fdom _)\", \"(_ %/ _)\" or \"(mu _ _) (_ <= _)\". Requires a loaded file. The query neither advances the proof nor enters the ec_commit transcript.","inputSchema":{"type":"object","properties":{"pattern":{"type":"string","description":"an EasyCrypt search pattern"}},"required":["pattern"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["uuid","changed"]},"annotations":{"readOnlyHint":true}}]}} diff --git a/tests/mcp/expected/try-revert.out b/tests/mcp/expected/try-revert.out new file mode 100644 index 000000000..b2f298025 --- /dev/null +++ b/tests/mcp/expected/try-revert.out @@ -0,0 +1,7 @@ +{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2025-11-25","capabilities":{"tools":{}},"serverInfo":{"name":"easycrypt","version":"VERSION"}}} +{"jsonrpc":"2.0","id":2,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"uuid":3,"changed":true},"isError":false}} +{"jsonrpc":"2.0","id":3,"result":{"content":[{"type":"text","text":": line 1 (7-25): unknown lemma `nosuchlemma'\nsource: apply nosuchlemma.\nCurrent goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"uuid":3,"changed":false,"reverted":true},"isError":true}} +{"jsonrpc":"2.0","id":4,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"uuid":3,"changed":false},"isError":false}} +{"jsonrpc":"2.0","id":5,"result":{"content":[{"type":"text","text":""}],"structuredContent":{"uuid":3,"changed":false},"isError":false}} +{"jsonrpc":"2.0","id":6,"result":{"content":[{"type":"text","text":"Current goal (remaining: 2)\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1\n"}],"structuredContent":{"uuid":4,"changed":true},"isError":false}} +{"jsonrpc":"2.0","id":7,"result":{"content":[{"type":"text","text":"Current goal (remaining: 2)\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1\n"}],"structuredContent":{"uuid":4,"changed":false},"isError":false}} diff --git a/tests/mcp/expected/version-negotiation.out b/tests/mcp/expected/version-negotiation.out new file mode 100644 index 000000000..6e0aa6aa9 --- /dev/null +++ b/tests/mcp/expected/version-negotiation.out @@ -0,0 +1,3 @@ +{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2025-11-25","capabilities":{"tools":{}},"serverInfo":{"name":"easycrypt","version":"VERSION"}}} +{"jsonrpc":"2.0","id":2,"result":{"protocolVersion":"2025-06-18","capabilities":{"tools":{}},"serverInfo":{"name":"easycrypt","version":"VERSION"}}} +{"jsonrpc":"2.0","id":3,"result":{"protocolVersion":"2025-11-25","capabilities":{"tools":{}},"serverInfo":{"name":"easycrypt","version":"VERSION"}}} diff --git a/tests/mcp/scripts/eof.script b/tests/mcp/scripts/eof.script new file mode 100644 index 000000000..b50d488d2 --- /dev/null +++ b/tests/mcp/scripts/eof.script @@ -0,0 +1,3 @@ +# exit: 0 +# End of input is a clean shutdown, exit 0, with no farewell message. +{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{}}} diff --git a/tests/mcp/scripts/exit.script b/tests/mcp/scripts/exit.script new file mode 100644 index 000000000..cbec912ad --- /dev/null +++ b/tests/mcp/scripts/exit.script @@ -0,0 +1,7 @@ +# exit: 0 +# `exit.' ends the session: the response still goes out, then the +# process stops. The ping after it is never read, so it must not +# appear in the golden. +{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{}}} +{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"ec_step","arguments":{"phrase":"exit."}}} +{"jsonrpc":"2.0","id":3,"method":"ping"} diff --git a/tests/mcp/scripts/happy-path.script b/tests/mcp/scripts/happy-path.script new file mode 100644 index 000000000..c5961b668 --- /dev/null +++ b/tests/mcp/scripts/happy-path.script @@ -0,0 +1,13 @@ +# exit: 0 +# A whole session over MCP: load a file up to its `proof.', split, +# inspect the goals and the tree, close both branches, and read the +# proof body back out of ec_commit. +{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{}}} +{"jsonrpc":"2.0","method":"notifications/initialized"} +{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"ec_load","arguments":{"file":"../llm/fixtures/simple.ec","line":6}}} +{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"ec_step","arguments":{"phrase":"split."}}} +{"jsonrpc":"2.0","id":4,"method":"tools/call","params":{"name":"ec_goals","arguments":{"all":true}}} +{"jsonrpc":"2.0","id":5,"method":"tools/call","params":{"name":"ec_tree","arguments":{}}} +{"jsonrpc":"2.0","id":6,"method":"tools/call","params":{"name":"ec_focus","arguments":{"path":"2"}}} +{"jsonrpc":"2.0","id":7,"method":"tools/call","params":{"name":"ec_step","arguments":{"phrase":"trivial. trivial."}}} +{"jsonrpc":"2.0","id":8,"method":"tools/call","params":{"name":"ec_commit","arguments":{}}} diff --git a/tests/mcp/scripts/initialize.script b/tests/mcp/scripts/initialize.script new file mode 100644 index 000000000..ea3ac3291 --- /dev/null +++ b/tests/mcp/scripts/initialize.script @@ -0,0 +1,6 @@ +# exit: 0 +# The lifecycle handshake: initialize, the client's initialized +# notification (no reply), then a ping. +{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{},"clientInfo":{"name":"golden","version":"0"}}} +{"jsonrpc":"2.0","method":"notifications/initialized"} +{"jsonrpc":"2.0","id":2,"method":"ping"} diff --git a/tests/mcp/scripts/load-missing.script b/tests/mcp/scripts/load-missing.script new file mode 100644 index 000000000..e7aa4ef5a --- /dev/null +++ b/tests/mcp/scripts/load-missing.script @@ -0,0 +1,8 @@ +# exit: 0 +# A file the tool cannot find is an EasyCrypt-level failure, so it +# comes back as an isError result and NOT as a -32602: the arguments +# satisfy the schema, it is the world that does not. Same for a file +# whose extension the loader does not know. +{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{}}} +{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"ec_load","arguments":{"file":"../llm/fixtures/nosuchfile.ec"}}} +{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"ec_load","arguments":{"file":"../llm/fixtures/notec.txt"}}} diff --git a/tests/mcp/scripts/notifications.script b/tests/mcp/scripts/notifications.script new file mode 100644 index 000000000..9eaadab7e --- /dev/null +++ b/tests/mcp/scripts/notifications.script @@ -0,0 +1,13 @@ +# exit: 0 +# Notifications never draw a reply, whatever they are: the three the +# spec has us tolerate, an unknown one, and a message whose id is +# null (which MCP forbids, so we read it as "no id" and stay silent). +# The two pings bracket them, so the golden shows nothing in between. +{"jsonrpc":"2.0","id":1,"method":"ping"} +{"jsonrpc":"2.0","method":"notifications/initialized"} +{"jsonrpc":"2.0","method":"notifications/cancelled","params":{"requestId":1}} +{"jsonrpc":"2.0","method":"notifications/roots/list_changed"} +{"jsonrpc":"2.0","method":"notifications/nobody/knows"} +{"jsonrpc":"2.0","id":null,"method":"ping"} + +{"jsonrpc":"2.0","id":2,"method":"ping"} diff --git a/tests/mcp/scripts/protocol-errors.script b/tests/mcp/scripts/protocol-errors.script new file mode 100644 index 000000000..4d18d3ba6 --- /dev/null +++ b/tests/mcp/scripts/protocol-errors.script @@ -0,0 +1,22 @@ +# exit: 0 +# Protocol-level failures, which are JSON-RPC errors and never +# isError results: malformed JSON (-32700), a batch array and a +# malformed envelope (-32600), an unknown method (-32601), and the +# -32602 family -- unknown tool, missing and ill-typed arguments, +# `col' without `line', and an unparsable ec_focus path. +{not json at all +[{"jsonrpc":"2.0","id":1,"method":"ping"}] +{"jsonrpc":"2.0","id":2,"method":"server/discover"} +{"jsonrpc":"2.0","id":3} +{"jsonrpc":"2.0","id":4,"method":42} +"just a string" +{"jsonrpc":"2.0","id":5,"method":"tools/call","params":{"name":"ec_nosuchtool","arguments":{}}} +{"jsonrpc":"2.0","id":6,"method":"tools/call","params":{"arguments":{}}} +{"jsonrpc":"2.0","id":7,"method":"tools/call","params":{"name":17,"arguments":{}}} +{"jsonrpc":"2.0","id":8,"method":"tools/call","params":{"name":"ec_step","arguments":{}}} +{"jsonrpc":"2.0","id":9,"method":"tools/call","params":{"name":"ec_goals","arguments":{"all":"yes"}}} +{"jsonrpc":"2.0","id":10,"method":"tools/call","params":{"name":"ec_load","arguments":{"file":"../llm/fixtures/simple.ec","col":3}}} +{"jsonrpc":"2.0","id":11,"method":"tools/call","params":{"name":"ec_load","arguments":{"file":"../llm/fixtures/simple.ec","line":"6"}}} +{"jsonrpc":"2.0","id":12,"method":"tools/call","params":{"name":"ec_focus","arguments":{"path":"1.oops"}}} +{"jsonrpc":"2.0","id":13,"method":"tools/call","params":{"name":"ec_focus","arguments":{"path":"0"}}} +{"jsonrpc":"2.0","id":14,"method":"tools/call","params":"not an object"} diff --git a/tests/mcp/scripts/prover-error.script b/tests/mcp/scripts/prover-error.script new file mode 100644 index 000000000..a599dfea9 --- /dev/null +++ b/tests/mcp/scripts/prover-error.script @@ -0,0 +1,11 @@ +# exit: 0 +# An EasyCrypt-level failure is data, not a protocol error: a +# successful response carrying isError, the prover's message and the +# goal state at the point of failure. ec_undo then reports there is +# nothing left to undo, which is the same kind of failure. +{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{}}} +{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"ec_load","arguments":{"file":"../llm/fixtures/simple.ec","line":6}}} +{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"ec_step","arguments":{"phrase":"apply nosuchlemma."}}} +{"jsonrpc":"2.0","id":4,"method":"tools/call","params":{"name":"ec_focus","arguments":{"path":"7"}}} +{"jsonrpc":"2.0","id":5,"method":"tools/call","params":{"name":"ec_revert","arguments":{"target":"0"}}} +{"jsonrpc":"2.0","id":6,"method":"tools/call","params":{"name":"ec_undo","arguments":{}}} diff --git a/tests/mcp/scripts/revert.script b/tests/mcp/scripts/revert.script new file mode 100644 index 000000000..2f37151c0 --- /dev/null +++ b/tests/mcp/scripts/revert.script @@ -0,0 +1,15 @@ +# exit: 0 +# ec_revert addressed both ways: by the uuid ec_load reported, and by +# a name given to ec_checkpoint. Each revert is followed by ec_goals, +# so the golden records where the session actually landed. +{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{}}} +{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"ec_load","arguments":{"file":"../llm/fixtures/simple.ec","line":6}}} +{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"ec_checkpoint","arguments":{"name":"start"}}} +{"jsonrpc":"2.0","id":4,"method":"tools/call","params":{"name":"ec_step","arguments":{"phrase":"split."}}} +{"jsonrpc":"2.0","id":5,"method":"tools/call","params":{"name":"ec_revert","arguments":{"target":"3"}}} +{"jsonrpc":"2.0","id":6,"method":"tools/call","params":{"name":"ec_goals","arguments":{}}} +{"jsonrpc":"2.0","id":7,"method":"tools/call","params":{"name":"ec_step","arguments":{"phrase":"split. trivial."}}} +{"jsonrpc":"2.0","id":8,"method":"tools/call","params":{"name":"ec_revert","arguments":{"target":"start"}}} +{"jsonrpc":"2.0","id":9,"method":"tools/call","params":{"name":"ec_goals","arguments":{}}} +{"jsonrpc":"2.0","id":10,"method":"tools/call","params":{"name":"ec_commit","arguments":{}}} +{"jsonrpc":"2.0","id":11,"method":"tools/call","params":{"name":"ec_revert","arguments":{"target":"nosuchname"}}} diff --git a/tests/mcp/scripts/tools-list.script b/tests/mcp/scripts/tools-list.script new file mode 100644 index 000000000..ea2b8ad25 --- /dev/null +++ b/tests/mcp/scripts/tools-list.script @@ -0,0 +1,5 @@ +# exit: 0 +# The tool declarations, verbatim. This golden pins the agent-facing +# contract: names, descriptions, input schemas, output schemas and +# annotations. Editing any tool description re-records this file. +{"jsonrpc":"2.0","id":1,"method":"tools/list"} diff --git a/tests/mcp/scripts/try-revert.script b/tests/mcp/scripts/try-revert.script new file mode 100644 index 000000000..2d4e7e3f4 --- /dev/null +++ b/tests/mcp/scripts/try-revert.script @@ -0,0 +1,12 @@ +# exit: 0 +# ec_try rolls back. The phrase advances the proof (`split.') before +# failing, so the rollback has real work to do: the failure reply +# reports reverted true and changed false, and the ec_goals that +# follows proves the pre-call goal is back. +{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{}}} +{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"ec_load","arguments":{"file":"../llm/fixtures/simple.ec","line":6}}} +{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"ec_try","arguments":{"phrase":"split. apply nosuchlemma."}}} +{"jsonrpc":"2.0","id":4,"method":"tools/call","params":{"name":"ec_goals","arguments":{}}} +{"jsonrpc":"2.0","id":5,"method":"tools/call","params":{"name":"ec_commit","arguments":{}}} +{"jsonrpc":"2.0","id":6,"method":"tools/call","params":{"name":"ec_try","arguments":{"phrase":"split."}}} +{"jsonrpc":"2.0","id":7,"method":"tools/call","params":{"name":"ec_goals","arguments":{}}} diff --git a/tests/mcp/scripts/version-negotiation.script b/tests/mcp/scripts/version-negotiation.script new file mode 100644 index 000000000..f638af715 --- /dev/null +++ b/tests/mcp/scripts/version-negotiation.script @@ -0,0 +1,7 @@ +# exit: 0 +# Version negotiation. An unsupported revision gets the latest one we +# speak; a supported older revision is echoed back; a missing or +# non-string protocolVersion also falls back to the latest. +{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2099-01-01","capabilities":{},"clientInfo":{"name":"golden","version":"0"}}} +{"jsonrpc":"2.0","id":2,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{}}} +{"jsonrpc":"2.0","id":3,"method":"initialize","params":{"capabilities":{}}} From a4ad7be5d9595ede598e04cad9e48470c3530815 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Fri, 21 Aug 2026 15:43:00 +0200 Subject: [PATCH 30/51] [llm] parity check: two front-ends, one core The goldens freeze what each front-end answers; nothing yet pinned that they answer the *same thing*. `scripts/testing/mcp-parity', run by `make test-mcp' after the goldens, plays one representative operation per tool family -- load, step, goals, tree, focus, undo, checkpoint, step, revert, search, commit, and a failing phrase -- against two sessions started from the same directory on the same fixture, one driven with `llm -eval' and one with a JSON-RPC script, and asserts for each step that * the REPL's [uuid:N] envelope tag equals the MCP result's structuredContent.uuid, and * the REPL's reply body -- what it prints between the OK/ERROR line and -- equals the MCP result's content[0].text. The body comparison is exact up to one trailing newline, which is the only licensed difference: the REPL terminates a body that lacks one so that starts a line of its own, and MCP, having no sentinel, does not. Both wires are parsed structurally (blocks closed by a lone ; one JSON object per line) rather than pattern-matched, so a body containing something envelope-shaped cannot fool the checker. The search step is the interesting one: its payload is notices (the lemma listing, which arrives through the notifier) followed by the goal body, so it pins the notices/body join the two front-ends implement separately. Two asymmetries are structural and are documented in tests/mcp/README.md rather than papered over: the REPL's [loaded:...] / [focus: 1/N] tags ride on the status line and have no MCP counterpart (structuredContent carries uuid and changed only), and the REPL has never rendered notices on an ERROR reply while the MCP failure result does -- so the two agree on failures only when nothing was emitted, which holds for the phrase the check plays. Verified the checker bites: pointing the tree step at ec_tree {"full":true} while the REPL still says TREE makes it FAIL with both bodies printed. Gate: make test-llm (21) and make test-mcp (12 goldens + 12 parity) green, twice in a row. --- Makefile | 3 + scripts/testing/mcp-parity | 189 +++++++++++++++++++++++++++++++++++++ tests/mcp/README.md | 52 ++++++++++ 3 files changed, 244 insertions(+) create mode 100755 scripts/testing/mcp-parity diff --git a/Makefile b/Makefile index 0fdb6d262..e0b8bb5e2 100644 --- a/Makefile +++ b/Makefile @@ -17,6 +17,8 @@ LLMCHECK := scripts/testing/llm-golden LLMCHECK += --bin=./ec.native MCPCHECK := scripts/testing/mcp-golden MCPCHECK += --bin=./ec.native +MCPPARITY := scripts/testing/mcp-parity +MCPPARITY += --bin=./ec.native NIX ?= nix --extra-experimental-features "nix-command flakes" PROFILE ?= dev @@ -61,6 +63,7 @@ test-llm: build test-mcp: build $(MCPCHECK) + $(MCPPARITY) check: unit stdlib examples @true diff --git a/scripts/testing/mcp-parity b/scripts/testing/mcp-parity new file mode 100755 index 000000000..3901f637b --- /dev/null +++ b/scripts/testing/mcp-parity @@ -0,0 +1,189 @@ +#! /usr/bin/env python3 + +# -------------------------------------------------------------------- +# Parity check: `easycrypt llm' and `easycrypt mcp' are two front-ends +# over one core, so the same operation must produce the same answer on +# both wires. +# +# mcp-parity [--bin PATH] [-v] +# +# One representative operation per tool family is played, in order, +# against two sessions -- a REPL one driven with `llm -eval', an MCP +# one driven with a JSON-RPC script -- started from the same working +# directory on the same fixture. For each step the checker asserts: +# +# * the engine uuid matches: the REPL's `[uuid:N]' envelope tag +# against the MCP result's structuredContent.uuid; +# * the payload matches: the REPL's reply body (what it prints +# between the OK/ERROR line and `') against the MCP result's +# content[0].text. +# +# The payload comparison is up to one trailing newline, which the REPL +# appends to a body that lacks one so that `' starts a line of its +# own. That is the only licensed difference; see tests/mcp/README.md +# for the two structural asymmetries this check deliberately does not +# span (envelope tags, and notices on failures). +# -------------------------------------------------------------------- + +import json +import os +import subprocess +import sys + +# -------------------------------------------------------------------- +# The operations, as (label, REPL line, MCP tool name, MCP arguments). +# One per tool family, plus a failing phrase, played in this order +# against both sessions. + +STEPS = [ + ("load", 'LOAD "fixtures/simple.ec" 6', + "ec_load", {"file": "fixtures/simple.ec", "line": 6}), + ("step", 'split.', + "ec_step", {"phrase": "split."}), + ("goals", 'GOALS ALL', + "ec_goals", {"all": True}), + ("tree", 'TREE', + "ec_tree", {}), + ("focus", 'FOCUS 2', + "ec_focus", {"path": "2"}), + ("undo", 'UNDO', + "ec_undo", {}), + ("checkpoint", 'CHECKPOINT c0', + "ec_checkpoint", {"name": "c0"}), + ("step/2", 'trivial.', + "ec_step", {"phrase": "trivial."}), + ("revert", 'REVERT c0', + "ec_revert", {"target": "c0"}), + ("search", 'SEARCH (b2i _)', + "ec_search", {"pattern": "(b2i _)"}), + ("commit", 'COMMIT', + "ec_commit", {}), + ("failure", 'apply nosuchlemma.', + "ec_step", {"phrase": "apply nosuchlemma."}), +] + + +# -------------------------------------------------------------------- +def repl_replies(binary, cwd): + """Run the REPL script and return one (uuid, body) per reply. + + The REPL wire is a sequence of blocks, each opened by an + `OK [uuid:N]' or `ERROR [uuid:N]' line and closed by a lone + `'. The opening READY block is dropped.""" + + script = "\n".join(line for (_, line, _, _) in STEPS) + out = subprocess.run( + [binary, "llm", "-eval", script], + cwd=cwd, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, + ).stdout.decode() + + replies, head, body = [], None, [] + for line in out.split("\n")[:-1]: + if head is None: + head = line + elif line == "": + replies.append((head, "".join(l + "\n" for l in body))) + head, body = None, [] + else: + body.append(line) + + def uuid_of(head): + return int(head.split("[uuid:")[1].split("]")[0]) + + return [(uuid_of(h), b) for (h, b) in replies][1:] + + +# -------------------------------------------------------------------- +def mcp_results(binary, cwd): + """Run the MCP script and return one (uuid, text) per tools/call.""" + + script = "".join( + json.dumps({ + "jsonrpc": "2.0", "id": i + 1, "method": "tools/call", + "params": {"name": tool, "arguments": args}, + }) + "\n" + for (i, (_, _, tool, args)) in enumerate(STEPS) + ) + out = subprocess.run( + [binary, "mcp"], input=script.encode(), + cwd=cwd, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, + ).stdout.decode() + + results = [] + for line in out.splitlines(): + result = json.loads(line)["result"] + results.append((result["structuredContent"]["uuid"], + result["content"][0]["text"])) + return results + + +# -------------------------------------------------------------------- +def main(): + root = os.path.dirname(os.path.dirname(os.path.dirname( + os.path.abspath(__file__)))) + binary = os.path.join(root, "_build", "default", "src", "ec.exe") + verbose = False + + args = sys.argv[1:] + while args: + if args[0] == "--bin": + binary, args = args[1], args[2:] + elif args[0].startswith("--bin="): + binary, args = args[0][len("--bin="):], args[1:] + elif args[0] in ("-v", "--verbose"): + verbose, args = True, args[1:] + elif args[0] in ("-h", "--help"): + print("usage: mcp-parity [--bin PATH] [-v]") + return 0 + else: + print(f"mcp-parity: unknown option: {args[0]}", file=sys.stderr) + return 2 + + binary = os.path.abspath(binary) + if not os.access(binary, os.X_OK): + print(f"mcp-parity: no such executable: {binary}", file=sys.stderr) + return 2 + + # Both sessions run from tests/llm, so they name the fixture the + # same way and no path difference can leak into a reply. + cwd = os.path.join(root, "tests", "llm") + + repl = repl_replies(binary, cwd) + mcp = mcp_results(binary, cwd) + + nfail = 0 + + if len(repl) != len(STEPS) or len(mcp) != len(STEPS): + print(f"FAIL (reply count: {len(STEPS)} steps, " + f"{len(repl)} REPL replies, {len(mcp)} MCP results)") + return 1 + + for ((label, line, tool, _), (ruuid, body), (muuid, text)) in \ + zip(STEPS, repl, mcp): + # The REPL ends a non-empty body with a newline so that `' + # starts a line; MCP has no sentinel and so does not. + normalized = text if text.endswith("\n") or text == "" else text + "\n" + + problems = [] + if ruuid != muuid: + problems.append(f"uuid: REPL {ruuid}, MCP {muuid}") + if normalized != body: + problems.append(f"body:\n REPL {body!r}\n MCP {normalized!r}") + + if problems: + print(f"FAIL {label} ({line!r} vs {tool})") + for problem in problems: + print(" " + problem) + nfail += 1 + else: + print(f"PASS {label} (uuid {ruuid})") + if verbose: + print("".join(" | " + l + "\n" for l in body.splitlines())) + + print("----") + print(f"{len(STEPS) - nfail} passed, {nfail} failed") + return 1 if nfail else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/mcp/README.md b/tests/mcp/README.md index 8f2b1a0b2..ead0ed7df 100644 --- a/tests/mcp/README.md +++ b/tests/mcp/README.md @@ -17,6 +17,7 @@ up in both sets of goldens — that is the point. | `expected/*.out` | recorded stdout, one file per script | | `../llm/fixtures/*` | the EasyCrypt files the scripts load (shared with the REPL harness, never duplicated) | | `../../scripts/testing/mcp-golden` | the runner | +| `../../scripts/testing/mcp-parity` | the REPL/MCP parity checker (see below) | ## Running @@ -27,6 +28,7 @@ make test-mcp # build + run every scenario scripts/testing/mcp-golden # run every scenario scripts/testing/mcp-golden happy-path protocol-errors scripts/testing/mcp-golden --bin /path/to/ec.exe +scripts/testing/mcp-parity -v # the parity check, alone ``` The runner defaults to `_build/default/src/ec.exe`, resolved relative @@ -64,6 +66,56 @@ gate for changes to the protocol layer. | `exit` | `exit.` answers "session terminated", then the process stops | | `eof` | end of input is a clean shutdown, exit 0 | +## Parity + +`make test-mcp` runs `scripts/testing/mcp-parity` after the goldens. +Where the goldens freeze *what* the MCP server answers, the parity +check pins *why the two front-ends can be trusted to agree*: they are +two wire layers over one core, so the same operation must produce the +same answer on both. + +It plays one representative operation per tool family — load, step, +goals, tree, focus, undo, checkpoint, step again, revert, search, +commit, and a failing phrase — in that order, against two sessions +started from the same directory (`tests/llm`, so both name the fixture +identically and no path difference can leak into a reply): a REPL +session driven with `llm -eval`, and an MCP session driven with a +JSON-RPC script. For each step it asserts two things. + +**The uuid matches.** The REPL's `[uuid:N]` envelope tag against the +MCP result's `structuredContent.uuid`. + +**The payload matches.** The REPL's reply body — everything it prints +between the `OK`/`ERROR` line and `` — against the MCP result's +`content[0].text`, *up to one trailing newline*. That slack is the +whole of the licensed difference: the REPL terminates a body that lacks +a newline so that `` starts a line of its own, and MCP, having no +sentinel, does not. The checker appends that newline and then demands +byte equality. + +The comparison is derived from the two envelopes rather than pattern +matched out of them: the REPL wire is a sequence of blocks opened by a +status line and closed by a lone ``, and the MCP wire is one JSON +object per line. Both are parsed structurally, so the checker cannot +be fooled by a body that happens to contain something envelope-shaped. + +Two asymmetries are structural, and the check deliberately does not +span them: + +* **Envelope tags.** The REPL's `[loaded:file:N]` and `[focus: 1/N]` + annotations ride on the status line, not in the body; MCP's envelope + is `structuredContent`, which by the plan's result shape carries + `uuid` and `changed` only. So an MCP client does not see them at all. + That is a gap worth closing one day — the natural home is a + `structuredContent` field — but it is not a parity violation: no body + differs. +* **Notices on failures.** The REPL has never rendered the engine's + notice buffer on an `ERROR` reply; the MCP failure result does + include it. The two therefore agree only when the failing operation + emitted no notices, which is the case for the failing phrase the + check plays. Should a future step want a noisy failure, this is the + invariant to weaken — knowingly, and here. + ## Expected exit status Each `.script` declares its expected process exit status on its first From e3dce8552362ceb180740ba83a62fc81526d13ff Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Fri, 21 Aug 2026 15:48:23 +0200 Subject: [PATCH 31/51] [llm] real-client verification of `easycrypt mcp' MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Neither the goldens nor the parity check involve an MCP client; they speak the wire themselves. Two manual checks close that gap, both documented in tests/mcp/README.md and neither wired into CI (they need network access). `scripts/testing/mcp-inspector-check' drives the server with the reference client, `npx @modelcontextprotocol/inspector --cli', over tools/list and a tools/call. Observed, against ./ec.native: $ npx --yes @modelcontextprotocol/inspector --cli ./ec.native mcp \ --method tools/list 11 tools: ec_load, ec_step, ec_try, ec_goals, ec_tree, ec_focus, ec_undo, ec_revert, ec_checkpoint, ec_commit, ec_search $ ... --method tools/call --tool-name ec_load \ --tool-arg file=tests/llm/fixtures/simple.ec --tool-arg line=6 {"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------\n1 = 1 /\\ 2 = 2\n"}], "structuredContent":{"uuid":3,"changed":true},"isError":false} `tests/mcp/claude-code.mcp.json' is a ready-to-paste project config. Claude Code 2.1.238, registered at local scope (a project-scoped .mcp.json needs interactive approval), reports easycrypt: /.../ec.native mcp - ✔ Connected and a headless `claude -p' saw all eleven tools and drove a session: ec_load -> uuid 3, ec_step "split. trivial. trivial." -> uuid 6 (the multi-sentence step of the first commit, over a real client), ec_commit -> uuid 6. Connection + tools listing: gate met. FINDING, and it is the reason this gate exists. The same headless run shows the payload never reaches the agent: > Call ec_load ... Then quote the ENTIRE raw tool result, verbatim. {"uuid":3,"changed":true} That's the complete content -- no additional human-readable text accompanied it. Isolated against a throwaway probe server returning identical text under four result shapes, Claude Code's rule is: when a tools/call result carries structuredContent, the model is handed that object and `content' is dropped -- with or without an outputSchema. `content' alone arrives; text placed *inside* structuredContent arrives. So it is the presence of structuredContent, not of outputSchema, that suppresses the payload, and the Inspector passes because it displays both. Our shape puts metadata in structuredContent and the goal state, search results and proof body in `content', so `easycrypt mcp' is fully usable from the Inspector and blind from Claude Code. The fix is a change of result shape -- move the payload into structuredContent beside uuid and changed, or drop structuredContent and fold the metadata into the text -- which is a design decision this commit does not take. The evidence and both options are recorded in tests/mcp/README.md. Gate: make test-llm (21) and make test-mcp (12 + 12) green. --- scripts/testing/mcp-inspector-check | 72 ++++++++++++++++++++++++++ tests/mcp/README.md | 78 +++++++++++++++++++++++++++++ tests/mcp/claude-code.mcp.json | 8 +++ 3 files changed, 158 insertions(+) create mode 100755 scripts/testing/mcp-inspector-check create mode 100644 tests/mcp/claude-code.mcp.json diff --git a/scripts/testing/mcp-inspector-check b/scripts/testing/mcp-inspector-check new file mode 100755 index 000000000..9b869110a --- /dev/null +++ b/scripts/testing/mcp-inspector-check @@ -0,0 +1,72 @@ +#! /bin/sh + +# -------------------------------------------------------------------- +# Manual smoke test against a real MCP client. +# +# mcp-inspector-check [--bin PATH] +# +# Drives `easycrypt mcp' with the reference client, the MCP Inspector's +# CLI mode, rather than with our own golden harness: the goldens only +# prove the server is consistent with itself, this proves a client that +# knows nothing about EasyCrypt can complete the handshake, read the +# tool declarations and call a tool. +# +# NOT wired into CI, and deliberately so: it downloads +# @modelcontextprotocol/inspector through npx, so it needs node and +# network access, and it tracks a version we do not pin. Run it by hand +# after touching the protocol layer (src/ecMcp.ml). +# +# Two checks, both of which must print their result and exit 0: +# +# 1. tools/list -- the handshake and the tool table; +# 2. tools/call -- ec_load on a test fixture, whose result must +# carry the goal `1 = 1 /\ 2 = 2' and uuid 3. +# -------------------------------------------------------------------- + +set -eu + +root=$(CDPATH= cd -- "$(dirname -- "$0")/../.." && pwd) +bin="$root/_build/default/src/ec.exe" + +while [ $# -gt 0 ]; do + case "$1" in + --bin) + [ $# -ge 2 ] || { echo "mcp-inspector-check: --bin needs an argument" >&2; exit 2; } + bin=$2; shift 2 ;; + --bin=*) + bin=${1#--bin=}; shift ;; + -h|--help) + echo "usage: mcp-inspector-check [--bin PATH]"; exit 0 ;; + *) + echo "mcp-inspector-check: unknown option: $1" >&2; exit 2 ;; + esac +done + +case "$bin" in + /*) ;; + *) bin=$(CDPATH= cd -- "$(dirname -- "$bin")" && pwd)/$(basename -- "$bin") ;; +esac + +if [ ! -x "$bin" ]; then + echo "mcp-inspector-check: no such executable: $bin" >&2 + exit 2 +fi + +if ! command -v npx > /dev/null 2>&1; then + echo "mcp-inspector-check: npx not found; install node, or skip" >&2 + exit 2 +fi + +inspector="npx --yes @modelcontextprotocol/inspector --cli" +fixture="$root/tests/llm/fixtures/simple.ec" + +echo "== tools/list ==============================================" +$inspector "$bin" mcp --method tools/list + +echo +echo "== tools/call ec_load ======================================" +$inspector "$bin" mcp \ + --method tools/call \ + --tool-name ec_load \ + --tool-arg "file=$fixture" \ + --tool-arg line=6 diff --git a/tests/mcp/README.md b/tests/mcp/README.md index ead0ed7df..97cbfefb3 100644 --- a/tests/mcp/README.md +++ b/tests/mcp/README.md @@ -15,9 +15,11 @@ up in both sets of goldens — that is the point. |------|----------| | `scripts/*.script` | the JSON-RPC messages piped into the server | | `expected/*.out` | recorded stdout, one file per script | +| `claude-code.mcp.json` | a ready-to-paste client configuration | | `../llm/fixtures/*` | the EasyCrypt files the scripts load (shared with the REPL harness, never duplicated) | | `../../scripts/testing/mcp-golden` | the runner | | `../../scripts/testing/mcp-parity` | the REPL/MCP parity checker (see below) | +| `../../scripts/testing/mcp-inspector-check` | manual smoke test against a real client (see below) | ## Running @@ -116,6 +118,82 @@ span them: check plays. Should a future step want a noisy failure, this is the invariant to weaken — knowingly, and here. +## Real clients + +Neither the goldens nor the parity check involve an MCP client: they +speak the wire themselves, so they prove the server is consistent with +itself and with the REPL, not that a client can use it. Two manual +checks close that gap. Neither is in CI — both need network access — +and both should be run after touching `src/ecMcp.ml`. + +**The reference client.** `scripts/testing/mcp-inspector-check` drives +the server with the MCP Inspector's CLI mode, `npx +@modelcontextprotocol/inspector --cli`, over `tools/list` and a +`tools/call` of `ec_load`: + +``` +scripts/testing/mcp-inspector-check --bin ./ec.native +``` + +**Claude Code.** `claude-code.mcp.json` is a project configuration to +drop next to a proof development. It names `easycrypt` on the `PATH`, +so it stays free of absolute paths: + +```json +{"mcpServers": {"easycrypt": {"command": "easycrypt", "args": ["mcp"]}}} +``` + +A project-scoped `.mcp.json` needs interactive approval, so for a +headless check register the server at local scope instead and ask for +its health: + +``` +claude mcp add-json easycrypt '{"command":"/abs/path/ec.exe","args":["mcp"]}' --scope local +claude mcp list # easycrypt: ... - ✔ Connected +claude mcp remove easycrypt -s local +``` + +### Known gap: the payload does not reach a Claude Code agent + +That headless check found something the wire-level tests cannot see. +Claude Code, when a `tools/call` result carries `structuredContent`, +hands the model **only** that object and drops `content` entirely. Our +result shape puts the metadata in `structuredContent` (`uuid`, +`changed`) and the payload — the goal state, the search results, the +proof body — in `content`, so an agent driving this server through +Claude Code sees the uuid and nothing else: + +``` +> Call ec_load with file=.../simple.ec and line=6. Then quote the + ENTIRE raw tool result you received, verbatim. + + {"uuid":3,"changed":true} + + That's the complete content — no additional human-readable text + accompanied it. +``` + +Isolated against a four-tool probe server that returns the same text +under four different result shapes, the client's rule is: + +| result shape | payload reaches the model? | +|--------------|----------------------------| +| `content` + `structuredContent`, with `outputSchema` | no | +| `content` + `structuredContent`, without `outputSchema` | no | +| `content` only | yes | +| `content` + `structuredContent` *containing* the text | yes | + +So the presence of `structuredContent`, not of `outputSchema`, is what +suppresses `content`. The Inspector shows both, which is why the +Inspector check passes while the agent starves. + +Fixing this means changing the result shape, which is a design decision +outside this harness: either move the payload into `structuredContent` +(a `text` field beside `uuid` and `changed`, matching row 4), or drop +`structuredContent` and fold the metadata into the text (row 3). Until +then, `easycrypt mcp` is fully usable from the Inspector and from any +client that reads `content`, and effectively blind from Claude Code. + ## Expected exit status Each `.script` declares its expected process exit status on its first diff --git a/tests/mcp/claude-code.mcp.json b/tests/mcp/claude-code.mcp.json new file mode 100644 index 000000000..4ce195032 --- /dev/null +++ b/tests/mcp/claude-code.mcp.json @@ -0,0 +1,8 @@ +{ + "mcpServers": { + "easycrypt": { + "command": "easycrypt", + "args": ["mcp"] + } + } +} From 380ff7d1939987ce6ff6d7bf6bbb25adb5c54ac3 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Fri, 21 Aug 2026 15:54:14 +0200 Subject: [PATCH 32/51] [llm] mcp: repeat the reply text inside structuredContent Claude Code, the client this server is primarily for, hands the model a tools/call result's `structuredContent' alone and drops `content' entirely whenever both are present -- with or without an outputSchema. The probe matrix behind that statement is in the message of e3dce8552: four result shapes, identical text, and only the two that put the text somewhere other than a `content' shadowed by `structuredContent' reach the model. Our shape put the metadata in `structuredContent' and the payload -- goal state, commit body, search results -- in `content', so an agent driving `easycrypt mcp' through Claude Code saw `{"uuid":3,"changed":true}' and nothing else, while the Inspector, which displays both halves, showed nothing wrong. Each result now carries the text twice: `content' is unchanged, and `structuredContent' gains a `text' field holding exactly the string already in content[0].text -- one `~text' argument written into both halves by Result_of.make, so the copies cannot drift. Every tool's outputSchema declares `text' (string) required alongside `uuid' and `changed'; ec_try keeps `reverted' optional. The duplication is deliberate rather than a migration step. Dropping `content' would break spec-abiding clients that read it, and the parity check compares the REPL body against content[0].text; keeping both costs one repeated string per reply and serves either kind of client. Goldens: seven of twelve scenarios change, all in the same way -- structuredContent gains `text', and content[0].text, uuid, changed, reverted and isError are byte-identical to before (verified field by field against the recorded goldens); tools-list picks up the schema change. The five untouched scenarios play no tools/call. mcp-parity is unmodified and still green. tests/mcp/README.md: the "Known gap" section becomes "Result shape", which states the duplication, keeps the probe matrix as the reason, and turns the Claude Code check from "is it connected" into "can the agent quote the goal". Re-verified against the real client, Claude Code 2.1.238, the binary registered at local scope and removed afterwards: $ claude -p 'Call ec_load ... line=6. Then call ec_goals. Then quote the ENTIRE raw result object you received from ec_goals.' {"text":"Current goal\n\nType variables: \n\n-----...-----\n 1 = 1 /\\ 2 = 2\n","uuid":3,"changed":false} Recap: the open goal is `1 = 1 /\ 2 = 2' ... The agent quotes the goal, which is the whole point; note that the object it received is still `structuredContent' alone, confirming the client rule rather than working around it. Gates: dune build clean; make test-llm (21) green; make test-mcp (12 goldens + 12 parity) green twice. --- src/ecMcp.ml | 25 ++++++-- tests/mcp/README.md | 98 +++++++++++++++++------------ tests/mcp/expected/exit.out | 2 +- tests/mcp/expected/happy-path.out | 14 ++--- tests/mcp/expected/load-missing.out | 4 +- tests/mcp/expected/prover-error.out | 10 +-- tests/mcp/expected/revert.out | 20 +++--- tests/mcp/expected/tools-list.out | 2 +- tests/mcp/expected/try-revert.out | 12 ++-- 9 files changed, 110 insertions(+), 77 deletions(-) diff --git a/src/ecMcp.ml b/src/ecMcp.ml index 3d7d5c352..33846a453 100644 --- a/src/ecMcp.ml +++ b/src/ecMcp.ml @@ -110,10 +110,19 @@ module Schema = struct `List (List.map (fun s -> `String s) required))]) @ [("additionalProperties", `Bool false)]) - (* Every tool answers with the same structured payload: the engine - state the call left behind, and whether it moved. *) + (* Every tool answers with the same structured payload: the reply + text, the engine state the call left behind, and whether it moved. + + [text] repeats [content[0].text] verbatim. The duplication is + deliberate: Claude Code, our primary client, hands the model the + [structuredContent] object alone and drops [content] whenever both + are present, so a payload that lives only in [content] never + reaches the agent. See tests/mcp/README.md. *) let output ?(reverted = false) () = let base = [ + ("text", str ~description:"the reply body -- goal state, proof \ + body, search results, error text; the \ + same string as content[0].text" ()); ("uuid", int ~description:"engine state identifier after the call; \ pass it to ec_revert to come back here" ()); ("changed", `Assoc [("type", `String "boolean"); @@ -132,7 +141,8 @@ module Schema = struct in `Assoc [("type", `String "object"); ("properties", `Assoc base); - ("required", `List [`String "uuid"; `String "changed"])] + ("required", `List [`String "text"; `String "uuid"; + `String "changed"])] end (* -------------------------------------------------------------------- *) @@ -477,11 +487,18 @@ let run ~relocdir ~boot ~projini (mcpopts : EcOptions.mcp_option) = let content text = `List [`Assoc [("type", `String "text"); ("text", `String text)]] + (* [text] appears twice, once in each half of the result, and the + two copies are the same string by construction. Clients that read + [content] are served by the first; Claude Code, which drops + [content] as soon as [structuredContent] is present, is served + only by the second. *) let make ~text ~uuid ~changed ~is_error ~extra = `Assoc [ ("content", content text); ("structuredContent", - `Assoc ([("uuid", `Int uuid); ("changed", `Bool changed)] @ extra)); + `Assoc ([("text", `String text); + ("uuid", `Int uuid); + ("changed", `Bool changed)] @ extra)); ("isError", `Bool is_error); ] diff --git a/tests/mcp/README.md b/tests/mcp/README.md index 97cbfefb3..fa161c019 100644 --- a/tests/mcp/README.md +++ b/tests/mcp/README.md @@ -68,6 +68,48 @@ gate for changes to the protocol layer. | `exit` | `exit.` answers "session terminated", then the process stops | | `eof` | end of input is a clean shutdown, exit 0 | +## Result shape + +Every `tools/call` result carries the reply text **twice**: + +```json +{"content": [{"type": "text", "text": "Current goal\n..."}], + "structuredContent": {"text": "Current goal\n...", "uuid": 3, + "changed": true}, + "isError": false} +``` + +The two strings are the same by construction — `Result_of.make` takes +one `~text` and writes it into both halves — and `outputSchema` +declares `text` required alongside `uuid` and `changed` (and optional +`reverted`, on `ec_try`). + +The duplication is deliberate, and it is empirical rather than +aesthetic. Claude Code, the client this server is primarily for, hands +the model the `structuredContent` object **alone** and drops `content` +entirely whenever both are present. Isolated against a four-tool probe +server returning the same text under four result shapes (the run is +recorded in the message of commit `e3dce8552`): + +| result shape | payload reaches the model? | +|--------------|----------------------------| +| `content` + `structuredContent`, with `outputSchema` | no | +| `content` + `structuredContent`, without `outputSchema` | no | +| `content` only | yes | +| `content` + `structuredContent` *containing* the text | yes | + +So it is the presence of `structuredContent`, not of `outputSchema`, +that suppresses `content` — and before the text was duplicated, an +agent driving this server through Claude Code saw `{"uuid":3, +"changed":true}` and nothing else, while the Inspector, which displays +both halves, showed no problem at all. + +Row 4 is the shape we ship. Keeping `content` as well as filling +`structuredContent.text` costs one repeated string per reply and keeps +the server correct for spec-abiding clients that read `content`, for +clients that read only the structured half, and for the parity check, +which compares the REPL body against `content[0].text`. + ## Parity `make test-mcp` runs `scripts/testing/mcp-parity` after the goldens. @@ -106,11 +148,11 @@ span them: * **Envelope tags.** The REPL's `[loaded:file:N]` and `[focus: 1/N]` annotations ride on the status line, not in the body; MCP's envelope - is `structuredContent`, which by the plan's result shape carries - `uuid` and `changed` only. So an MCP client does not see them at all. - That is a gap worth closing one day — the natural home is a - `structuredContent` field — but it is not a parity violation: no body - differs. + is `structuredContent`, which carries `text`, `uuid` and `changed`, + none of which reproduces them. So an MCP client does not see them at + all. That is a gap worth closing one day — the natural home is a + further `structuredContent` field — but it is not a parity violation: + no body differs. * **Notices on failures.** The REPL has never rendered the engine's notice buffer on an `ERROR` reply; the MCP failure result does include it. The two therefore agree only when the failing operation @@ -153,46 +195,20 @@ claude mcp list # easycrypt: ... - ✔ Connected claude mcp remove easycrypt -s local ``` -### Known gap: the payload does not reach a Claude Code agent - -That headless check found something the wire-level tests cannot see. -Claude Code, when a `tools/call` result carries `structuredContent`, -hands the model **only** that object and drops `content` entirely. Our -result shape puts the metadata in `structuredContent` (`uuid`, -`changed`) and the payload — the goal state, the search results, the -proof body — in `content`, so an agent driving this server through -Claude Code sees the uuid and nothing else: +Health is not the interesting question, though: it was this headless +check that found the payload never reaching the agent (see **Result +shape** above), which no wire-level test can see. So ask the session to +*use* the server and quote back what it got — that, and not the +connection, is what the client-side check is for: ``` -> Call ec_load with file=.../simple.ec and line=6. Then quote the - ENTIRE raw tool result you received, verbatim. - - {"uuid":3,"changed":true} - - That's the complete content — no additional human-readable text - accompanied it. +claude -p 'Call ec_load with file=/tests/llm/fixtures/simple.ec + and line=6, then ec_goals. Quote the goal text verbatim.' \ + --allowedTools mcp__easycrypt__ec_load mcp__easycrypt__ec_goals ``` -Isolated against a four-tool probe server that returns the same text -under four different result shapes, the client's rule is: - -| result shape | payload reaches the model? | -|--------------|----------------------------| -| `content` + `structuredContent`, with `outputSchema` | no | -| `content` + `structuredContent`, without `outputSchema` | no | -| `content` only | yes | -| `content` + `structuredContent` *containing* the text | yes | - -So the presence of `structuredContent`, not of `outputSchema`, is what -suppresses `content`. The Inspector shows both, which is why the -Inspector check passes while the agent starves. - -Fixing this means changing the result shape, which is a design decision -outside this harness: either move the payload into `structuredContent` -(a `text` field beside `uuid` and `changed`, matching row 4), or drop -`structuredContent` and fold the metadata into the text (row 3). Until -then, `easycrypt mcp` is fully usable from the Inspector and from any -client that reads `content`, and effectively blind from Claude Code. +An agent that can quote `1 = 1 /\ 2 = 2` is reading the payload; one +that answers with a bare uuid is not. ## Expected exit status diff --git a/tests/mcp/expected/exit.out b/tests/mcp/expected/exit.out index 2c9c64214..366d40ffc 100644 --- a/tests/mcp/expected/exit.out +++ b/tests/mcp/expected/exit.out @@ -1,2 +1,2 @@ {"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2025-11-25","capabilities":{"tools":{}},"serverInfo":{"name":"easycrypt","version":"VERSION"}}} -{"jsonrpc":"2.0","id":2,"result":{"content":[{"type":"text","text":"session terminated"}],"structuredContent":{"uuid":0,"changed":false},"isError":false}} +{"jsonrpc":"2.0","id":2,"result":{"content":[{"type":"text","text":"session terminated"}],"structuredContent":{"text":"session terminated","uuid":0,"changed":false},"isError":false}} diff --git a/tests/mcp/expected/happy-path.out b/tests/mcp/expected/happy-path.out index 4ea16c67e..23d4391d0 100644 --- a/tests/mcp/expected/happy-path.out +++ b/tests/mcp/expected/happy-path.out @@ -1,8 +1,8 @@ {"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2025-11-25","capabilities":{"tools":{}},"serverInfo":{"name":"easycrypt","version":"VERSION"}}} -{"jsonrpc":"2.0","id":2,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"uuid":3,"changed":true},"isError":false}} -{"jsonrpc":"2.0","id":3,"result":{"content":[{"type":"text","text":"Current goal (remaining: 2)\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1\n"}],"structuredContent":{"uuid":4,"changed":true},"isError":false}} -{"jsonrpc":"2.0","id":4,"result":{"content":[{"type":"text","text":"Current goal (remaining: 2)\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1\n\n\n\n Goal #2\n ------------------------------------------------------------------------\n 2 = 2\n"}],"structuredContent":{"uuid":4,"changed":false},"isError":false}} -{"jsonrpc":"2.0","id":5,"result":{"content":[{"type":"text","text":"[1] 1 = 1 <- focused\n[2] 2 = 2\n"}],"structuredContent":{"uuid":4,"changed":false},"isError":false}} -{"jsonrpc":"2.0","id":6,"result":{"content":[{"type":"text","text":"Current goal (remaining: 2)\n\nType variables: \n\n------------------------------------------------------------------------\n2 = 2\n"}],"structuredContent":{"uuid":5,"changed":true},"isError":false}} -{"jsonrpc":"2.0","id":7,"result":{"content":[{"type":"text","text":"No more goals\n"}],"structuredContent":{"uuid":7,"changed":true},"isError":false}} -{"jsonrpc":"2.0","id":8,"result":{"content":[{"type":"text","text":"split.\n- trivial.\n- trivial.\n"}],"structuredContent":{"uuid":7,"changed":false},"isError":false}} +{"jsonrpc":"2.0","id":2,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n","uuid":3,"changed":true},"isError":false}} +{"jsonrpc":"2.0","id":3,"result":{"content":[{"type":"text","text":"Current goal (remaining: 2)\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1\n"}],"structuredContent":{"text":"Current goal (remaining: 2)\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1\n","uuid":4,"changed":true},"isError":false}} +{"jsonrpc":"2.0","id":4,"result":{"content":[{"type":"text","text":"Current goal (remaining: 2)\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1\n\n\n\n Goal #2\n ------------------------------------------------------------------------\n 2 = 2\n"}],"structuredContent":{"text":"Current goal (remaining: 2)\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1\n\n\n\n Goal #2\n ------------------------------------------------------------------------\n 2 = 2\n","uuid":4,"changed":false},"isError":false}} +{"jsonrpc":"2.0","id":5,"result":{"content":[{"type":"text","text":"[1] 1 = 1 <- focused\n[2] 2 = 2\n"}],"structuredContent":{"text":"[1] 1 = 1 <- focused\n[2] 2 = 2\n","uuid":4,"changed":false},"isError":false}} +{"jsonrpc":"2.0","id":6,"result":{"content":[{"type":"text","text":"Current goal (remaining: 2)\n\nType variables: \n\n------------------------------------------------------------------------\n2 = 2\n"}],"structuredContent":{"text":"Current goal (remaining: 2)\n\nType variables: \n\n------------------------------------------------------------------------\n2 = 2\n","uuid":5,"changed":true},"isError":false}} +{"jsonrpc":"2.0","id":7,"result":{"content":[{"type":"text","text":"No more goals\n"}],"structuredContent":{"text":"No more goals\n","uuid":7,"changed":true},"isError":false}} +{"jsonrpc":"2.0","id":8,"result":{"content":[{"type":"text","text":"split.\n- trivial.\n- trivial.\n"}],"structuredContent":{"text":"split.\n- trivial.\n- trivial.\n","uuid":7,"changed":false},"isError":false}} diff --git a/tests/mcp/expected/load-missing.out b/tests/mcp/expected/load-missing.out index 021b5bf4a..d1fb8737e 100644 --- a/tests/mcp/expected/load-missing.out +++ b/tests/mcp/expected/load-missing.out @@ -1,3 +1,3 @@ {"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2025-11-25","capabilities":{"tools":{}},"serverInfo":{"name":"easycrypt","version":"VERSION"}}} -{"jsonrpc":"2.0","id":2,"result":{"content":[{"type":"text","text":"LOAD: no such file: ../llm/fixtures/nosuchfile.ec"}],"structuredContent":{"uuid":0,"changed":false},"isError":true}} -{"jsonrpc":"2.0","id":3,"result":{"content":[{"type":"text","text":"unknown file extension: .txt\nNo active proof.\n"}],"structuredContent":{"uuid":0,"changed":false},"isError":true}} +{"jsonrpc":"2.0","id":2,"result":{"content":[{"type":"text","text":"LOAD: no such file: ../llm/fixtures/nosuchfile.ec"}],"structuredContent":{"text":"LOAD: no such file: ../llm/fixtures/nosuchfile.ec","uuid":0,"changed":false},"isError":true}} +{"jsonrpc":"2.0","id":3,"result":{"content":[{"type":"text","text":"unknown file extension: .txt\nNo active proof.\n"}],"structuredContent":{"text":"unknown file extension: .txt\nNo active proof.\n","uuid":0,"changed":false},"isError":true}} diff --git a/tests/mcp/expected/prover-error.out b/tests/mcp/expected/prover-error.out index 337f01463..2cc6d95de 100644 --- a/tests/mcp/expected/prover-error.out +++ b/tests/mcp/expected/prover-error.out @@ -1,6 +1,6 @@ {"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2025-11-25","capabilities":{"tools":{}},"serverInfo":{"name":"easycrypt","version":"VERSION"}}} -{"jsonrpc":"2.0","id":2,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"uuid":3,"changed":true},"isError":false}} -{"jsonrpc":"2.0","id":3,"result":{"content":[{"type":"text","text":": line 1 (0-18): unknown lemma `nosuchlemma'\nsource: apply nosuchlemma.\nCurrent goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"uuid":3,"changed":false},"isError":true}} -{"jsonrpc":"2.0","id":4,"result":{"content":[{"type":"text","text":"FOCUS: index 7 out of range (1..1)\nCurrent goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"uuid":3,"changed":false},"isError":true}} -{"jsonrpc":"2.0","id":5,"result":{"content":[{"type":"text","text":"No active proof.\n"}],"structuredContent":{"uuid":0,"changed":true},"isError":false}} -{"jsonrpc":"2.0","id":6,"result":{"content":[{"type":"text","text":"nothing to undo\nNo active proof.\n"}],"structuredContent":{"uuid":0,"changed":false},"isError":true}} +{"jsonrpc":"2.0","id":2,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n","uuid":3,"changed":true},"isError":false}} +{"jsonrpc":"2.0","id":3,"result":{"content":[{"type":"text","text":": line 1 (0-18): unknown lemma `nosuchlemma'\nsource: apply nosuchlemma.\nCurrent goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"text":": line 1 (0-18): unknown lemma `nosuchlemma'\nsource: apply nosuchlemma.\nCurrent goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n","uuid":3,"changed":false},"isError":true}} +{"jsonrpc":"2.0","id":4,"result":{"content":[{"type":"text","text":"FOCUS: index 7 out of range (1..1)\nCurrent goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"text":"FOCUS: index 7 out of range (1..1)\nCurrent goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n","uuid":3,"changed":false},"isError":true}} +{"jsonrpc":"2.0","id":5,"result":{"content":[{"type":"text","text":"No active proof.\n"}],"structuredContent":{"text":"No active proof.\n","uuid":0,"changed":true},"isError":false}} +{"jsonrpc":"2.0","id":6,"result":{"content":[{"type":"text","text":"nothing to undo\nNo active proof.\n"}],"structuredContent":{"text":"nothing to undo\nNo active proof.\n","uuid":0,"changed":false},"isError":true}} diff --git a/tests/mcp/expected/revert.out b/tests/mcp/expected/revert.out index e62225ec1..60c85a271 100644 --- a/tests/mcp/expected/revert.out +++ b/tests/mcp/expected/revert.out @@ -1,11 +1,11 @@ {"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2025-11-25","capabilities":{"tools":{}},"serverInfo":{"name":"easycrypt","version":"VERSION"}}} -{"jsonrpc":"2.0","id":2,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"uuid":3,"changed":true},"isError":false}} -{"jsonrpc":"2.0","id":3,"result":{"content":[{"type":"text","text":"checkpoint 'start' set at uuid 3"}],"structuredContent":{"uuid":3,"changed":false},"isError":false}} -{"jsonrpc":"2.0","id":4,"result":{"content":[{"type":"text","text":"Current goal (remaining: 2)\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1\n"}],"structuredContent":{"uuid":4,"changed":true},"isError":false}} -{"jsonrpc":"2.0","id":5,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"uuid":3,"changed":true},"isError":false}} -{"jsonrpc":"2.0","id":6,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"uuid":3,"changed":false},"isError":false}} -{"jsonrpc":"2.0","id":7,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n2 = 2\n"}],"structuredContent":{"uuid":5,"changed":true},"isError":false}} -{"jsonrpc":"2.0","id":8,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"uuid":3,"changed":true},"isError":false}} -{"jsonrpc":"2.0","id":9,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"uuid":3,"changed":false},"isError":false}} -{"jsonrpc":"2.0","id":10,"result":{"content":[{"type":"text","text":""}],"structuredContent":{"uuid":3,"changed":false},"isError":false}} -{"jsonrpc":"2.0","id":11,"result":{"content":[{"type":"text","text":"REVERT: 'nosuchname' is not a valid uuid or checkpoint name\nCurrent goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"uuid":3,"changed":false},"isError":true}} +{"jsonrpc":"2.0","id":2,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n","uuid":3,"changed":true},"isError":false}} +{"jsonrpc":"2.0","id":3,"result":{"content":[{"type":"text","text":"checkpoint 'start' set at uuid 3"}],"structuredContent":{"text":"checkpoint 'start' set at uuid 3","uuid":3,"changed":false},"isError":false}} +{"jsonrpc":"2.0","id":4,"result":{"content":[{"type":"text","text":"Current goal (remaining: 2)\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1\n"}],"structuredContent":{"text":"Current goal (remaining: 2)\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1\n","uuid":4,"changed":true},"isError":false}} +{"jsonrpc":"2.0","id":5,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n","uuid":3,"changed":true},"isError":false}} +{"jsonrpc":"2.0","id":6,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n","uuid":3,"changed":false},"isError":false}} +{"jsonrpc":"2.0","id":7,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n2 = 2\n"}],"structuredContent":{"text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n2 = 2\n","uuid":5,"changed":true},"isError":false}} +{"jsonrpc":"2.0","id":8,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n","uuid":3,"changed":true},"isError":false}} +{"jsonrpc":"2.0","id":9,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n","uuid":3,"changed":false},"isError":false}} +{"jsonrpc":"2.0","id":10,"result":{"content":[{"type":"text","text":""}],"structuredContent":{"text":"","uuid":3,"changed":false},"isError":false}} +{"jsonrpc":"2.0","id":11,"result":{"content":[{"type":"text","text":"REVERT: 'nosuchname' is not a valid uuid or checkpoint name\nCurrent goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"text":"REVERT: 'nosuchname' is not a valid uuid or checkpoint name\nCurrent goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n","uuid":3,"changed":false},"isError":true}} diff --git a/tests/mcp/expected/tools-list.out b/tests/mcp/expected/tools-list.out index 9de3be4a9..ab9b959cb 100644 --- a/tests/mcp/expected/tools-list.out +++ b/tests/mcp/expected/tools-list.out @@ -1 +1 @@ -{"jsonrpc":"2.0","id":1,"result":{"tools":[{"name":"ec_load","description":"Reset the session and compile FILE from the top, stopping after the last sentence that ends on or before LINE (and column COL when given). This is the entry point: every other tool needs a loaded file, and tactics need the position to land inside a proof. Set nosmt to weaken SMT calls while replaying a prefix that was already verified, which is much faster on large files. Set trace to have the reply describe the last loaded sentence as BEFORE / TACTIC / AFTER / SUMMARY blocks. The reply reports where compilation stopped and the resulting goal state; note the uuid it returns, reverting to it is the instant way back to the start of the proof.","inputSchema":{"type":"object","properties":{"file":{"type":"string","description":"path to the .ec/.eca file"},"line":{"type":"integer","description":"stop after the last sentence ending on or before this line; omit to compile the whole file"},"col":{"type":"integer","description":"column bound within `line'; requires `line'"},"nosmt":{"type":"boolean","description":"weaken SMT calls while compiling the prefix","default":false},"trace":{"type":"boolean","description":"report the proof state around the last loaded sentence","default":false}},"required":["file"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["uuid","changed"]},"annotations":{"destructiveHint":true}},{"name":"ec_step","description":"Run EasyCrypt sentences -- tactics, declarations, require, print, ... -- against the current session. Every complete sentence in the argument is executed, in order, exactly as if the text had been appended to the source file, and a single reply describes the state they leave behind; sentences may span several lines. Requires a file loaded with ec_load, and, for tactics, an open proof. On success the reply carries the new goal state; on failure the prover's error text comes back with isError set, the sentences before the failing one stay applied and the engine is left wherever that sentence left it -- use ec_try when you want a guaranteed rollback. Successful non-query phrases are recorded for ec_commit.","inputSchema":{"type":"object","properties":{"phrase":{"type":"string","description":"one or more complete EasyCrypt sentences, each ending with `.'"}},"required":["phrase"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["uuid","changed"]},"annotations":{"destructiveHint":false,"idempotentHint":false}},{"name":"ec_try","description":"Like ec_step, but the engine is rolled back to the state it had before the call whenever a sentence fails, including input that failed only after having already advanced the proof. The failure reply sets structuredContent.reverted to true, and its uuid and goal text describe the restored state, not the point of failure. Use this to probe a tactic without having to ec_revert afterwards; use ec_step when you mean to keep whatever progress the phrase makes. A successful phrase behaves exactly as under ec_step and is recorded for ec_commit.","inputSchema":{"type":"object","properties":{"phrase":{"type":"string","description":"one complete EasyCrypt sentence, ending with `.'"}},"required":["phrase"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"},"reverted":{"type":"boolean","description":"set when the phrase failed and the engine was rolled back to its pre-call state"}},"required":["uuid","changed"]},"annotations":{"destructiveHint":false}},{"name":"ec_goals","description":"Print the current proof state: the focused subgoal alone, or, with all set, every open subgoal. Requires an open proof, and does not advance the engine.","inputSchema":{"type":"object","properties":{"all":{"type":"boolean","description":"print every open subgoal instead of the focused one","default":false}},"additionalProperties":false},"outputSchema":{"type":"object","properties":{"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["uuid","changed"]},"annotations":{"readOnlyHint":true}},{"name":"ec_tree","description":"List the open subgoals as a tree of dotted-path labels -- [1], [1.2], [2.1.1] -- showing how the splits nest, and marking the focused one. Those labels are exactly what ec_focus accepts. Set full for whole goal bodies rather than one-line conclusions. The labels are not stable across focus changes: the tree always shows the focused goal first, so re-read it after every ec_focus. Does not advance the engine.","inputSchema":{"type":"object","properties":{"full":{"type":"boolean","description":"print full goal bodies instead of one-line conclusions","default":false}},"additionalProperties":false},"outputSchema":{"type":"object","properties":{"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["uuid","changed"]},"annotations":{"readOnlyHint":true}},{"name":"ec_focus","description":"Rotate the focus onto the subgoal at dotted path PATH, as printed by ec_tree (\"2\", \"1.2\", \"1.1.1\"); a single integer selects the k-th goal of the flat listing, and the special value \"next\" moves to the next open subgoal. Subsequent tactics act on the focused goal. Selecting an internal frame instead of a leaf goal is an error.","inputSchema":{"type":"object","properties":{"path":{"type":"string","description":"\"N\", a dotted path \"N1.N2...\", or \"next\""}},"required":["path"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["uuid","changed"]},"annotations":{"destructiveHint":false}},{"name":"ec_undo","description":"Undo the last engine step, returning to the immediately preceding state. The ec_commit transcript is trimmed to match. Fails when there is nothing left to undo.","inputSchema":{"type":"object","properties":{},"additionalProperties":false},"outputSchema":{"type":"object","properties":{"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["uuid","changed"]},"annotations":{"destructiveHint":false}},{"name":"ec_revert","description":"Return the session to an earlier state, named either by a uuid reported in some previous structuredContent or by a name given to ec_checkpoint. Reverting is instant, unlike re-running ec_load, so going back to the uuid ec_load returned is the cheap way to restart a proof from scratch after a failed experiment. The ec_commit transcript is trimmed to match.","inputSchema":{"type":"object","properties":{"target":{"type":"string","description":"a uuid (as a decimal string) or a checkpoint name"}},"required":["target"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["uuid","changed"]},"annotations":{"destructiveHint":true}},{"name":"ec_checkpoint","description":"Record the current uuid under NAME, so that ec_revert can address it by name later. Worth doing before a branching experiment, when carrying the bare uuid around is awkward. Does not change the proof state.","inputSchema":{"type":"object","properties":{"name":{"type":"string","description":"checkpoint name"}},"required":["name"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["uuid","changed"]},"annotations":{"readOnlyHint":true}},{"name":"ec_commit","description":"Emit the phrases recorded since the last ec_load as a proof body, with bullets inserted at every multi-child split: the result compiles under `pragma +strict_bullets' and can be pasted straight into the source file. Queries (search, print, locate, ec_search) are never recorded, so looking things up mid-proof does not pollute the body, and ec_undo / ec_revert trim the transcript. Still works after `qed.'. Does not change the proof state.","inputSchema":{"type":"object","properties":{},"additionalProperties":false},"outputSchema":{"type":"object","properties":{"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["uuid","changed"]},"annotations":{"readOnlyHint":true}},{"name":"ec_search","description":"Search the environment for lemmas matching an EasyCrypt search pattern. This is pattern syntax, not keyword search: use _ as the wildcard, as in \"(fdom _)\", \"(_ %/ _)\" or \"(mu _ _) (_ <= _)\". Requires a loaded file. The query neither advances the proof nor enters the ec_commit transcript.","inputSchema":{"type":"object","properties":{"pattern":{"type":"string","description":"an EasyCrypt search pattern"}},"required":["pattern"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["uuid","changed"]},"annotations":{"readOnlyHint":true}}]}} +{"jsonrpc":"2.0","id":1,"result":{"tools":[{"name":"ec_load","description":"Reset the session and compile FILE from the top, stopping after the last sentence that ends on or before LINE (and column COL when given). This is the entry point: every other tool needs a loaded file, and tactics need the position to land inside a proof. Set nosmt to weaken SMT calls while replaying a prefix that was already verified, which is much faster on large files. Set trace to have the reply describe the last loaded sentence as BEFORE / TACTIC / AFTER / SUMMARY blocks. The reply reports where compilation stopped and the resulting goal state; note the uuid it returns, reverting to it is the instant way back to the start of the proof.","inputSchema":{"type":"object","properties":{"file":{"type":"string","description":"path to the .ec/.eca file"},"line":{"type":"integer","description":"stop after the last sentence ending on or before this line; omit to compile the whole file"},"col":{"type":"integer","description":"column bound within `line'; requires `line'"},"nosmt":{"type":"boolean","description":"weaken SMT calls while compiling the prefix","default":false},"trace":{"type":"boolean","description":"report the proof state around the last loaded sentence","default":false}},"required":["file"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["text","uuid","changed"]},"annotations":{"destructiveHint":true}},{"name":"ec_step","description":"Run EasyCrypt sentences -- tactics, declarations, require, print, ... -- against the current session. Every complete sentence in the argument is executed, in order, exactly as if the text had been appended to the source file, and a single reply describes the state they leave behind; sentences may span several lines. Requires a file loaded with ec_load, and, for tactics, an open proof. On success the reply carries the new goal state; on failure the prover's error text comes back with isError set, the sentences before the failing one stay applied and the engine is left wherever that sentence left it -- use ec_try when you want a guaranteed rollback. Successful non-query phrases are recorded for ec_commit.","inputSchema":{"type":"object","properties":{"phrase":{"type":"string","description":"one or more complete EasyCrypt sentences, each ending with `.'"}},"required":["phrase"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["text","uuid","changed"]},"annotations":{"destructiveHint":false,"idempotentHint":false}},{"name":"ec_try","description":"Like ec_step, but the engine is rolled back to the state it had before the call whenever a sentence fails, including input that failed only after having already advanced the proof. The failure reply sets structuredContent.reverted to true, and its uuid and goal text describe the restored state, not the point of failure. Use this to probe a tactic without having to ec_revert afterwards; use ec_step when you mean to keep whatever progress the phrase makes. A successful phrase behaves exactly as under ec_step and is recorded for ec_commit.","inputSchema":{"type":"object","properties":{"phrase":{"type":"string","description":"one complete EasyCrypt sentence, ending with `.'"}},"required":["phrase"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"},"reverted":{"type":"boolean","description":"set when the phrase failed and the engine was rolled back to its pre-call state"}},"required":["text","uuid","changed"]},"annotations":{"destructiveHint":false}},{"name":"ec_goals","description":"Print the current proof state: the focused subgoal alone, or, with all set, every open subgoal. Requires an open proof, and does not advance the engine.","inputSchema":{"type":"object","properties":{"all":{"type":"boolean","description":"print every open subgoal instead of the focused one","default":false}},"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["text","uuid","changed"]},"annotations":{"readOnlyHint":true}},{"name":"ec_tree","description":"List the open subgoals as a tree of dotted-path labels -- [1], [1.2], [2.1.1] -- showing how the splits nest, and marking the focused one. Those labels are exactly what ec_focus accepts. Set full for whole goal bodies rather than one-line conclusions. The labels are not stable across focus changes: the tree always shows the focused goal first, so re-read it after every ec_focus. Does not advance the engine.","inputSchema":{"type":"object","properties":{"full":{"type":"boolean","description":"print full goal bodies instead of one-line conclusions","default":false}},"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["text","uuid","changed"]},"annotations":{"readOnlyHint":true}},{"name":"ec_focus","description":"Rotate the focus onto the subgoal at dotted path PATH, as printed by ec_tree (\"2\", \"1.2\", \"1.1.1\"); a single integer selects the k-th goal of the flat listing, and the special value \"next\" moves to the next open subgoal. Subsequent tactics act on the focused goal. Selecting an internal frame instead of a leaf goal is an error.","inputSchema":{"type":"object","properties":{"path":{"type":"string","description":"\"N\", a dotted path \"N1.N2...\", or \"next\""}},"required":["path"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["text","uuid","changed"]},"annotations":{"destructiveHint":false}},{"name":"ec_undo","description":"Undo the last engine step, returning to the immediately preceding state. The ec_commit transcript is trimmed to match. Fails when there is nothing left to undo.","inputSchema":{"type":"object","properties":{},"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["text","uuid","changed"]},"annotations":{"destructiveHint":false}},{"name":"ec_revert","description":"Return the session to an earlier state, named either by a uuid reported in some previous structuredContent or by a name given to ec_checkpoint. Reverting is instant, unlike re-running ec_load, so going back to the uuid ec_load returned is the cheap way to restart a proof from scratch after a failed experiment. The ec_commit transcript is trimmed to match.","inputSchema":{"type":"object","properties":{"target":{"type":"string","description":"a uuid (as a decimal string) or a checkpoint name"}},"required":["target"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["text","uuid","changed"]},"annotations":{"destructiveHint":true}},{"name":"ec_checkpoint","description":"Record the current uuid under NAME, so that ec_revert can address it by name later. Worth doing before a branching experiment, when carrying the bare uuid around is awkward. Does not change the proof state.","inputSchema":{"type":"object","properties":{"name":{"type":"string","description":"checkpoint name"}},"required":["name"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["text","uuid","changed"]},"annotations":{"readOnlyHint":true}},{"name":"ec_commit","description":"Emit the phrases recorded since the last ec_load as a proof body, with bullets inserted at every multi-child split: the result compiles under `pragma +strict_bullets' and can be pasted straight into the source file. Queries (search, print, locate, ec_search) are never recorded, so looking things up mid-proof does not pollute the body, and ec_undo / ec_revert trim the transcript. Still works after `qed.'. Does not change the proof state.","inputSchema":{"type":"object","properties":{},"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["text","uuid","changed"]},"annotations":{"readOnlyHint":true}},{"name":"ec_search","description":"Search the environment for lemmas matching an EasyCrypt search pattern. This is pattern syntax, not keyword search: use _ as the wildcard, as in \"(fdom _)\", \"(_ %/ _)\" or \"(mu _ _) (_ <= _)\". Requires a loaded file. The query neither advances the proof nor enters the ec_commit transcript.","inputSchema":{"type":"object","properties":{"pattern":{"type":"string","description":"an EasyCrypt search pattern"}},"required":["pattern"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["text","uuid","changed"]},"annotations":{"readOnlyHint":true}}]}} diff --git a/tests/mcp/expected/try-revert.out b/tests/mcp/expected/try-revert.out index b2f298025..665049196 100644 --- a/tests/mcp/expected/try-revert.out +++ b/tests/mcp/expected/try-revert.out @@ -1,7 +1,7 @@ {"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2025-11-25","capabilities":{"tools":{}},"serverInfo":{"name":"easycrypt","version":"VERSION"}}} -{"jsonrpc":"2.0","id":2,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"uuid":3,"changed":true},"isError":false}} -{"jsonrpc":"2.0","id":3,"result":{"content":[{"type":"text","text":": line 1 (7-25): unknown lemma `nosuchlemma'\nsource: apply nosuchlemma.\nCurrent goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"uuid":3,"changed":false,"reverted":true},"isError":true}} -{"jsonrpc":"2.0","id":4,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"uuid":3,"changed":false},"isError":false}} -{"jsonrpc":"2.0","id":5,"result":{"content":[{"type":"text","text":""}],"structuredContent":{"uuid":3,"changed":false},"isError":false}} -{"jsonrpc":"2.0","id":6,"result":{"content":[{"type":"text","text":"Current goal (remaining: 2)\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1\n"}],"structuredContent":{"uuid":4,"changed":true},"isError":false}} -{"jsonrpc":"2.0","id":7,"result":{"content":[{"type":"text","text":"Current goal (remaining: 2)\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1\n"}],"structuredContent":{"uuid":4,"changed":false},"isError":false}} +{"jsonrpc":"2.0","id":2,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n","uuid":3,"changed":true},"isError":false}} +{"jsonrpc":"2.0","id":3,"result":{"content":[{"type":"text","text":": line 1 (7-25): unknown lemma `nosuchlemma'\nsource: apply nosuchlemma.\nCurrent goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"text":": line 1 (7-25): unknown lemma `nosuchlemma'\nsource: apply nosuchlemma.\nCurrent goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n","uuid":3,"changed":false,"reverted":true},"isError":true}} +{"jsonrpc":"2.0","id":4,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n","uuid":3,"changed":false},"isError":false}} +{"jsonrpc":"2.0","id":5,"result":{"content":[{"type":"text","text":""}],"structuredContent":{"text":"","uuid":3,"changed":false},"isError":false}} +{"jsonrpc":"2.0","id":6,"result":{"content":[{"type":"text","text":"Current goal (remaining: 2)\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1\n"}],"structuredContent":{"text":"Current goal (remaining: 2)\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1\n","uuid":4,"changed":true},"isError":false}} +{"jsonrpc":"2.0","id":7,"result":{"content":[{"type":"text","text":"Current goal (remaining: 2)\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1\n"}],"structuredContent":{"text":"Current goal (remaining: 2)\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1\n","uuid":4,"changed":false},"isError":false}} From e10cd7051e7d59a3a2621436aec18536f41020ca Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Fri, 21 Aug 2026 16:06:43 +0200 Subject: [PATCH 33/51] [llm] document the MCP front-end Adds a "Using the MCP mode" section to the agent guide: how to launch, the eleven tools, the uuid/state model (shared with the REPL section), the JSON-RPC-error / isError split, the result shape and why the reply text is duplicated, the ec_step and ec_try contracts, and a ready-to-paste client configuration. `mcp -help` now prints that section, as `llm -help` prints the whole guide: it reads the same file and cuts from the heading to the next one at the same level, falling back to the whole guide if the heading is gone. That is why `EcLlm.llm_guide_path` becomes public. README gains a paragraph on the LLM-agent interface, pointing at the guide. --- README.md | 7 ++ doc/llm/CLAUDE.md | 158 +++++++++++++++++++++++++++++++++++++++++++++- src/ecLlm.mli | 6 ++ src/ecMcp.ml | 60 ++++++++++-------- 4 files changed, 203 insertions(+), 28 deletions(-) diff --git a/README.md b/README.md index 9ef3d2917..c493052ec 100644 --- a/README.md +++ b/README.md @@ -166,6 +166,13 @@ with proof scripts). At present, the only available front-end is based on Emacs' [Proof General](https://github.com/ProofGeneral/PG). However, a front-end for VSCode is currently in development. +Besides these, EasyCrypt ships an interface aimed at LLM agents rather +than at humans: `easycrypt llm`, an interactive REPL speaking a +machine-friendly protocol, and `easycrypt mcp`, a +[Model Context Protocol](https://modelcontextprotocol.io/) server over +stdio. Both drive the same proof engine, and both are documented in +[doc/llm/CLAUDE.md](doc/llm/CLAUDE.md). + ### Proof-General (Emacs) EasyCrypt mode has been integrated upstream. Please, go diff --git a/doc/llm/CLAUDE.md b/doc/llm/CLAUDE.md index 0bd38c7a4..b1b2f720b 100644 --- a/doc/llm/CLAUDE.md +++ b/doc/llm/CLAUDE.md @@ -10,7 +10,11 @@ reasoning. The `llm` subcommand provides an interactive REPL with a machine-friendly protocol designed for LLM agents. The LLM sends -commands over stdin and receives structured responses on stdout. +commands over stdin and receives structured responses on stdout. The +same engine is also served over the Model Context Protocol by the `mcp` +subcommand — see [the MCP section](#using-the-mcp-mode) below. The +two are front-ends over one core, so everything said here about state, +uuids and the proof workflow holds there as well. ``` easycrypt llm [OPTIONS] @@ -302,6 +306,158 @@ SEARCH (fdom _) SEARCH (_ %/ _) ``` +## Using the MCP mode + +The `mcp` subcommand serves the same proof engine over the [Model +Context Protocol](https://modelcontextprotocol.io/) instead of the +text protocol above: JSON-RPC 2.0 messages, one per line, over stdio. +Use it from a client that already speaks MCP; use `llm` for the raw +protocol, as a debug console, or for `-eval` scripting. + +``` +easycrypt mcp [OPTIONS] +``` + +The same loader and prover options as `llm` are available (`-I`, +`-timeout`, `-p`, `-stdlib`, etc.). Use `-help` to print this section +and exit: + +``` +easycrypt mcp -help +``` + +Only protocol messages appear on stdout; everything the engine has to +say goes to stderr. The server speaks the `initialize` / +`notifications/initialized` handshake, implements `initialize`, +`ping`, `tools/list` and `tools/call`, and tolerates notifications as +no-ops. It advertises the `tools` capability and nothing else: no +resources, no prompts, no sampling. + +### Tools + +Eleven tools. Required arguments are marked; the others default as +noted. + +| Tool | Arguments | Description | +|------|-----------|-------------| +| `ec_load` | `file` (req), `line`, `col`, `nosmt` (false), `trace` (false) | Reset the session and compile `file` from the top, stopping after the last sentence that ends on or before `line` | +| `ec_step` | `phrase` (req) | Run EasyCrypt sentences — tactics, declarations, `require`, `print`, ... — against the current session | +| `ec_try` | `phrase` (req) | Like `ec_step`, but roll the engine back to its pre-call state whenever a sentence fails | +| `ec_goals` | `all` (false) | Print the focused subgoal, or, with `all`, every open subgoal | +| `ec_tree` | `full` (false) | List the open subgoals as a tree of dotted-path labels, marking the focused one | +| `ec_focus` | `path` (req) | Rotate the focus onto the subgoal at dotted path `path`, or onto the next one with `"next"` | +| `ec_undo` | — | Undo the last engine step | +| `ec_revert` | `target` (req) | Return the session to an earlier state, named by a uuid or by a checkpoint name | +| `ec_checkpoint` | `name` (req) | Record the current uuid under `name`, for a later `ec_revert` | +| `ec_commit` | — | Emit the phrases recorded since the last `ec_load` as a bulleted proof body | +| `ec_search` | `pattern` (req) | Search the environment for lemmas matching an EasyCrypt search pattern | + +`tools/list` carries a fuller, agent-facing `description` and a JSON +Schema for every tool; those are the authoritative texts. The tools +mirror the REPL meta-commands — `-nosmt`, `-trace`, dotted paths, +checkpoints, bullets and search patterns all behave exactly as +described above, and `NEXT` folds into `ec_focus` with path `"next"` — +plus `ec_try`, which has no REPL equivalent. The meta-commands that are +pure console affordances have no tool: multi-line input needs no +``/`` (a `phrase` may simply contain newlines), `QUIET` +has no purpose when the client decides what to display, `HELP` is this +section, and the session ends when the client closes stdin — or when a +phrase is `exit.`, which answers `session terminated` and stops the +process. + +### Running sentences + +`ec_step` takes one or more complete EasyCrypt sentences in a single +`phrase`, exactly as a REPL line does: all of them are executed, in +order, as if the text had been appended to the source file, and one +reply describes the state they leave behind. If one of them fails, the +reply is that failure, the sentences before it stay applied, and the +engine is left wherever the failing sentence left it. + +`ec_try` runs the same input under a rollback contract: whenever a +sentence fails, the engine is returned to the state it had before the +call — including input that failed only after having already advanced +the proof. The failure result sets `structuredContent.reverted` to +`true`, and its `uuid` and text describe that restored state, not the +point of failure. Use `ec_try` to probe a tactic without having to +`ec_revert` afterwards, and `ec_step` when you mean to keep whatever +progress the phrase makes. A successful phrase behaves identically +under both, and is recorded for `ec_commit` in both. + +### State and uuids + +The state model is the REPL's, unchanged. One client is one process is +one engine state: there is no multiplexing, and tool calls run strictly +in arrival order even when a client pipelines them. Every result +reports in its `structuredContent` the `uuid` the call left behind — +the same monotonically increasing state identifier the REPL prints as +`[uuid:N]`, advancing only on calls that change engine state — and +`ec_revert` accepts either one of those uuids or a name given to +`ec_checkpoint`. Note the uuid returned by `ec_load`: reverting to it +is the instant way back to the start of the proof. + +### Errors + +Two kinds of failure, deliberately kept apart: + +* **Protocol faults** — malformed JSON, an unknown method, an unknown + tool, an argument that violates the declared schema — are JSON-RPC + errors (`-32700`, `-32600`, `-32601`, `-32602`). +* **EasyCrypt failures** — a tactic that does not apply, a file that + does not compile, an SMT timeout, a file that is not there — are + *successful* responses carrying `"isError": true` and the prover's + error text. + +The second kind is data: read those messages and act on them, the way +the REPL's `ERROR` replies are meant to be read. + +### Result shape + +Every `tools/call` result, error or not, has the same shape: + +```json +{"content": [{"type": "text", "text": "Current goal\n..."}], + "structuredContent": {"text": "Current goal\n...", "uuid": 3, + "changed": true}, + "isError": false} +``` + +`text` is the body the REPL would print between its envelope and +``, `uuid` is the resulting state, and `changed` says whether the +engine advanced; `ec_try` adds `reverted` on failure, and each tool +declares an `outputSchema` matching that structured half. The text +appears twice on purpose: some clients hand the model +`structuredContent` alone and drop `content` whenever both are present, +so a payload living only in `content` would never reach the agent (the +measurement is in `tests/mcp/README.md`). + +What has no counterpart here are the REPL's status-line annotations: +`[loaded:file:N]` and the `[focus: k/N]` tag do not ride along, so ask +`ec_tree` when you need to know which of several goals the next tactic +will hit. + +### Client configuration + +As a project-scoped `.mcp.json`, dropped next to a proof development: + +```json +{ + "mcpServers": { + "easycrypt": { + "command": "easycrypt", + "args": ["mcp"] + } + } +} +``` + +Add loader options to `args` as needed, e.g. `["mcp", "-I", +"theories"]`. The equivalent one-liner, for Claude Code: + +``` +claude mcp add easycrypt -- easycrypt mcp +``` + ## EasyCrypt proof strategy ### General approach diff --git a/src/ecLlm.mli b/src/ecLlm.mli index d426c4c6e..28b9896d5 100644 --- a/src/ecLlm.mli +++ b/src/ecLlm.mli @@ -2,6 +2,12 @@ (* The LLM coding-agent REPL: an interactive proof-development protocol over stdin/stdout. Driven via the [easycrypt llm] command. *) +(* Path to the bundled agent guide ([doc/llm/CLAUDE.md] in a source + tree, its installed copy otherwise). [llm -help] and the [HELP] + command print the whole of it; exposed because [mcp -help] prints one + section of the same file. *) +val llm_guide_path : unit -> string + (* Run the REPL until [QUIT] or EOF, then exit the process. Never returns. [projini] resolves the [easycrypt.project] context of a file path, so [LOAD] can apply the project's load path and prover diff --git a/src/ecMcp.ml b/src/ecMcp.ml index 33846a453..e0eb15b59 100644 --- a/src/ecMcp.ml +++ b/src/ecMcp.ml @@ -56,32 +56,38 @@ exception Invalid_params of string exception Tool_error of string (* -------------------------------------------------------------------- *) -(* TODO(phase 3): replace with the MCP section of doc/llm/CLAUDE.md, - the way [llm -help] prints the whole guide. *) -let usage = {|easycrypt mcp -- Model Context Protocol server (stdio, JSON-RPC 2.0) - -Exposes the EasyCrypt proof engine to an MCP client as a set of tools. -Reads newline-delimited JSON-RPC messages on stdin and writes responses -on stdout; diagnostics go to stderr. One client = one process = one -proof session. Standard loader and prover options (-I, -timeout, -p, --stdlib, ...) are accepted, as for `easycrypt llm'. - -Tools: - ec_load compile a file up to a position and start a session - ec_step run one or more EasyCrypt sentences - ec_try run sentences, rolling back if any of them fails - ec_goals print the current goal state - ec_tree list the open subgoals as a labelled tree - ec_focus focus the subgoal at a dotted path (or `next') - ec_undo undo the last step - ec_revert return to a uuid or to a named checkpoint - ec_checkpoint name the current state for a later ec_revert - ec_commit emit the recorded phrases as a bulleted proof body - ec_search search for lemmas matching a pattern - -Client configuration and the full protocol description live in -doc/llm/CLAUDE.md. -|} +(* [-help]. Where [llm -help] prints the whole agent guide, we print the + one section of it that describes this server: from its heading down + to the next heading of the same level. A guide in which that heading + cannot be found is printed whole, rather than not at all. *) +let usage_section = "## Using the MCP mode" + +let extract_usage (guide : string) = + let is_heading line = + String.length line >= 3 && String.sub line 0 3 = "## " in + let rec seek = function + | [] -> None + | line :: rest when String.trim line = usage_section -> + Some (line :: keep rest) + | _ :: rest -> seek rest + and keep = function + | [] -> [] + | line :: _ when is_heading line -> [] + | line :: rest -> line :: keep rest + in + match seek (String.split_on_char '\n' guide) with + | None -> guide + | Some lines -> String.concat "\n" lines + +let print_usage () = + let path = EcLlm.llm_guide_path () in + try + let ic = open_in_bin path in + let guide = really_input_string ic (in_channel_length ic) in + close_in ic; + print_string (extract_usage guide) + with Sys_error e -> + Printf.eprintf "cannot read LLM guide: %s\n%!" e (* -------------------------------------------------------------------- *) (* JSON schema fragments for the tool declarations. *) @@ -429,7 +435,7 @@ let focus_target (arg : string) = (* -------------------------------------------------------------------- *) let run ~relocdir ~boot ~projini (mcpopts : EcOptions.mcp_option) = if mcpopts.mcpo_help then begin - print_string usage; + print_usage (); exit 0 end; From d1389160cecf4293612559b055e67d34409d3828 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Fri, 21 Aug 2026 16:18:25 +0200 Subject: [PATCH 34/51] [llm] wire the llm/mcp golden harnesses into check and CI `make check` now runs test-llm and test-mcp alongside unit, stdlib and examples, and the CI library-check matrix gains both targets. Both harnesses need only the built binary (their fixtures avoid SMT), so they run in the same docker job as the existing targets. Closes the last open item of PLAN-easycrypt-mcp.md Phase 2 ("wire both into the existing test harness/CI"). --- .github/workflows/ci.yml | 2 +- Makefile | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 7688d008b..1a583304d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -113,7 +113,7 @@ jobs: strategy: fail-fast: false matrix: - target: [unit, stdlib, examples] + target: [unit, stdlib, examples, test-llm, test-mcp] steps: - uses: actions/checkout@v4 - uses: actions/download-artifact@v4 diff --git a/Makefile b/Makefile index e0b8bb5e2..2fb3307a9 100644 --- a/Makefile +++ b/Makefile @@ -65,7 +65,7 @@ test-mcp: build $(MCPCHECK) $(MCPPARITY) -check: unit stdlib examples +check: unit stdlib examples test-llm test-mcp @true nix-build: From e480a783bf75cb226dd66c2bb2d3ead09fb604e9 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Fri, 21 Aug 2026 20:05:32 +0200 Subject: [PATCH 35/51] [llm] SEARCH takes a pattern, not a command stream EcLlmCore.search spliced its argument into "search %s." and handed the result to [step], which runs *every* sentence its input holds. A pattern carrying a sentence-ending '.' therefore escaped the query and executed arbitrary EasyCrypt: SEARCH (_ /\ _). split. admit ran [split] and [admit], advanced the uuid past the search, and left both in the body COMMIT emits -- a query silently editing the proof. Filtering '.' out of the pattern would be the wrong fix: qualified names (A.B.lem, RField.ofint) are legitimate patterns and are full of dots. So the core now composes the phrase and parses it itself (EcIo.from_string + xparse), accepting it only when the reader yields exactly one toplevel item, that item's action is Gsearch, and nothing but end-of-input follows. A pattern that closes the sentence early leaves trailing input and is refused before the engine is touched. Since a search can no longer reach [exit.], it can no longer end the session, so it returns a [result] rather than an [answer]; the two front-ends drop their now-unreachable Quit arm (Wire.answer -> Wire.reply, answer -> outcome). No golden was re-recorded: legitimate patterns behave exactly as before, so all 21 llm and 12+12 mcp cases pass untouched. One golden is added -- search-injection pins the uuid at the LOAD value across the rejected pattern, shows COMMIT staying empty, and checks that a dotted qualified pattern (RField.ofint _) still searches normally. --- src/ecLlm.ml | 2 +- src/ecLlmCore.ml | 49 ++++++++++++++++++++- src/ecLlmCore.mli | 8 +++- src/ecMcp.ml | 2 +- tests/llm/expected/search-injection.out | 53 +++++++++++++++++++++++ tests/llm/scripts/search-injection.script | 9 ++++ 6 files changed, 119 insertions(+), 4 deletions(-) create mode 100644 tests/llm/expected/search-injection.out create mode 100644 tests/llm/scripts/search-injection.script diff --git a/src/ecLlm.ml b/src/ecLlm.ml index 8a2c1ead6..5e0d1a276 100644 --- a/src/ecLlm.ml +++ b/src/ecLlm.ml @@ -341,7 +341,7 @@ let run ~relocdir ~boot ~projini (llmopts : EcOptions.llm_option) = | Checkpoint n -> Wire.reply (EcLlmCore.checkpoint st ~name:n) | Revert s -> Wire.reply (EcLlmCore.revert st s) | Quiet on -> do_quiet on - | Search q -> Wire.answer (EcLlmCore.search st ~pattern:q) + | Search q -> Wire.reply (EcLlmCore.search st ~pattern:q) | Load args -> Wire.reply (EcLlmCore.load st ~file:args.Parse.ld_file diff --git a/src/ecLlmCore.ml b/src/ecLlmCore.ml index dbc5d7b45..f8f249af7 100644 --- a/src/ecLlmCore.ml +++ b/src/ecLlmCore.ml @@ -1054,5 +1054,52 @@ let revert (st : state) spec = Ok (mk_reply_goals st ~pre) end +(* SEARCH is handed a search pattern, not EasyCrypt input. Composing + ["search " ^ pattern ^ "."] and running it through [step] made every + sentence-ending '.' inside the pattern a statement separator, so a + pattern like [(_ /\ _). split. admit] executed [split] and [admit] + too. Parse the composed phrase here instead, and run it only if it + is exactly one toplevel item whose action is a [search]: a pattern + that closes the sentence on its own leaves trailing input, which + this rejects. Screening the pattern for '.' would be wrong -- + qualified names (A.B.lem) are legitimate patterns. *) let search (st : state) ~pattern = - step st (Printf.sprintf "search %s." pattern) + let pre = EcCommands.uuid () in + Buffer.clear st.notices; + let src = Printf.sprintf "search %s." pattern in + let reject = "SEARCH: the argument must be a single search pattern" in + let is_search (p : EP.global) = + not p.EP.gl_fail + && match EcLocation.unloc p.EP.gl_action with + | EP.Gsearch _ -> true + | _ -> false + in + let parsed = + let reader = EcIo.from_string src in + let next () = + match EcIo.xparse reader with + | exception End_of_file -> `End + | (_, prog) -> + match EcLocation.unloc prog with + | EP.P_Prog ([ ], true ) -> `End + | EP.P_Prog ([p], false) -> `Item p + | _ -> `Other + in + let result = + try + match next () with + | `Item p when is_search p -> + (match next () with + | `End -> Ok p + | `Item _ | `Other -> Error reject) + | `Item _ | `Other | `End -> Error reject + with e -> Error (Goals.format_error e) + in + EcIo.finalize reader; result + in + match parsed with + | Error msg -> Error (mk_failure st ~pre msg) + | Ok p -> + match process_action st ~src p with + | () -> Ok (mk_reply_goals st ~pre) + | exception e -> Error (mk_failure st ~pre (Goals.format_error ~src e)) diff --git a/src/ecLlmCore.mli b/src/ecLlmCore.mli index ba85d13b3..f6f40e590 100644 --- a/src/ecLlmCore.mli +++ b/src/ecLlmCore.mli @@ -107,7 +107,13 @@ val undo : state -> (reply, failure) result val revert : state -> string -> (reply, failure) result val checkpoint : state -> name:string -> (reply, failure) result val commit : state -> (reply, failure) result -val search : state -> pattern:string -> answer + +(* SEARCH. [pattern] is a search pattern, not EasyCrypt input: the + composed phrase is parsed here and refused unless it is exactly one + toplevel [search] item, so a pattern carrying a sentence-ending '.' + cannot smuggle further commands past it. Hence a [result] and not an + [answer]: SEARCH runs a query and can never end the session. *) +val search : state -> pattern:string -> (reply, failure) result (* -------------------------------------------------------------------- *) (* Front-end helpers: for replies a front-end produces on its own (the diff --git a/src/ecMcp.ml b/src/ecMcp.ml index e0eb15b59..70ffd1c15 100644 --- a/src/ecMcp.ml +++ b/src/ecMcp.ml @@ -625,7 +625,7 @@ let run ~relocdir ~boot ~projini (mcpopts : EcOptions.mcp_option) = outcome (EcLlmCore.commit st) | "ec_search" -> - answer (EcLlmCore.search st + outcome (EcLlmCore.search st ~pattern:(Args.string_req name args "pattern")) | _ -> diff --git a/tests/llm/expected/search-injection.out b/tests/llm/expected/search-injection.out new file mode 100644 index 000000000..009dc8ba5 --- /dev/null +++ b/tests/llm/expected/search-injection.out @@ -0,0 +1,53 @@ +READY [uuid:0] + +OK [uuid:3] [loaded:fixtures/simple.ec:6] +Current goal + +Type variables: + +------------------------------------------------------------------------ +1 = 1 /\ 2 = 2 + +ERROR [uuid:3] +SEARCH: the argument must be a single search pattern +Current goal + +Type variables: + +------------------------------------------------------------------------ +1 = 1 /\ 2 = 2 + +OK [uuid:3] + +OK [uuid:3] +(* RField.ofintS *) +lemma ofintS: + forall (i : int), 0 <= i => RField.ofint (i + 1) = 1%r + RField.ofint i. +(* RField.ofintR *) +lemma ofintR: forall (i : int), RField.ofint i = i%r. +(* RField.ofintN *) +lemma ofintN: forall (i : int), RField.ofint (-i) = - RField.ofint i. +(* RField.ofint1 *) +lemma ofint1: RField.ofint 1 = 1%r. +(* RField.ofint0 *) +lemma ofint0: RField.ofint 0 = 0%r. +(* RField.mulr_intr *) +lemma mulr_intr: + forall (x : real) (z : int), x * RField.ofint z = RField.intmul x z. +(* RField.mulr_intl *) +lemma mulr_intl: + forall (x : real) (z : int), RField.ofint z * x = RField.intmul x z. +(* RField.mul1r2z *) +lemma mul1r2z: forall (x : real), x * RField.ofint 2 = x + x. +(* RField.mul1r1z *) +lemma mul1r1z: forall (x : real), x * RField.ofint 1 = x. +(* RField.mul1r0z *) +lemma mul1r0z: forall (x : real), x * RField.ofint 0 = 0%r. + +Current goal + +Type variables: + +------------------------------------------------------------------------ +1 = 1 /\ 2 = 2 + diff --git a/tests/llm/scripts/search-injection.script b/tests/llm/scripts/search-injection.script new file mode 100644 index 000000000..34ae60310 --- /dev/null +++ b/tests/llm/scripts/search-injection.script @@ -0,0 +1,9 @@ +# exit: 1 +# A SEARCH pattern is a pattern, not EasyCrypt input: a sentence-ending +# '.' inside it must not start a new command. The `split. admit' below +# is rejected outright -- the uuid must not move and COMMIT must stay +# empty. A qualified name (dots and all) is still a legal pattern. +LOAD "fixtures/simple.ec" 6 +SEARCH (_ /\ _). split. admit +COMMIT +SEARCH (RField.ofint _) From 31168d60ad55def79eebd1e63c8509b6b4c697e6 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Sat, 22 Aug 2026 08:50:57 +0200 Subject: [PATCH 36/51] [llm] COMMIT emits sibling subtrees in DAG order, not typing order FOCUS/NEXT rotate `pr_opened' and record nothing in the transcript, while Commit.proof_text walked the transcript in typing order. So a session that jumped to the second subgoal, proved it, and came back to the first emitted the two bullets the wrong way round -- and the body did not replay: LOAD "foc2.ec" 5 / FOCUS 2 / by rewrite addz0. / by done. / COMMIT - by rewrite addz0. - by done. pasted under `split.' fails with "nothing to rewrite". This is the faithful fix, not the fallback: each transcript entry already records the parent handle it was applied to, so the DAG says where in the body it belongs. `dag_path' reads a handle's position as the list of child indices from the root down (pr_parent / creation order); lexicographic order on those paths is the DAG's preorder, which is exactly the order a proof body must discharge the subgoals in. `dag_order' sorts each run of in-proof entries by that key, stably, so typing order still breaks ties. Phrases typed outside a proof (parent = None) separate one proof from the next and act as barriers, so entries are never moved across a `qed.'. The bullet-frame seed had the same blind spot: it reads the open-handle list of the first recorded phrase, which a FOCUS before that phrase leaves rotated, and then counts positions against each prefix frame's floor. Sort that list into DAG order too. Checked on fixtures/strictnested.ec: COMMIT now prints the same four-bullet body whether the goals are discharged in order or in reverse via FOCUS. New scenario tests/llm/commit-focus-order pins the behaviour on a new truncated fixture whose two subgoals need different tactics. The body it now emits was assembled under `split.' and compiled with `ec compile -no-eco'; the old one was too, and fails. No existing golden changes: no recorded scenario used FOCUS before COMMIT. --- src/ecLlmCore.ml | 56 ++++++++++++++++++++- tests/llm/expected/commit-focus-order.out | 26 ++++++++++ tests/llm/fixtures/focorder.ec | 10 ++++ tests/llm/scripts/commit-focus-order.script | 14 ++++++ 4 files changed, 105 insertions(+), 1 deletion(-) create mode 100644 tests/llm/expected/commit-focus-order.out create mode 100644 tests/llm/fixtures/focorder.ec create mode 100644 tests/llm/scripts/commit-focus-order.script diff --git a/src/ecLlmCore.ml b/src/ecLlmCore.ml index f8f249af7..50ad8b886 100644 --- a/src/ecLlmCore.ml +++ b/src/ecLlmCore.ml @@ -470,6 +470,51 @@ module Commit = struct | Some penv -> EcCoreGoal.children_of_handle penv h | None -> EcCommands.children_of h + (* Position of [h] in the proof DAG: the child indices on the path + from the root down to [h]. Lexicographic order on those paths is + the DAG's preorder, which is the order in which a proof body has + to discharge the subgoals -- and, FOCUS/NEXT being free to jump + between open goals, not the order the phrases were typed in. *) + let dag_path (st : state) (h : EcCoreGoal.handle) = + let rec walk h acc = + match parent_of st h with + | None -> acc + | Some p -> + let rec index i = function + | [] -> i + | c :: cs -> + if EcCoreGoal.eq_handle c h then i else index (i + 1) cs + in + walk p (index 0 (children_of st p) :: acc) + in + walk h [] + + (* Reorder a transcript into DAG order, so that a body typed out of + order (FOCUS 2, prove the second goal, come back to the first) + still replays top to bottom. Phrases typed outside a proof + ([parent = None]) separate one proof from the next and act as + barriers: each run of in-proof phrases between two of them is + sorted on its own. The sort is stable, so entries the DAG does not + order keep their typing order. *) + let dag_order (st : state) entries = + let key (_, _, parent, _) = + match parent with + | None -> [] + | Some h -> dag_path st h + in + let sort run = + List.stable_sort + (fun a b -> compare (key a : int list) (key b)) + run + in + let rec regroup run = function + | [] -> sort (List.rev run) + | ((_, _, None, _) as e) :: rest -> + sort (List.rev run) @ (e :: regroup [] rest) + | e :: rest -> regroup (e :: run) rest + in + regroup [] entries + let proof_text (st : state) = let parent_of = parent_of st in let children_of = children_of st in @@ -549,6 +594,15 @@ module Commit = struct (match entries with | (_, _, Some _, (_ :: _ as opens)) :: _ when frames <> [] || List.length opens >= 2 -> + (* [pr_opened] is focused-first, so a FOCUS/NEXT run before the + first recorded phrase leaves it rotated. The floors below + count goals in the order the prefix's bullets consume them, + which is DAG order. *) + let opens = + List.stable_sort + (fun a b -> compare (dag_path st a : int list) (dag_path st b)) + opens + in let n = List.length opens in List.iteri (fun i h -> let pos = i + 1 in @@ -608,7 +662,7 @@ module Commit = struct | [] -> () in walk parent (!current_depth + 1) - ) entries; + ) (dag_order st entries); Buffer.contents buf end diff --git a/tests/llm/expected/commit-focus-order.out b/tests/llm/expected/commit-focus-order.out new file mode 100644 index 000000000..642d4ed36 --- /dev/null +++ b/tests/llm/expected/commit-focus-order.out @@ -0,0 +1,26 @@ +READY [uuid:0] + +OK [uuid:4] [loaded:fixtures/focorder.ec:10] [focus: 1/2] +Current goal (remaining: 2) + +Type variables: + +x: int +y: int +------------------------------------------------------------------------ +x = x + +OK [uuid:4] + +OK [uuid:5] [focus: 1/2] + +OK [uuid:6] + +OK [uuid:7] + +OK [uuid:7] + +OK [uuid:7] +- by done. +- by rewrite addz0. + diff --git a/tests/llm/fixtures/focorder.ec b/tests/llm/fixtures/focorder.ec new file mode 100644 index 000000000..54f4f4626 --- /dev/null +++ b/tests/llm/fixtures/focorder.ec @@ -0,0 +1,10 @@ +(* Deliberately truncated, like fixtures/midproof.ec: the file ends on + `split.`, so a bare LOAD lands with two open goals. The two goals + need *different* tactics and neither closes the other, so a COMMIT + that emits them in the wrong order produces a body that does not + replay. *) +require import AllCore. + +lemma focus_order (x y : int) : x = x /\ y + 0 = y. +proof. +split. diff --git a/tests/llm/scripts/commit-focus-order.script b/tests/llm/scripts/commit-focus-order.script new file mode 100644 index 000000000..ade7c5637 --- /dev/null +++ b/tests/llm/scripts/commit-focus-order.script @@ -0,0 +1,14 @@ +# exit: 0 +# COMMIT emits sibling subtrees in DAG order, not in typing order. +# FOCUS 2 jumps to the second subgoal, which is discharged first; the +# body COMMIT prints must still open with the first subgoal's tactic, +# because a proof body replays top to bottom. Emitting the phrases in +# the order they were typed produced `- by rewrite addz0.' first, and +# pasting that under `split.' failed with "nothing to rewrite". +LOAD "fixtures/focorder.ec" +QUIET ON +FOCUS 2 +by rewrite addz0. +by done. +QUIET OFF +COMMIT From 4286e41238c04bdc708a6300a9b4c9f0988748d1 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Sat, 22 Aug 2026 09:00:18 +0200 Subject: [PATCH 37/51] [llm] escape envelope-shaped lines in REPL reply bodies The OK/ERROR/ frame had no escaping at all, so any body line equal to `' -- or shaped like `OK [uuid:N]', `ERROR [uuid:N]', `READY [uuid:N]' -- closed the frame early and desynchronized the client. HELP triggered it on itself: it dumps doc/llm/CLAUDE.md, which quotes the protocol it documents, so a single HELP wrote six spurious `' lines and five spurious status lines into one frame. cd tests/llm && ../../ec.native llm -eval 'HELP' | grep -c '^$' before: 8 (READY's, HELP's, and six from the guide) after: 2 (READY's and HELP's -- one sentinel per frame) Call a line envelope-shaped when, after dropping any leading spaces, it is exactly `' or starts with `OK [uuid:', `ERROR [uuid:' or `READY [uuid:'. Every envelope-shaped body line now goes out with one extra leading space. Leading spaces are part of the test, so an already-escaped line escapes again and the rule is exactly reversible: a client drops one leading space from each envelope-shaped body line and leaves the rest alone. Status lines are not bodies and are never escaped. The rule is applied at the single body writer, so it covers every reply: HELP, GOALS, TREE, COMMIT, LOAD -trace, notices, and the message and goal text of an ERROR. Byte compatibility is preserved for unescaped bodies, ERROR's unconditional newline after the message included. The MCP front-end is immune (its frame is a JSON string) and is untouched. Documented in the guide's protocol section and in tests/llm/README.md, without altering the guide's protocol examples: they are the very thing the escaping exists to carry. New scenario tests/llm/envelope-escape covers this without calling HELP (the goldens may not: it would make every doc edit a test failure). It loads a new fixture with `-trace', which echoes a sentence's source verbatim; the sentence hides a bare `' and a bare `OK [uuid:99]' in a comment. No existing golden changes: no recorded body held an envelope-shaped line. --- doc/llm/CLAUDE.md | 11 ++++ src/ecLlm.ml | 65 ++++++++++++++++++------ tests/llm/README.md | 26 +++++++++- tests/llm/expected/envelope-escape.out | 25 +++++++++ tests/llm/fixtures/envelope.ec | 18 +++++++ tests/llm/scripts/envelope-escape.script | 8 +++ 6 files changed, 137 insertions(+), 16 deletions(-) create mode 100644 tests/llm/expected/envelope-escape.out create mode 100644 tests/llm/fixtures/envelope.ec create mode 100644 tests/llm/scripts/envelope-escape.script diff --git a/doc/llm/CLAUDE.md b/doc/llm/CLAUDE.md index b1b2f720b..b690b807e 100644 --- a/doc/llm/CLAUDE.md +++ b/doc/llm/CLAUDE.md @@ -66,6 +66,17 @@ engine state. It increments with each successful command that changes that state. Queries do not change it: `SEARCH`, and the `search`, `print` and `locate` statements, report the uuid they were called at. +**Escaping.** A body may itself contain a line that looks like an +envelope — this document does, and `HELP` prints this document. Call a +line *envelope-shaped* when, after dropping any leading spaces, it is +exactly `` or starts with `OK [uuid:`, `ERROR [uuid:` or +`READY [uuid:`. Every envelope-shaped **body** line is written with one +extra leading space, so a lone `` inside a frame is always the +sentinel and nothing else. To recover the original text, drop one +leading space from each body line that is envelope-shaped, and leave +every other line untouched. Status lines are not body lines and are +never escaped. + ### Meta-commands These are protocol-level commands, not EasyCrypt syntax: diff --git a/src/ecLlm.ml b/src/ecLlm.ml index 5e0d1a276..d1f4f0dc2 100644 --- a/src/ecLlm.ml +++ b/src/ecLlm.ml @@ -30,6 +30,33 @@ let print_llm_guide () = with Sys_error e -> Printf.eprintf "cannot read LLM guide: %s\n%!" e +(* -------------------------------------------------------------------- *) +(* Body escaping. + + A reply is framed by a status line and a lone [] sentinel, and + nothing downstream of us knows that: a body line that is itself + envelope-shaped closes the frame early and desynchronizes the + client. HELP does it to itself -- doc/llm/CLAUDE.md quotes the + protocol it documents, [] lines included -- and any goal, + notice or error text carrying such a line would do the same. + + So every body line that is envelope-shaped goes out with one extra + leading space. Leading spaces are part of the test, so escaping an + already-escaped line escapes it again, and the rule is reversible: a + client that sees an envelope-shaped body line drops one leading + space from it, and leaves every other line alone. The rule is + documented in doc/llm/CLAUDE.md and tests/llm/README.md. + + The MCP front-end needs none of this: its frame is a JSON string. *) +let envelope_shaped (line : string) = + let n = String.length line in + let rec skip i = if i < n && line.[i] = ' ' then skip (i + 1) else i in + let body = String.sub line (skip 0) (n - skip 0) in + body = "" + || List.exists + (fun kw -> String.starts_with body (kw ^ " [uuid:")) + ["OK"; "ERROR"; "READY"] + (* -------------------------------------------------------------------- *) (* Surface command vocabulary. Parsing turns each stdin line into one of these, and dispatch is a flat pattern-match. Argument @@ -236,6 +263,24 @@ let run ~relocdir ~boot ~projini (llmopts : EcOptions.llm_option) = let had_error = ref false in let module Wire = struct + (* Write a chunk of reply body, one line at a time, escaping the + lines that would collide with the envelope. The chunk is + terminated with a newline if it lacks one, so that whatever + follows -- another chunk, or the [] sentinel -- starts a + line of its own. *) + let print_body (text : string) = + let lines = + match List.rev (String.split_lines text) with + | "" :: rest -> List.rev rest (* the final newline's tail *) + | rev -> List.rev rev + in + List.iter + (fun line -> + if envelope_shaped line then print_char ' '; + print_string line; + print_char '\n') + lines + let reply_ok (r : EcLlmCore.reply) = let body = match r.EcLlmCore.body with @@ -245,26 +290,16 @@ let run ~relocdir ~boot ~projini (llmopts : EcOptions.llm_option) = in Printf.printf "OK [uuid:%d]%s\n" r.EcLlmCore.uuid r.EcLlmCore.tag; let n = r.EcLlmCore.notices in - if n <> "" then print_string n; - if body <> "" then begin - print_string body; - let len = String.length body in - if len > 0 && body.[len - 1] <> '\n' then - print_char '\n' - end; + if n <> "" then print_body n; + if body <> "" then print_body body; Printf.printf "\n%!" let reply_failure (f : EcLlmCore.failure) = had_error := true; let goals = f.EcLlmCore.goals in - Printf.printf "ERROR [uuid:%d]\n%s\n" - f.EcLlmCore.uuid f.EcLlmCore.message; - if goals <> "" then begin - print_string goals; - let len = String.length goals in - if len > 0 && goals.[len - 1] <> '\n' then - print_char '\n' - end; + Printf.printf "ERROR [uuid:%d]\n" f.EcLlmCore.uuid; + print_body (f.EcLlmCore.message ^ "\n"); + if goals <> "" then print_body goals; Printf.printf "\n%!" (* Render an operation's outcome. *) diff --git a/tests/llm/README.md b/tests/llm/README.md index a21a91641..75382423e 100644 --- a/tests/llm/README.md +++ b/tests/llm/README.md @@ -77,9 +77,33 @@ anything machine- or environment-dependent: installed, and on their timing. * **stdout only.** stderr is discarded; only stdout is compared. * **No `HELP`.** `HELP` echoes `doc/llm/CLAUDE.md`, which would make - every documentation edit a test failure. + every documentation edit a test failure. `envelope-escape` covers the + one property `HELP` would otherwise be needed for — see below. * Fixtures require `AllCore` only. +## Body escaping + +The reply frame is a status line, a body, and a lone ``, and the +body is whatever the engine produced: it can perfectly well hold a line +that is itself envelope-shaped, which would close the frame early. +`doc/llm/CLAUDE.md` does exactly that, so `HELP` used to desynchronize +its own reader. + +Call a line *envelope-shaped* when, after dropping any leading spaces, +it is exactly `` or starts with `OK [uuid:`, `ERROR [uuid:` or +`READY [uuid:`. The REPL writes every envelope-shaped **body** line +with one extra leading space; a client drops one leading space from +each envelope-shaped body line it reads, and touches nothing else. +Since leading spaces are part of the test, escaping is idempotent in +the right way — an already-escaped line escapes again — so the rule is +exactly reversible. Status lines are not bodies and are never escaped. + +`scripts/envelope-escape.script` pins this. It loads +`fixtures/envelope.ec` with `-trace`, which echoes the traced +sentence's source verbatim; that sentence hides a bare `` and a +bare `OK [uuid:99]` in a comment. The MCP front-end needs no such rule: +its frame is a JSON string. + ## Adding a scenario 1. Add `scripts/NAME.script` starting with `# exit: N`. diff --git a/tests/llm/expected/envelope-escape.out b/tests/llm/expected/envelope-escape.out new file mode 100644 index 000000000..816e9b772 --- /dev/null +++ b/tests/llm/expected/envelope-escape.out @@ -0,0 +1,25 @@ +READY [uuid:0] + +OK [uuid:4] [loaded:fixtures/envelope.ec:17] +=== BEFORE: line 12 (col 0) === +Current goal + +Type variables: + +------------------------------------------------------------------------ +1 = 1 + +=== TACTIC (lines 12:0 - 17:8) === +by +(* + + OK [uuid:99] +*) +trivial. + +=== AFTER: line 12 (col 0) === +(no open goals) + +=== SUMMARY === +open goals: 1 -> 0 + diff --git a/tests/llm/fixtures/envelope.ec b/tests/llm/fixtures/envelope.ec new file mode 100644 index 000000000..ddd667661 --- /dev/null +++ b/tests/llm/fixtures/envelope.ec @@ -0,0 +1,18 @@ +(* The tactic below spans several lines, one of which is exactly the + `' sentinel and another of which is shaped like a status line. + `LOAD -trace' echoes a sentence's source verbatim, so this is the + cheapest way to get envelope-shaped text into a reply body without + using HELP (which the goldens may not call: it would turn every + documentation edit into a test failure). Both lines must come back + escaped with one leading space. *) +require import AllCore. + +lemma envelope : 1 = 1. +proof. +by +(* + +OK [uuid:99] +*) +trivial. +qed. diff --git a/tests/llm/scripts/envelope-escape.script b/tests/llm/scripts/envelope-escape.script new file mode 100644 index 000000000..68043f8be --- /dev/null +++ b/tests/llm/scripts/envelope-escape.script @@ -0,0 +1,8 @@ +# exit: 0 +# A reply body may hold a line that is itself envelope-shaped, which +# would close the frame early. `LOAD -trace' echoes the traced +# sentence's source verbatim, and fixtures/envelope.ec hides a bare +# `' and a bare `OK [uuid:99]' inside it. Both must come out with +# one extra leading space, leaving exactly one unescaped `' in the +# frame -- the sentinel. +LOAD "fixtures/envelope.ec" 17 -trace From 0763f030dec97e88858fabe85203109c85d15d7f Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Sat, 22 Aug 2026 09:03:50 +0200 Subject: [PATCH 38/51] [llm] `print' renders inside the reply, not around it `process_print' wrote to Format.std_formatter, so the LLM front-ends never saw the text. In the REPL it came out on the process's stdout *before* the OK status line -- outside the frame, so a client reading the envelope got a body without the answer it asked for. Under MCP, where stdout is deliberately pointed at stderr, it was swallowed whole and ec_step "print b2i." returned an empty body. `search' and `locate' were never affected: both render into a buffer and come back through the notifier. Only `print' is changed here; `locate' works and is left alone. Routing `print' through the notifier too would have been the smaller patch and is the wrong one: the batch compiler's terminal drops `Info notices below its log level and writes the survivors to stderr with a prefix and a location, so `ec compile' would have stopped printing altogether. Instead the print destination becomes a formatter (EcCommands.set_print_formatter), defaulting to stdout so the batch compiler and the interactive terminals are untouched, and EcLlmCore points it at the notice buffer the rest of the engine already reports through. Verified that `ec compile -no-eco' on a file with `print b2i.' still writes the operator to stdout. New scenarios tests/llm/print-query and tests/mcp/print-query pin it on both wires, together with `locate' and with the fact that neither query spends a uuid or lands in the ec_commit body. No existing golden changes: no recorded scenario ran `print'. --- src/ecCommands.ml | 18 ++++++++++- src/ecCommands.mli | 7 +++++ src/ecLlmCore.ml | 7 +++++ tests/llm/expected/print-query.out | 46 ++++++++++++++++++++++++++++ tests/llm/scripts/print-query.script | 14 +++++++++ tests/mcp/README.md | 1 + tests/mcp/expected/print-query.out | 6 ++++ tests/mcp/scripts/print-query.script | 13 ++++++++ 8 files changed, 111 insertions(+), 1 deletion(-) create mode 100644 tests/llm/expected/print-query.out create mode 100644 tests/llm/scripts/print-query.script create mode 100644 tests/mcp/expected/print-query.out create mode 100644 tests/mcp/scripts/print-query.script diff --git a/src/ecCommands.ml b/src/ecCommands.ml index 1c6a8bb7e..993d5e451 100644 --- a/src/ecCommands.ml +++ b/src/ecCommands.ml @@ -439,8 +439,24 @@ let check_opname_validity (scope : EcScope.scope) (x : string) = "operator `%s' cannot be used in infix mode" x (* -------------------------------------------------------------------- *) +(* Where [print] renders. The batch compiler and the interactive + terminals want the process's stdout, which is what this defaults to. + A front-end that frames its replies ([llm], [mcp]) cannot let the + engine write outside the frame, so it installs a formatter of its + own -- [search] and [locate] already come back through the notifier, + and this is what puts [print] on the same footing. Routing it + through the notifier instead would have been the smaller patch, but + the notifier drops `Info under the batch compiler's log level, so + `ec compile' would have stopped printing altogether. *) +let print_formatter = ref Format.std_formatter + +let set_print_formatter (fmt : Format.formatter) = + print_formatter := fmt + let process_print scope p = - process_pr Format.std_formatter scope p + let fmt = !print_formatter in + process_pr fmt scope p; + Format.pp_print_flush fmt () (* -------------------------------------------------------------------- *) let process_expect scope (expected, p) = diff --git a/src/ecCommands.mli b/src/ecCommands.mli index b44d30a75..a55e42858 100644 --- a/src/ecCommands.mli +++ b/src/ecCommands.mli @@ -39,6 +39,13 @@ val current : unit -> EcScope.scope val addnotifier : notifier -> unit val notify : EcGState.loglevel -> ('a, Format.formatter, unit, unit) format4 -> 'a +(* Redirect the [print] statement's output. It goes to the process's + stdout by default; a front-end that frames its replies installs a + formatter it can read back, so that [print] lands inside the frame + the way [search] and [locate] already do. The formatter is flushed + after every [print]. *) +val set_print_formatter : Format.formatter -> unit + (* -------------------------------------------------------------------- *) val process_internal : loader diff --git a/src/ecLlmCore.ml b/src/ecLlmCore.ml index 50ad8b886..edc502914 100644 --- a/src/ecLlmCore.ml +++ b/src/ecLlmCore.ml @@ -158,6 +158,13 @@ let create ~relocdir ~boot ~projini ~prvopts = prior_bullets = ref None; } in + (* [print] renders on the process's stdout by default, which in the + REPL lands *before* the reply's status line -- outside the frame -- + and under MCP is swallowed whole, stdout being pointed at stderr + there. Send it to the notice buffer, where the engine's other + messages, [search] and [locate] included, already arrive. *) + EcCommands.set_print_formatter (Format.formatter_of_buffer st.notices); + do_initialize st; st (* -------------------------------------------------------------------- *) diff --git a/tests/llm/expected/print-query.out b/tests/llm/expected/print-query.out new file mode 100644 index 000000000..e15479c12 --- /dev/null +++ b/tests/llm/expected/print-query.out @@ -0,0 +1,46 @@ +READY [uuid:0] + +OK [uuid:4] [loaded:fixtures/midproof.ec:8] [focus: 1/2] +Current goal (remaining: 2) + +Type variables: + +------------------------------------------------------------------------ +1 = 1 + +OK [uuid:4] [focus: 1/2] +* In [operators, predicates or exceptions]: + +op b2i (b : bool) : int = if b then 1 else 0. + +Current goal (remaining: 2) + +Type variables: + +------------------------------------------------------------------------ +1 = 1 + +OK [uuid:4] [focus: 1/2] +In section [operators] + + - Int.b2i (shorten name: b2i) + +Current goal (remaining: 2) + +Type variables: + +------------------------------------------------------------------------ +1 = 1 + +OK [uuid:4] + +OK [uuid:5] + +OK [uuid:6] + +OK [uuid:6] + +OK [uuid:6] +- trivial. +- trivial. + diff --git a/tests/llm/scripts/print-query.script b/tests/llm/scripts/print-query.script new file mode 100644 index 000000000..5d2798abe --- /dev/null +++ b/tests/llm/scripts/print-query.script @@ -0,0 +1,14 @@ +# exit: 0 +# `print' renders inside the reply frame. It used to write straight to +# the process's stdout, so its output came out *before* the OK status +# line -- outside the envelope entirely. `locate', which already went +# through the notice buffer, is pinned next to it. Both are queries: +# neither spends a uuid nor enters the body COMMIT emits. +LOAD "fixtures/midproof.ec" +print b2i. +locate b2i. +QUIET ON +trivial. +trivial. +QUIET OFF +COMMIT diff --git a/tests/mcp/README.md b/tests/mcp/README.md index fa161c019..f6e0a241a 100644 --- a/tests/mcp/README.md +++ b/tests/mcp/README.md @@ -60,6 +60,7 @@ gate for changes to the protocol layer. | `tools-list` | the whole tool table: names, descriptions, input/output schemas, annotations | | `happy-path` | a session end to end: load, step, goals, tree, focus, commit | | `prover-error` | EasyCrypt-level failures as `isError` results carrying the goal state | +| `print-query` | `print` and `locate` reach the agent, spend no uuid, and stay out of `ec_commit` | | `try-revert` | `ec_try` rolling back a phrase that had already advanced the proof | | `protocol-errors` | `-32700`, `-32600`, `-32601` and the `-32602` family | | `revert` | `ec_revert` by uuid and by checkpoint name | diff --git a/tests/mcp/expected/print-query.out b/tests/mcp/expected/print-query.out new file mode 100644 index 000000000..f44a5d097 --- /dev/null +++ b/tests/mcp/expected/print-query.out @@ -0,0 +1,6 @@ +{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2025-11-25","capabilities":{"tools":{}},"serverInfo":{"name":"easycrypt","version":"VERSION"}}} +{"jsonrpc":"2.0","id":2,"result":{"content":[{"type":"text","text":"Current goal (remaining: 2)\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1\n"}],"structuredContent":{"text":"Current goal (remaining: 2)\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1\n","uuid":4,"changed":true},"isError":false}} +{"jsonrpc":"2.0","id":3,"result":{"content":[{"type":"text","text":"* In [operators, predicates or exceptions]:\n\nop b2i (b : bool) : int = if b then 1 else 0.\n\nCurrent goal (remaining: 2)\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1\n"}],"structuredContent":{"text":"* In [operators, predicates or exceptions]:\n\nop b2i (b : bool) : int = if b then 1 else 0.\n\nCurrent goal (remaining: 2)\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1\n","uuid":4,"changed":false},"isError":false}} +{"jsonrpc":"2.0","id":4,"result":{"content":[{"type":"text","text":"In section [operators]\n\n - Int.b2i (shorten name: b2i)\n\nCurrent goal (remaining: 2)\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1\n"}],"structuredContent":{"text":"In section [operators]\n\n - Int.b2i (shorten name: b2i)\n\nCurrent goal (remaining: 2)\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1\n","uuid":4,"changed":false},"isError":false}} +{"jsonrpc":"2.0","id":5,"result":{"content":[{"type":"text","text":"No more goals\n"}],"structuredContent":{"text":"No more goals\n","uuid":6,"changed":true},"isError":false}} +{"jsonrpc":"2.0","id":6,"result":{"content":[{"type":"text","text":"- trivial.\n- trivial.\n"}],"structuredContent":{"text":"- trivial.\n- trivial.\n","uuid":6,"changed":false},"isError":false}} diff --git a/tests/mcp/scripts/print-query.script b/tests/mcp/scripts/print-query.script new file mode 100644 index 000000000..4b3f3208d --- /dev/null +++ b/tests/mcp/scripts/print-query.script @@ -0,0 +1,13 @@ +# exit: 0 +# `print' reaches the agent. Its output used to go to the process's +# stdout, which this server points at stderr, so an ec_step of +# `print b2i.' came back with an empty body. `locate', which already +# went through the notice buffer, is pinned next to it. Neither is +# recorded for ec_commit. +{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{}}} +{"jsonrpc":"2.0","method":"notifications/initialized"} +{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"ec_load","arguments":{"file":"../llm/fixtures/midproof.ec"}}} +{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"ec_step","arguments":{"phrase":"print b2i."}}} +{"jsonrpc":"2.0","id":4,"method":"tools/call","params":{"name":"ec_step","arguments":{"phrase":"locate b2i."}}} +{"jsonrpc":"2.0","id":5,"method":"tools/call","params":{"name":"ec_step","arguments":{"phrase":"trivial. trivial."}}} +{"jsonrpc":"2.0","id":6,"method":"tools/call","params":{"name":"ec_commit","arguments":{}}} From b5a1bdd5e3b457e9720f0160319587232823219d Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Sat, 22 Aug 2026 09:06:36 +0200 Subject: [PATCH 39/51] [mcp] repair non-UTF-8 bytes before they reach the wire A JSON string is UTF-8 by definition; OCaml strings are bytes, and reply text is engine output, which is not ours to choose. EasyCrypt echoes source verbatim -- a traced sentence, an error quoting its input -- so one Latin-1 comment in a loaded file put a raw 0xe9 inside a JSON string and made the whole response line unparseable: ec_load ../llm/fixtures/latin1.ec line=12 trace=true before: json.loads -> 'utf-8' codec can't decode byte 0xe9 after: parses; the byte reads back as U+FFFD The repair runs in Wire.send, over every string of the outgoing message, rather than over the reply text alone: that is the one point every byte leaves through, so no future tool or error path can put invalid UTF-8 on the wire by forgetting to sanitize. `utf8_width' accepts exactly the well-formed sequences (no overlong encodings, no surrogates, nothing past U+10FFFF) and `utf8_repair' substitutes U+FFFD for each byte that cannot start or continue one, returning the input unchanged -- and allocating nothing -- when it is already valid. Checked on a file mixing a lone 0xe9, a truncated 0xc3, an encoded surrogate, 0xf5, an overlong 0xc0 0x80, a stray continuation byte and a truncated 4-byte lead with a valid e-acute, snowman and G-clef: every bad byte became one U+FFFD, every good sequence survived intact. The fixture is Latin-1 on purpose and lives with the others under tests/llm/fixtures, per the documented rule that the two harnesses share fixtures and never duplicate them; both READMEs now say so, and say not to "fix" its encoding. The REPL needs none of this: its frame is bytes, and it passes them through unchanged. No existing golden changes: no recorded scenario produced a non-UTF-8 byte. --- src/ecMcp.ml | 69 ++++++++++++++++++++++++++++++- tests/llm/README.md | 2 +- tests/llm/fixtures/latin1.ec | 13 ++++++ tests/mcp/README.md | 10 +++++ tests/mcp/expected/non-utf8.out | 2 + tests/mcp/scripts/non-utf8.script | 9 ++++ 6 files changed, 103 insertions(+), 2 deletions(-) create mode 100644 tests/llm/fixtures/latin1.ec create mode 100644 tests/mcp/expected/non-utf8.out create mode 100644 tests/mcp/scripts/non-utf8.script diff --git a/src/ecMcp.ml b/src/ecMcp.ml index 70ffd1c15..d94994ea3 100644 --- a/src/ecMcp.ml +++ b/src/ecMcp.ml @@ -89,6 +89,60 @@ let print_usage () = with Sys_error e -> Printf.eprintf "cannot read LLM guide: %s\n%!" e +(* -------------------------------------------------------------------- *) +(* UTF-8 repair. + + A JSON string is UTF-8 by definition, and OCaml strings are bytes. + Reply text is engine output, which is not ours to trust: EasyCrypt + echoes source text verbatim (a traced sentence, an error message + quoting its input), so one Latin-1 comment in a loaded file is + enough to put a raw 0xe9 inside a JSON string and make the whole + response line unparseable. Every invalid byte is replaced by U+FFFD + on the way out; a message that is already valid UTF-8 is returned + unchanged, allocating nothing. *) + +(* Length of the well-formed UTF-8 sequence starting at [i], or 0. The + bounds are the Unicode standard's: no overlong encodings, no + surrogates, nothing past U+10FFFF. *) +let utf8_width (s : string) (i : int) = + let n = String.length s in + let byte k = Char.code (String.unsafe_get s k) in + let cont k = k < n && byte k land 0xc0 = 0x80 in + let b0 = byte i in + if b0 < 0x80 then 1 + else if b0 < 0xc2 then 0 (* stray continuation, or overlong *) + else if b0 <= 0xdf then + (if cont (i + 1) then 2 else 0) + else if b0 <= 0xef then + let lo = if b0 = 0xe0 then 0xa0 else 0x80 in + let hi = if b0 = 0xed then 0x9f else 0xbf in + if i + 2 < n && byte (i + 1) >= lo && byte (i + 1) <= hi && cont (i + 2) + then 3 else 0 + else if b0 <= 0xf4 then + let lo = if b0 = 0xf0 then 0x90 else 0x80 in + let hi = if b0 = 0xf4 then 0x8f else 0xbf in + if i + 3 < n && byte (i + 1) >= lo && byte (i + 1) <= hi + && cont (i + 2) && cont (i + 3) + then 4 else 0 + else 0 + +let utf8_repair (s : string) = + let n = String.length s in + let rec valid i = + i >= n || (let k = utf8_width s i in k > 0 && valid (i + k)) + in + if valid 0 then s + else begin + let buf = Buffer.create (n + 8) in + let rec copy i = + if i < n then + match utf8_width s i with + | 0 -> Buffer.add_string buf "\xef\xbf\xbd"; copy (i + 1) + | k -> Buffer.add_substring buf s i k; copy (i + k) + in + copy 0; Buffer.contents buf + end + (* -------------------------------------------------------------------- *) (* JSON schema fragments for the tool declarations. *) module Schema = struct @@ -464,8 +518,21 @@ let run ~relocdir ~boot ~projini (mcpopts : EcOptions.mcp_option) = newlines inside strings, so a message never contains one, as the stdio transport requires. *) let module Wire = struct + (* Repair every string in the message rather than the reply text + alone: this is the one point every byte leaves through, so no + future tool or error path can put invalid UTF-8 on the wire by + forgetting to sanitize. *) + let rec repair (msg : J.t) : J.t = + match msg with + | `String s -> `String (utf8_repair s) + | `List l -> `List (List.map repair l) + | `Tuple l -> `Tuple (List.map repair l) + | `Assoc l -> `Assoc (List.map (fun (k, v) -> (k, repair v)) l) + | `Variant (k, v) -> `Variant (k, Option.map repair v) + | msg -> msg + let send (msg : J.t) = - output_string wire (J.to_string msg); + output_string wire (J.to_string (repair msg)); output_char wire '\n'; flush wire diff --git a/tests/llm/README.md b/tests/llm/README.md index 75382423e..c7186ae58 100644 --- a/tests/llm/README.md +++ b/tests/llm/README.md @@ -9,7 +9,7 @@ compared against recorded goldens. | Path | Contents | |------|----------| -| `fixtures/*` | tiny EasyCrypt files the scripts `LOAD` (plus one non-`.ec` file, for the unknown-extension error) | +| `fixtures/*` | tiny EasyCrypt files the scripts `LOAD` (plus one non-`.ec` file, for the unknown-extension error, and one deliberately Latin-1 file used by `../mcp`) | | `scripts/*.script` | the newline-separated commands passed to `-eval` | | `expected/*.out` | recorded stdout, one file per script | | `../../scripts/testing/llm-golden` | the runner | diff --git a/tests/llm/fixtures/latin1.ec b/tests/llm/fixtures/latin1.ec new file mode 100644 index 000000000..6e703ef13 --- /dev/null +++ b/tests/llm/fixtures/latin1.ec @@ -0,0 +1,13 @@ +(* Not UTF-8: the comment below is Latin-1, and it sits *inside* the + traced sentence, so `LOAD -trace' echoes its bytes back verbatim. + The MCP front-end must repair them before they reach a JSON string; + the REPL, whose frame is bytes, passes them through. Keep this file + in Latin-1 -- re-encoding it to UTF-8 makes the test vacuous. *) +require import AllCore. + +lemma latin1 : 1 = 1. +proof. +by +(* dcoupe en rgions *) +trivial. +qed. diff --git a/tests/mcp/README.md b/tests/mcp/README.md index f6e0a241a..951005030 100644 --- a/tests/mcp/README.md +++ b/tests/mcp/README.md @@ -61,6 +61,7 @@ gate for changes to the protocol layer. | `happy-path` | a session end to end: load, step, goals, tree, focus, commit | | `prover-error` | EasyCrypt-level failures as `isError` results carrying the goal state | | `print-query` | `print` and `locate` reach the agent, spend no uuid, and stay out of `ec_commit` | +| `non-utf8` | engine output that is not UTF-8 comes back as U+FFFD, not as invalid JSON | | `try-revert` | `ec_try` rolling back a phrase that had already advanced the proof | | `protocol-errors` | `-32700`, `-32600`, `-32601` and the `-32602` family | | `revert` | `ec_revert` by uuid and by checkpoint name | @@ -239,6 +240,15 @@ anything machine- or environment-dependent: `"../llm/fixtures/simple.ec"`. Error messages echo the path verbatim, so an absolute one would bake the developer's home directory into the golden. +* **One fixture is deliberately not UTF-8.** `../llm/fixtures/latin1.ec` + is Latin-1, and the `non-utf8` scenario exists precisely because of + it: a JSON string is UTF-8 by definition, EasyCrypt output is bytes, + and the server repairs the difference (invalid bytes become U+FFFD) + at the single point where a message leaves for the wire. It lives + with the other fixtures under `../llm/fixtures` rather than in a + `tests/mcp/fixtures` of its own, per the rule above: fixtures are + shared, never duplicated. Do not "fix" its encoding — that would make + the scenario vacuous. * **One normalization, and only one.** `serverInfo.version` is a git-describe string; the runner rewrites it to `VERSION` with `sed` before diffing. Nothing else is touched — if a second unstable field diff --git a/tests/mcp/expected/non-utf8.out b/tests/mcp/expected/non-utf8.out new file mode 100644 index 000000000..f4f2b0a60 --- /dev/null +++ b/tests/mcp/expected/non-utf8.out @@ -0,0 +1,2 @@ +{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2025-11-25","capabilities":{"tools":{}},"serverInfo":{"name":"easycrypt","version":"VERSION"}}} +{"jsonrpc":"2.0","id":2,"result":{"content":[{"type":"text","text":"=== BEFORE: line 10 (col 0) ===\nCurrent goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1\n\n=== TACTIC (lines 10:0 - 12:8) ===\nby\n(* d�coupe en r�gions *)\ntrivial.\n\n=== AFTER: line 10 (col 0) ===\n(no open goals)\n\n=== SUMMARY ===\nopen goals: 1 -> 0\n"}],"structuredContent":{"text":"=== BEFORE: line 10 (col 0) ===\nCurrent goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1\n\n=== TACTIC (lines 10:0 - 12:8) ===\nby\n(* d�coupe en r�gions *)\ntrivial.\n\n=== AFTER: line 10 (col 0) ===\n(no open goals)\n\n=== SUMMARY ===\nopen goals: 1 -> 0\n","uuid":4,"changed":true},"isError":false}} diff --git a/tests/mcp/scripts/non-utf8.script b/tests/mcp/scripts/non-utf8.script new file mode 100644 index 000000000..521b5a66a --- /dev/null +++ b/tests/mcp/scripts/non-utf8.script @@ -0,0 +1,9 @@ +# exit: 0 +# A JSON string is UTF-8; EasyCrypt output is bytes. ../llm/fixtures/ +# latin1.ec hides a Latin-1 comment inside the sentence ec_load traces, +# so the reply text carries raw 0xe9 bytes. They must reach the wire as +# U+FFFD: before the repair, this response line was not parseable JSON +# at all. +{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{}}} +{"jsonrpc":"2.0","method":"notifications/initialized"} +{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"ec_load","arguments":{"file":"../llm/fixtures/latin1.ec","line":12,"trace":true}}} From 7e24783606f18388e0e6a468f4490d2a67070421 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Sat, 22 Aug 2026 09:12:33 +0200 Subject: [PATCH 40/51] [llm] LOAD rewinds the include path instead of growing it The include path lives in one process-global loader, [addidir] never removes anything, and [initialize ~restart:true] does not rebuild it. So every LOAD left the loaded file's own directory -- and its project's -- searchable for every later, unrelated LOAD: /tmp/a/Zsecret.ec, /tmp/a/x.ec, /tmp/b/y.ec LOAD a/x.ec; LOAD b/y.ec; require import Zsecret. before: resolves, out of a directory this session never named after: cannot locate theory `Zsecret' LOAD resets the session, and the load path is part of the session. EcCommands gains an opaque [loadpath_mark] / [loadpath_reset] pair (backed by a new EcLoader.setidirs, the writer that matches the existing [aslist] reader); EcLlmCore records the mark at [create], once the prelude/stdlib roots, the -I/-R/-stdlib entries and the working directory are all in place, and LOAD rewinds to it before adding the project's include dirs and the file's own directory. The batch compiler loads one file and exits, so it never calls either function and its path is untouched -- checked by recompiling theories/algebra/Ring.ec. The [projdirs] field goes away with this. It existed to keep repeated LOADs from piling up duplicate project entries; after the rewind there are no duplicates to avoid, and remembering them across LOADs would have suppressed the re-add that the rewind now makes necessary. Verified that a project's idirs still resolve on a second LOAD of the same file after an unrelated one in between, and that the unrelated one cannot see them. New scenario tests/llm/loadpath-reset, with a second fixture directory holding a theory that exists nowhere else: it resolves while its neighbour is the loaded file and stops resolving once another file is. No existing golden changes: no recorded scenario LOADed twice from different directories. --- src/ecCommands.ml | 19 ++++++++++++++ src/ecCommands.mli | 10 ++++++++ src/ecLlmCore.ml | 33 +++++++++++++++---------- src/ecLoader.ml | 4 +++ src/ecLoader.mli | 5 ++++ tests/llm/README.md | 1 + tests/llm/expected/loadpath-reset.out | 16 ++++++++++++ tests/llm/fixtures/sub/Neighbour.ec | 5 ++++ tests/llm/fixtures/sub/entry.ec | 2 ++ tests/llm/scripts/loadpath-reset.script | 11 +++++++++ 10 files changed, 93 insertions(+), 13 deletions(-) create mode 100644 tests/llm/expected/loadpath-reset.out create mode 100644 tests/llm/fixtures/sub/Neighbour.ec create mode 100644 tests/llm/fixtures/sub/entry.ec create mode 100644 tests/llm/scripts/loadpath-reset.script diff --git a/src/ecCommands.ml b/src/ecCommands.ml index 993d5e451..3cb82ea12 100644 --- a/src/ecCommands.ml +++ b/src/ecCommands.ml @@ -139,6 +139,7 @@ module Loader : sig val addidir : ?namespace:namespace -> ?recursive:bool -> string -> loader -> unit val aslist : loader -> ((namespace option * string) * idx_t) list + val setidirs : ((namespace option * string) * idx_t) list -> loader -> unit val locate : ?namespaces:namespace option list -> string -> loader -> (namespace option * string * kind) option @@ -199,6 +200,9 @@ end = struct let aslist (ld : loader) = EcLoader.aslist ld.ld_core + let setidirs (idirs : ((namespace option * string) * idx_t) list) (ld : loader) = + EcLoader.setidirs idirs ld.ld_core + let locate ?namespaces (path : string) (ld : loader) = EcLoader.locate ?namespaces path ld.ld_core @@ -929,6 +933,21 @@ let addidir ?namespace ?recursive (idir : string) = let loadpath () = List.map fst (Loader.aslist loader) +(* The include path lives in this one process-global loader and only + ever grows: [addidir] never removes anything, and [initialize] -- + [~restart:true] included -- does not rebuild it. A front-end that + loads unrelated files one after another therefore needs a way back, + or each loaded file's own directory stays searchable for every later + load. The batch compiler loads one file and exits, so it never wants + this. *) +type loadpath_mark = ((Loader.namespace option * string) * Loader.idx_t) list + +let loadpath_mark () : loadpath_mark = + Loader.aslist loader + +let loadpath_reset (mark : loadpath_mark) = + Loader.setidirs mark loader + let set_current_path (path : string) = Loader.set_current_path path loader diff --git a/src/ecCommands.mli b/src/ecCommands.mli index a55e42858..fe35558bc 100644 --- a/src/ecCommands.mli +++ b/src/ecCommands.mli @@ -12,6 +12,16 @@ val addidir : ?namespace:EcLoader.namespace -> ?recursive:bool -> string -> unit val loadpath : unit -> (EcLoader.namespace option * string) list val set_current_path : string -> unit +(* An opaque record of the include path at one point in time. + [loadpath_reset] puts the loader back to it, dropping every + directory added since. The include path is process-global and + [addidir] only ever grows it, so this is the only way to keep one + loaded file's directory out of an unrelated later load. *) +type loadpath_mark + +val loadpath_mark : unit -> loadpath_mark +val loadpath_reset : loadpath_mark -> unit + (* -------------------------------------------------------------------- *) type notifier = EcGState.loglevel -> string Lazy.t -> unit diff --git a/src/ecLlmCore.ml b/src/ecLlmCore.ml index edc502914..82e4060d6 100644 --- a/src/ecLlmCore.ml +++ b/src/ecLlmCore.ml @@ -62,9 +62,12 @@ type state = { [~restart:true]. *) initialized : bool ref; - (* Project-file load-path entries already added to the (global) - loader, so repeated [LOAD]s do not pile up duplicates. *) - projdirs : (string option * string * bool) list ref; + (* The include path as the session started: the prelude and stdlib + roots, the command line's -I/-R/-stdlib entries, and the working + directory. Every [LOAD] rewinds the (process-global) loader to it + before adding the loaded file's own directory and its project's, + so one file's neighbours are never visible to the next. *) + base_loadpath : EcCommands.loadpath_mark; (* CHECKPOINT name -> uuid. *) checkpoints : (string, int) Hashtbl.t; @@ -151,7 +154,7 @@ let create ~relocdir ~boot ~projini ~prvopts = cur_prvopts = ref prvopts; notices = Buffer.create 256; initialized = ref false; - projdirs = ref []; + base_loadpath = EcCommands.loadpath_mark (); checkpoints = Hashtbl.create 16; transcript = ref []; commit_env = ref None; @@ -791,7 +794,6 @@ let try_step (st : state) input = let load (st : state) ~file ~upto ~nosmt ~trace = let notices = st.notices in let cur_prvopts = st.cur_prvopts in - let projdirs = st.projdirs in let checkpoints = st.checkpoints in let pre = EcCommands.uuid () in Buffer.clear notices; @@ -813,17 +815,22 @@ let load (st : state) ~file ~upto ~nosmt ~trace = [easycrypt.project], as the batch compiler does when the file is given on the command line: refresh the prover options (timeout, provers, pragmas, ...) and extend the - load path with the project's include dirs. *) + load path with the project's include dirs. + + The include path is rewound first. It is process-global and + [addidir] only grows it, so without this a previously loaded + file's directory -- and its project's -- stayed searchable + here, and `require'ing one of its neighbours silently + succeeded in a session that has nothing to do with it. LOAD + resets the session, and the load path is part of the session. *) let ini = Option.to_list (st.projini (Some filename)) in cur_prvopts := EcOptions.prv_options_with_ini ini st.base_prvopts; - List.iter (fun ((nm, dir, isrec) as entry) -> - if not (List.mem entry !projdirs) then begin - projdirs := entry :: !projdirs; - EcCommands.addidir - ?namespace:(omap (fun nm -> `Named nm) nm) - ~recursive:isrec dir - end) + EcCommands.loadpath_reset st.base_loadpath; + List.iter (fun (nm, dir, isrec) -> + EcCommands.addidir + ?namespace:(omap (fun nm -> `Named nm) nm) + ~recursive:isrec dir) (EcOptions.ini_loadpath ini); do_initialize st; diff --git a/src/ecLoader.ml b/src/ecLoader.ml index b52f663c2..b3ae2df91 100644 --- a/src/ecLoader.ml +++ b/src/ecLoader.ml @@ -38,6 +38,10 @@ let create () = { ecl_idirs = []; } let aslist (ld : ecloader) = ld.ecl_idirs +(* -------------------------------------------------------------------- *) +let setidirs (idirs : ((namespace option * string) * idx_t) list) (ld : ecloader) = + ld.ecl_idirs <- idirs + (* -------------------------------------------------------------------- *) let dup (ld : ecloader) = { ecl_idirs = ld.ecl_idirs; } diff --git a/src/ecLoader.mli b/src/ecLoader.mli index 2338f6ad7..6ed7747f8 100644 --- a/src/ecLoader.mli +++ b/src/ecLoader.mli @@ -20,4 +20,9 @@ val aslist : ecloader -> ((namespace option * string) * idx_t) list val dup : ecloader -> ecloader val forsys : ecloader -> ecloader val addidir : ?namespace:namespace -> ?recursive:bool -> string -> ecloader -> unit + +(* Replace the include path wholesale, [aslist] being its reader. + [addidir] only ever grows the path, so this is what lets a caller + come back to an earlier one. *) +val setidirs : ((namespace option * string) * idx_t) list -> ecloader -> unit val locate : ?namespaces:(namespace option) list -> string -> ecloader -> (namespace option * string * kind) option diff --git a/tests/llm/README.md b/tests/llm/README.md index c7186ae58..7fc0c3a67 100644 --- a/tests/llm/README.md +++ b/tests/llm/README.md @@ -10,6 +10,7 @@ compared against recorded goldens. | Path | Contents | |------|----------| | `fixtures/*` | tiny EasyCrypt files the scripts `LOAD` (plus one non-`.ec` file, for the unknown-extension error, and one deliberately Latin-1 file used by `../mcp`) | +| `fixtures/sub/*` | a second directory, so a scenario can check that one `LOAD`'s include path does not survive into the next | | `scripts/*.script` | the newline-separated commands passed to `-eval` | | `expected/*.out` | recorded stdout, one file per script | | `../../scripts/testing/llm-golden` | the runner | diff --git a/tests/llm/expected/loadpath-reset.out b/tests/llm/expected/loadpath-reset.out new file mode 100644 index 000000000..d3e550490 --- /dev/null +++ b/tests/llm/expected/loadpath-reset.out @@ -0,0 +1,16 @@ +READY [uuid:0] + +OK [uuid:1] [loaded:fixtures/sub/entry.ec:2] +No active proof. + +OK [uuid:2] +No active proof. + +OK [uuid:1] [loaded:fixtures/simple.ec:3] +No active proof. + +ERROR [uuid:1] +: line 1 (0-25): cannot locate theory `Neighbour' +source: require import Neighbour. +No active proof. + diff --git a/tests/llm/fixtures/sub/Neighbour.ec b/tests/llm/fixtures/sub/Neighbour.ec new file mode 100644 index 000000000..186cf91f3 --- /dev/null +++ b/tests/llm/fixtures/sub/Neighbour.ec @@ -0,0 +1,5 @@ +(* A theory that exists only in this subdirectory. It is reachable from + fixtures/sub/entry.ec, its neighbour, and must be reachable from + nowhere else: a LOAD of a file in another directory has to leave the + include path with no memory of this one. *) +op neighbour : int = 3. diff --git a/tests/llm/fixtures/sub/entry.ec b/tests/llm/fixtures/sub/entry.ec new file mode 100644 index 000000000..e520b6f62 --- /dev/null +++ b/tests/llm/fixtures/sub/entry.ec @@ -0,0 +1,2 @@ +(* Loaded first, only so that its directory joins the include path. *) +require import AllCore. diff --git a/tests/llm/scripts/loadpath-reset.script b/tests/llm/scripts/loadpath-reset.script new file mode 100644 index 000000000..b1fdc420b --- /dev/null +++ b/tests/llm/scripts/loadpath-reset.script @@ -0,0 +1,11 @@ +# exit: 1 +# The include path is process-global and grows with every LOAD, so a +# previously loaded file's directory used to stay searchable for later, +# unrelated LOADs. Here `Neighbour' lives next to fixtures/sub/entry.ec +# and nowhere else: it must resolve while that file is the session, and +# stop resolving once fixtures/simple.ec is (loaded up to its own +# `require', so that the second attempt is outside any proof). +LOAD "fixtures/sub/entry.ec" +require import Neighbour. +LOAD "fixtures/simple.ec" 3 +require import Neighbour. From 82faf102356cc9e40d0080e5e40259497bcaf52c Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Sat, 22 Aug 2026 09:34:13 +0200 Subject: [PATCH 41/51] [llm] ec_try restores the exact pre-call state, forward included try_step rolled back with `EcCommands.undo pre', and [undo] only pops: it cannot undo an [undo]. Input that lowered the uuid before failing therefore left the session at that lower state while the reply said reverted true. ec_load simple.ec 6; ec_step "split. trivial. trivial." (uuid 6) ec_try "undo 3. apply nosuchlemma." before: uuid 3, goals back at `1 = 1 /\ 2 = 2', reverted true after: uuid 6, "No more goals", changed false, reverted true This is the faithful fix, not the fallback: refusing to run undo-like input under ec_try was unnecessary, because the engine's undo context is an immutable record (current scope, undo stack, uuid) held in a ref, so a mark of it restores forward as readily as backward. EcCommands gains undo_mark / undo_restore for that; try_step takes a mark on entry and restores it on failure. The session's own bookkeeping had the same one-directional flaw: Transcript.trim only drops entries, so the `undo 3.' above also emptied the ec_commit transcript for good. The transcript, COMMIT's proof environment and the prefix's bullet stack are now saved and put back verbatim rather than trimmed, and the scenario checks ec_commit still prints the whole body afterwards. Checkpoints need nothing: EasyCrypt input cannot reach them, and the uuids they name are valid again once the engine is back. `changed' on a try_step failure is now false by construction -- the restore always reaches [pre] -- so the two .mli comments that described the case where it could not are updated. New scenario tests/mcp/try-undo. (The REPL has no TRY command; ec_try is MCP-only, so there is no llm-side scenario to add.) No existing golden changes: try-revert's phrase only ever advanced. --- src/ecCommands.ml | 15 ++++++++++++ src/ecCommands.mli | 11 +++++++++ src/ecLlmCore.ml | 38 +++++++++++++++++++++---------- src/ecLlmCore.mli | 23 ++++++++++--------- tests/mcp/README.md | 1 + tests/mcp/expected/try-undo.out | 6 +++++ tests/mcp/scripts/try-undo.script | 14 ++++++++++++ 7 files changed, 85 insertions(+), 23 deletions(-) create mode 100644 tests/mcp/expected/try-undo.out create mode 100644 tests/mcp/scripts/try-undo.script diff --git a/src/ecCommands.ml b/src/ecCommands.ml index 3cb82ea12..c446bff80 100644 --- a/src/ecCommands.ml +++ b/src/ecCommands.ml @@ -1150,6 +1150,21 @@ let undo (olduuid : int) = context := Some (pop_context (oget !context)) done +(* -------------------------------------------------------------------- *) +(* [undo] only pops, so it cannot undo an [undo]: input that lowered the + uuid before failing leaves it at the wrong state, not the one it + started from. A caller that has to put the engine back *exactly* + where it was takes a mark first. The context is an immutable record + -- current scope, undo stack, uuid -- so this is a snapshot, not a + replay: restoring it moves forward as readily as backward. *) +type undo_mark = context + +let undo_mark () : undo_mark = + oget !context + +let undo_restore (mark : undo_mark) = + context := Some mark + (* -------------------------------------------------------------------- *) let doc_comment (doc : [`Global | `Item] * string) : unit = let current = oget !context in diff --git a/src/ecCommands.mli b/src/ecCommands.mli index fe35558bc..e1e656fec 100644 --- a/src/ecCommands.mli +++ b/src/ecCommands.mli @@ -69,6 +69,17 @@ val process : ?src:string -> ?timed:bool -> ?break:bool -> val undo : int -> unit val reset : unit -> unit + +(* An opaque snapshot of the engine's undo context: current scope, undo + stack and uuid. [undo] only pops, so it cannot undo an [undo] -- + input that lowered the uuid before failing lands somewhere else + entirely. [undo_restore] puts the engine back exactly, forward as + well as backward. Pragmas and the printing state are global and + outside the context, as they already are for [undo]. *) +type undo_mark + +val undo_mark : unit -> undo_mark +val undo_restore : undo_mark -> unit val uuid : unit -> int val mode : unit -> string diff --git a/src/ecLlmCore.ml b/src/ecLlmCore.ml index 82e4060d6..ae12e674d 100644 --- a/src/ecLlmCore.ml +++ b/src/ecLlmCore.ml @@ -762,24 +762,38 @@ let step (st : state) input = (* -------------------------------------------------------------------- *) (* [step] with an automatic rollback on failure. A phrase can fail - after having advanced the engine, so the pre-entry uuid is the only - faithful notion of "unchanged": restore it the way REVERT does. + after having advanced the engine, so nothing short of the state on + entry is a faithful notion of "unchanged". + + Rolling back with [undo pre] was not enough: [undo] only pops, so + input that *lowered* the uuid before failing -- [undo 3.] followed + by a bad tactic -- left the engine at that lower state while the + reply claimed [reverted = true]. Take a mark of the whole engine + context instead, which restores forward as readily as backward, and + restore the session's own bookkeeping (transcript, COMMIT's proof + environment, the prefix's bullet stack) alongside it rather than + trimming it, since trimming is likewise one-directional. Checkpoints + need nothing: EasyCrypt input cannot reach them, and the uuids they + name are valid again once the engine is back. + The failure is then re-stamped, because its uuid, goal text and - [changed] flag described the state at the point of failure, which no - longer exists. [changed] is recomputed against the same pre-entry - uuid, so it reports the *net* effect of the call: normally [false], - the rollback having undone whatever the input managed to do. It - stays [true] in the one case where the rollback cannot reach [pre], - namely a phrase that reset the engine ([pragma Reset]) and landed - below it -- then the state really did change. *) + [changed] flag described the point of failure, which no longer + exists. [changed] is [false] by construction now: the restore always + reaches [pre]. *) let try_step (st : state) input = - let pre = EcCommands.uuid () in + let pre = EcCommands.uuid () in + let mark = EcCommands.undo_mark () in + let transcript = !(st.transcript) in + let commit_env = !(st.commit_env) in + let bullets = !(st.prior_bullets) in match step st input with | Quit -> Quit | Done (Ok _) as answer -> answer | Done (Error failure) -> - EcCommands.undo pre; - Transcript.trim st pre; + EcCommands.undo_restore mark; + st.transcript := transcript; + st.commit_env := commit_env; + st.prior_bullets := bullets; let uuid = EcCommands.uuid () in Done (Error { failure with uuid; diff --git a/src/ecLlmCore.mli b/src/ecLlmCore.mli index f6f40e590..48e6e0440 100644 --- a/src/ecLlmCore.mli +++ b/src/ecLlmCore.mli @@ -35,11 +35,10 @@ type reply = { [reverted] is set by [try_step] only: it says the engine was rolled back to the state it had before the operation ran, so [uuid] and [goals] describe that restored state, not the point of failure. - [changed] tells whether the engine uuid advanced -- a failing - operation may well have moved the engine before failing. It reports - the *net* effect of the call, so under [try_step] it is [false] for - a phrase that advanced, failed and was rolled back: after the - rollback there is nothing left to have changed. *) + [changed] tells whether the engine uuid moved -- a failing operation + may well have moved it before failing. It reports the *net* effect + of the call, so under [try_step] it is always [false]: the rollback + is exact, and after it there is nothing left to have changed. *) type failure = { uuid : int; message : string; @@ -92,12 +91,14 @@ val load : preceded it applied. *) val step : state -> string -> answer -(* [step], but a failure leaves no trace: the engine is rolled back to - the uuid it had on entry (as REVERT does) and the failure comes back - with [reverted = true]. Successes and [Quit] behave exactly as in - [step]. Input that fails after having already advanced the engine - -- a phrase with a side effect, or an earlier sentence of a - multi-sentence input -- is rolled back whole. *) +(* [step], but a failure leaves no trace: the observable session -- + uuid, goals, COMMIT transcript -- is put back exactly as it was on + entry, and the failure comes back with [reverted = true] and + [changed = false]. Successes and [Quit] behave exactly as in [step]. + Input that fails after having already moved the engine is rolled + back whole, and "moved" includes moving *down*: a phrase whose first + sentence is [undo 3.] is restored just as faithfully as one that + advanced. *) val try_step : state -> string -> answer val goals : state -> all:bool -> (reply, failure) result diff --git a/tests/mcp/README.md b/tests/mcp/README.md index 951005030..5db111487 100644 --- a/tests/mcp/README.md +++ b/tests/mcp/README.md @@ -63,6 +63,7 @@ gate for changes to the protocol layer. | `print-query` | `print` and `locate` reach the agent, spend no uuid, and stay out of `ec_commit` | | `non-utf8` | engine output that is not UTF-8 comes back as U+FFFD, not as invalid JSON | | `try-revert` | `ec_try` rolling back a phrase that had already advanced the proof | +| `try-undo` | `ec_try` rolling *forward* again after a phrase whose `undo` lowered the uuid | | `protocol-errors` | `-32700`, `-32600`, `-32601` and the `-32602` family | | `revert` | `ec_revert` by uuid and by checkpoint name | | `load-missing` | a missing file and an unknown extension: `isError`, *not* `-32602` | diff --git a/tests/mcp/expected/try-undo.out b/tests/mcp/expected/try-undo.out new file mode 100644 index 000000000..94cf99fbd --- /dev/null +++ b/tests/mcp/expected/try-undo.out @@ -0,0 +1,6 @@ +{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2025-11-25","capabilities":{"tools":{}},"serverInfo":{"name":"easycrypt","version":"VERSION"}}} +{"jsonrpc":"2.0","id":2,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n","uuid":3,"changed":true},"isError":false}} +{"jsonrpc":"2.0","id":3,"result":{"content":[{"type":"text","text":"No more goals\n"}],"structuredContent":{"text":"No more goals\n","uuid":6,"changed":true},"isError":false}} +{"jsonrpc":"2.0","id":4,"result":{"content":[{"type":"text","text":": line 1 (8-26): unknown lemma `nosuchlemma'\nsource: apply nosuchlemma.\nNo more goals\n"}],"structuredContent":{"text":": line 1 (8-26): unknown lemma `nosuchlemma'\nsource: apply nosuchlemma.\nNo more goals\n","uuid":6,"changed":false,"reverted":true},"isError":true}} +{"jsonrpc":"2.0","id":5,"result":{"content":[{"type":"text","text":"No more goals\n"}],"structuredContent":{"text":"No more goals\n","uuid":6,"changed":false},"isError":false}} +{"jsonrpc":"2.0","id":6,"result":{"content":[{"type":"text","text":"split.\n- trivial.\n- trivial.\n"}],"structuredContent":{"text":"split.\n- trivial.\n- trivial.\n","uuid":6,"changed":false},"isError":false}} diff --git a/tests/mcp/scripts/try-undo.script b/tests/mcp/scripts/try-undo.script new file mode 100644 index 000000000..d4fc598c8 --- /dev/null +++ b/tests/mcp/scripts/try-undo.script @@ -0,0 +1,14 @@ +# exit: 0 +# ec_try rolls back input that moved the engine *down*. The proof is +# closed at uuid 6; the phrase then runs `undo 3.' before failing, so +# the rollback has to move forward again -- which the old `undo pre' +# could not do, leaving the session three states back while reporting +# reverted true. The ec_goals and ec_commit that follow prove the +# closed proof and its transcript are both back. +{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{}}} +{"jsonrpc":"2.0","method":"notifications/initialized"} +{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"ec_load","arguments":{"file":"../llm/fixtures/simple.ec","line":6}}} +{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"ec_step","arguments":{"phrase":"split. trivial. trivial."}}} +{"jsonrpc":"2.0","id":4,"method":"tools/call","params":{"name":"ec_try","arguments":{"phrase":"undo 3. apply nosuchlemma."}}} +{"jsonrpc":"2.0","id":5,"method":"tools/call","params":{"name":"ec_goals","arguments":{}}} +{"jsonrpc":"2.0","id":6,"method":"tools/call","params":{"name":"ec_commit","arguments":{}}} From 1fef2350e588ca2f10a4fb70f866f5439eeac4db Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Sat, 22 Aug 2026 10:20:41 +0200 Subject: [PATCH 42/51] [llm] correct the documented meaning of FOCUS and NEXT The guide and the ec_focus tool description both claimed that a single-integer `FOCUS k' selects the k-th goal of the flat listing and that `NEXT' is shorthand for `FOCUS 2'. Both are false, and both mislead an agent precisely when the proof tree is nested -- the case where FOCUS is worth using at all. A FOCUS path walks the tree, one component per level, so a single integer names the k-th TOP-LEVEL node. NEXT is an unrelated operation: it moves to the next open subgoal in GOALS ALL order, whatever the nesting. The two coincide only for a flat tree, which is why the error went unnoticed. Verified against the binary on fixtures/nested.ec at line 6 after three `split.' (tree: [1.1.1] [1.1.2] [1.2] [2], four goals under two top-level nodes): FOCUS 2 -> 4 = 4 (second top-level node) NEXT -> 2 = 2 (second open goal, [1.1.2]) FOCUS 3 -> ERROR: FOCUS: index 3 out of range (1..2) FOCUS 1 -> ERROR: FOCUS: path must select a leaf goal, not a frame Both guide passages and the ec_focus description now say this, with that worked example. tests/mcp/expected/tools-list.out is re-recorded because the tool description is part of the tools/list payload; no behaviour changed. --- doc/llm/CLAUDE.md | 24 +++++++++++++++++++----- src/ecMcp.ml | 15 ++++++++++----- tests/mcp/expected/tools-list.out | 2 +- 3 files changed, 30 insertions(+), 11 deletions(-) diff --git a/doc/llm/CLAUDE.md b/doc/llm/CLAUDE.md index b690b807e..261039dae 100644 --- a/doc/llm/CLAUDE.md +++ b/doc/llm/CLAUDE.md @@ -90,8 +90,8 @@ These are protocol-level commands, not EasyCrypt syntax: | `GOALS ALL` | Print all subgoals | | `TREE` | List open subgoals with dotted-path labels showing nesting, marking the focused one | | `TREE ALL` | Same as `TREE`, but with full goal bodies | -| `FOCUS P` | Rotate focus to the leaf addressed by path `P` (`N` or `N1.N2.N3...`) | -| `NEXT` | Rotate focus to the next subgoal (equivalent to `FOCUS 2`) | +| `FOCUS P` | Focus the leaf at `TREE` path `P` (`N` or `N1.N2.N3...`); the path walks the tree, so a single `N` picks the `N`-th **top-level node**, not the `N`-th goal | +| `NEXT` | Focus the next open subgoal, in `GOALS ALL` order. A different operation from `FOCUS 2` — see below | | `COMMIT` | Emit recorded REPL phrases as a bulleted proof body (works under `+strict_bullets`) | | `CHECKPOINT ` | Save current uuid under a name for later `REVERT` | | `SEARCH ` | Search for lemmas matching a pattern (read-only: the uuid does not move) | @@ -246,9 +246,23 @@ FOCUS 2 ← work on `w = 3` FOCUS 1.1.1 ← back to `x = 0` ``` -`FOCUS k` (a single integer) targets the k-th open goal in the flat -listing. `NEXT` is shorthand for `FOCUS 2`. Selecting an internal -frame errors (`FOCUS: path must select a leaf goal, not a frame`). +A `FOCUS` path always walks the tree, one component per level, so a +single integer `k` names the **k-th top-level node** — not the k-th +open goal. The tree above has four open goals but only two top-level +nodes, `[1]` (a frame) and `[2]` (a leaf), so: + +``` +FOCUS 2 ← `w = 3`, the second top-level node +FOCUS 3 ← ERROR: FOCUS: index 3 out of range (1..2) +FOCUS 1 ← ERROR: FOCUS: path must select a leaf goal, not a frame +``` + +`NEXT` is a different operation, and **not** shorthand for `FOCUS 2`: +it moves to the next open subgoal in `GOALS ALL` order, whatever the +nesting. From the tree above, `NEXT` focuses `y = 1` (`[1.1.2]`) while +`FOCUS 2` focuses `w = 3`. The two agree only when the tree is flat — +one `split.`, two leaves at the top level — which is the common case, +and the reason the difference is easy to miss. Replies carry a `[focus: k/N]` tag when more than one goal is open (e.g. `OK [uuid:42] [focus: 1/3]`) so you always know which goal the diff --git a/src/ecMcp.ml b/src/ecMcp.ml index d94994ea3..371f05dc2 100644 --- a/src/ecMcp.ml +++ b/src/ecMcp.ml @@ -337,11 +337,16 @@ let tools : J.t list = ~name:"ec_focus" ~description: "Rotate the focus onto the subgoal at dotted path PATH, as \ - printed by ec_tree (\"2\", \"1.2\", \"1.1.1\"); a single \ - integer selects the k-th goal of the flat listing, and the \ - special value \"next\" moves to the next open subgoal. \ - Subsequent tactics act on the focused goal. Selecting an \ - internal frame instead of a leaf goal is an error." + printed by ec_tree (\"2\", \"1.2\", \"1.1.1\"). The path walks \ + the tree, one component per level, so a single integer selects \ + the k-th TOP-LEVEL node -- not the k-th open goal: with four \ + goals nested under two top-level nodes, \"3\" is out of range. \ + Selecting a node that is an internal frame rather than a leaf \ + goal is an error. The special value \"next\" is a different \ + operation, not a synonym for \"2\": it moves to the next open \ + subgoal in ec_goals-with-all order, whatever the nesting, and \ + the two coincide only when the tree is flat. Subsequent \ + tactics act on the focused goal." ~input:(Schema.obj ~required:["path"] [ ("path", Schema.str ~description:"\"N\", a dotted path \"N1.N2...\", or \ diff --git a/tests/mcp/expected/tools-list.out b/tests/mcp/expected/tools-list.out index ab9b959cb..7e4b76bd3 100644 --- a/tests/mcp/expected/tools-list.out +++ b/tests/mcp/expected/tools-list.out @@ -1 +1 @@ -{"jsonrpc":"2.0","id":1,"result":{"tools":[{"name":"ec_load","description":"Reset the session and compile FILE from the top, stopping after the last sentence that ends on or before LINE (and column COL when given). This is the entry point: every other tool needs a loaded file, and tactics need the position to land inside a proof. Set nosmt to weaken SMT calls while replaying a prefix that was already verified, which is much faster on large files. Set trace to have the reply describe the last loaded sentence as BEFORE / TACTIC / AFTER / SUMMARY blocks. The reply reports where compilation stopped and the resulting goal state; note the uuid it returns, reverting to it is the instant way back to the start of the proof.","inputSchema":{"type":"object","properties":{"file":{"type":"string","description":"path to the .ec/.eca file"},"line":{"type":"integer","description":"stop after the last sentence ending on or before this line; omit to compile the whole file"},"col":{"type":"integer","description":"column bound within `line'; requires `line'"},"nosmt":{"type":"boolean","description":"weaken SMT calls while compiling the prefix","default":false},"trace":{"type":"boolean","description":"report the proof state around the last loaded sentence","default":false}},"required":["file"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["text","uuid","changed"]},"annotations":{"destructiveHint":true}},{"name":"ec_step","description":"Run EasyCrypt sentences -- tactics, declarations, require, print, ... -- against the current session. Every complete sentence in the argument is executed, in order, exactly as if the text had been appended to the source file, and a single reply describes the state they leave behind; sentences may span several lines. Requires a file loaded with ec_load, and, for tactics, an open proof. On success the reply carries the new goal state; on failure the prover's error text comes back with isError set, the sentences before the failing one stay applied and the engine is left wherever that sentence left it -- use ec_try when you want a guaranteed rollback. Successful non-query phrases are recorded for ec_commit.","inputSchema":{"type":"object","properties":{"phrase":{"type":"string","description":"one or more complete EasyCrypt sentences, each ending with `.'"}},"required":["phrase"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["text","uuid","changed"]},"annotations":{"destructiveHint":false,"idempotentHint":false}},{"name":"ec_try","description":"Like ec_step, but the engine is rolled back to the state it had before the call whenever a sentence fails, including input that failed only after having already advanced the proof. The failure reply sets structuredContent.reverted to true, and its uuid and goal text describe the restored state, not the point of failure. Use this to probe a tactic without having to ec_revert afterwards; use ec_step when you mean to keep whatever progress the phrase makes. A successful phrase behaves exactly as under ec_step and is recorded for ec_commit.","inputSchema":{"type":"object","properties":{"phrase":{"type":"string","description":"one complete EasyCrypt sentence, ending with `.'"}},"required":["phrase"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"},"reverted":{"type":"boolean","description":"set when the phrase failed and the engine was rolled back to its pre-call state"}},"required":["text","uuid","changed"]},"annotations":{"destructiveHint":false}},{"name":"ec_goals","description":"Print the current proof state: the focused subgoal alone, or, with all set, every open subgoal. Requires an open proof, and does not advance the engine.","inputSchema":{"type":"object","properties":{"all":{"type":"boolean","description":"print every open subgoal instead of the focused one","default":false}},"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["text","uuid","changed"]},"annotations":{"readOnlyHint":true}},{"name":"ec_tree","description":"List the open subgoals as a tree of dotted-path labels -- [1], [1.2], [2.1.1] -- showing how the splits nest, and marking the focused one. Those labels are exactly what ec_focus accepts. Set full for whole goal bodies rather than one-line conclusions. The labels are not stable across focus changes: the tree always shows the focused goal first, so re-read it after every ec_focus. Does not advance the engine.","inputSchema":{"type":"object","properties":{"full":{"type":"boolean","description":"print full goal bodies instead of one-line conclusions","default":false}},"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["text","uuid","changed"]},"annotations":{"readOnlyHint":true}},{"name":"ec_focus","description":"Rotate the focus onto the subgoal at dotted path PATH, as printed by ec_tree (\"2\", \"1.2\", \"1.1.1\"); a single integer selects the k-th goal of the flat listing, and the special value \"next\" moves to the next open subgoal. Subsequent tactics act on the focused goal. Selecting an internal frame instead of a leaf goal is an error.","inputSchema":{"type":"object","properties":{"path":{"type":"string","description":"\"N\", a dotted path \"N1.N2...\", or \"next\""}},"required":["path"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["text","uuid","changed"]},"annotations":{"destructiveHint":false}},{"name":"ec_undo","description":"Undo the last engine step, returning to the immediately preceding state. The ec_commit transcript is trimmed to match. Fails when there is nothing left to undo.","inputSchema":{"type":"object","properties":{},"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["text","uuid","changed"]},"annotations":{"destructiveHint":false}},{"name":"ec_revert","description":"Return the session to an earlier state, named either by a uuid reported in some previous structuredContent or by a name given to ec_checkpoint. Reverting is instant, unlike re-running ec_load, so going back to the uuid ec_load returned is the cheap way to restart a proof from scratch after a failed experiment. The ec_commit transcript is trimmed to match.","inputSchema":{"type":"object","properties":{"target":{"type":"string","description":"a uuid (as a decimal string) or a checkpoint name"}},"required":["target"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["text","uuid","changed"]},"annotations":{"destructiveHint":true}},{"name":"ec_checkpoint","description":"Record the current uuid under NAME, so that ec_revert can address it by name later. Worth doing before a branching experiment, when carrying the bare uuid around is awkward. Does not change the proof state.","inputSchema":{"type":"object","properties":{"name":{"type":"string","description":"checkpoint name"}},"required":["name"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["text","uuid","changed"]},"annotations":{"readOnlyHint":true}},{"name":"ec_commit","description":"Emit the phrases recorded since the last ec_load as a proof body, with bullets inserted at every multi-child split: the result compiles under `pragma +strict_bullets' and can be pasted straight into the source file. Queries (search, print, locate, ec_search) are never recorded, so looking things up mid-proof does not pollute the body, and ec_undo / ec_revert trim the transcript. Still works after `qed.'. Does not change the proof state.","inputSchema":{"type":"object","properties":{},"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["text","uuid","changed"]},"annotations":{"readOnlyHint":true}},{"name":"ec_search","description":"Search the environment for lemmas matching an EasyCrypt search pattern. This is pattern syntax, not keyword search: use _ as the wildcard, as in \"(fdom _)\", \"(_ %/ _)\" or \"(mu _ _) (_ <= _)\". Requires a loaded file. The query neither advances the proof nor enters the ec_commit transcript.","inputSchema":{"type":"object","properties":{"pattern":{"type":"string","description":"an EasyCrypt search pattern"}},"required":["pattern"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["text","uuid","changed"]},"annotations":{"readOnlyHint":true}}]}} +{"jsonrpc":"2.0","id":1,"result":{"tools":[{"name":"ec_load","description":"Reset the session and compile FILE from the top, stopping after the last sentence that ends on or before LINE (and column COL when given). This is the entry point: every other tool needs a loaded file, and tactics need the position to land inside a proof. Set nosmt to weaken SMT calls while replaying a prefix that was already verified, which is much faster on large files. Set trace to have the reply describe the last loaded sentence as BEFORE / TACTIC / AFTER / SUMMARY blocks. The reply reports where compilation stopped and the resulting goal state; note the uuid it returns, reverting to it is the instant way back to the start of the proof.","inputSchema":{"type":"object","properties":{"file":{"type":"string","description":"path to the .ec/.eca file"},"line":{"type":"integer","description":"stop after the last sentence ending on or before this line; omit to compile the whole file"},"col":{"type":"integer","description":"column bound within `line'; requires `line'"},"nosmt":{"type":"boolean","description":"weaken SMT calls while compiling the prefix","default":false},"trace":{"type":"boolean","description":"report the proof state around the last loaded sentence","default":false}},"required":["file"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["text","uuid","changed"]},"annotations":{"destructiveHint":true}},{"name":"ec_step","description":"Run EasyCrypt sentences -- tactics, declarations, require, print, ... -- against the current session. Every complete sentence in the argument is executed, in order, exactly as if the text had been appended to the source file, and a single reply describes the state they leave behind; sentences may span several lines. Requires a file loaded with ec_load, and, for tactics, an open proof. On success the reply carries the new goal state; on failure the prover's error text comes back with isError set, the sentences before the failing one stay applied and the engine is left wherever that sentence left it -- use ec_try when you want a guaranteed rollback. Successful non-query phrases are recorded for ec_commit.","inputSchema":{"type":"object","properties":{"phrase":{"type":"string","description":"one or more complete EasyCrypt sentences, each ending with `.'"}},"required":["phrase"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["text","uuid","changed"]},"annotations":{"destructiveHint":false,"idempotentHint":false}},{"name":"ec_try","description":"Like ec_step, but the engine is rolled back to the state it had before the call whenever a sentence fails, including input that failed only after having already advanced the proof. The failure reply sets structuredContent.reverted to true, and its uuid and goal text describe the restored state, not the point of failure. Use this to probe a tactic without having to ec_revert afterwards; use ec_step when you mean to keep whatever progress the phrase makes. A successful phrase behaves exactly as under ec_step and is recorded for ec_commit.","inputSchema":{"type":"object","properties":{"phrase":{"type":"string","description":"one complete EasyCrypt sentence, ending with `.'"}},"required":["phrase"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"},"reverted":{"type":"boolean","description":"set when the phrase failed and the engine was rolled back to its pre-call state"}},"required":["text","uuid","changed"]},"annotations":{"destructiveHint":false}},{"name":"ec_goals","description":"Print the current proof state: the focused subgoal alone, or, with all set, every open subgoal. Requires an open proof, and does not advance the engine.","inputSchema":{"type":"object","properties":{"all":{"type":"boolean","description":"print every open subgoal instead of the focused one","default":false}},"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["text","uuid","changed"]},"annotations":{"readOnlyHint":true}},{"name":"ec_tree","description":"List the open subgoals as a tree of dotted-path labels -- [1], [1.2], [2.1.1] -- showing how the splits nest, and marking the focused one. Those labels are exactly what ec_focus accepts. Set full for whole goal bodies rather than one-line conclusions. The labels are not stable across focus changes: the tree always shows the focused goal first, so re-read it after every ec_focus. Does not advance the engine.","inputSchema":{"type":"object","properties":{"full":{"type":"boolean","description":"print full goal bodies instead of one-line conclusions","default":false}},"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["text","uuid","changed"]},"annotations":{"readOnlyHint":true}},{"name":"ec_focus","description":"Rotate the focus onto the subgoal at dotted path PATH, as printed by ec_tree (\"2\", \"1.2\", \"1.1.1\"). The path walks the tree, one component per level, so a single integer selects the k-th TOP-LEVEL node -- not the k-th open goal: with four goals nested under two top-level nodes, \"3\" is out of range. Selecting a node that is an internal frame rather than a leaf goal is an error. The special value \"next\" is a different operation, not a synonym for \"2\": it moves to the next open subgoal in ec_goals-with-all order, whatever the nesting, and the two coincide only when the tree is flat. Subsequent tactics act on the focused goal.","inputSchema":{"type":"object","properties":{"path":{"type":"string","description":"\"N\", a dotted path \"N1.N2...\", or \"next\""}},"required":["path"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["text","uuid","changed"]},"annotations":{"destructiveHint":false}},{"name":"ec_undo","description":"Undo the last engine step, returning to the immediately preceding state. The ec_commit transcript is trimmed to match. Fails when there is nothing left to undo.","inputSchema":{"type":"object","properties":{},"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["text","uuid","changed"]},"annotations":{"destructiveHint":false}},{"name":"ec_revert","description":"Return the session to an earlier state, named either by a uuid reported in some previous structuredContent or by a name given to ec_checkpoint. Reverting is instant, unlike re-running ec_load, so going back to the uuid ec_load returned is the cheap way to restart a proof from scratch after a failed experiment. The ec_commit transcript is trimmed to match.","inputSchema":{"type":"object","properties":{"target":{"type":"string","description":"a uuid (as a decimal string) or a checkpoint name"}},"required":["target"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["text","uuid","changed"]},"annotations":{"destructiveHint":true}},{"name":"ec_checkpoint","description":"Record the current uuid under NAME, so that ec_revert can address it by name later. Worth doing before a branching experiment, when carrying the bare uuid around is awkward. Does not change the proof state.","inputSchema":{"type":"object","properties":{"name":{"type":"string","description":"checkpoint name"}},"required":["name"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["text","uuid","changed"]},"annotations":{"readOnlyHint":true}},{"name":"ec_commit","description":"Emit the phrases recorded since the last ec_load as a proof body, with bullets inserted at every multi-child split: the result compiles under `pragma +strict_bullets' and can be pasted straight into the source file. Queries (search, print, locate, ec_search) are never recorded, so looking things up mid-proof does not pollute the body, and ec_undo / ec_revert trim the transcript. Still works after `qed.'. Does not change the proof state.","inputSchema":{"type":"object","properties":{},"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["text","uuid","changed"]},"annotations":{"readOnlyHint":true}},{"name":"ec_search","description":"Search the environment for lemmas matching an EasyCrypt search pattern. This is pattern syntax, not keyword search: use _ as the wildcard, as in \"(fdom _)\", \"(_ %/ _)\" or \"(mu _ _) (_ <= _)\". Requires a loaded file. The query neither advances the proof nor enters the ec_commit transcript.","inputSchema":{"type":"object","properties":{"pattern":{"type":"string","description":"an EasyCrypt search pattern"}},"required":["pattern"],"additionalProperties":false},"outputSchema":{"type":"object","properties":{"text":{"type":"string","description":"the reply body -- goal state, proof body, search results, error text; the same string as content[0].text"},"uuid":{"type":"integer","description":"engine state identifier after the call; pass it to ec_revert to come back here"},"changed":{"type":"boolean","description":"whether the engine state advanced"}},"required":["text","uuid","changed"]},"annotations":{"readOnlyHint":true}}]}} From 9c586e142c02c436a8477338c974ca8c63c25f7d Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Sat, 22 Aug 2026 10:47:27 +0200 Subject: [PATCH 43/51] [llm] COMMIT renders each proof of a session on its own A session that holds more than one proof came out mangled: the DAG snapshot COMMIT queries was a single session-wide ref, kept at the newest proofenv, so once a second lemma was started every handle of the first resolved to no parent and that lemma was emitted flat, without its bullets. The bullet state was session-wide too, so the second lemma's phrases were indented at the first's leftover depth and picked tokens around the first's reservations. Key the snapshot per phrase instead: the transcript's 4-tuples become a record with an [en_penv] field holding the proof DAG as of just after that phrase ran. A proofenv is immutable and cumulative within one proof, so each entry's snapshot answers for its own proof and keeps answering after qed -- which is what the old single ref was for -- without an entry of one lemma ever being read against another's DAG. [state.commit_env] goes away with it, try_step included. Rendering is then cut at proof boundaries: [Commit.blocks] splits the transcript on the phrases typed outside a proof (the lemma statement, the qed), and each run in between is rendered by [render_run] with its own sibling map, current depth, and depth-to-token cache. The LOAD prefix's bullet frames belong to the proof that was in progress when the REPL took over, so only the first run receives them. Two scenarios pin it, both new: commit-two-lemmas (plain) and commit-two-lemmas-strict (the first proof continuing the prefix's +strict_bullets stack). Both bodies were reassembled with their prefix and compiled with `ec compile -no-eco`. No existing golden changes. --- src/ecLlmCore.ml | 431 ++++++++++-------- .../llm/expected/commit-two-lemmas-strict.out | 48 ++ tests/llm/expected/commit-two-lemmas.out | 48 ++ .../scripts/commit-two-lemmas-strict.script | 19 + tests/llm/scripts/commit-two-lemmas.script | 19 + 5 files changed, 364 insertions(+), 201 deletions(-) create mode 100644 tests/llm/expected/commit-two-lemmas-strict.out create mode 100644 tests/llm/expected/commit-two-lemmas.out create mode 100644 tests/llm/scripts/commit-two-lemmas-strict.script create mode 100644 tests/llm/scripts/commit-two-lemmas.script diff --git a/src/ecLlmCore.ml b/src/ecLlmCore.ml index ae12e674d..ca0846326 100644 --- a/src/ecLlmCore.ml +++ b/src/ecLlmCore.ml @@ -36,6 +36,29 @@ type answer = exception Init_error of string +(* -------------------------------------------------------------------- *) +(* One recorded REPL phrase, as [Commit] needs to see it again. + + [en_penv] is the proof DAG as of just after the phrase ran, and it + is per entry rather than per session on purpose: a session holding + several lemmas has one DAG per lemma, handles are only meaningful in + their own, and a single newest-wins snapshot answered "no parent" + for every handle of every earlier proof -- which rendered those + proofs flat, without their bullets. *) +type entry = { + (* Engine uuid right before the phrase; UNDO/REVERT trim on it. *) + en_uuid : int; + en_src : string; + (* Focused handle right before the phrase; [None] iff outside a + proof, which is also what separates one proof from the next. *) + en_parent : EcCoreGoal.handle option; + (* Full open-handle list (focused first) right before the phrase, + used to seed the sibling map when the first recorded phrase of a + proof already sits inside a frame opened by the LOAD prefix. *) + en_opens : EcCoreGoal.handle list; + en_penv : EcCoreGoal.proofenv option; +} + (* -------------------------------------------------------------------- *) (* Session state. The proof engine ([EcCommands]) is a global mutable singleton, so at most one [state] may exist per process. *) @@ -72,24 +95,9 @@ type state = { (* CHECKPOINT name -> uuid. *) checkpoints : (string, int) Hashtbl.t; - (* Transcript of REPL-typed phrases that succeeded. Each entry is - [(uuid_before, src, parent, opens_at_entry)]: - - [parent]: focused handle right before the phrase ([None] iff - outside a proof); - - [opens_at_entry]: full open-handle list (focused first), used - by [Commit] to seed the sibling map when the first recorded - phrase already sits inside a frame opened by the LOAD prefix. + (* Transcript of REPL-typed phrases that succeeded, newest first. Trimmed by UNDO/REVERT; cleared on LOAD/Restart. *) - transcript : - (int * string * EcCoreGoal.handle option - * EcCoreGoal.handle list) list ref; - - (* Proof environment snapshot, refreshed at every recorded phrase. - [COMMIT] queries the proof DAG through it rather than through the - active proof, which is gone once [qed] has run. A [proofenv] is - immutable and cumulative, so the last snapshot knows about every - handle any transcript entry can mention. *) - commit_env : EcCoreGoal.proofenv option ref; + transcript : entry list ref; (* The bullet stack of the active proof at the moment REPL input took over. Captured the first time [disable_repl_bullets] clears @@ -157,7 +165,6 @@ let create ~relocdir ~boot ~projini ~prvopts = base_loadpath = EcCommands.loadpath_mark (); checkpoints = Hashtbl.create 16; transcript = ref []; - commit_env = ref None; prior_bullets = ref None; } in @@ -389,14 +396,11 @@ module Transcript = struct let trim (st : state) target = let transcript = st.transcript in transcript := - List.filter - (fun (uuid_before, _, _, _) -> uuid_before < target) - !transcript + List.filter (fun e -> e.en_uuid < target) !transcript let clear (st : state) = st.transcript := []; - st.prior_bullets := None; - st.commit_env := None + st.prior_bullets := None end (* -------------------------------------------------------------------- *) @@ -406,7 +410,6 @@ end which together let [Commit] reconstruct bullet structure. *) let process_action (st : state) ?(record=false) ~src (p : EP.global) = let transcript = st.transcript in - let commit_env = st.commit_env in let loc = p.EP.gl_action.EcLocation.pl_loc in let pre_uuid = EcCommands.uuid () in let opens_pre = @@ -441,15 +444,21 @@ let process_action (st : state) ?(record=false) ~src (p : EP.global) = raise (EcScope.toperror_of_exn ~gloc:loc (EcScope.HiScopeError (None, "this command is expected to fail"))); - if record && !succeeded && not p.EP.gl_fail && not is_query then begin - transcript := (pre_uuid, src, parent, opens_pre) :: !transcript; - (* Keep the newest non-empty snapshot: a phrase that closes the - proof ([qed]) leaves no active proof, and precisely then we - still need the environment the previous phrases built. *) - match EcCommands.current_proofenv () with - | None -> () - | Some _ as penv -> commit_env := penv - end + if record && !succeeded && not p.EP.gl_fail && not is_query then + (* The DAG is snapshot here, per phrase: a [proofenv] is immutable + and cumulative *within one proof*, so this entry's snapshot + answers for every handle its own proof can mention -- and, being + its own, keeps answering after [qed] has discarded the proof and + a later lemma has replaced it. A phrase that ends the proof + leaves none active and so records [None]; such a phrase is + outside a proof ([en_parent = None]) and its DAG is never + consulted. *) + transcript := + { en_uuid = pre_uuid; + en_src = src; + en_parent = parent; + en_opens = opens_pre; + en_penv = EcCommands.current_proofenv (); } :: !transcript (* -------------------------------------------------------------------- *) (* COMMIT: replay the transcript against the proof DAG (parent_of / @@ -467,16 +476,17 @@ module Commit = struct let chr = chars.(i mod 3) in String.concat "" (List.init rep (fun _ -> chr)) - (* DAG queries go through the snapshot recorded at the last phrase, - so COMMIT still sees the structure after [qed]. Fall back to the - live proof when no phrase was recorded under a proof. *) - let parent_of (st : state) h = - match !(st.commit_env) with + (* DAG queries go through the snapshot the entry recorded, so COMMIT + still sees the structure of a proof [qed] has since discarded -- + and sees the *right* one when the session holds several. Fall back + to the live proof for an entry that recorded no snapshot. *) + let parent_of penv h = + match penv with | Some penv -> EcCoreGoal.parent_of_handle penv h | None -> EcCommands.parent_of h - let children_of (st : state) h = - match !(st.commit_env) with + let children_of penv h = + match penv with | Some penv -> EcCoreGoal.children_of_handle penv h | None -> EcCommands.children_of h @@ -485,9 +495,9 @@ module Commit = struct the DAG's preorder, which is the order in which a proof body has to discharge the subgoals -- and, FOCUS/NEXT being free to jump between open goals, not the order the phrases were typed in. *) - let dag_path (st : state) (h : EcCoreGoal.handle) = + let dag_path penv (h : EcCoreGoal.handle) = let rec walk h acc = - match parent_of st h with + match parent_of penv h with | None -> acc | Some p -> let rec index i = function @@ -495,42 +505,50 @@ module Commit = struct | c :: cs -> if EcCoreGoal.eq_handle c h then i else index (i + 1) cs in - walk p (index 0 (children_of st p) :: acc) + walk p (index 0 (children_of penv p) :: acc) in walk h [] - (* Reorder a transcript into DAG order, so that a body typed out of - order (FOCUS 2, prove the second goal, come back to the first) - still replays top to bottom. Phrases typed outside a proof - ([parent = None]) separate one proof from the next and act as - barriers: each run of in-proof phrases between two of them is - sorted on its own. The sort is stable, so entries the DAG does not - order keep their typing order. *) - let dag_order (st : state) entries = - let key (_, _, parent, _) = - match parent with - | None -> [] - | Some h -> dag_path st h + (* Cut the transcript at its proof boundaries. A phrase typed outside + a proof ([en_parent = None]) is a [`Barrier] -- the lemma statement + that opens a proof, the [qed] that closes it -- and each run of + in-proof phrases between two of them belongs to exactly one proof. + Both the DAG sort and the bullet state below are per proof, and + this is what delimits one. *) + let blocks entries = + let close acc run = if run = [] then acc else `Run (List.rev run) :: acc in + let rec walk acc run = function + | [] -> List.rev (close acc run) + | ({ en_parent = None; _ } as e) :: rest -> + walk (`Barrier e :: close acc run) [] rest + | e :: rest -> walk acc (e :: run) rest in - let sort run = - List.stable_sort - (fun a b -> compare (key a : int list) (key b)) - run + walk [] [] entries + + (* Reorder one proof's phrases into DAG order, so that a body typed + out of order (FOCUS 2, prove the second goal, come back to the + first) still replays top to bottom. The sort is stable, so entries + the DAG does not order keep their typing order. *) + let dag_order run = + let key e = + match e.en_parent with + | None -> [] + | Some h -> dag_path e.en_penv h in - let rec regroup run = function - | [] -> sort (List.rev run) - | ((_, _, None, _) as e) :: rest -> - sort (List.rev run) @ (e :: regroup [] rest) - | e :: rest -> regroup (e :: run) rest + List.stable_sort + (fun a b -> compare (key a : int list) (key b)) + run + + let bullet_to_string (b : EcParsetree.bullet) = + let ch = + match b.b_kind with + | `Minus -> "-" + | `Plus -> "+" + | `Star -> "*" in - regroup [] entries + String.concat "" (List.init b.b_count (fun _ -> ch)) let proof_text (st : state) = - let parent_of = parent_of st in - let children_of = children_of st in - let transcript = st.transcript in - let prior_bullets = st.prior_bullets in - let entries = List.rev !transcript in let buf = Buffer.create 1024 in let emit_indent depth = for _ = 1 to depth do Buffer.add_string buf " " done @@ -541,138 +559,150 @@ module Commit = struct let compare = compare end) in - let sibling_depth : int Hmap.t ref = ref Hmap.empty in - let current_depth = ref 0 in - let bullet_to_string (b : EcParsetree.bullet) = - let ch = - match b.b_kind with - | `Minus -> "-" - | `Plus -> "+" - | `Star -> "*" + (* Render one proof's phrases. Every piece of bullet state -- + the sibling map, the current depth, the token reserved at each + depth -- is local to this call, so one lemma can neither inherit + another's indentation nor exhaust its token supply. + [frames] are the bullet frames the LOAD prefix left open; they + belong to the proof that was in progress when the REPL took + over, hence to the first run only. *) + let render_run ~(frames : EcBullets.frame list) run = + let sibling_depth : int Hmap.t ref = ref Hmap.empty in + let current_depth = ref 0 in + let in_use_tokens = + List.map + (fun (f : EcBullets.frame) -> bullet_to_string f.bf_bullet) + frames + in + let depth_cache : (int, string) Hashtbl.t = Hashtbl.create 8 in + let next_tok_idx = ref 0 in + let assigned_tokens = ref [] in + (* Depths 1..k address the next sibling of a frame the prefix + already opened, and strict bullets accepts nothing but that + frame's own token there. Deeper levels get fresh tokens, so + pre-populate the cache before any fresh pick happens. *) + List.iteri (fun i (f : EcBullets.frame) -> + let t = bullet_to_string f.bf_bullet in + Hashtbl.replace depth_cache (i + 1) t; + assigned_tokens := t :: !assigned_tokens) + frames; + let bullet_for_depth d = + match Hashtbl.find_opt depth_cache d with + | Some t -> t + | None -> + let rec pick () = + let t = token_at_index !next_tok_idx in + incr next_tok_idx; + if List.mem t in_use_tokens || List.mem t !assigned_tokens + then pick () + else t + in + let t = pick () in + assigned_tokens := t :: !assigned_tokens; + Hashtbl.add depth_cache d t; + t in - String.concat "" (List.init b.b_count (fun _ -> ch)) + (* Seed: the goals already open when this proof's first recorded + phrase ran were left there by the LOAD prefix, so COMMIT must + place each of them at the depth the prefix's own bullets put it + at. A frame with floor [f] is discharged once [f] goals remain, + hence it still owns the first [n - f] goals of the focused-first + list; a goal covered by [c] frames sits at depth [c + 1]. + Nothing to seed when the prefix left no frame and a single goal + (the REPL just continues on the prefix's own focus). *) + (match run with + | { en_parent = Some _; en_opens = (_ :: _ as opens); en_penv; _ } :: _ + when frames <> [] || List.length opens >= 2 -> + (* [pr_opened] is focused-first, so a FOCUS/NEXT run before the + first recorded phrase leaves it rotated. The floors below + count goals in the order the prefix's bullets consume them, + which is DAG order. *) + let opens = + List.stable_sort + (fun a b -> + compare (dag_path en_penv a : int list) (dag_path en_penv b)) + opens + in + let n = List.length opens in + List.iteri (fun i h -> + let pos = i + 1 in + let covering = + List.length + (List.filter + (fun (f : EcBullets.frame) -> pos <= n - f.bf_floor) + frames) + in + sibling_depth := Hmap.add h (covering + 1) !sibling_depth) + opens + | _ -> ()); + List.iter (fun e -> + match e.en_parent with + | None -> assert false (* [blocks] keeps these out *) + | Some parent -> + let parent_of = parent_of e.en_penv in + let children_of = children_of e.en_penv in + (* Walk upward via pr_parent until we hit a registered + sibling ancestor. If found, emit its bullet and consume + the registration. *) + let rec find_ancestor h = + match Hmap.find_opt h !sibling_depth with + | Some d -> Some (h, d) + | None -> + match parent_of h with + | Some p -> find_ancestor p + | None -> None + in + (match find_ancestor parent with + | Some (h, d) -> + emit_indent (d - 1); + Buffer.add_string buf (bullet_for_depth d); + Buffer.add_char buf ' '; + current_depth := d; + sibling_depth := Hmap.remove h !sibling_depth + | None -> + emit_indent !current_depth); + Buffer.add_string buf e.en_src; + Buffer.add_char buf '\n'; + (* Register fresh siblings: walk the subtree rooted at + [parent], finding every multi-child split, and register + each such child at the right depth. Single-child links + are continuations and don't bump depth; multi-child + links do. A compound phrase like [split; split.] can + produce nested splits within one phrase. *) + let rec walk h d = + match children_of h with + | [c] -> walk c d + | (_ :: _ :: _) as cs -> + List.iter + (fun c -> + sibling_depth := + Hmap.add c d !sibling_depth; + walk c (d + 1)) + cs + | [] -> () + in + walk parent (!current_depth + 1)) + (dag_order run) in - (* Bullet frames the LOAD prefix left open, OUTERMOST first (the - stack stores the innermost frame at its head). Frame [t_d] is - the one whose siblings live at emitted depth [d]. *) - let frames : EcBullets.frame list = - match !prior_bullets with + let prefix_frames : EcBullets.frame list = + (* The stack stores the innermost frame at its head; [render_run] + wants them OUTERMOST first, frame [t_d] being the one whose + siblings live at emitted depth [d]. *) + match !(st.prior_bullets) with | None -> [] | Some stack -> List.rev stack in - let in_use_tokens = - List.map - (fun (f : EcBullets.frame) -> bullet_to_string f.bf_bullet) - frames - in - let depth_cache : (int, string) Hashtbl.t = Hashtbl.create 8 in - let next_tok_idx = ref 0 in - let assigned_tokens = ref [] in - (* Depths 1..k address the next sibling of a frame the prefix - already opened, and strict bullets accepts nothing but that - frame's own token there. Deeper levels get fresh tokens, so - pre-populate the cache before any fresh pick happens. *) - List.iteri (fun i (f : EcBullets.frame) -> - let t = bullet_to_string f.bf_bullet in - Hashtbl.replace depth_cache (i + 1) t; - assigned_tokens := t :: !assigned_tokens) - frames; - let bullet_for_depth d = - match Hashtbl.find_opt depth_cache d with - | Some t -> t - | None -> - let rec pick () = - let t = token_at_index !next_tok_idx in - incr next_tok_idx; - if List.mem t in_use_tokens || List.mem t !assigned_tokens - then pick () - else t - in - let t = pick () in - assigned_tokens := t :: !assigned_tokens; - Hashtbl.add depth_cache d t; - t - in - (* Seed: the goals already open when the first recorded phrase ran - were left there by the LOAD prefix, so COMMIT must place each of - them at the depth the prefix's own bullets put it at. A frame - with floor [f] is discharged once [f] goals remain, hence it - still owns the first [n - f] goals of the focused-first list; - a goal covered by [c] frames sits at depth [c + 1]. - Nothing to seed when the prefix left no frame and a single goal - (the REPL just continues on the prefix's own focus). *) - (match entries with - | (_, _, Some _, (_ :: _ as opens)) :: _ - when frames <> [] || List.length opens >= 2 -> - (* [pr_opened] is focused-first, so a FOCUS/NEXT run before the - first recorded phrase leaves it rotated. The floors below - count goals in the order the prefix's bullets consume them, - which is DAG order. *) - let opens = - List.stable_sort - (fun a b -> compare (dag_path st a : int list) (dag_path st b)) - opens - in - let n = List.length opens in - List.iteri (fun i h -> - let pos = i + 1 in - let covering = - List.length - (List.filter - (fun (f : EcBullets.frame) -> pos <= n - f.bf_floor) - frames) - in - sibling_depth := Hmap.add h (covering + 1) !sibling_depth) - opens - | _ -> ()); - List.iter (fun (_uuid, src, parent_opt, _opens) -> - match parent_opt with - | None -> - Buffer.add_string buf src; - Buffer.add_char buf '\n' - | Some parent -> - (* Walk upward via pr_parent until we hit a registered - sibling ancestor. If found, emit its bullet and consume - the registration. *) - let rec find_ancestor h = - match Hmap.find_opt h !sibling_depth with - | Some d -> Some (h, d) - | None -> - match parent_of h with - | Some p -> find_ancestor p - | None -> None - in - (match find_ancestor parent with - | Some (h, d) -> - emit_indent (d - 1); - Buffer.add_string buf (bullet_for_depth d); - Buffer.add_char buf ' '; - current_depth := d; - sibling_depth := Hmap.remove h !sibling_depth - | None -> - emit_indent !current_depth); - Buffer.add_string buf src; - Buffer.add_char buf '\n'; - (* Register fresh siblings: walk the subtree rooted at - [parent], finding every multi-child split, and register - each such child at the right depth. Single-child links - are continuations and don't bump depth; multi-child - links do. A compound phrase like [split; split.] can - produce nested splits within one phrase. *) - let rec walk h d = - match children_of h with - | [c] -> walk c d - | (_ :: _ :: _) as cs -> - List.iter - (fun c -> - sibling_depth := - Hmap.add c d !sibling_depth; - walk c (d + 1)) - cs - | [] -> () - in - walk parent (!current_depth + 1) - ) (dag_order st entries); + let first_run = ref true in + List.iter + (function + | `Barrier e -> + Buffer.add_string buf e.en_src; + Buffer.add_char buf '\n' + | `Run run -> + let frames = if !first_run then prefix_frames else [] in + first_run := false; + render_run ~frames run) + (blocks (List.rev !(st.transcript))); Buffer.contents buf end @@ -770,8 +800,9 @@ let step (st : state) input = by a bad tactic -- left the engine at that lower state while the reply claimed [reverted = true]. Take a mark of the whole engine context instead, which restores forward as readily as backward, and - restore the session's own bookkeeping (transcript, COMMIT's proof - environment, the prefix's bullet stack) alongside it rather than + restore the session's own bookkeeping (the transcript, whose entries + carry COMMIT's proof-DAG snapshots, and the prefix's bullet stack) + alongside it rather than trimming it, since trimming is likewise one-directional. Checkpoints need nothing: EasyCrypt input cannot reach them, and the uuids they name are valid again once the engine is back. @@ -784,7 +815,6 @@ let try_step (st : state) input = let pre = EcCommands.uuid () in let mark = EcCommands.undo_mark () in let transcript = !(st.transcript) in - let commit_env = !(st.commit_env) in let bullets = !(st.prior_bullets) in match step st input with | Quit -> Quit @@ -792,7 +822,6 @@ let try_step (st : state) input = | Done (Error failure) -> EcCommands.undo_restore mark; st.transcript := transcript; - st.commit_env := commit_env; st.prior_bullets := bullets; let uuid = EcCommands.uuid () in Done (Error { failure with diff --git a/tests/llm/expected/commit-two-lemmas-strict.out b/tests/llm/expected/commit-two-lemmas-strict.out new file mode 100644 index 000000000..ae6b6b1ac --- /dev/null +++ b/tests/llm/expected/commit-two-lemmas-strict.out @@ -0,0 +1,48 @@ +READY [uuid:0] + +OK [uuid:6] [loaded:fixtures/strict.ec:11] [focus: 1/3] +Current goal (remaining: 3) + +Type variables: + +------------------------------------------------------------------------ +1 = 1 + +OK [uuid:6] + +OK [uuid:7] [focus: 1/2] + +OK [uuid:8] + +OK [uuid:9] + +OK [uuid:10] +added lemma: `strict_and' + +OK [uuid:11] + +OK [uuid:12] + +OK [uuid:13] [focus: 1/2] + +OK [uuid:14] + +OK [uuid:15] + +OK [uuid:16] +added lemma: `strict_two' + +OK [uuid:16] + +OK [uuid:16] + + trivial. + + trivial. +- trivial. +qed. +lemma strict_two : 3 = 3 /\ 4 = 4. +proof. +split. +- trivial. +- trivial. +qed. + diff --git a/tests/llm/expected/commit-two-lemmas.out b/tests/llm/expected/commit-two-lemmas.out new file mode 100644 index 000000000..6771d6334 --- /dev/null +++ b/tests/llm/expected/commit-two-lemmas.out @@ -0,0 +1,48 @@ +READY [uuid:0] + +OK [uuid:3] [loaded:fixtures/simple.ec:6] +Current goal + +Type variables: + +------------------------------------------------------------------------ +1 = 1 /\ 2 = 2 + +OK [uuid:3] + +OK [uuid:4] [focus: 1/2] + +OK [uuid:5] + +OK [uuid:6] + +OK [uuid:7] +added lemma: `simple_and' + +OK [uuid:8] + +OK [uuid:9] + +OK [uuid:10] [focus: 1/2] + +OK [uuid:11] + +OK [uuid:12] + +OK [uuid:13] +added lemma: `two_and' + +OK [uuid:13] + +OK [uuid:13] +split. +- trivial. +- trivial. +qed. +lemma two_and : 3 = 3 /\ 4 = 4. +proof. +split. +- trivial. +- trivial. +qed. + diff --git a/tests/llm/scripts/commit-two-lemmas-strict.script b/tests/llm/scripts/commit-two-lemmas-strict.script new file mode 100644 index 000000000..38140bd8d --- /dev/null +++ b/tests/llm/scripts/commit-two-lemmas-strict.script @@ -0,0 +1,19 @@ +# exit: 0 +# The same, but the first proof continues the LOAD prefix's own bullet +# stack (fixtures/strict.ec, under `pragma +strict_bullets`). The +# second lemma opens no such frame, so it must start again at depth 0 +# with the first token, not inherit the prefix's depth or its tokens. +LOAD "fixtures/strict.ec" +QUIET ON +trivial. +trivial. +trivial. +qed. +lemma strict_two : 3 = 3 /\ 4 = 4. +proof. +split. +trivial. +trivial. +qed. +QUIET OFF +COMMIT diff --git a/tests/llm/scripts/commit-two-lemmas.script b/tests/llm/scripts/commit-two-lemmas.script new file mode 100644 index 000000000..d89b204b9 --- /dev/null +++ b/tests/llm/scripts/commit-two-lemmas.script @@ -0,0 +1,19 @@ +# exit: 0 +# Two complete lemmas in one session. Bullet structure and indentation +# are per proof: the first lemma keeps its bullets once the second is +# started, and the second neither inherits the first's depth nor its +# reserved tokens. +LOAD "fixtures/simple.ec" 6 +QUIET ON +split. +trivial. +trivial. +qed. +lemma two_and : 3 = 3 /\ 4 = 4. +proof. +split. +trivial. +trivial. +qed. +QUIET OFF +COMMIT From a447e155f195d5f1c1a3031814466b7f6332dd96 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Sat, 22 Aug 2026 10:49:32 +0200 Subject: [PATCH 44/51] [llm] QUIT and `exit.` no longer defeat the -eval exit status A scripted run is supposed to exit nonzero when any command produced an ERROR reply, but QUIT and an `exit.` phrase both called `exit 0` outright, bypassing the check. A script ending in QUIT -- the natural thing to write -- therefore reported success no matter how many of its commands had failed. Give the REPL a single [terminate] and route all three exits through it: end of input, QUIT, and the [EcLlmCore.Quit] answer that an `exit.` phrase produces. Interactive sessions are unaffected; [terminate] still exits 0 unless -eval is in effect. Two new scenarios pin the contract through their `# exit:` headers: error-exit-quit (error, then QUIT) and error-exit-phrase (error, then `exit.`). Both end on a trailing GOALS whose reply is absent from the golden, so they also witness that the session really did stop there. No existing golden changes. --- doc/llm/CLAUDE.md | 6 ++++++ src/ecLlm.ml | 23 +++++++++++++++------- tests/llm/README.md | 5 ++++- tests/llm/expected/error-exit-phrase.out | 19 ++++++++++++++++++ tests/llm/expected/error-exit-quit.out | 6 ++++++ tests/llm/scripts/error-exit-phrase.script | 8 ++++++++ tests/llm/scripts/error-exit-quit.script | 8 ++++++++ 7 files changed, 67 insertions(+), 8 deletions(-) create mode 100644 tests/llm/expected/error-exit-phrase.out create mode 100644 tests/llm/expected/error-exit-quit.out create mode 100644 tests/llm/scripts/error-exit-phrase.script create mode 100644 tests/llm/scripts/error-exit-quit.script diff --git a/doc/llm/CLAUDE.md b/doc/llm/CLAUDE.md index 261039dae..d5681c31b 100644 --- a/doc/llm/CLAUDE.md +++ b/doc/llm/CLAUDE.md @@ -37,6 +37,12 @@ GOALS COMMIT' ``` +A `-eval` run exits 1 if any command produced an `ERROR` reply, and 0 +otherwise. The status covers the whole run however it ends: ending the +script with `QUIT`, or with an `exit.` phrase, reports the errors that +came before just as end-of-input does. Interactive sessions (no +`-eval`) always exit 0. + ### Protocol **Startup.** EasyCrypt prints a `READY` message and waits for input: diff --git a/src/ecLlm.ml b/src/ecLlm.ml index d1f4f0dc2..78e5cbc19 100644 --- a/src/ecLlm.ml +++ b/src/ecLlm.ml @@ -259,9 +259,21 @@ let run ~relocdir ~boot ~projini (llmopts : EcOptions.llm_option) = let quiet = ref false in (* ------------------------------------------------------------------ *) - (* OK/ERROR/ wire envelope: the only printers. *) + (* Exit status. Scripted runs (-eval) report in-band errors through + it, so that automation does not mistake an ERROR reply for success; + interactive sessions always exit 0. + + Every way out of the REPL goes through [terminate]: end of input, + QUIT, and an [exit.] phrase alike. Routing QUIT and [exit.] around + it is how the contract used to be lost -- and a script ending in + QUIT, which is the natural thing to write, is exactly the case + that lost it. *) let had_error = ref false in + let terminate () = + exit (if llmopts.llmo_eval <> None && !had_error then 1 else 0) + in + let module Wire = struct (* Write a chunk of reply body, one line at a time, escaping the lines that would collide with the envelope. The chunk is @@ -309,7 +321,7 @@ let run ~relocdir ~boot ~projini (llmopts : EcOptions.llm_option) = (* Same, for operations that may end the session. *) let answer = function - | EcLlmCore.Quit -> exit 0 + | EcLlmCore.Quit -> terminate () | EcLlmCore.Done outcome -> reply outcome let reply_error msg = @@ -363,7 +375,7 @@ let run ~relocdir ~boot ~projini (llmopts : EcOptions.llm_option) = let run (cmd : Parse.command) = match cmd with | Blank -> () - | Quit -> exit 0 + | Quit -> terminate () | Help -> do_help () | Undo -> Wire.reply (EcLlmCore.undo st) | Goals `One -> Wire.reply (EcLlmCore.goals st ~all:false) @@ -420,7 +432,4 @@ let run ~relocdir ~boot ~projini (llmopts : EcOptions.llm_option) = | End_of_file -> () end; - (* Scripted runs (-eval) report in-band errors through the exit - status, so that automation does not mistake an ERROR reply for - success. Interactive sessions keep exiting 0. *) - exit (if llmopts.llmo_eval <> None && !had_error then 1 else 0) + terminate () diff --git a/tests/llm/README.md b/tests/llm/README.md index 7fc0c3a67..92d6f8a09 100644 --- a/tests/llm/README.md +++ b/tests/llm/README.md @@ -60,7 +60,10 @@ prose. The first `# exit: N` line wins; a script without one fails. `ec.exe llm -eval` exits 1 if any command produced an `ERROR` reply and 0 otherwise, so scenarios that deliberately exercise error paths -declare `# exit: 1`. +declare `# exit: 1`. That holds however the run ends: `error-exit` +falls off the end of the script, `error-exit-quit` ends on `QUIT` and +`error-exit-phrase` on an `exit.` phrase, and all three declare +`# exit: 1`. ## Determinism rules diff --git a/tests/llm/expected/error-exit-phrase.out b/tests/llm/expected/error-exit-phrase.out new file mode 100644 index 000000000..6a5e443e7 --- /dev/null +++ b/tests/llm/expected/error-exit-phrase.out @@ -0,0 +1,19 @@ +READY [uuid:0] + +OK [uuid:3] [loaded:fixtures/simple.ec:6] +Current goal + +Type variables: + +------------------------------------------------------------------------ +1 = 1 /\ 2 = 2 + +ERROR [uuid:3] +parse error +Current goal + +Type variables: + +------------------------------------------------------------------------ +1 = 1 /\ 2 = 2 + diff --git a/tests/llm/expected/error-exit-quit.out b/tests/llm/expected/error-exit-quit.out new file mode 100644 index 000000000..8346241ef --- /dev/null +++ b/tests/llm/expected/error-exit-quit.out @@ -0,0 +1,6 @@ +READY [uuid:0] + +ERROR [uuid:0] +nothing to undo +No active proof. + diff --git a/tests/llm/scripts/error-exit-phrase.script b/tests/llm/scripts/error-exit-phrase.script new file mode 100644 index 000000000..699f8beca --- /dev/null +++ b/tests/llm/scripts/error-exit-phrase.script @@ -0,0 +1,8 @@ +# exit: 1 +# An `exit.` phrase ends the session too, and likewise must not swallow +# the ERROR that came before it. As above, the trailing GOALS is there +# to show the session did stop at `exit.`. +LOAD "fixtures/simple.ec" 6 +nosuchtactic. +exit. +GOALS diff --git a/tests/llm/scripts/error-exit-quit.script b/tests/llm/scripts/error-exit-quit.script new file mode 100644 index 000000000..c72a25787 --- /dev/null +++ b/tests/llm/scripts/error-exit-quit.script @@ -0,0 +1,8 @@ +# exit: 1 +# The same error, followed by QUIT -- the natural way to end a script. +# QUIT leaves through the same exit-status logic as end of input, so +# the ERROR is still reported. The trailing GOALS proves QUIT really +# ended the session: its reply is absent from the golden. +UNDO +QUIT +GOALS From fa93fe6c2895440ce79c19a703e7024acc0caf5d Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Sat, 22 Aug 2026 10:51:28 +0200 Subject: [PATCH 45/51] [llm] LOAD honours `upto` for every sentence kind, not just commands The prefix loop only consulted [past_upto] for [P_Prog] items, so a [P_Undo] or a [P_DocComment] positioned after the requested stop point still ran. An `undo N.` on the line following `upto` therefore rewound the very prefix the caller asked for, and LOAD returned a state that is not the state at that line -- while still tagging its reply `[loaded:file:LINE]`, since [P_Undo] does not move [last_loc] either. Factor the check into [stop_at_upto] and apply it to every item the reader yields. [P_Exit] already stopped the loop unconditionally. New fixture fixtures/undoafter.ec (an `undo 3.` on the line after the stop point) and scenario load-upto-undo, whose golden shows the two goals `split.` opened and the uuid the prefix reaches. No existing golden changes. --- src/ecLlmCore.ml | 12 ++++++++++++ tests/llm/expected/load-upto-undo.out | 24 ++++++++++++++++++++++++ tests/llm/fixtures/undoafter.ec | 10 ++++++++++ tests/llm/scripts/load-upto-undo.script | 8 ++++++++ 4 files changed, 54 insertions(+) create mode 100644 tests/llm/expected/load-upto-undo.out create mode 100644 tests/llm/fixtures/undoafter.ec create mode 100644 tests/llm/scripts/load-upto-undo.script diff --git a/src/ecLlmCore.ml b/src/ecLlmCore.ml index ca0846326..7297decb7 100644 --- a/src/ecLlmCore.ml +++ b/src/ecLlmCore.ml @@ -894,6 +894,16 @@ let load (st : state) ~file ~upto ~nosmt ~trace = | Some c -> ec > c) in + (* [upto] stops the prefix at the requested position whatever kind + of sentence sits past it. This is applied to every item the + reader yields, not only to the [P_Prog] commands: an `undo N.` + on the line after [upto] used to run all the same, silently + rewinding the very prefix the caller asked for, so that LOAD + returned a state that was not the state at that line. *) + let stop_at_upto (item : _ EcLocation.located) = + if past_upto (EcLocation.loc item) then raise Exit + in + let last_loc = ref None in (* For -trace: lazy whole-file bytes, used to slice the exact @@ -948,11 +958,13 @@ let load (st : state) ~file ~upto ~nosmt ~trace = List.iter (step src) commands; if locterm then raise Exit | EP.P_Undo i -> + stop_at_upto prog; last_src := src; EcCommands.undo i | EP.P_Exit -> raise Exit | EP.P_DocComment doc -> + stop_at_upto prog; last_src := src; EcCommands.doc_comment doc done with diff --git a/tests/llm/expected/load-upto-undo.out b/tests/llm/expected/load-upto-undo.out new file mode 100644 index 000000000..f49a766c8 --- /dev/null +++ b/tests/llm/expected/load-upto-undo.out @@ -0,0 +1,24 @@ +READY [uuid:0] + +OK [uuid:4] [loaded:fixtures/undoafter.ec:8] [focus: 1/2] +Current goal (remaining: 2) + +Type variables: + +------------------------------------------------------------------------ +1 = 1 + +OK [uuid:4] [focus: 1/2] +Current goal (remaining: 2) + +Type variables: + +------------------------------------------------------------------------ +1 = 1 + + + + Goal #2 + ------------------------------------------------------------------------ + 2 = 2 + diff --git a/tests/llm/fixtures/undoafter.ec b/tests/llm/fixtures/undoafter.ec new file mode 100644 index 000000000..bea4944cd --- /dev/null +++ b/tests/llm/fixtures/undoafter.ec @@ -0,0 +1,10 @@ +(* An `undo` sits on line 9, right after the line a LOAD stops at. + Stopping at line 8 must leave the two goals `split.` opened: the + `undo` is past the stop point and must not run. *) +require import AllCore. + +lemma undo_after : 1 = 1 /\ 2 = 2. +proof. +split. +undo 3. +trivial. diff --git a/tests/llm/scripts/load-upto-undo.script b/tests/llm/scripts/load-upto-undo.script new file mode 100644 index 000000000..c2fea9828 --- /dev/null +++ b/tests/llm/scripts/load-upto-undo.script @@ -0,0 +1,8 @@ +# exit: 0 +# LOAD stops at the requested position whatever sentence sits after it. +# fixtures/undoafter.ec has `undo 3.` on the line following the stop +# point: the two goals `split.` opened must still be there, and the +# uuid must be the one that prefix reaches -- not the lower one the +# `undo` would rewind to. +LOAD "fixtures/undoafter.ec" 8 +GOALS ALL From 771ec846b5f00ee1abd823f3e9b70adfdca0bcef Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Sat, 22 Aug 2026 10:52:51 +0200 Subject: [PATCH 46/51] [llm] drop the -upto/-lastgoals machinery the batch llm mode left behind Replacing the old batch `llm` command with the REPL removed the only caller of both. Neither has had a command-line spec since: there is no "upto" or "lastgoals" in ecOptions, `State.upto` was hard-coded to [None] at all three construction sites, and no caller of [EcTerminal.from_channel] passed [?lastgoals]. Removed, all of it dead by construction: * ec.ml: the `State.upto` field and its three [None] sites; the [past_upto] helper (always [false]); the `-upto` early exit in the command loop (unreachable); the `state.upto = None` conjunct of the .eco freshness guard (always [true]). * ecTerminal.ml/.mli: the [?lastgoals] parameter of [from_channel] and the class it wraps, and the goal dump it guarded in [finish] (always [false], so never taken). Deliberately left alone: everything else reachable from compile, cli or docgen. [EcCommands.pp_current_goal_or_noproof] stays -- the LLM core is its remaining user -- as do [T.finalize] and the rest of the terminal interface. No behaviour change on any path, and no golden changes. --- src/ec.ml | 23 +---------------------- src/ecTerminal.ml | 7 ++----- src/ecTerminal.mli | 1 - 3 files changed, 3 insertions(+), 28 deletions(-) diff --git a/src/ec.ml b/src/ec.ml index f798081e9..68b704e91 100644 --- a/src/ec.ml +++ b/src/ec.ml @@ -465,7 +465,6 @@ let main () = (*---*) gccompact : int option; (*---*) docgen : bool; (*---*) outdirp : string option; - (*---*) upto : (int * int option) option; mutable trace : trace1 list option; } @@ -544,7 +543,6 @@ let main () = ; gccompact = None ; docgen = false ; outdirp = None - ; upto = None ; trace = None } end @@ -580,7 +578,6 @@ let main () = ; gccompact = cmpopts.cmpo_compact ; docgen = false ; outdirp = None - ; upto = None ; trace = trace0 } end @@ -631,7 +628,6 @@ let main () = ; gccompact = None ; docgen = true ; outdirp = docopts.doco_outdirp - ; upto = None ; trace = None } end @@ -651,7 +647,7 @@ let main () = EcCommands.set_current_path current_path); (* Check if the .eco is up-to-date and exit if so *) - (if not state.docgen && state.upto = None then + (if not state.docgen then oiter (fun input -> if EcCommands.check_eco input then exit 0) state.input); @@ -738,16 +734,6 @@ let main () = (* Warn about GC-regressed OCaml versions (5.0-5.3) *) warn_ocaml_version terminal; - (* Check if a location is past the -upto point *) - let past_upto (loc : EcLocation.t) = - match state.upto with - | None -> false - | Some (line, col) -> - let (sl, sc) = loc.loc_start in - sl > line || (sl = line && match col with - | None -> true - | Some c -> sc >= c) in - try if T.interactive terminal then Sys.catch_break true; @@ -817,13 +803,6 @@ let main () = (fun p -> let loc = p.EP.gl_action.EcLocation.pl_loc in - (* -upto: if this command starts past the target, print goals and exit *) - if past_upto loc then begin - T.finalize terminal; - EcCommands.pp_current_goal_or_noproof ~all:true Format.std_formatter; - exit 0 - end; - let timed = p.EP.gl_debug = Some `Timed in let break = p.EP.gl_debug = Some `Break in let ignore_fail = ref false in diff --git a/src/ecTerminal.ml b/src/ecTerminal.ml index c5f85bc81..ecda3f97e 100644 --- a/src/ecTerminal.ml +++ b/src/ecTerminal.ml @@ -148,7 +148,6 @@ type progress = [ `Human | `Script | `Silent ] class from_channel ?(gcstats : bool = true) ?(progress : progress option) - ?(lastgoals : bool = false) ~(name : string) (stream : in_channel) : terminal @@ -291,8 +290,6 @@ class from_channel let msg = String.strip (EcPException.tostring e) in self#_clean_progress_line (); - if lastgoals then - EcCommands.pp_current_goal_or_noproof ~all:true Format.std_formatter; self#_notice ?subloc ~immediate:true `Critical msg; self#_update_progress; self#_clean_progress_line ~erase:false (); @@ -317,5 +314,5 @@ class from_channel Format.pp_set_margin Format.err_formatter i end -let from_channel ?gcstats ?progress ?lastgoals ~name stream = - new from_channel ?gcstats ?progress ?lastgoals ~name stream +let from_channel ?gcstats ?progress ~name stream = + new from_channel ?gcstats ?progress ~name stream diff --git a/src/ecTerminal.mli b/src/ecTerminal.mli index faacff0e7..0a96a56d2 100644 --- a/src/ecTerminal.mli +++ b/src/ecTerminal.mli @@ -22,7 +22,6 @@ type progress = [ `Human | `Script | `Silent ] val from_channel : ?gcstats:bool -> ?progress:progress - -> ?lastgoals:bool -> name:string -> in_channel -> terminal From 9c284825dd766a21ffb860104fbb157ea20b6321 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Sat, 22 Aug 2026 10:54:57 +0200 Subject: [PATCH 47/51] [llm] cover ec_load's nosmt and trace options on the MCP side Both options change what the engine does, and neither had any MCP coverage: no tests/mcp script set them, and mcp-parity's STEPS did not play them either. The REPL side has had load-nosmt, load-trace and load-trace-notinproof all along, so this was an MCP-side gap. Two scenarios, reusing tests/llm/fixtures (no new fixture): * load-options: ec_load with nosmt on fixtures/simple.ec, then with trace on fixtures/midproof.ec -- the happy path of each. * load-trace-error: trace whose target sentence is outside a proof. The call is an isError result, and the ec_step that follows pins what the REPL's load-trace-notinproof pins: the prefix is in effect exactly as after a plain load, the deferred sentence included, so `b2i' still resolves. mcp-parity gains a load/nosmt and a load/trace step at the end of STEPS, where their session reset harms nothing, so the trace body -- the one reply body the core builds itself instead of handing back the goals -- is now compared between the two front-ends. 14 parity steps. Only new goldens; no existing golden changes. --- scripts/testing/mcp-parity | 11 +++++++++++ tests/mcp/README.md | 7 +++++-- tests/mcp/expected/load-options.out | 3 +++ tests/mcp/expected/load-trace-error.out | 3 +++ tests/mcp/scripts/load-options.script | 10 ++++++++++ tests/mcp/scripts/load-trace-error.script | 11 +++++++++++ 6 files changed, 43 insertions(+), 2 deletions(-) create mode 100644 tests/mcp/expected/load-options.out create mode 100644 tests/mcp/expected/load-trace-error.out create mode 100644 tests/mcp/scripts/load-options.script create mode 100644 tests/mcp/scripts/load-trace-error.script diff --git a/scripts/testing/mcp-parity b/scripts/testing/mcp-parity index 3901f637b..52756123d 100755 --- a/scripts/testing/mcp-parity +++ b/scripts/testing/mcp-parity @@ -60,6 +60,17 @@ STEPS = [ "ec_commit", {}), ("failure", 'apply nosuchlemma.', "ec_step", {"phrase": "apply nosuchlemma."}), + # The two load options that change what the engine does. Both + # reset the session, so they come last. `trace' is the one whose + # reply body the core builds itself, rather than handing back the + # goals -- the most front-end-independent body there is, and the + # one most worth pinning across both wires. + ("load/nosmt", 'LOAD "fixtures/simple.ec" 6 -nosmt', + "ec_load", {"file": "fixtures/simple.ec", + "line": 6, "nosmt": True}), + ("load/trace", 'LOAD "fixtures/midproof.ec" -trace', + "ec_load", {"file": "fixtures/midproof.ec", + "trace": True}), ] diff --git a/tests/mcp/README.md b/tests/mcp/README.md index 5db111487..31a8ab1a0 100644 --- a/tests/mcp/README.md +++ b/tests/mcp/README.md @@ -67,6 +67,8 @@ gate for changes to the protocol layer. | `protocol-errors` | `-32700`, `-32600`, `-32601` and the `-32602` family | | `revert` | `ec_revert` by uuid and by checkpoint name | | `load-missing` | a missing file and an unknown extension: `isError`, *not* `-32602` | +| `load-options` | `ec_load` with `nosmt`, and with `trace`: the two options that change what the engine does | +| `load-trace-error` | `trace` on a sentence outside a proof: `isError`, and the prefix still in effect | | `notifications` | notifications, known and unknown, draw no reply | | `exit` | `exit.` answers "session terminated", then the process stops | | `eof` | end of input is a clean shutdown, exit 0 | @@ -123,8 +125,9 @@ same answer on both. It plays one representative operation per tool family — load, step, goals, tree, focus, undo, checkpoint, step again, revert, search, -commit, and a failing phrase — in that order, against two sessions -started from the same directory (`tests/llm`, so both name the fixture +commit, a failing phrase, and finally a `nosmt` load and a `trace` load +(both reset the session, hence last) — in that order, against two +sessions started from the same directory (`tests/llm`, so both name the fixture identically and no path difference can leak into a reply): a REPL session driven with `llm -eval`, and an MCP session driven with a JSON-RPC script. For each step it asserts two things. diff --git a/tests/mcp/expected/load-options.out b/tests/mcp/expected/load-options.out new file mode 100644 index 000000000..4e14c5854 --- /dev/null +++ b/tests/mcp/expected/load-options.out @@ -0,0 +1,3 @@ +{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2025-11-25","capabilities":{"tools":{}},"serverInfo":{"name":"easycrypt","version":"VERSION"}}} +{"jsonrpc":"2.0","id":2,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n"}],"structuredContent":{"text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n","uuid":3,"changed":true},"isError":false}} +{"jsonrpc":"2.0","id":3,"result":{"content":[{"type":"text","text":"=== BEFORE: line 8 (col 0) ===\nCurrent goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n\n=== TACTIC (lines 8:0 - 8:6) ===\nsplit.\n\n=== AFTER: line 8 (col 0) ===\nType variables: \n\n------------------------------------------------------------------------\n1 = 1\n\n\nType variables: \n\n------------------------------------------------------------------------\n2 = 2\n\n\n=== SUMMARY ===\nopen goals: 1 -> 2\n"}],"structuredContent":{"text":"=== BEFORE: line 8 (col 0) ===\nCurrent goal\n\nType variables: \n\n------------------------------------------------------------------------\n1 = 1 /\\ 2 = 2\n\n=== TACTIC (lines 8:0 - 8:6) ===\nsplit.\n\n=== AFTER: line 8 (col 0) ===\nType variables: \n\n------------------------------------------------------------------------\n1 = 1\n\n\nType variables: \n\n------------------------------------------------------------------------\n2 = 2\n\n\n=== SUMMARY ===\nopen goals: 1 -> 2\n","uuid":4,"changed":true},"isError":false}} diff --git a/tests/mcp/expected/load-trace-error.out b/tests/mcp/expected/load-trace-error.out new file mode 100644 index 000000000..3579a4103 --- /dev/null +++ b/tests/mcp/expected/load-trace-error.out @@ -0,0 +1,3 @@ +{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2025-11-25","capabilities":{"tools":{}},"serverInfo":{"name":"easycrypt","version":"VERSION"}}} +{"jsonrpc":"2.0","id":2,"result":{"content":[{"type":"text","text":"trace: target sentence is not in a proof context\nNo active proof.\n"}],"structuredContent":{"text":"trace: target sentence is not in a proof context\nNo active proof.\n","uuid":1,"changed":true},"isError":true}} +{"jsonrpc":"2.0","id":3,"result":{"content":[{"type":"text","text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\nb2i true = 1\n"}],"structuredContent":{"text":"Current goal\n\nType variables: \n\n------------------------------------------------------------------------\nb2i true = 1\n","uuid":2,"changed":true},"isError":false}} diff --git a/tests/mcp/scripts/load-options.script b/tests/mcp/scripts/load-options.script new file mode 100644 index 000000000..7c47ca671 --- /dev/null +++ b/tests/mcp/scripts/load-options.script @@ -0,0 +1,10 @@ +# exit: 0 +# The two ec_load options that change what the engine does, which the +# other scenarios never set. `nosmt' weakens SMT calls while replaying +# the prefix; `trace' has the reply describe the last loaded sentence +# as BEFORE/TACTIC/AFTER/SUMMARY instead of just showing the goals. +# The REPL side of both is tests/llm's load-nosmt and load-trace. +{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{}}} +{"jsonrpc":"2.0","method":"notifications/initialized"} +{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"ec_load","arguments":{"file":"../llm/fixtures/simple.ec","line":6,"nosmt":true}}} +{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"ec_load","arguments":{"file":"../llm/fixtures/midproof.ec","trace":true}}} diff --git a/tests/mcp/scripts/load-trace-error.script b/tests/mcp/scripts/load-trace-error.script new file mode 100644 index 000000000..47ae5f48a --- /dev/null +++ b/tests/mcp/scripts/load-trace-error.script @@ -0,0 +1,11 @@ +# exit: 0 +# `trace' whose target sentence is outside any proof: tracing fails, so +# the call is an isError result -- but the prefix must be in effect +# exactly as after a plain ec_load, the deferred sentence included. The +# ec_step below resolves `b2i', which only the traced-and-failed +# `require import AllCore.' can have brought in. Mirrors tests/llm's +# load-trace-notinproof. +{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{}}} +{"jsonrpc":"2.0","method":"notifications/initialized"} +{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"ec_load","arguments":{"file":"../llm/fixtures/simple.ec","line":3,"trace":true}}} +{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"ec_step","arguments":{"phrase":"lemma preserved : b2i true = 1."}}} From a57852ff628f649c185ef2831fa924c7a5f94dad Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Sat, 22 Aug 2026 12:26:10 +0200 Subject: [PATCH 48/51] [llm] `pragma restart.` drops the checkpoint table too A checkpoint records an engine uuid, and a restart destroys the uuid space it was taken in. [step]'s Restart handler cleared the transcript but not the checkpoints, so a name set before `pragma restart.` still resolved afterwards -- to a uuid of the session that no longer exists. [load]'s handler already cleared both; the two had drifted. Factor the reset the two paths share (engine re-init, checkpoints, transcript) into [reset_session], and use it at all three sites, LOAD's own reset included, so they cannot drift again. New golden restart-checkpoint pins the fixed behaviour: the REVERT now fails with "'c' is not a valid uuid or checkpoint name" instead of reaching for a destroyed uuid. No existing golden changed. --- src/ecLlmCore.ml | 23 ++++++++++------- tests/llm/expected/restart-checkpoint.out | 28 +++++++++++++++++++++ tests/llm/scripts/restart-checkpoint.script | 10 ++++++++ 3 files changed, 52 insertions(+), 9 deletions(-) create mode 100644 tests/llm/expected/restart-checkpoint.out create mode 100644 tests/llm/scripts/restart-checkpoint.script diff --git a/src/ecLlmCore.ml b/src/ecLlmCore.ml index 7297decb7..b77611486 100644 --- a/src/ecLlmCore.ml +++ b/src/ecLlmCore.ml @@ -403,6 +403,17 @@ module Transcript = struct st.prior_bullets := None end +(* -------------------------------------------------------------------- *) +(* Full session reset. Both LOAD and a [pragma restart.] destroy the + uuid space, so every piece of bookkeeping keyed on engine uuids goes + with it: a surviving checkpoint would name a state of a session that + no longer exists, and REVERT would resolve it. Shared by [step] and + [load] so the two paths cannot drift apart again. *) +let reset_session (st : state) : unit = + do_initialize st; + Hashtbl.clear st.checkpoints; + Transcript.clear st + (* -------------------------------------------------------------------- *) (* Process a single EasyCrypt command, respecting [gl_fail]. When [~record:true], append a transcript entry on success: the parent @@ -780,8 +791,7 @@ let step (st : state) input = | Text _ as b -> Done (Ok (mk_reply st ~pre b)) with | EcCommands.Restart -> - do_initialize st; - Transcript.clear st; + reset_session st; Done (Ok (mk_reply st ~pre (Text "Session restarted"))) | e -> Done (Error (mk_failure st ~pre (Goals.format_error ~src:!last_src e))) @@ -837,7 +847,6 @@ let try_step (st : state) input = let load (st : state) ~file ~upto ~nosmt ~trace = let notices = st.notices in let cur_prvopts = st.cur_prvopts in - let checkpoints = st.checkpoints in let pre = EcCommands.uuid () in Buffer.clear notices; let filename = file in @@ -876,9 +885,7 @@ let load (st : state) ~file ~upto ~nosmt ~trace = ~recursive:isrec dir) (EcOptions.ini_loadpath ini); - do_initialize st; - Hashtbl.clear checkpoints; - Transcript.clear st; + reset_session st; EcCommands.addidir (Filename.dirname filename); EcCommands.set_current_path (Filename.dirname filename); @@ -1083,9 +1090,7 @@ let load (st : state) ~file ~upto ~nosmt ~trace = with | EcCommands.Restart -> - do_initialize st; - Hashtbl.clear checkpoints; - Transcript.clear st; + reset_session st; Ok (mk_reply st ~pre (Text "Session restarted")) | Trace_failed e -> let msg = Goals.format_error ~src:!last_src e in diff --git a/tests/llm/expected/restart-checkpoint.out b/tests/llm/expected/restart-checkpoint.out new file mode 100644 index 000000000..75439f3f4 --- /dev/null +++ b/tests/llm/expected/restart-checkpoint.out @@ -0,0 +1,28 @@ +READY [uuid:0] + +OK [uuid:3] [loaded:fixtures/simple.ec:6] +Current goal + +Type variables: + +------------------------------------------------------------------------ +1 = 1 /\ 2 = 2 + +OK [uuid:4] [focus: 1/2] +Current goal (remaining: 2) + +Type variables: + +------------------------------------------------------------------------ +1 = 1 + +OK [uuid:4] +checkpoint 'c' set at uuid 4 + +OK [uuid:0] +Session restarted + +ERROR [uuid:0] +REVERT: 'c' is not a valid uuid or checkpoint name +No active proof. + diff --git a/tests/llm/scripts/restart-checkpoint.script b/tests/llm/scripts/restart-checkpoint.script new file mode 100644 index 000000000..82c7faf4f --- /dev/null +++ b/tests/llm/scripts/restart-checkpoint.script @@ -0,0 +1,10 @@ +# exit: 1 +# A checkpoint names a uuid, and `pragma restart.' destroys the uuid +# space it was taken in. The restart must therefore drop the checkpoint +# table too: REVERT may not resolve the name at all afterwards, let +# alone reach a state of the session that no longer exists. +LOAD "fixtures/simple.ec" 6 +split. +CHECKPOINT c +pragma restart. +REVERT c From 8f959912036cea04a7ab8f13a025142d0603a74b Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Sat, 22 Aug 2026 12:37:24 +0200 Subject: [PATCH 49/51] [llm] document the front-end-facing half of ecCommands' interface The eight vals this branch added for the LLM front-ends were appended to the pretty-printing block with no comment, so nothing said they were front-end-only, nothing distinguished the pure queries from the two that rewrite the global context, and pp_tree's `(int * bool * string)' components were anybody's guess. Give them a section of their own with a header saying what they are for and which of them mutate, and one comment per val. pp_tree's tuple becomes a `goal_entry' record -- three components, and only two destructuring sites to update, so the maintainer's records-over-tuples rule costs nothing here. Pure refactor: no behaviour change, and no golden changed (test-llm 32/32, test-mcp 14/14, both byte-identical). --- src/ecCommands.ml | 16 +++++++------ src/ecCommands.mli | 57 +++++++++++++++++++++++++++++++++++++++++++--- src/ecLlmCore.ml | 12 +++++----- 3 files changed, 69 insertions(+), 16 deletions(-) diff --git a/src/ecCommands.ml b/src/ecCommands.ml index c446bff80..d63dd8b63 100644 --- a/src/ecCommands.ml +++ b/src/ecCommands.ml @@ -1332,12 +1332,14 @@ let pp_all_goals () = | _ -> [] (* -------------------------------------------------------------------- *) -(* Render the open-subgoals tree. Each entry is (index, is_focused, - text). [index] is 1-based, [is_focused] marks the focused goal - (always at index 1 with EC's current focus model), and [text] is - either a one-line conclusion digest (when [~all = false]) or the - full goal body (when [~all = true]). *) -let pp_tree ?(all = false) () : (int * bool * string) list = +type goal_entry = { + ge_index : int; + ge_focused : bool; + ge_text : string; +} + +(* Render the open goals of the active proof, focused first. *) +let pp_tree ?(all = false) () : goal_entry list = let scope = current () in match S.xgoal scope with | Some { S.puc_active = Some ({ puc_jdg = S.PSCheck pf }, _) } -> begin @@ -1358,6 +1360,6 @@ let pp_tree ?(all = false) () : (int * bool * string) list = else Format.asprintf "%a" (EcPrinting.pp_form ppe) g_concl in - (i + 1, i = 0, text)) goals + { ge_index = i + 1; ge_focused = i = 0; ge_text = text; }) goals end | _ -> [] diff --git a/src/ecCommands.mli b/src/ecCommands.mli index e1e656fec..d4daa6ff9 100644 --- a/src/ecCommands.mli +++ b/src/ecCommands.mli @@ -92,15 +92,66 @@ val pp_current_goal : ?all:bool -> Format.formatter -> unit val pp_current_goal_or_noproof : ?all:bool -> Format.formatter -> unit val pp_maybe_current_goal : Format.formatter -> unit val pp_all_goals : unit -> string list + +(* -------------------------------------------------------------------- *) +(* Proof-state introspection and navigation, for the LLM front-ends + ([EcLlmCore] and the REPL and MCP servers on top of it). Batch + compilation needs none of it: it never asks what the open goals are, + never walks between them, and never rewrites a proof's bullet state. + + [focus_goal] and [disable_repl_bullets] MUTATE the global context; + every other val here is a query that leaves it alone. *) + +(* One open subgoal, as [pp_tree] reports it. *) +type goal_entry = { + (* 1-based position in the open-goal list. *) + ge_index : int; + (* The focused goal -- always the one at index 1, EC's focus model + keeping the focused goal at the head. *) + ge_focused : bool; + (* The goal's conclusion on one line, or its full body under [~all]. *) + ge_text : string; +} + +(* Is a proof active in the current scope? *) val in_proof : unit -> bool -val disable_repl_bullets : unit -> EcBullets.stack option -val pp_tree : ?all:bool -> unit -> (int * bool * string) list -val focus_goal : int -> (int, string) result + +(* Every open goal of the active proof, rendered, focused first (the + order [open_handles] uses). [] when no proof is active. *) +val pp_tree : ?all:bool -> unit -> goal_entry list + +(* Handles of the active proof's open goals, focused first; [] when no + proof is active. Same goals as [pp_tree], unrendered. *) val open_handles : unit -> EcCoreGoal.handle list + +(* The active proof's environment, the one the DAG queries below read. + It is immutable and cumulative, so a snapshot keeps answering for its + own proof after [qed] has discarded it -- which is how COMMIT still + reconstructs the structure of a finished proof. [None] when no proof + is active. *) val current_proofenv : unit -> EcCoreGoal.proofenv option + +(* Proof-DAG navigation in the *active* proof; both answer emptily when + no proof is active. Use [EcCoreGoal.children_of_handle] / + [parent_of_handle] on a [current_proofenv] snapshot to query a proof + other than the active one. *) val children_of : EcCoreGoal.handle -> EcCoreGoal.handle list val parent_of : EcCoreGoal.handle -> EcCoreGoal.handle option +(* MUTATES the context: rotates the active proof's focus onto the open + goal at 1-based index [k], and pushes the result as a new undo level, + so UNDO/REVERT roll the rotation back like any other step. Returns + the number of open goals. *) +val focus_goal : int -> (int, string) result + +(* MUTATES the context: turns bullet enforcement off for phrases typed + at a prompt, by clearing the [strict_bullets] pragma and dropping the + active proof's bullet stack. Spends no undo level. Returns the stack + it dropped -- which COMMIT reads to pick bullet tokens that do not + collide with the ones already open -- and [None] on the idempotent + later calls, the stack being gone by then. *) +val disable_repl_bullets : unit -> EcBullets.stack option + (* -------------------------------------------------------------------- *) val pragma_verbose : bool -> unit val pragma_g_prall : bool -> unit diff --git a/src/ecLlmCore.ml b/src/ecLlmCore.ml index b77611486..936316ad9 100644 --- a/src/ecLlmCore.ml +++ b/src/ecLlmCore.ml @@ -284,8 +284,10 @@ module FrameTree = struct if handles = [] then [] else let leaves = - List.mapi (fun i (h, (_, focused, text)) -> - let leaf = Leaf { idx = i + 1; focused; text } in + List.mapi (fun i (h, (e : EcCommands.goal_entry)) -> + let leaf = + Leaf { idx = i + 1; focused = e.ge_focused; text = e.ge_text } + in (split_chain h, leaf)) (List.combine handles texts) in @@ -325,11 +327,9 @@ module FrameTree = struct (Printf.sprintf "[%s] %s%s\n" label (one_line text) marker) | Some entries -> - let (_, _, full) = - List.nth entries (idx - 1) - in + let e : EcCommands.goal_entry = List.nth entries (idx - 1) in Buffer.add_string buf - (Printf.sprintf "[%s]%s\n%s\n" label marker full)) + (Printf.sprintf "[%s]%s\n%s\n" label marker e.ge_text)) | Frame children -> List.iteri (fun i child -> emit ~depth:(depth + 1) ~path:((i + 1) :: path) child) From 0ab9a87d1147808924d4555906697028974f871c Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Sat, 22 Aug 2026 12:38:22 +0200 Subject: [PATCH 50/51] [llm] count open goals for the focus tag instead of rendering them [Goals.focus_tag] built the " [focus: 1/N]" annotation by calling [EcCommands.pp_tree], which pretty-prints the conclusion of every open subgoal, and then kept nothing but [List.length] of the result. Every reply that ends on the goals paid for that -- and on a wide split it is the whole open-goal set, rendered and dropped. Count [EcCommands.open_handles] instead. The two are equal by construction, not by coincidence: both match the active proof under the same [puc_active = Some ({ puc_jdg = PSCheck pf }, _)] guard and answer [] otherwise, and [all_hd_opened] is [pr_opened] while [all_opened] is [pr_opened] mapped through [get_pregoal_by_id] -- same list, same length, including the empty case ([opened] returns [None] exactly when [pr_opened] is []). Pure refactor: no golden changed. 21 goldens carry a focus tag, over the values 1/2, 1/3 and 1/4, and test-llm (32/32) and test-mcp (14/14) are byte-identical. --- src/ecLlmCore.ml | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/src/ecLlmCore.ml b/src/ecLlmCore.ml index 936316ad9..ad2859e7a 100644 --- a/src/ecLlmCore.ml +++ b/src/ecLlmCore.ml @@ -201,11 +201,14 @@ module Goals = struct Buffer.contents buf (* Inline focus annotation ([focus: 1/N]) appended to reply tags - whenever the active proof has >=2 open subgoals. *) + whenever the active proof has >=2 open subgoals. Counted on the + handles: [pp_tree] enumerates the same goals under the same guard, + but renders each one's conclusion on the way, and every reply that + ends on the goals asks for this tag. *) let focus_tag () = - match EcCommands.pp_tree () with - | _ :: _ :: _ as entries -> - Printf.sprintf " [focus: 1/%d]" (List.length entries) + match EcCommands.open_handles () with + | _ :: _ :: _ as handles -> + Printf.sprintf " [focus: 1/%d]" (List.length handles) | _ -> "" end From 4a070db8889b812f89aff84f0f98afd9beeb4a32 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Strub Date: Sun, 23 Aug 2026 13:13:43 +0200 Subject: [PATCH 51/51] [llm] share the argument checks both front-ends were duplicating Dotted-path parsing and the LOAD file-existence check existed twice, once per front-end, with the rejection wording copied along with them: ecLlm's parse_focus and ecMcp's focus_target both split on `.' and mapped int_of_string, and each front-end spelled its own "no such file" message. Two copies of a user-visible contract drift. Both now live in EcLlmCore as parse_goal_path and check_load_file. They return the message instead of raising, because how a rejection travels is genuinely the front-end's business -- a line-parse error at the prompt, a JSON-RPC error or an isError result over MCP -- while what is accepted and what is said about it is not. parse_goal_path takes the command's name so each caller keeps its own spelling (`FOCUS' or `ec_focus') in the message. check_load_file stays a separate call rather than moving inside [load]: load resets the session before it opens the file, so a missing path has to be caught before the call, not during it. Left deliberately duplicated: JSON-typed argument validation belongs to the MCP layer and line-oriented parsing to the REPL layer; only the front-end-agnostic middle moved. Pure refactor. Error wording is byte-identical and no golden changed. --- src/ecLlm.ml | 20 ++++++-------------- src/ecLlmCore.ml | 28 ++++++++++++++++++++++++++++ src/ecLlmCore.mli | 17 +++++++++++++++++ src/ecMcp.ml | 37 ++++++++++++++++--------------------- 4 files changed, 67 insertions(+), 35 deletions(-) diff --git a/src/ecLlm.ml b/src/ecLlm.ml index 78e5cbc19..60d2302d5 100644 --- a/src/ecLlm.ml +++ b/src/ecLlm.ml @@ -110,17 +110,9 @@ module Parse = struct let parse_focus arg = if arg = "" then raise (Parse_error "FOCUS: missing argument"); - let parts = String.split_on_char '.' arg in - let path = - try List.map int_of_string parts - with Failure _ -> - raise (Parse_error - (Printf.sprintf "FOCUS: not a path of integers: %s" arg)) - in - if List.exists (fun k -> k < 1) path then - raise (Parse_error - (Printf.sprintf "FOCUS: path indices must be >= 1: %s" arg)); - Focus path + match EcLlmCore.parse_goal_path ~what:"FOCUS" arg with + | Ok path -> Focus path + | Error msg -> raise (Parse_error msg) let parse_checkpoint name = if name = "" then @@ -175,9 +167,9 @@ module Parse = struct reader would otherwise raise [Sys_error] far downstream, and the REPL would report it as an anomaly after having already reset the scope. *) - if not (Sys.file_exists filename) then - failwith - (Printf.sprintf "LOAD: no such file: %s" filename); + (match EcLlmCore.check_load_file filename with + | Ok () -> () + | Error msg -> failwith msg); (* Parse optional LINE[:COL] and flags (-nosmt, -trace). *) let upto, nosmt, trace = diff --git a/src/ecLlmCore.ml b/src/ecLlmCore.ml index ad2859e7a..0a65abf01 100644 --- a/src/ecLlmCore.ml +++ b/src/ecLlmCore.ml @@ -738,6 +738,34 @@ let make_reply (st : state) ?tag (body : body) = let make_failure (st : state) (message : string) = mk_failure st ~pre:(EcCommands.uuid ()) message +(* -------------------------------------------------------------------- *) +(* Argument checks the two front-ends share. What a command accepts, and + what it says when it refuses, is the same at the prompt as over MCP; + only the way a rejection travels -- a line-parse error there, a + JSON-RPC error or an [isError] result here -- is the front-end's. *) + +(* A dotted goal path, as FOCUS accepts it: "2", "1.2.1". [what] is the + command's name as the calling front-end spells it, and opens the + error message. *) +let parse_goal_path ~(what : string) (arg : string) + : (int list, string) result += + match List.map int_of_string (String.split_on_char '.' arg) with + | exception Failure _ -> + Error (Printf.sprintf "%s: not a path of integers: %s" what arg) + | path when List.exists (fun k -> k < 1) path -> + Error (Printf.sprintf "%s: path indices must be >= 1: %s" what arg) + | path -> Ok path + +(* Reject a LOAD path naming no file. [load] resets the session before + it opens the file, so a front-end that skipped this would report the + reader's [Sys_error] against a session it had already destroyed -- + which is why the check belongs to the caller, and only its wording + here. *) +let check_load_file (file : string) : (unit, string) result = + if Sys.file_exists file then Ok () + else Error (Printf.sprintf "LOAD: no such file: %s" file) + (* -------------------------------------------------------------------- *) (* Process EasyCrypt input typed at the prompt. The input is a file fragment, not a single phrase: every sentence it holds runs, in diff --git a/src/ecLlmCore.mli b/src/ecLlmCore.mli index 48e6e0440..4a9a92e2f 100644 --- a/src/ecLlmCore.mli +++ b/src/ecLlmCore.mli @@ -127,3 +127,20 @@ val current_goals : state -> string val clear_notices : state -> unit val make_reply : state -> ?tag:string -> body -> reply val make_failure : state -> string -> failure + +(* -------------------------------------------------------------------- *) +(* Argument checks the two front-ends share. What a command accepts, and + what it says when it refuses, is the same at the prompt as over MCP; + only the way a rejection travels -- a line-parse error there, a + JSON-RPC error or an [isError] result here -- is the front-end's, so + these return the message rather than raising. *) + +(* A dotted goal path, as FOCUS accepts it: "2", "1.2.1". [what] is the + command's name as the calling front-end spells it ("FOCUS" at the + prompt, "ec_focus" over MCP), and opens the error message. *) +val parse_goal_path : what:string -> string -> (int list, string) result + +(* Reject a LOAD path naming no file. [load] itself does not check: + it resets the session before opening the file, so the check has to + happen in the front-end, before the call. *) +val check_load_file : string -> (unit, string) result diff --git a/src/ecMcp.ml b/src/ecMcp.ml index 371f05dc2..f9fc30645 100644 --- a/src/ecMcp.ml +++ b/src/ecMcp.ml @@ -475,21 +475,15 @@ module Args = struct end (* The [ec_focus] path is a string in the schema, so its shape is ours - to check: "next", or a dotted sequence of positive integers. *) + to check: "next", or a dotted sequence of positive integers. Only + "next" is MCP's own -- the REPL spells it as a separate command -- + so the path itself goes through the shared parser. *) let focus_target (arg : string) = if String.lowercase_ascii arg = "next" then `Next - else begin - let path = - try List.map int_of_string (String.split_on_char '.' arg) - with Failure _ -> - raise (Invalid_params - (Printf.sprintf "ec_focus: not a path of integers: %s" arg)) - in - if List.exists (fun k -> k < 1) path then - raise (Invalid_params - (Printf.sprintf "ec_focus: path indices must be >= 1: %s" arg)); - `Path path - end + else + match EcLlmCore.parse_goal_path ~what:"ec_focus" arg with + | Ok path -> `Path path + | Error msg -> raise (Invalid_params msg) (* -------------------------------------------------------------------- *) let run ~relocdir ~boot ~projini (mcpopts : EcOptions.mcp_option) = @@ -624,11 +618,12 @@ let run ~relocdir ~boot ~projini (mcpopts : EcOptions.mcp_option) = (* Tool dispatch. Argument checking happens here, before the engine is touched: the - core trusts what it is handed (it does not test that a LOAD path - exists, for one), and a raw [int_of_string] message has no business - reaching an agent. Schema violations raise [Invalid_params] and - become JSON-RPC errors; checks a tool makes on its own behalf raise - [Tool_error] and become [isError] results. *) + core trusts what it is handed, [EcLlmCore.load] resetting the + session before it so much as opens the file. Schema violations + raise [Invalid_params] and become JSON-RPC errors; checks a tool + makes on its own behalf raise [Tool_error] and become [isError] + results. The checks the REPL makes too live in [EcLlmCore] and are + only reported here. *) (* Set by a phrase that ends the session ([exit.]): the response still goes out, then the process stops. *) @@ -656,9 +651,9 @@ let run ~relocdir ~boot ~projini (mcpopts : EcOptions.mcp_option) = let trace = Args.bool_opt name args "trace" ~default:false in if line = None && col <> None then raise (Invalid_params "ec_load: `col' requires `line'"); - if not (Sys.file_exists file) then - raise (Tool_error - (Printf.sprintf "LOAD: no such file: %s" file)); + (match EcLlmCore.check_load_file file with + | Ok () -> () + | Error msg -> raise (Tool_error msg)); let upto = Option.map (fun line -> (line, col)) line in outcome (EcLlmCore.load st ~file ~upto ~nosmt ~trace)