Sharded PyCG silently drops timed-out shards: call graph is load-dependent and not self-describing

[rahlk]

Problem

Sharded PyCG silently drops edges from shards that exceed their wall-clock timeout, so the L2+
call graph is neither reproducible nor self-describing. Three runs over the same 2,364-file
fixture (a dependency-closed Odoo subset), same flags --no-venv --pycg-shard --ray:

run	edges	jedi	pycg	jedi,pycg	externals
-a 2	48,595	26,309	18,962	3,324	4,492
-a 4	40,224	28,046	10,529	1,649	2,881
-a 2 (repeat)	43,431	27,720	13,751	1,960	3,340

The two -a 2 runs are byte-identical invocations and differ by 5,164 edges — 11%. PyCG's
contribution swings 44% between runs. This also breaks the stated
analysis.json(-a 1) ⊆ … ⊆ (-a 4) invariant, though that is a symptom rather than the defect.

The mechanism is documented in the code itself
(semantic_analysis/pycg/pycg_analysis.py:674): "A runaway shard contributes zero edges, so
splitting it recovers…". A shard exceeding shard_timeout is a runaway; its files are
re-partitioned at a tighter budget and retried, and once _PYCG_MAX_DECOMP_ROUNDS is hit or
splitting stops helping, the remaining files are counted into irreducible_files and skipped —
"accept Jedi-only rather than loop forever" (:723-731). Which shards run slow depends on machine
load and Ray scheduling, so the surviving edge set varies run to run.

Scope boundary

Makes incompleteness visible and reproducible. Does not aim to make PyCG faster, raise the
default timeout, or eliminate runaways.

Goals

Reporting the incompleteness was the original framing and is rejected: a
self-describing non-deterministic output is still non-deterministic. Remove the
load-dependence at its source instead.

• Decide shard outcomes on a function of the input, not the clock: a shard is a runaway when PyCG's fixpoint stopped at --pycg-max-iter rather than converging (has_converged())
• Keep adaptive decomposition — a runaway is still re-partitioned at a tighter budget, which is what recovers recall
• Keep a runaway's partial edges when it cannot be split further, instead of dropping the shard to zero: a capped fixpoint is a sound under-approximation, so those edges are real
• Remove --pycg-shard-timeout: it bounded the fixpoint a second time after --pycg-max-iter had already bounded it, and that second bound is the defect

Caveats and known risks

• L2 call graph is nondeterministic run-to-run (PyCG fixpoint under --pycg-max-iter) #99 closed the same symptom from a different cause (the PyCG fixpoint under --pycg-max-iter). Its determinism gate sorts edges so identical sets serialise identically, which cannot detect sets that genuinely differ — that gate will not catch this. Its fixture (requests, 35 files) is also below the sharding threshold, so it structurally cannot reach this path.
• Removing --pycg-shard-timeout is a breaking CLI change.
• Recall changes in both directions: partial edges are now kept (edges appear that previously did not), and slow-but-converging shards are no longer cancelled (more edges). Neither is a regression — both are edges PyCG actually derived.
• Nothing is left unbounded: --pycg-max-iter (default 50) already guarantees PyCG terminates, which is why the wall-clock bound was redundant.

Definition of done

• Two runs on the same input produce identical call graphs — not "differ and say so"
• The Odoo fixture (2,364 files, --no-venv --pycg-shard --ray) yields the same edge count across repeated -a 2 runs on a loaded and an idle machine
• No wall-clock deadline, ray.cancel, or SIGALRM bound remains in any shard-collection path
• --pycg-shard-timeout is gone from the CLI, the options model, and the analyzer