Skip to content

fix: serve pg_class and pg_namespace from SQLite instead of Rust handlers (#87) - #101

Merged
erans merged 19 commits into
mainfrom
worktree-fix+pg-class-from-sqlite-87
Aug 11, 2026
Merged

fix: serve pg_class and pg_namespace from SQLite instead of Rust handlers (#87)#101
erans merged 19 commits into
mainfrom
worktree-fix+pg-class-from-sqlite-87

Conversation

@erans

@erans erans commented Aug 11, 2026

Copy link
Copy Markdown
Owner

Fixes #87

The bug

\dt reported "Did not find any tables" on a database that has tables. Three
mechanisms combined to turn one psql predicate into a universal row filter:

  1. RegexTranslator runs before catalog interception and rewrites
    n.nspname !~ '^pg_toast' into NOT regexp('^pg_toast', n.nspname).
  2. WhereEvaluator has no case for regexp(). Unknown functions fall through
    to true — "default to including the row".
  3. NOT inverts that default into an exclusion. !true is false. Every row
    fails.

The dedicated PGRegexMatch/PGRegexNotMatch arms in WhereEvaluator are dead
code on this path — the operator is gone before the AST reaches them.

Two further defects sat behind it, so a narrow regex fix would not have worked:
PgClassHandler synthesized only its own columns, so the LEFT JOIN pg_namespace was never materialized and n.nspname evaluated to None
(\dt's n.nspname <> 'pg_catalog' passed only by accident, because
None != Some(..)); and joins were not executed at all.

The fix

Rather than patch the WHERE evaluator, delete the hand-rolled Rust handlers for
pg_class and pg_namespace and let SQLite execute those queries against real
views. Joins, WHERE, regexp(), ORDER BY, and projection all become the
engine's job. Migration v28 enriches the pg_class view from 25 to 33 columns
so nothing is lost, computing relnatts from pragma_table_info and
relhastriggers from sqlite_master rather than hardcoding them.

The alternative of bailing out of interception on hard queries was rejected: the
bail-list is itself a guess-list, and a query that succeeds while returning
wrong data never triggers a fallback. Deleting the handler leaves one code path
and no decision logic to drift — a missing column is now a loud SQLite error
rather than silent wrong rows.

Design: docs/superpowers/specs/2026-08-10-pg-class-sqlite-engine-design.md

Also fixed

Found while implementing; each was silently returning wrong answers.

  • ~ was silently ignored on catalog tables, returning unfiltered results
    with no error.
  • Cross-catalog joins through pg_class.oid never matched. pg_class
    served hash31 OIDs while pg_constraint, pg_index, pg_attrdef, and
    pg_depend persist unicode-formula OIDs. Measured on a fresh database:
    pg_class.oid(customers) = 197947 but pg_constraint.conrelid = 51945, so
    FK discovery — the first ORM example in CLAUDE.md — returned zero rows. Four
    different OID formulas existed; there is now one,
    crate::utils::generate_table_oid, which every producer delegates to.
  • --in-memory catalog queries hard-errored. Migrations ran on a throwaway
    :memory: connection while each session opened its own private empty
    database. Now uses a shared-cache URI with a keepalive connection.
  • \dt no longer lists pgsqlite's own pg_* relations, because v28 assigns
    them to the pg_catalog namespace (oid 11) and information_schema_* to a
    new information_schema namespace (oid 13000). No special-casing in
    pgsqlite — psql's own nspname predicate does the hiding.
  • generate_table_oid and generate_oid overflowed u32 on a leading
    codepoint above 4294, panicking in debug builds and silently wrapping in
    release. Both now compute in u64.
  • The Rust and SQL OID formulas were not the same function. Rust used
    name.len() (UTF-8 bytes); SQLite uses length() (characters). café
    produced 996619 in Rust and 996612 in SQL, so any non-ASCII table name got
    a pg_class.oid matching none of its persisted join keys. Every ASCII value
    is unchanged — customers is still 197947.

Three existing view bugs fixed in passing: relkind_full is not a real
PostgreSQL column (dropped), relreplident should be 'd' not 'v', and
relispartition should be 'f' not 't'.

Behavior changes worth knowing

  • --in-memory sessions now share one database. Previously each session got
    its own (empty, unmigrated) one, which is why catalog queries failed. This is
    the fix, but it is a real semantic change.
  • pg_class OIDs changed value from hash31 to the unicode formula. This
    removes an inconsistency — the new values are the ones the rest of the
    catalog already stored on disk — but any external tooling that memoized a
    pg_class OID will see it move.
  • Non-ASCII table names get a new OID. Any such OID already persisted was
    written by the byte-counting version and will not match until a data
    migration. The prior state was already broken (view and persisted value
    disagreed), so this is a fix-forward.
  • relhasindex is hardcoded 't' and relfilenode '0' in the view, where
    the deleted handler computed them from PRAGMA index_list and the oid. Both
    inherited from the v26 view and both fail safe.
  • \dt now hides pgsqlite's internals while information_schema.tables still
    shows them — it builds its own row set with a hardcoded 'public'. That gap
    is information_schema.tables lists pgsqlite's materialized pg_* and information_schema_* relations #88.

Testing

1302 passed, 0 failed, 195 binaries. Baseline before this branch was 1296/194.

New coverage: psql's exact \dt query (asserting customers appears and
that pg_constraint/pg_attrdef/pg_index/pg_depend do not); ~ and !~
each pinned with a positive and a negative case so neither can pass vacuously;
pg_class JOIN pg_constraint ON conrelid = oid returning rows; all 33 columns
selectable with relnatts matching real column counts; the OID formula pinned
against values computed independently through the sqlite3 CLI; and cross-session
pg_class visibility on the in-memory URI.

\dt was also verified end-to-end through real psql 18 against the built binary.

Follow-ups filed

#93 OID collisions · #94 WhereEvaluator unknown-predicate-under-NOT ·
#95 migrate the remaining catalog handlers · #96 JOIN routing bypassed by the
system-function gate · #97 pooling path registers no UDFs · #98
obj_description() can never match · #99 regclass non-canonical OID ·
#100 PgAttributeHandler missing COUNT(*)

🤖 Generated with Claude Code

erans and others added 19 commits August 10, 2026 11:43
\dt returns no tables because psql's `n.nspname !~ '^pg_toast'` predicate is
rewritten to `NOT regexp(...)` before catalog interception, WhereEvaluator
defaults unknown functions to true, and NOT inverts that into a universal
row filter.

Design deletes PgClassHandler and lets SQLite execute pg_class queries, since
the views and required UDFs already exist. Also records two further defects
found while investigating: `~` is silently ignored on catalog tables, and
pg_class OIDs (hash31) already disagree with the persisted OIDs (unicode
formula) used by pg_constraint/pg_index/pg_attrdef.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
8 tasks, TDD throughout: failing test for #87, trusted_schema guarantee,
migration v28 (enriched pg_class view + information_schema namespace),
removal of PgClassHandler, regression tests for the regex operators and
cross-catalog joins, then follow-up issues.

Also corrects the spec: the new migration is v28, not v26 -- v26 and v27
are both registered, and v26 owns the current pg_class view.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pragma_table_info() is a virtual table; SQLite refuses virtual tables
inside views unless trusted_schema is ON. It defaults to ON in the C
API (so this already works), but the sqlite3 CLI disables it, and a
future default change or defensive-mode setting could silently break
Task 3's pg_class.relnatts column. Pin it explicitly on every
connection-init site that registers UDFs: the per-session connection
pool, the shared initial/migration connection, and the --migrate CLI
path.

Add a regression test that creates a view calling pragma_table_info()
and queries it, pinning the guarantee end-to-end over the wire
protocol.
Adds a pg_class view with the full 33 PostgreSQL columns and a
pg_namespace view with pg_catalog/public/information_schema rows, so
SQLite can serve pg_class directly instead of the hand-rolled
PgClassHandler evaluator. Internal pg_*/information_schema_* relations
are assigned to their proper namespaces via the relname pattern.

PgClassHandler still intercepts pg_class queries and shadows the new
view until it is removed in a follow-up task; most new tests in
tests/pg_class_view_test.rs are expected to fail until then. Also
updates tests/migration_test.rs's hardcoded migration counts (27->28)
to match the new migration.
Implementing Task 3 revealed pg_namespace is intercepted separately by
handle_pg_namespace_query, which hardcodes two rows and shadows v28's new
information_schema namespace. Human ruling: remove that branch in Task 4.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…lers (#87)

Removes the pg_class and pg_namespace interception branches in
CatalogInterceptor::check_table_factor along with the now-orphaned
PgClassHandler module and handle_pg_namespace_query function, so these
queries fall through to SQLite and execute against the v28 catalog
views instead of the hand-rolled Rust responders. Also removes
is_count_star_projection/count_response, which were only reachable
from the deleted pg_namespace handler.

This is the fix for \dt reporting "Did not find any tables": the
RegexTranslator-rewritten pg_table_is_visible/regexp predicates in
the \dt query were being evaluated by WhereEvaluator's default-true
fallback for unknown functions, inverted by NOT into an always-false
filter. Routing to SQLite avoids that evaluator entirely.
… stale pg_namespace test expectations

Code review response for the pg_class/pg_namespace-from-SQLite change (#87):

- --in-memory mode: switch to a shared-cache SQLite URI
  (file:pgsqlite_mem?mode=memory&cache=shared) and keep the migration
  connection alive for the DbHandler's lifetime, since SQLite destroys a
  shared-cache in-memory database as soon as its last connection closes.
  Without this, the pg_class/pg_namespace views created at migration time
  vanished before the first session connection opened, breaking \dt and
  every other catalog query in --in-memory mode.

- Unify all table-identity OID generation on one canonical formula
  (crate::utils::generate_table_oid), replacing the old hash31 formula in
  pg_attribute.rs, pg_trigger.rs, and pg_sequence.rs. Those had drifted
  from the formula used by the v28 pg_class view and
  constraint_populator.rs, breaking pg_class.oid joins to
  pg_attribute.attrelid/pg_trigger.tgrelid/pg_sequence's table OID in the
  default file-backed mode.

- Repoint catalog_alias_test.rs's pg_namespace-only assertions to
  pg_roles/pg_database, since pg_namespace is now a SQLite view (3 rows,
  including information_schema) rather than a Rust-intercepted catalog,
  and the tests exist to exercise CatalogInterceptor's projection logic.

- Apply the same shared-cache-URI fix to catalog_where_simple_test.rs,
  which constructs DbHandler directly and bypasses main.rs.

- Reword a stale "PgClassHandler" doc comment in migration/registry.rs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pg_constraint.rs:256 has a fourth OID formula (DefaultHasher) and synthesizes
conrelid at query time, so pg_class JOIN pg_constraint returns zero rows.
Measured: pg_class.oid=197947 vs pg_constraint.conrelid=51945.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…nt joins to SQLite (#87)

pg_constraint.rs synthesized conrelid/confrelid with Rust's DefaultHasher,
diverging from the canonical crate::utils::generate_table_oid formula that
pg_class, pg_attribute, and constraint_populator already agree on. Delegate
to the shared implementation instead, matching pg_attribute.rs and
constraint_populator.rs.

That alone was not sufficient: the query interceptor routed any
pg_class-main JOIN into another SQLite-backed catalog table (pg_constraint,
pg_index, pg_depend, ...) through PgConstraintHandler's single-table WHERE
evaluator, which strips qualified identifiers like c.oid/c.relname down to
bare column names and evaluates them against pg_constraint's own row data -
so the join predicate could never match regardless of OID consistency.
Extend the existing "let SQLite handle it" fallback (already used when
pg_index or pg_attribute is the main table) to also apply when pg_class is
the main table and the join target is one of those real SQLite-backed
catalog tables, so the query executes as genuine SQL against the pg_class
view and the persisted pg_constraint table.

Add tests/catalog_join_test.rs covering both the OID formula and the
cross-catalog join.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…fallback (#87)

Review finding on the pg_class/pg_constraint JOIN fix: pg_stats and
pg_tablespace are handled entirely by the catalog interceptor with no
backing SQLite view (migrations v19 and v24 say so explicitly). They had
leaked into the "let SQLite handle this JOIN" fallback list, which is only
correct for tables that really exist as SQLite views/tables. On the default
connection this was masked by substring interception further down
db_handler.rs, but on the connection-pooling path
(query_router.rs -> read_only_handler.rs) there is no such rescue, so a
pg_class/pg_tablespace or pg_class/pg_stats JOIN would fail with
"no such table" under PGSQLITE_USE_POOLING=true.

Remove both from the fallback list and extract the now-shared 7-name list
into is_sqlite_backed_catalog() so the two call sites (main-table check and
the new join-target check) can't drift again - the drift is what let
pg_stats/pg_tablespace end up in the list in the first place.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Full test suite (1296 tests, 194 binaries) passes with zero failures;
earlier commits on this branch already fixed the expected pg_class OID
and pg_namespace fallout, so no test files needed changes. Only
CLAUDE.md's stale "Current Migrations (v1-v25)" section needed updating
to v28, documenting v26-v27 and the new v28 pg_class/pg_namespace work.
…ly (#87)

generate_table_oid used name.len() (UTF-8 byte count) where the v28 pg_class/
pg_namespace view SQL uses SQLite's length(name) (character count), so any
non-ASCII table name got a pg_class.oid that didn't match its persisted
conrelid/indrelid/adrelid/attrelid, causing silent zero-row joins. The
per-character arithmetic also overflowed u32 for high-codepoint leading
characters (e.g. "日本語"), panicking in debug builds.

Switch length to name.chars().count() and widen the intermediate arithmetic
to u64 before reducing with % 1_000_000 + 16_384, which keeps the result
well within u32 range. All ASCII OIDs (e.g. "customers" -> 197947) are
byte-identical to before. Non-ASCII OIDs necessarily change to agree with
the SQL side; any such OID already persisted on disk was produced by the old
byte-counting formula and needs a follow-up data migration, not attempted
here.

Adds unit tests pinning the ASCII value, verifying non-ASCII values against
the sqlite3 CLI running the identical v28 SQL expression, and covering a
high-codepoint leading character that used to panic.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#87)

- object_resolver.rs: correct the comments on resolve_table_oid and its
  private generate_table_oid helper, which claimed to use "the same
  algorithm as pg_class view". They actually use a DefaultHasher and
  intentionally diverge from crate::utils::generate_table_oid; changing the
  formula would require a data migration for persisted
  __pgsqlite_comments.object_oid rows, so only the comment is corrected.
- query_interceptor.rs: convert the println! debug traces that fire on
  every catalog query to debug!, using the tracing macro already imported
  in this file.
- db_handler.rs: convert the eprintln! in query_with_session, which printed
  every session query's text to stdout, to debug!.
- catalog_alias_test.rs: rename four test functions still named
  test_pg_namespace_* even though pg_namespace is now served directly from
  SQLite and their bodies query pg_roles instead (test bodies unchanged).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
No test exercised the shared-cache in-memory URI
(file:pgsqlite_mem?mode=memory&cache=shared) that src/main.rs uses for
--in-memory against pg_class specifically. Reverting that URI to bare
":memory:" would leave the whole suite green while re-breaking the hard
error ("no such table: pg_class") that a second session's private, empty
in-memory database used to produce.

Adds a regression test that opens a DbHandler on the shared-cache URI,
creates a table via one (temporary, memory-mode) session connection, then
queries pg_class from an independently created second session connection
and asserts the table is visible there. Verified the test actually guards
the regression by temporarily switching its URI to bare ":memory:", which
reproduces the exact "no such table: pg_class" failure, then reverting.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The regression test added for the --in-memory fix hardcoded its own copy of
the shared-cache URI, so reverting src/main.rs to a bare ":memory:" left the
test green. It guarded DbHandler's keepalive and cross-session visibility,
but not the decision that was actually broken.

Move that decision into Config::resolve_db_path(), backed by
in_memory_db_uri()/IN_MEMORY_DB_NAME, and have both main.rs and the test go
through it. Verified by temporarily making resolve_db_path return ":memory:":
the test fails with "--in-memory must not resolve to a bare :memory:".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
generate_table_oid was widened to u64 earlier in this branch because
char1 * 1_000_000 overflows u32 once the leading codepoint exceeds 4294 —
a panic in debug builds, a silent wrap in release. Its sibling generate_oid
in the same module still had the identical defect, and reaches persisted
catalog OIDs through migration v5's populate_catalog_tables, so a database
containing such a table name panicked on the upgrade path.

Widen the same way. This is strictly panic-eliminating: measured against the
pre-widening body with checked arithmetic, every input that already produced
a value produces the same value, non-ASCII names included (café -> 1007190
before and after). Only previously-overflowing inputs change, from panic to
a defined value. Pinned in tests.

Also drop the module's doc comment claiming generate_oid "uses the same
formula as the pg_class view" — it is a different, six-position formula for
constraint and sequence OIDs, and only generate_table_oid matches the view.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@erans
erans merged commit a264216 into main Aug 11, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

\dt reports "Did not find any tables" on a database that has tables

1 participant