Skip to content

epic: SQE catalog — replace sqlx-sqlite with sqlite-rs for internal metadata #621

Description

@iheitlager

Use Case

sqlite-rs as SQE's internal catalog storage — metadata for tables/indexes when no remote catalog is configured.

Context

SQE uses SQLite via sqlx-sqlite for the Iceberg SQL catalog. Currently depends on the C amalgamation via libsqlite3-sys. sqlite-rs could eliminate:

Current State

SQE's SQLite usage:

  • All access via sqlx, inside vendored Iceberg catalog
  • Four consumers: sql-sqlite catalog backend, embedded warehouse, ATTACH ... TYPE sqlite, test-sqlite coordinator
  • Low QPS (metadata path) — correctness critical, not perf critical

Gaps

  1. No consumable API — Need Connection/Statement facade
  2. Bound parameters — sqlx uses query(...).bind(...) throughout
  3. Send pagerPager is Rc/RefCell, blocks tokio pool

Path

  1. Probe: sqlite-rs query/dump on SQE warehouse, diff against sqlite3
  2. Corpus: Add SQE catalog statements to oracle suite
  3. API facade: Connection/Statement with prepare(), bind()
  4. Bounded integration: embedded warehouse only (single-process)
  5. Send pager: thread-safe for sqlx pool
  6. sqlx driver: full swap

Success

  • Reads/writes SQE warehouses identically to sqlite3
  • Public API with bind support
  • Send + Sync pager
  • sqlx-sqlite-rs driver passes catalog tests

Metadata

Metadata

Assignees

No one assigned

    Labels

    epicTracking issue spanning multiple tickets

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions