Improve large diagram scalability with viewport virtualization

[rat]

Summary

Improve DrawDB's behavior with very large diagrams by virtualizing canvas and side-panel rendering, coalescing viewport updates, and moving JSON/DDB validation off the main thread when Web Workers are available.

The diagram contexts remain the single source of truth. Culling only controls which React/SVG components are mounted, so off-screen entities remain available to search, export, editing, undo/redo, and other whole-diagram operations.

Problem

The canvas previously mounted every table, note, area, and relationship at the same time. Several components also scanned the complete diagram or rebuilt relationship paths during viewport interaction. On large imports this creates substantial DOM, React, SVG, and V8 heap pressure and can eventually crash Chromium's renderer even when system RAM and swap are available, because the renderer/V8 heap has per-process limits.

The side-panel lists and issue detection also performed work proportional to the complete graph during normal interaction, while JSON/DDB parsing and validation ran synchronously on the main thread.

Changes

• Add a lightweight incremental uniform-grid spatial index for tables, notes, and areas.
• Render only entities intersecting the viewport plus screen-space overscan. Relationships are included when either endpoint is visible, and selected/linking entities are pinned so interaction state is preserved.
• Keep global diagram arrays intact and use ID maps/relationship adjacency lookups instead of partial diagram state or repeated full-array searches.
• Batch pan and zoom state updates with requestAnimationFrame and cancel superseded frames without touching persistence or history.
• Memoize table and relationship components and reuse stable lookup data so unchanged entities and SVG paths are not recreated on every viewport update.
• Build global search metadata independently from mounted canvas components. Selecting an off-screen result moves the viewport to it before opening the entity.
• Virtualize large table and relationship side-panel lists while keeping selected rows mounted and navigable.
• Replace repeated graph scans in issue detection with indexed lookups and a linear graph traversal.
• Parse and validate JSON/DDB imports in a dedicated Worker when supported. The input ArrayBuffer is transferred rather than cloned, progress and cancellation are exposed in the UI, and unsupported/failed Worker environments retain a main-thread fallback.
• Terminate pending Workers, animation frames, observers, and derived indexes during cleanup.
• Add development-only diagnostics for total/rendered entity counts, viewport update duration, import duration, and approximate heap usage when performance.memory is available. Diagram contents and entity names are never logged.

Compatibility

• Import/export schemas and existing JSON, DDB, DBML, and SQL flows are unchanged.
• Canvas virtualization is a rendering layer only; exports still operate on complete diagram data.
• Selection, multi-selection, dragging, linking, pan/zoom, history, and side panels continue to use the original contexts/actions.
• Relationship double-click explicitly opens the relationship editor; it never performs deletion.
• No desktop runtime, browser flags, telemetry, network calls, or new runtime dependencies are introduced.

Validation

• npm run lint
• npm test — 3 regression test files pass, covering spatial lookup, off-screen global search, relationship double-click behavior, pan/zoom invariants, complete export data, import counts/references, and scalable cycle detection.
• npm run build
• Manual Chrome smoke test with a representative real-world diagram containing 275 tables, 4,058 fields, 1,021 relationships, 40 areas, 2 notes, and 95 enums: import completed without a renderer crash and the editor remained interactive.
• The deterministic spatial-index regression fixture keeps 1,000 logical tables while returning only the single table near its test viewport.

Limitations

This substantially reduces mounted component count and main-thread work, but a web application cannot remove Chromium/V8's intrinsic per-renderer heap limits. Operations that necessarily consume the complete diagram, such as export and auto-arrange, still require whole-diagram data; they now remain independent from the virtualized render set.

[vercel]

@rat is attempting to deploy a commit to the dottle's projects Team on Vercel.

A member of the Team first needs to authorize it.