docground

Extract with sight. Prove with math.

docground is a TypeScript SDK that extracts structured data from documents using multimodal vision-language models. Define your output as a Zod schema, pass a PNG, JPEG or a real PDF, and receive typed data, per-field visual groundings, extracted tables, and optional invariant checks.

---

One-line demo

Set GEMINI_API_KEY, OPENAI_API_KEY or ANTHROPIC_API_KEY (or pass apiKey in options):

import { z } from "zod";
import { DocumentExtractor } from "docground";

const Invoice = z.object({ invoiceNumber: z.string(), total: z.number() });
const result = await new DocumentExtractor().extract("invoice.pdf", Invoice, {
  provider: "gemini",
  prove: true,
  invariants: [{ id: "balance", expression: "subtotal + tax === total" }],
});

You get result.data, result.groundings, result.tables, result.invariants and result.provenance.

---

Why docground?

Capability	Classic OCR	docground
Schema-guaranteed output	❌	✅ via Zod
Native PDF multimodal input	❌	✅ (Gemini, Anthropic)
Per-field visual grounding	❌	✅ [ymin, xmin, ymax, xmax]
Multi-page table stitching	❌	✅ across tiles & pages
Invariant checks / auto-repair	❌	✅ built-in math engine
Provider switch (Gemini/OpenAI/Anthropic)	❌	✅ one-line

---

Architecture

┌─────────────┐      ┌──────────────┐      ┌─────────────────┐
│   Input     │      │  Document    │      │   VLM Provider  │
│ PNG/JPG/PDF │─────▶│  Extractor   │─────▶│ Gemini / OpenAI │
└─────────────┘      └──────┬───────┘      │    / Anthropic  │
                            │              └────────┬────────┘
                            │                       │
              ┌─────────────┼─────────────┐         │
              │             │             │         │
              ▼             ▼             ▼         │
       ┌──────────┐  ┌──────────┐  ┌──────────┐     │
       │  Tiler   │  │  Cutter  │  │ Invariant│◀────┘
       └──────────┘  │ (sharp)  │  │  Engine  │
                     └──────────┘  └────┬─────┘
                                        │
                                        ▼
                              ┌─────────────────┐
                              │  Table Stitcher │
                              └─────────────────┘

---

Installation

npm install docground
# or
yarn add docground
# or
pnpm add docground

---

Quick start

import { z } from "zod";
import { DocumentExtractor } from "docground";

const LabReport = z.object({
  patientName: z.string(),
  biomarkers: z.array(
    z.object({ marker: z.string(), value: z.number(), unit: z.string() })
  ),
});

const result = await new DocumentExtractor().extract(
  "lab-report.pdf",
  LabReport,
  { provider: "gemini" }
);

console.log(result.data);
console.log(result.groundings);

Zero-Config Schema Discovery

Don't know the structure of a PDF? Let docground inspect it first and suggest a Zod schema.

const extractor = new DocumentExtractor();

const { detectedType, summary, inferredZodCode } = await extractor.inferSchema(
  "mystery-document.pdf"
);

console.log("Type:", detectedType);   // e.g. "invoice", "business case"
console.log("Summary:", summary);
console.log("Suggested schema:");
console.log(inferredZodCode);         // ready-to-use `const SomethingSchema = z.object({ ... })`

inferSchema performs a fast visual pass over the document, classifies the document type and emits the most likely fields with their Zod-compatible types. You can then refine the generated code, paste it into your project and call extract() with it.

---

Public API Reference

new DocumentExtractor(config?)

Creates an extractor with an optional default provider.

new DocumentExtractor({
  provider: "gemini",
  apiKey: process.env.GEMINI_API_KEY,
  model: "gemini-3.7-flash",
});

extract(input, schema, options?)

Extracts structured data from a file path, Buffer, Uint8Array or data URI.

Option	Description
provider	gemini, openai, anthropic or custom
model	Model ID; falls back to provider default
apiKey	API key override
prove	Enable invariant checks
invariants	Array of { id, expression, tolerance? } checks
autoRepair	Re-run extraction when invariants fail
tiling	auto, off, or explicit TileOptions
prompt	Extra instructions for the model
maxTokens	Max output tokens
temperature	Sampling temperature
seed	Deterministic seed
onProgress	Progress callback (p) => void

inferSchema(input, options?)

Inspects a file, Buffer, Uint8Array or data URI and suggests a Zod schema.

const discovery = await new DocumentExtractor().inferSchema("unknown.pdf", {
  provider: "gemini",
  maxTokens: 4096,
});

Return field	Description
detectedType: string	Real-world document type, e.g. invoice, business case, contract
summary: string	Brief description of what the document contains
inferredZodCode: string	Generated const ...Schema = z.object({ ... }) string you can refine and use

ExtractionResult<T>

Field	Description
data: T	Typed payload (parsed with your Zod schema)
confidence: number	Global confidence in [0, 1]
fields	Per-field value + confidence + bbox
tables	Extracted and stitched tables
groundings	Visual grounding records
invariants	Invariant report when prove: true
provenance	Provider, model, duration, tiles used, repairs

Invariants

Use plain JavaScript expressions with comparisons or aggregates:

invariants: [
  { id: "balance", expression: "subtotal + tax === total" },
  { id: "items_sum", expression: "sum(items.lineTotal) === subtotal" },
  { id: "vat_positive", expression: "tax > 0" },
];

---

Supported providers & formats

Provider	Images	PDFs	Notes
Gemini	✅	✅ Native	Send application/pdf inline
Anthropic	✅	✅ Native	Document block with application/pdf
OpenAI	✅	❌	Convert PDF to images before sending
Custom	✅	✅	Implement VLMProvider interface

---

Recommended Vision Models

docground requires a multimodal (VLM) model with both visual/spatial understanding and Structured Outputs / JSON mode. The following model IDs are known to work as of August 2026; always pin to a model snapshot in production to avoid unexpected behavior.

• Google Gemini — gemini-3.7-flash, gemini-3.6-flash, gemini-3.5-flash-lite, gemini-3.1-pro.

  • Native PDF input, 1M context and strong document/infographic reading.
  • gemini-3.7-flash is the current Flash workhorse for vision and agentic work; gemini-3.6-flash is the prior generation; gemini-3.5-flash-lite is the lower-cost GA option.

• OpenAI — gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna.

  • As of August 2026, these are OpenAI's production vision models; all support low/high/original/auto image detail.
  • Use original detail for OCR and coordinate-sensitive work; PDFs must be converted to images before sending.

• Anthropic — claude-fable-5, claude-opus-5, claude-sonnet-5.

  • claude-sonnet-5 is the current default for most vision and extraction work; claude-opus-5 handles complex agentic coding and dense layouts; claude-fable-5 is the highest-capability option.
  • Supports both application/pdf document blocks and standard images.

You can also wire any open-weight VLM (e.g. Qwen, LLaVA, InternVL) by implementing the VLMProvider interface and passing it as customVLM.

---

Environment variables

Create a .env file from .env.example:

GEMINI_API_KEY=...
OPENAI_API_KEY=...
ANTHROPIC_API_KEY=...

# Optional: override the default model per provider
GEMINI_MODEL=gemini-3.7-flash
OPENAI_MODEL=gpt-5.6-luna
ANTHROPIC_MODEL=claude-sonnet-5

---

Examples

npx tsx examples/01_invoice_with_grounding.ts
npx tsx examples/02_clinical_lab_report.ts
npx tsx examples/03_legal_pdf_extraction.ts

---

Development

npm install
npm run generate:fixtures
npm run typecheck
npm run test
npm run build

---

License

MIT © DocGround