document-assistant

A retrieval-augmented question answering system that runs entirely on your own machine. You point it at documents or source files, it indexes them locally, and you ask questions against that index. Embeddings are computed with sentence-transformers and answers are generated by a local Ollama model, so nothing is sent to a hosted API.

How it works

Ingestion and querying are separate paths through the same pipeline (src/rag_pipeline.py):

1. DocumentLoader reads the file and tags it as prose or code.
2. Prose goes to TextSplitter (512-character chunks, 50 overlap). Code goes to CodeSplitter, which uses larger 1000-character chunks with 100 overlap.
3. Embedder encodes each chunk with all-MiniLM-L6-v2.
4. ChromaStore persists vectors to ./chroma_db.
5. On a query, Retriever pulls the top 5 matches and converts Chroma's distance to a similarity score.
6. OllamaClient builds a prompt from those chunks and asks the model to cite its sources using [Source N] markers.

Supported files

PDF, DOCX, Markdown, and plain text are treated as prose. Twenty source extensions — including .py, .js, .ts, .java, .cpp, .go, .rs, .json, .yaml, and .html — are treated as code and chunked differently. Anything else is rejected.

Requirements

• Python 3.10 or later (the code uses list[dict] and str | Path)
• Ollama running locally with a model pulled

Setup

pip install -r requirements.txt
ollama pull llama3.2

Usage

python cli.py ingest ./docs -d      # index a directory
python cli.py ingest report.pdf     # index one file
python cli.py query "How does retrieval work?"
python cli.py stats                 # collection size and config
python cli.py clear                 # drop the index

Two servers are available:

python cli.py serve --api           # FastAPI on :8000
python cli.py serve --ui            # Gradio interface

The API exposes GET /health, GET /stats, POST /ingest, POST /query, and DELETE /documents.

Configuration

Settings are read from the environment with a RAG_ prefix, or from a .env file. Defaults live in src/config.py:

Variable	Default
RAG_OLLAMA_MODEL	llama3.2
RAG_OLLAMA_BASE_URL	http://localhost:11434
RAG_CHROMA_COLLECTION	documents
RAG_CHROMA_PERSIST_DIR	./chroma_db
RAG_CHUNK_SIZE	512
RAG_CHUNK_OVERLAP	50
RAG_TOP_K	5

Limitations

• There are no tests. tests/ contains an empty __init__.py and nothing else.
• Retrieval is dense-only. There is no keyword or hybrid search, so exact identifier lookups in code are weaker than semantic queries.
• Answer quality has not been measured. There is no evaluation harness and no benchmark numbers behind any of this.
• Chunking is fixed-size with overlap. It does not respect function or section boundaries, so a chunk can split mid-definition.
• Re-ingesting a file adds new chunks rather than replacing the old ones. Deduplication is by content hash at the document level only.
• PDF extraction quality depends on PyPDF2. Scanned documents will not work; there is no OCR step.
• The API has no authentication. Bind it to localhost or put something in front of it.

License

MIT