Local Multi-Model LLM Inference with Desktop UI, CLI & API

Preprint Paper Link (https://doi.org/10.32388/7MLDM5)
🤩 Star this repository - It helps others discover SnapLLM 🤩

How_To_Start_Server_Locally.mp4

Developed by Mahesh Vaikri

What is SnapLLM?

SnapLLM is a local inference engine built on llama.cpp and stable-diffusion.cpp. It can keep multiple models loaded and select an already-resident model without reloading its weights. Actual load, switch, and generation latency depends on model size, hardware, memory pressure, and build configuration; benchmark your own workload before setting latency targets.

The Problem

One common single-resident-model workflow:

Load Model A
Use Model A
Unload Model A
Load Model B
Use Model B

The SnapLLM Solution

Load Model A
Load Model B
Select A or B without an avoidable reload while both remain resident

Features

Core Capabilities

Feature	Description
Resident Model Selection	Select between loaded models without reloading their weights
Reload After Eviction	Memory mapping can benefit from the operating-system page cache
Multi-Model Management	Load and manage multiple models simultaneously
GPU/CPU Hybrid	Automatic layer distribution based on available VRAM
OpenAI-Compatible API	Drop-in replacement for OpenAI API
Multi-Modal Support	LLM, Vision (VLM), and Stable Diffusion models
KV Cache Persistence	Persist context state and use indexed cache lookup; generation work still depends on query and context size
Web UI	React dashboard served by Vite at `localhost:9780`; the API listens on `localhost:6930`
Desktop Application	Beautiful React-based UI for model management

Supported Model Types

Text LLMs: Llama 1/2/3, Mistral, Mixtral, Qwen, Gemma, Phi, DeepSeek, and all GGUF models
Vision Models: Gemma 3 Vision, Qwen-VL, LLaVA (with mmproj files)
Diffusion Models: Stable Diffusion 1.5, SDXL, SD3, FLUX (via stable-diffusion.cpp)

Performance

SnapLLM reports operation timing at runtime where available. Published end-to-end latency guarantees require a reproducible benchmark that records the model, quantization, prompt, build flags, hardware, cache state, and sampling configuration. No such guarantee is made by this README.

Installation

Prerequisites

Windows: Visual Studio 2022 with C++ workload
Linux: GCC 11+ or Clang 14+
CUDA: 12.x (for GPU acceleration)
CMake: 3.18+
Node.js: 18+ (for desktop app)

Build from Source

Windows (GPU)

# Clone the repository
git clone https://github.com/maheshvaikri-code/snapllm.git
cd snapllm

# Build with CUDA support
build_gpu.bat

Windows (CPU only)

build_cpu.bat

Linux (GPU)

chmod +x build.sh
./build.sh gpu

Verify Installation

# Check the build
build_gpu/bin/snapllm --help

# Start the server
build_gpu/bin/snapllm --server --port 6930

How to Start the Server

Windows (recommended)

Start_Server.bat

This launcher auto-detects the newest build and starts the server on 127.0.0.1:6930. If the desktop app is built, it will launch the UI; otherwise it opens the browser.

Windows (manual)

build_gpu\bin\snapllm.exe --server --host 127.0.0.1 --port 6930

Start with a model preloaded

build_gpu\bin\snapllm.exe --server --host 127.0.0.1 --port 6930 --load-model mymodel D:\Models\mymodel.gguf

Linux/macOS

chmod +x start_server.sh
./start_server.sh

Docker

The Compose service binds inside its container on 0.0.0.0, publishes only to host loopback, and requires a runtime API key:

mkdir -p workspace models
export SNAPLLM_API_KEY='replace-with-a-random-value-at-least-32-characters'
docker compose up --build

Docker Hub image

Tagged releases publish a CPU image to Docker Hub as <dockerhub-user>/snapllm:<version> and latest through the protected docker-publish GitHub environment. Configure repository secrets DOCKERHUB_USERNAME and DOCKERHUB_TOKEN, then create a release tag. The published image listens on 0.0.0.0:6930 inside the container and includes a health check for /health. The -cuda image uses the NVIDIA CUDA 12.6 runtime and requires NVIDIA Container Toolkit plus --gpus all; the unqualified image is CPU-only.

The container runs as an unprivileged user with a read-only root filesystem, dropped Linux capabilities, a read-only model mount, and a writable workspace mount. Changing the host port mapping to a public interface is a deployment decision: keep the API key enabled and put TLS plus network access controls in front of SnapLLM.

Optional launch overrides (scripts)

SNAPLLM_SERVER_EXE: Full path to the SnapLLM binary
SNAPLLM_HOST: Host bind address (default 127.0.0.1)
SNAPLLM_PORT: Server port (default 6930)
SNAPLLM_WORKSPACE_ROOT: Workspace root directory
SNAPLLM_API_KEY: Runtime-only API key; required when binding beyond loopback
SNAPLLM_CORS_ORIGINS: Comma-separated exact browser origins to allow

Quick Start

1. Start the Server

# Start with a model pre-loaded
./snapllm --server --port 6930 --load-model mymodel /path/to/model.gguf

# Or start empty and load models via API
./snapllm --server --port 6930

2. Load Models via API

# Load first model
curl -X POST http://localhost:6930/api/v1/models/load \
  -H "Content-Type: application/json" \
  -d '{
    "model_id": "llama3",
    "file_path": "/models/llama-3-8b-instruct.Q5_K_M.gguf"
  }'

# Load second model
curl -X POST http://localhost:6930/api/v1/models/load \
  -H "Content-Type: application/json" \
  -d '{
    "model_id": "gemma",
    "file_path": "/models/gemma-2-9b-it.Q5_K_M.gguf"
  }'

3. Select a Loaded Model

# Select gemma without reloading it if it is still resident
curl -X POST http://localhost:6930/api/v1/models/switch \
  -H "Content-Type: application/json" \
  -d '{"model_id": "gemma"}'

4. Chat with the Active Model

# OpenAI-compatible chat endpoint
curl -X POST http://localhost:6930/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma",
    "messages": [
      {"role": "user", "content": "Explain quantum computing in simple terms"}
    ],
    "max_tokens": 256
  }'

5. Use the Web UI

Start_Server.bat starts the API and, when the desktop binary is unavailable, starts the Vite web UI at http://localhost:9780. The API health endpoint remains available at http://localhost:6930/health.

To run the standalone desktop app (development mode):

cd desktop-app
npm install
npm run dev
# Open http://localhost:9780

API Reference

Base URL

http://localhost:6930

The default loopback listener can be used without an API key. For any non-loopback bind, set SNAPLLM_API_KEY to 32–4096 visible ASCII characters before starting the process. Send that value as either Authorization: Bearer <key> or X-API-Key: <key>. The key is read from the environment only and is not persisted or returned by the API.

All requests are checked against the configured Host. Browser requests also use an exact Origin allowlist; add trusted origins with repeatable --cors-origin options or SNAPLLM_CORS_ORIGINS. Wildcards and reflected origins are not accepted. Model and workspace paths are canonicalized and must remain inside their configured roots.

Endpoints Overview

Health & Status

Method	Endpoint	Description
GET	`/`	API information
GET	`/health`	Server health check
GET	`/v1/models`	List models (OpenAI format)
GET	`/api/v1/models`	List models (extended info)

Model Management

Method	Endpoint	Description
POST	`/api/v1/models/load`	Load a model
POST	`/api/v1/models/switch`	Select an already-loaded model
POST	`/api/v1/models/unload`	Unload a model
DELETE	`/api/v1/models/{id}`	Delete a model
POST	`/api/v1/models/scan`	Scan folder for models

Text Generation

Method	Endpoint	Description
POST	`/v1/chat/completions`	Chat completion (OpenAI)
POST	`/api/v1/generate`	Text generation
POST	`/api/v1/generate/batch`	Batch generation

Vision (Multimodal)

Method	Endpoint	Description
POST	`/api/v1/vision/generate`	Analyze images

Image Generation

Method	Endpoint	Description
POST	`/api/v1/diffusion/generate`	Generate images

Context API (vPID L2)

Method	Endpoint	Description
POST	`/api/v1/contexts/ingest`	Pre-compute KV cache
GET	`/api/v1/contexts`	List contexts
POST	`/api/v1/contexts/{id}/query`	Query with indexed cache lookup
DELETE	`/api/v1/contexts/{id}`	Delete context

Detailed API Examples

Load Model

POST /api/v1/models/load

Request:

{
  "model_id": "medical-llm",
  "file_path": "D:/Models/medicine-llm.Q8_0.gguf",
  "model_type": "auto"
}

Response:

{
  "status": "success",
  "message": "Model loaded: medical-llm",
  "model": "medical-llm",
  "model_type": "Text LLM",
  "load_time_ms": 2500.5,
  "active": true
}

Chat Completion

POST /v1/chat/completions

Request:

{
  "model": "medical-llm",
  "messages": [
    {"role": "system", "content": "You are a medical assistant."},
    {"role": "user", "content": "What are the symptoms of diabetes?"}
  ],
  "max_tokens": 512,
  "temperature": 0.7,
  "stream": false
}

Chat completions stream by default. Set "stream": false when a buffered JSON response is required.

Response:

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1234567890,
  "model": "medical-llm",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Common symptoms of diabetes include..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 25,
    "completion_tokens": 150,
    "total_tokens": 175,
    "tokens_per_second": 64.5
  }
}

Load Vision Model

POST /api/v1/models/load

Request:

{
  "model_id": "gemma-vision",
  "file_path": "/models/gemma-3-4b-it.gguf",
  "model_type": "vision",
  "mmproj_path": "/models/mmproj-gemma-3-4b.gguf"
}

Vision Analysis

POST /api/v1/vision/generate

Request:

{
  "prompt": "Describe this image in detail",
  "images": ["<base64-encoded-image>"],
  "max_tokens": 256
}

Response:

{
  "status": "success",
  "response": "The image shows a beautiful sunset over mountains...",
  "model": "gemma-vision",
  "generation_time_s": 2.5,
  "tokens_per_second": 57.5
}

Context Ingestion (vPID L2)

POST /api/v1/contexts/ingest

Request:

{
  "content": "Your large document text here...",
  "model_id": "llama3",
  "name": "company-handbook",
  "ttl_seconds": 86400
}

Response:

{
  "status": "success",
  "context_id": "ctx_abc123",
  "token_count": 5000,
  "storage_size_mb": 125.5,
  "tier": "hot",
  "ingest_time_ms": 1500.5
}

Query with Cached Context

POST /api/v1/contexts/ctx_abc123/query

Request:

{
  "query": "What is the vacation policy?",
  "max_tokens": 100
}

Response:

{
  "status": "success",
  "context_id": "ctx_abc123",
  "response": "According to the handbook, employees receive...",
  "cache_hit": true,
  "latency_ms": 15.2
}

Architecture

vPID (Virtual Processing-In-Disk) System

SnapLLM uses a vPID architecture to select already-loaded models without an avoidable weight reload:

+--------------------------------------------------------------------+
|                          SnapLLM Server                             |
+--------------------------------------------------------------------+
|  Model A (vPID 1)  |  Model B (vPID 2)  |  Model C (vPID 3)  | ... |
+---------+----------+---------+----------+---------+----------+-----+
          |                    |                    |
          +--------------------+--------------------+
                               |
                               v
+--------------------------------------------------------------------+
|                    Unified Memory Manager                           |
|   HOT (GPU VRAM)    |   WARM (CPU RAM)    |    COLD (SSD)           |
+--------------------------------------------------------------------+
|               HTTP API Server (OpenAI-compatible)                   |
+--------------------------------------------------------------------+

Request flow (simplified):

Client -> HTTP API -> Router -> Model Manager -> Active Model -> Response
                           |                      ^
                           v                      |
                             vPID Cache (L1/L2)

vPID Levels

Level	Name	Purpose
L1	Model Cache	Model workspace metadata and engine-managed cache data
L2	Context Cache	KV cache persistence with hash-indexed lookup

Memory Tiers

Tier	Storage	Relative access	Capacity constraint
HOT	GPU VRAM	Highest	Available VRAM
WARM	CPU RAM	Intermediate	Available system RAM
COLD	SSD	Lowest	Available storage

Per-Model Workspace Structure

SnapLLM_Workspace/
|-- index.json                          # Model registry
|-- medicine-llm/
|   `-- Q8_0/
|       `-- workspace.bin               # Engine-managed workspace
|-- legal-llama/
|   `-- Q4_K_M/
|       `-- workspace.bin
|-- gemma-3-4b/
|   `-- Q5_K_M/
|       `-- workspace.bin
|-- diffusion/
|   `-- sd15/
|       `-- workspace.bin
`-- contexts/                           # vPID L2 KV caches
    |-- hot/
    |-- warm/
    `-- cold/

Desktop Application

SnapLLM includes a beautiful, modern desktop application built with React and Vite.

Features

Dashboard: System overview and health metrics
Chat: Interactive chat with loaded models
Images: Stable Diffusion image generation
Vision: Multimodal image analysis
Models: Load, manage, and switch models
A/B Compare: Side-by-side model comparison
Quick Switch: select an already-loaded model
Contexts: vPID L2 KV cache management
Playground: API testing interface
Metrics: Performance analytics

Running the Desktop App

cd desktop-app
npm install
npm run dev
# Open http://localhost:9780

Demo Videos

ImageGenerationDemo.mp4

Use Cases

1. Multi-Domain Assistant

Load specialized models for different domains and select them as needed:

import requests

BASE_URL = "http://localhost:6930"

# Load domain-specific models
for model in [("medical", "medical.gguf"), ("legal", "legal.gguf"), ("coding", "coding.gguf")]:
    requests.post(f"{BASE_URL}/api/v1/models/load", json={
        "model_id": model[0],
        "file_path": f"/models/{model[1]}"
    })

def ask(domain, question):
    requests.post(f"{BASE_URL}/api/v1/models/switch", json={"model_id": domain})
    response = requests.post(f"{BASE_URL}/v1/chat/completions", json={
        "model": domain,
        "messages": [{"role": "user", "content": question}]
    })
    return response.json()["choices"][0]["message"]["content"]

# Select the loaded domain model
print(ask("medical", "What are symptoms of flu?"))
print(ask("legal", "What is intellectual property?"))
print(ask("coding", "Write a Python fibonacci function"))

2. A/B Model Comparison

Compare responses from different models side-by-side:

models = ["llama3", "gemma2", "mistral"]

def compare(question):
    results = {}
    for model in models:
        requests.post(f"{BASE_URL}/api/v1/models/switch", json={"model_id": model})
        response = requests.post(f"{BASE_URL}/v1/chat/completions", json={
            "model": model,
            "messages": [{"role": "user", "content": question}]
        })
        results[model] = response.json()["choices"][0]["message"]["content"]
    return results

comparison = compare("Explain machine learning in one paragraph")
for model, answer in comparison.items():
    print(f"--- {model} ---\n{answer}\n")

3. RAG with Context Caching (vPID L2)

Pre-compute KV cache for large documents:

# Ingest a large document (one-time O(n^2) operation)
response = requests.post(f"{BASE_URL}/api/v1/contexts/ingest", json={
    "content": open("large_document.txt").read(),
    "model_id": "llama3",
    "name": "company-handbook"
})
context_id = response.json()["context_id"]

# Query through the indexed context cache
def query_handbook(question):
    response = requests.post(f"{BASE_URL}/api/v1/contexts/{context_id}/query", json={
        "query": question,
        "max_tokens": 256
    })
    return response.json()["response"]

# Reuse pre-computed context state; generation still performs model work
print(query_handbook("What is the vacation policy?"))
print(query_handbook("How do I submit expenses?"))

CLI Reference

Server Options

./snapllm --server [OPTIONS]

Options:
  --port PORT              Server port (default: 6930)
  --host HOST              Bind address (default: 127.0.0.1)
  --cors-origin ORIGIN     Allow an exact browser origin (repeatable)
  --workspace-root PATH    Workspace directory
  --ui-dir PATH            Web UI directory (auto-detected)
  --load-model NAME PATH   Pre-load a model

Text LLM Options

./snapllm --load-model NAME PATH [OPTIONS]

Options:
  --prompt TEXT            Generate text from prompt
  --max-tokens N           Maximum tokens to generate
  --temperature N          Sampling temperature
  --multi-model-test       Run multi-model switching benchmark
  --list-models            List all loaded models
  --stats                  Show cache statistics

Image Generation Options

./snapllm --load-diffusion NAME PATH [OPTIONS]

Options:
  --generate-image PROMPT  Generate image from text
  --output PATH            Output image path
  --width N                Image width (default: 512)
  --height N               Image height (default: 512)
  --steps N                Sampling steps (default: 20)
  --cfg-scale N            CFG scale (default: 7.0)
  --seed N                 Random seed (-1 for random)
  --negative PROMPT        Negative prompt

Vision Options

./snapllm --multimodal MODEL MMPROJ [OPTIONS]

Options:
  --image PATH             Input image file
  --vision-prompt TEXT     Prompt with <__media__> marker
  --max-tokens N           Maximum tokens to generate

Configuration

Environment Variables

Variable	Description	Default
`SNAPLLM_HOME`	Workspace root directory	Platform default
`SNAPLLM_MODELS_PATH`	Default models directory	Platform default
`SNAPLLM_CONFIG_PATH`	Server config file path	Platform default
`SNAPLLM_API_KEY`	Runtime-only API key (required off loopback)	Unset
`SNAPLLM_CORS_ORIGINS`	Comma-separated exact browser origins	Unset
`SNAPLLM_MAX_ACTIVE_INFERENCES`	Optional concurrent inference slots; bounded by HTTP workers and defaults to 1 for GPU safety	`1`

Contributing

We welcome contributions! Here's how to get started:

Development Setup

# Clone the repository
git clone https://github.com/maheshvaikri-code/snapllm.git
cd snapllm

# Build debug version
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Debug
cmake --build .

# Run tests
ctest --output-on-failure

Code Style

C++: snake_case functions, PascalCase classes, I* prefix for interfaces
TypeScript: ESLint + Prettier
Commits: Conventional commits format

Submitting Changes

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

License

SnapLLM is released under the MIT License.

Security issues should be reported privately as described in SECURITY.md.

MIT License

Copyright (c) 2024-2026 SnapLLM Contributors

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.

Acknowledgments

SnapLLM is built on the shoulders of giants:

llama.cpp - The foundation for LLM inference
stable-diffusion.cpp - Diffusion model support
cpp-httplib - HTTP server library
NVIDIA CUDA - GPU acceleration

Switch models in a snap!

Follow SnapLLM in LinkedIn

Creator: Mahesh Vaikri
Developed by AroorA AI Labs
Made with care for the AI community

Report Bug | Request Feature |

Name		Name	Last commit message	Last commit date
.github		.github
bindings		bindings
config		config
desktop-app		desktop-app
docker		docker
docs		docs
examples		examples
external		external
include/snapllm		include/snapllm
logo_files		logo_files
packaging		packaging
scripts		scripts
src		src
tests		tests
video		video
.dockerignore		.dockerignore
.gitignore		.gitignore
BUILD_OUTPUTS.md		BUILD_OUTPUTS.md
CHANGELOG.md		CHANGELOG.md
CMakeLists.txt		CMakeLists.txt
CONTRIBUTING.md		CONTRIBUTING.md
DEVELOPER_GUIDE.md		DEVELOPER_GUIDE.md
Dockerfile		Dockerfile
Dockerfile.cuda		Dockerfile.cuda
Dockerfile.ui		Dockerfile.ui
LICENSE		LICENSE
LinuxBuildGPU_Install_Guide.md		LinuxBuildGPU_Install_Guide.md
QUICKSTART.md		QUICKSTART.md
README.md		README.md
RELEASE.md		RELEASE.md
SECURITY.md		SECURITY.md
Start_Server.bat		Start_Server.bat
build.sh		build.sh
build_cpu.bat		build_cpu.bat
build_desktop.bat		build_desktop.bat
build_gpu.bat		build_gpu.bat
docker-compose.yml		docker-compose.yml
package.bat		package.bat
package_release.bat		package_release.bat
package_release.sh		package_release.sh
start_frontend.bat		start_frontend.bat
start_server.sh		start_server.sh

Tier	Monthly	Benefits
Coffee	$5	Name in README
Supporter	$25	Priority issue response, early access
Pro	$100	Roadmap input, feature voting
Enterprise	$500	Custom feature requests, onboarding help
Corporate	$2000+	Logo placement, consulting hours

Latest commit

History

Folders and files

Repository files navigation

Local Multi-Model LLM Inference with Desktop UI, CLI & API

What is SnapLLM?

The Problem

The SnapLLM Solution

Features

Core Capabilities

Supported Model Types

Performance

Installation

Prerequisites

Build from Source

Windows (GPU)

Windows (CPU only)

Linux (GPU)

Verify Installation

How to Start the Server

Windows (recommended)

Windows (manual)

Start with a model preloaded

Linux/macOS

Docker

Docker Hub image

Optional launch overrides (scripts)

Quick Start

1. Start the Server

2. Load Models via API

3. Select a Loaded Model

4. Chat with the Active Model

5. Use the Web UI

API Reference

Base URL

Endpoints Overview

Health & Status

Model Management

Text Generation

Vision (Multimodal)

Image Generation

Context API (vPID L2)

Detailed API Examples

Load Model

Chat Completion

Load Vision Model

Vision Analysis

Context Ingestion (vPID L2)

Query with Cached Context

Architecture

vPID (Virtual Processing-In-Disk) System

vPID Levels

Memory Tiers

Per-Model Workspace Structure

Desktop Application

Features

Running the Desktop App

Demo Videos

Use Cases

1. Multi-Domain Assistant

2. A/B Model Comparison

3. RAG with Context Caching (vPID L2)

CLI Reference

Server Options

Text LLM Options

Image Generation Options

Vision Options

Configuration

Environment Variables

Contributing

Development Setup

Code Style

Submitting Changes

Sponsors

Become a Sponsor

Sponsor Tiers

Corporate Sponsors

Individual Sponsors

Other Ways to Support

License