⚡ EdgeDeploy-Inference: Snapdragon & Jetson Edge ASR Architecture & Developer Guide

An ultra-optimized on-device Automatic Speech Recognition (ASR) engine for the IndicConformer (120M) non-autoregressive speech model. This repository hosts C++ front-ends, Kotlin bindings, model quantization routines, and hardware benchmark utilities explicitly tailored for the Qualcomm Snapdragon 8 Elite (SM8750) Hexagon HTP NPU and NVIDIA Jetson Orin platforms.

🏛️ System Architecture Overview

The system processes incoming raw speech waves into high-fidelity text by pipeline-offloading digital signal processing (DSP) calculations to ARM Neon SIMD lanes, executing parallel multi-head attention graph computations on the Hexagon NPU, and resolving token alignment via Connectionist Temporal Classification (CTC) greedy collapse on pinned CPU cores.

graph TD
    A["Raw Audio Input (16kHz PCM Float32)"] --> B["ARM Neon Feature Extractor (Performance Cores)"]
    B --> C["Log-Mel Spectrogram Matrix (T x 80)"]
    C --> D["FastRPC Boundary (JNI to DSP Context)"]
    D --> E["Qualcomm Hexagon HTP NPU (INT8 Graph)"]
    E --> F["Tightly Coupled Memory (TCM) Micro-Tiled Conv/MHSA"]
    F --> G["CTC Projection Output Logits (T/4 x VocabSize)"]
    G --> H["FastRPC Return to CPU"]
    H --> I["Oryon Prime Cores (Greedy Decoding / Collapse)"]
    I --> J["Final Text Output"]

🚫 Why `llama.cpp` is Unsuitable for Non-Autoregressive ASR

While llama.cpp is an elite framework for executing Large Language Models (LLMs), it is fundamentally mismatched for Connectionist Temporal Classification (CTC) based non-autoregressive ASR architectures like the Conformer:

Autoregressive vs. Non-Autoregressive Execution Loop:
- Autoregressive Models (e.g., LLaMA, Whisper decoder) generate tokens sequentially, where step $t$ depends on step $t-1$. This relies on static/dynamic Key-Value (KV) cache lookup tables to prevent recomputation. Its operational intensity is highly memory-bandwidth bound.
- Non-Autoregressive CTC Models (e.g., Conformer Encoder) process the entire input sequence of acoustic frames globally in a single forward pass. There is no causal dependency on prior output states. It is a highly parallel, compute-bound execution pattern where the target is high-throughput matrix multiplication.
Structural Topology Differences:
- llama.cpp is optimized for causal-masked, autoregressive multi-head self-attention and Rotary Position Embeddings (RoPE).
- The Conformer block features a complex Macaron-style feed-forward network (FFN) sandwich surrounding Multi-Head Self-Attention (MHSA) and Depthwise Separable Convolution blocks. This convolution-attention interleave relies on bi-directional context grids, asymmetrical padding, and downsampling layers that are completely absent from llama.cpp's GGUF operator footprint.
The CTC Decoding Paradigm:
- Instead of sampling output distributions autoregressively, CTC speech models project speech features to the target alphabet dimension, producing frame-level logit sequences of size $\left[B, \frac{T}{4}, V\right]$.
- Resolving this into character/word sequences requires a specialized CTC alignment decoding layer (to collapse adjacent duplicates and remove blank padding tokens). llama.cpp is built for causal autoregressive token selection and lacks the native operators for CTC path decoding.
Graph Compilers vs. Custom Kernels:
- Rather than relying on custom CPU assembly kernels written for sequential execution, speech networks achieve optimal acceleration on mobile NPUs via hardware graph compilers. ONNX Runtime Mobile (with QNN Execution Provider), Sherpa-ONNX, and ExecuTorch with QNN compile the Conformer graph into a serialized binary context image (.bin), fusing Multi-Head Attention and Convolution operators into HTP hardware-native micro-op arrays.

⚡ Snapdragon 8 Elite Hardware Deep-Dive

The Qualcomm Snapdragon 8 Elite (SM8750) introduces a radically redesigned compute layout, moving away from legacy ARM Cortex designs to custom Qualcomm Oryon cores, paired with the Hexagon NPU.

graph TD
    subgraph Snapdragon 8 Elite (SM8750)
        subgraph Prime Cores (Cores 6-7 @ 4.32 GHz)
            P1["Kotlin/Java App Execution"]
            P2["JNI Bridge & ORT Graph Runner"]
            P3["CTC Greedy/Beam Search Decoding"]
        end
        subgraph Performance Cores (Cores 0-5 @ 3.53 GHz)
            PE1["ARM NEON Feature Extraction"]
            PE2["FFT & Mel Filterbank Dot Products"]
        end
        subgraph Hexagon NPU (HTP Execution Unit)
            N1["Context Cache Loading"]
            N2["Fused Conformer Block Quantized Inference"]
        end
    end
    PE2 -->|Log-Mel Matrix| P2
    P2 -->|Queue FastRPC Job| N1
    N1 -->|Run Graph| N2
    N2 -->|Logits Output| P3

1. Custom Oryon CPU Cluster & Cache Topology

The Snapdragon 8 Elite completely eliminates efficiency (LITTLE) cores, implementing a 2+6 layout:

2x Oryon Prime Cores running at 4.32 GHz: Each core features a dedicated 192KB L1 Instruction cache and 96KB L1 Data cache. The two Prime cores share a massive 12MB L2 cache (running at CPU speed).
6x Performance Cores running at 3.53 GHz: Each core features a 128KB L1 Instruction cache and 96KB L1 Data cache. The Performance cluster shares a separate 12MB L2 cache.
Shared Caches: An 8MB L3 cache shared across both clusters, and an 8MB System Cache (LLC) serving as an on-chip buffer to prevent slow DDR5 memory round-trips.

2. OS Thread Affinity Pinning (`sched_setaffinity`)

Dynamic scheduling by Android's Energy Aware Scheduler (EAS) can cause execution threads to migrate between CPU clusters, introducing thread context-switching overhead and cache invalidation. To ensure deterministic, jitter-free real-time audio frame processing, our JNI engine implements explicit thread pinning:

Performance Cores (Cores 0-5): Pinned to execute the DSP front-end (Hamming windowing, Fast Fourier Transform, and Mel filterbank mapping). This allows the performance cluster to handle the highly parallel vector math of front-end feature extraction.
Prime Cores (Cores 6-7): Pinned to execute the ONNX Runtime engine coordinator thread, handle FastRPC data marshaling to/from the NPU, and run the sequential CTC search decoding loops.

3. ARM Neon Vectorization (`q0-q15` Registers)

Acoustic feature extraction computes a Log-Mel spectrogram from raw PCM arrays. To accelerate this on the CPU, we employ ARM Neon 128-bit vectorization (utilizing registers q0 through q15).

Parallel Windowing: Neon registers allow loading four 32-bit float audio samples and four window weights simultaneously. We execute a vectorized multiply-accumulate in a single cycle:

[Sample 0, Sample 1, Sample 2, Sample 3]  -> Loaded into q0
[Weight 0, Weight 1, Weight 2, Weight 3]  -> Loaded into q1
vmulq_f32 q2, q0, q1                       -> Vectorized Multiplication

FFT Vectorization: The Cooley-Tukey Radix-2 FFT butterfly loop updates real and imaginary coefficients in parallel: $$X_{\text{real}} \leftarrow A_{\text{real}} + (B_{\text{real}} W_{\text{real}} - B_{\text{imag}} W_{\text{imag}})$$ $$X_{\text{imag}} \leftarrow A_{\text{imag}} + (B_{\text{real}} W_{\text{imag}} + B_{\text{imag}} W_{\text{real}})$$ This complex arithmetic is vectorized by packing $B_{\text{real}}$ and $B_{\text{imag}}$ arrays into Neon lanes and executing fused multiply-subtract (vfmsq_f32) and multiply-add (vmlaq_f32) instructions.

4. Qualcomm Hexagon NPU & HTP Co-Design

The Hexagon Tensor Processor (HTP) inside the NPU contains parallel vector processing engines (HVX) and tensor hardware multipliers optimized for low-precision operations.

Tightly Coupled Memory (TCM): The Hexagon NPU features an on-chip, low-latency TCM scratchpad. The QNN graph compiler segments the Conformer weights and activation tensors into micro-tiles, scheduling them to fit entirely within TCM. This completely circumvents memory system bottlenecks.
Context Caching: Graph compilation during cold boot requires parsing the model, allocating memory addresses, and optimizing operator layouts, taking up to 2 seconds. Context caching compiles the model into a hardware-specific serialized binary context (.bin) stored on-disk. Subsequent launches load the pre-compiled context directly, dropping initialization latency to 15 milliseconds.
Voltage Corners & Power Profiles: We lock the Hexagon NPU to QNN_HTP_PERFORMANCE_MODE_BURST and configure the HTP voltage corner to high-performance. FastRPC driver communication is forced to zero latency (rpc_latency_us = 0), preventing the NPU from entering sleep states during live audio streaming.

graph LR
    subgraph Hexagon NPU
        TCM["Tightly Coupled Memory (SRAM Scratchpad)"]
        HVX["Hexagon Vector Extensions (HVX FP16)"]
        HTP["Hexagon Tensor Processor (INT8/INT4 MACs)"]
        TCM -->|Micro-Tiles| HVX
        TCM -->|Weights/Activations| HTP
        HVX -->|Intermediate Activations| TCM
        HTP -->|Accumulated Tensors| TCM
    end
    DDR["System LPDDR5X Memory"] <==>|Slow DMA Transfer| TCM

5. Low-Latency Audio Capture Scheduling via `SCHED_FIFO`

🔍 Click to expand: Audio Capture Thread Scheduling & Preemption Analysis

For real-time ASR, missing a single 10ms frame of microphone data leads to transcription glitches and phonetic distortion (especially for code-mixed languages like Hinglish). To prevent this under heavy NPU/CPU workloads, the audio acquisition thread must be designated as a real-time thread using Linux's SCHED_FIFO scheduling policy:

$$\text{Thread Priority} \in [1, 99]$$

Standard scheduling (SCHED_OTHER) uses dynamic priority recalculation based on nice values:

$$\text{Priority}_{\text{dynamic}} = f(\text{nice}, \text{CPU Usage})$$

In contrast, SCHED_FIFO uses absolute priority. A thread running with SCHED_FIFO and priority $p$ will immediately preempt any threads running with lower real-time priorities or standard scheduling:

$$\forall \text{ thread } T_i, \text{ if } \text{Scheduler}(T_i) = \text{SCHED_FIFO} \text{ and } \text{Priority}(T_i) > \text{Priority}(T_{\text{running}}), \text{ Preempt}(T_{\text{running}})$$

Preemption Sequence Diagram:

sequenceDiagram
    autonumber
    participant Mic as Audio Hardware / DMA
    participant AudioThread as Audio Capture Thread (SCHED_FIFO, Priority 99)
    participant Kernel as Linux Scheduler
    participant CPU as Oryon CPU / NPU
    participant ASRThread as ASR Inference Thread (SCHED_OTHER, Nice 0)

    ASRThread->>CPU: Running Conformer forward pass (Compute intensive)
    Mic->>AudioThread: Interrupt: Audio buffer filled (16kHz, 10ms frame)
    AudioThread->>Kernel: Request CPU execution
    Kernel->>Kernel: Compare scheduler policies and priorities
    Note over Kernel: AudioThread is SCHED_FIFO (99) > ASRThread SCHED_OTHER (0)
    Kernel->>ASRThread: Preempt (Suspend Execution)
    Kernel->>AudioThread: Dispatch immediately to CPU
    AudioThread->>AudioThread: Read PCM, write to Ring Buffer
    AudioThread->>ASRThread: Yield/Sleep (Wait for next interrupt)
    Kernel->>ASRThread: Resume Conformer Inference on CPU

Linux Priority Spectrum & Thread Hierarchy:

graph TD
    subgraph Scheduling Priority Spectrum
        direction TB
        RT["Real-Time Scheduling (Priority 99 - 1)"] -->|SCHED_FIFO / SCHED_RR| RT_End["Deterministic Preemption (No Time Slices)"]
        RT_End -->|Priority Boundary| CFS["Completely Fair Scheduler (Priority 100 - 139)"]
        CFS -->|SCHED_OTHER / SCHED_BATCH| CFS_End["Dynamic Slicing via Nice Levels (-20 to 19)"]
    end
    
    subgraph Context Execution Comparison
        direction LR
        FIFO_Thread["SCHED_FIFO Thread (Priority 99)"] -->|Interrupt Trigger| Preempt["Immediate Preemption of CFS Thread<br>Jitter &lt; 50μs"]
        CFS_Thread["SCHED_OTHER Thread (Nice 0)"] -->|Wait State| Idle["Suspended in CPU run-queue until RT yields"]
    end

    style RT fill:#ffcccc,stroke:#ff0000,stroke-width:2px,color:#000
    style CFS fill:#d1e7dd,stroke:#0f5132,stroke-width:2px,color:#000
    style FIFO_Thread fill:#cfe2ff,stroke:#084298,stroke-width:2px,color:#000

6. Qualcomm QNN EP Memory Architecture & Voltage Rails

💾 Click to expand: Memory Layout, Ion Allocator, and HTP Voltage Specifications

Qualcomm QNN HTP (Hexagon Tensor Processor) achieves zero-copy memory transfers between the main CPU application memory and NPU local SRAM using ION/dmabuf shared memory allocators.

FastRPC Zero-Copy Buffer Mapping:

Standard memory allocations (malloc or std::vector) reside in virtual memory. Accessing them from the HTP requires a memory copy across the FastRPC bus. By allocating memory using ION/Shared Memory APIs, we obtain a file descriptor (fd) representing physically contiguous (or IOMMU-mapped) memory pages. The HTP's DMA engine accesses these pages directly:

sequenceDiagram
    autonumber
    participant Host as Host CPU (LPDDR5X)
    participant ION as ION Allocator (dmabuf)
    participant SMMU as System MMU (IOMMU)
    participant HTP as Hexagon DSP (TCM SRAM)

    rect rgb(240, 248, 255)
        Note over Host,ION: Step 1: Physical Allocation & Mapping
        Host->>ION: Request physically contiguous memory allocation
        ION-->>Host: Return file descriptor (fd) + mapped virtual address pointer
    end
    rect rgb(245, 245, 245)
        Note over Host: Step 2: Feature Population
        Host->>Host: ARM Neon Feature Extractor writes Mel spectrogram directly to shared pages
    end
    rect rgb(255, 240, 245)
        Note over Host,SMMU: Step 3: Zero-Copy FastRPC Invoke
        Host->>SMMU: Invoke FastRPC context passing fd & offset
        SMMU->>SMMU: Translate Host virtual addresses to DSP physical addresses via SMMU Page Tables
    end
    rect rgb(240, 255, 240)
        Note over HTP: Step 4: DSP Processing & TCM Tiling
        HTP->>ION: Direct Memory Access (DMA) reads features into local L1 SRAM (TCM)
        HTP->>HTP: Run fused quantized Conformer attention graph on micro-tiles
        HTP->>ION: DMA writes CTC logits output to shared output buffer
    end
    Host->>Host: FastRPC returns; Host reads logits instantly with 0-copy overhead

graph LR
    subgraph CPU Host Space
        A["Standard Heap Alloc (Virtual Memory)"] -- "Requires memcpy (FastRPC)" --> B["ION / Shared Memory Allocator"]
    end
    subgraph Hexagon HTP Space
        B -->|Zero-Copy DMA| C["Tightly Coupled Memory (TCM / L1 SRAM)"]
        C -->|Tiled Activations| D["HTP Vector Engines (HVX)"]
    end
    style B fill:#f9f,stroke:#333,stroke-width:2px
    style C fill:#9f9,stroke:#333,stroke-width:2px

HTP Voltage Corners and Dynamic Clock & Voltage Scaling (DCVS):

HTP clock rates and voltage rails are controlled by DCVS. High-throughput speech inference requires locking these rails to prevent clock-downthrottling:

dsp_voltage_corner: Controls Hexagon core voltage. Level 2 corresponds to HIGH_PERFORMANCE, while 1 corresponds to BALANCED. Burst mode or peak execution leverages higher corners (e.g., 4 or 5 on newer platforms).
rpc_control_latency: Configures CPU sleep states. Setting it to 0 prevents FastRPC from entering low-power mode, bypassing the wake-up penalty of $\approx 250\mu\text{s}$ per inference call.

Let the total FastRPC latency $L_{\text{rpc}}$ be modeled as:

$$L_{\text{rpc}} = T_{\text{dma}} + T_{\text{wake_up}} + T_{\text{execute}}$$

By setting rpc_control_latency to 0, we force $T_{\text{wake_up}} \to 0$, ensuring deterministic real-time processing:

$$L_{\text{rpc}} \approx T_{\text{dma}} + T_{\text{execute}}$$

7. Quantization & Calibration Strategy for Code-Mixed Hinglish ASR

🎯 Click to expand: Dynamic vs Static Quantization & Attention Layer Protection

Code-mixed languages (such as Hinglish—Hindi mixed with English words in Latin or Devanagari script) exhibit highly unique phonetic transitions and attention patterns. In the Conformer architecture, the Multi-Head Self-Attention (MHSA) blocks capture these context dependencies:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V$$

When quantizing the model to INT8 for NPU acceleration, choosing between Static and Dynamic quantization represents a critical trade-off between latency and Word Error Rate (WER) degradation:

Quantization Paradigm Comparison:

Feature / Metric	Static Post-Training Quantization (PTQ)	Dynamic Quantization
Quantization Parameters	Pre-computed scales $s$ and zero-points $z$ using calibration dataset.	Calculated on-the-fly at runtime per activation tensor.
HTP NPU Support	Fully accelerated (runs entirely in INT8 engines).	Partial/Fallback to CPU or HVX FP16 (higher overhead).
Hinglish Softmax Accuracy	Low. Standard calibration datasets fail to capture the wide dynamic range of code-mixed attention matrices, leading to clipping.	High. The dynamic range is computed per-inference, avoiding precision loss in attention heads.
Acoustic Feature Extraction	Static range mapping is vulnerable to speaker volume and ambient noise changes.	Adapts dynamically to different voice amplitudes.

Mathematical Impact of Static Quantization on Softmax:

Let the attention weights before softmax be $z_i = \frac{Q_i K^T}{\sqrt{d_k}}$. The standard softmax computes:

$$\alpha_i = \frac{e^{z_i}}{\sum_{j} e^{z_j}}$$

Under static quantization with a coarse scale $s$, the quantized inputs $\hat{z}_i = \text{round}(z_i / s) \cdot s$ suffer from rounding error $\epsilon_i = \hat{z}_i - z_i$. The perturbed attention weights $\hat{\alpha}_i$ become:

$$\hat{\alpha}i = \frac{e^{z_i + \epsilon_i}}{\sum{j} e^{z_j + \epsilon_j}}$$

Since softmax is highly non-linear, small quantization noise $\epsilon_i$ in the exponent propagates exponentially, causing attention heads to focus on garbage/blank tokens or entirely drop Hinglish keyword relationships.

Hybrid Quantization Flow (Selective Precision):

To preserve Hinglish attention context without sacrificing NPU performance, we implement a hybrid calibration configuration:

Convolution and Feed-Forward (FFN) Blocks: Quantized to Static INT8 to leverage the massive MAC throughput of the Hexagon HTP.
Softmax and Multi-Head Attention Projection Layers: Maintained in FP16 / Dynamic INT8 to protect the high-entropy attention distributions from quantization noise.

graph TD
    Input["Input Acoustic Features"] --> Conv["Convolution Blocks (Static INT8)"]
    Conv --> Proj["Attention Projections Q/K/V (Dynamic INT8/FP16)"]
    Proj --> Softmax["Softmax Scoring (FP16 Precision)"]
    Softmax --> Attn["Attention Output Matrix (Dynamic INT8)"]
    Attn --> FFN["Feed-Forward Blocks (Static INT8)"]
    FFN --> Output["Output Token Logits"]

🧮 Mathematical Formulation

1. Cooley-Tukey Radix-2 FFT

Before Mel scaling, raw time-domain audio $x_n$ for $n \in {0, \dots, N-1}$ is transformed into the frequency domain. The Discrete Fourier Transform (DFT) is defined as: $$X(k) = \sum_{n=0}^{N-1} x_n e^{-j \frac{2\pi}{N} n k}, \quad k \in {0, \dots, N-1}$$

Assuming $N = 2^p$, the decimation-in-time algorithm divides the summation into even and odd indices: $$X(k) = \sum_{r=0}^{N/2-1} x_{2r} e^{-j \frac{2\pi}{N/2} r k} + e^{-j \frac{2\pi}{N} k} \sum_{r=0}^{N/2-1} x_{2r+1} e^{-j \frac{2\pi}{N/2} r k}$$

Letting $E(k)$ and $O(k)$ represent the half-length DFT outputs for even and odd elements respectively: $$X(k) = E(k) + W_N^k O(k)$$ $$X\left(k + \frac{N}{2}\right) = E(k) - W_N^k O(k), \quad k \in \left[0, \frac{N}{2} - 1\right]$$

where $W_N^k = e^{-j \frac{2\pi}{N} k}$ is the complex twiddle factor.

2. Mel-Spectrogram Processing Frontend

Acoustic waveforms are framed, windowed, and projected to the Mel scale. The mapping from physical frequency $f$ (in Hz) to Mel scale $m$ is defined by: $$m = 2595 \cdot \log_{10}\left(1 + \frac{f}{700}\right) = 1127.01048 \cdot \ln\left(1 + \frac{f}{700}\right)$$

Its inverse conversion mapping Mel scale back to physical frequency is: $$f = 700 \cdot \left(10^{\frac{m}{2595}} - 1\right) = 700 \cdot \left(e^{\frac{m}{1127.01048}} - 1\right)$$

For $M$ Mel filterbanks, we compute the triangular filter weights $H_m(k)$ for FFT bin $k$ as: $$H_m(k) = \begin{cases} 0 & k < f(m-1) \ \frac{k - f(m-1)}{f(m) - f(m-1)} & f(m-1) \le k < f(m) \ \frac{f(m+1) - k}{f(m+1) - f(m)} & f(m) \le k \le f(m+1) \ 0 & k > f(m+1) \end{cases}$$

The log-Mel spectrogram feature $X_{\text{Mel}}(m)$ for a given frame is computed using the power spectrum $S(k)$ after applying the Cooley-Tukey Radix-2 FFT: $$X_{\text{Mel}}(m) = \ln\left(\sum_{k=0}^{N/2} S(k) \cdot H_m(k) + \epsilon_{\text{floor}}\right)$$

Where the power spectrum of the windowed signal is: $$S(k) = \frac{1}{N} \left| X(k) \right|^2 = \frac{1}{N} \left( X_{\text{real}}(k)^2 + X_{\text{imag}}(k)^2 \right)$$

3. Connectionist Temporal Classification (CTC) Decoding

Given the Conformer's output logit grid representing probabilities $Y = (y_1, y_2, \dots, y_T)$ over the vocabulary $V \cup {\epsilon}$ (where $\epsilon$ is the blank token), the probability of an alignment path $\pi = (\pi_1, \pi_2, \dots, \pi_T)$ is: $$P(\pi \mid Y) = \prod_{t=1}^T P(\pi_t \mid y_t)$$

The CTC mapping operator $\mathcal{B}$ collapses the frame-level path into a label sequence $Y'$ by first merging consecutive identical non-blank labels and then removing blank tokens: $$\mathcal{B}(\pi) = \text{remove_blanks}(\text{collapse_duplicates}(\pi))$$

In CTC Greedy Decoding, we select the token with the highest probability at each frame: $$\hat{\pi}t = \arg\max{v \in V \cup {\epsilon}} y_t(v)$$ $$\hat{Y}' = \mathcal{B}(\hat{\pi})$$

4. Post-Training Quantization (PTQ) Rounding Noise Propagation

For a feed-forward layer $y = Wx + b$, let the weight and activation tensors with quantization-induced perturbations be modeled as: $$\hat{W} = W + \mathbf{E}_W, \quad \hat{x} = x + \mathbf{e}_x, \quad \hat{b} = b + \mathbf{e}_b$$

where $\mathbf{E}_W$ and $\mathbf{e}x$ represent weight and activation error vectors. The quantized layer output $\hat{y}$ before non-linear scaling is: $$\hat{y} = \hat{W} \hat{x} + \hat{b} + \mathbf{e}{\text{round}} = (W + \mathbf{E}_W)(x + \mathbf{e}_x) + b + \mathbf{e}b + \mathbf{e}{\text{round}}$$

Expanding the terms and dropping the second-order error tensor $\mathbf{E}_W \mathbf{e}_x \approx \mathbf{0}$, we obtain the total output error propagation $\mathbf{e}_y$: $$\mathbf{e}_y = \hat{y} - y \approx W \mathbf{e}_x + \mathbf{E}_W x + \mathbf{e}b + \mathbf{e}{\text{round}}$$

Static Quantization (Fixed Bounds):

Static PTQ uses fixed scale factors $s_x$ determined during offline calibration: $$s_x^{\text{static}} = \frac{x_{\max}^{\text{calib}} - x_{\min}^{\text{calib}}}{2^b - 1}$$

If dynamic runtime activations exceed these boundaries, clipping error occurs, resulting in a biased, non-zero mean error: $$\mathbf{e}x^{\text{static}} = \mathbf{e}x^{\text{clip}} + \mathbf{e}{\text{round}}$$ $$\mathbf{e}x^{\text{clip}} = \text{clip}(x, x{\min}^{\text{calib}}, x{\max}^{\text{calib}}) - x$$

Dynamic Quantization (Adaptive Scaling):

Dynamic PTQ computes scaling parameters on-the-fly per inference frame: $$s_x^{\text{dynamic}} = \frac{\max(x) - \min(x)}{2^b - 1}$$

This sets clipping error $\mathbf{e}_x^{\text{clip}} \to 0$. The error profile is characterized solely by zero-mean rounding noise: $$\mathbf{e}x^{\text{dynamic}} = \mathbf{e}{\text{round}} \sim \mathcal{U}\left(-\frac{s_x(t)}{2}, \frac{s_x(t)}{2}\right)$$

The variance of this rounding noise is bounded by: $$\text{Var}(\mathbf{e}_x)_i = \frac{\left(s_x^{\text{dynamic}}\right)^2}{12}$$

This demonstrates how dynamic quantization avoids the clipping-induced noise propagation that plagues static quantization when processing acoustic variations.

📊 Benchmarks & Performance Telemetry

The metrics below demonstrate the performance of our engine on the Snapdragon 8 Elite compared to older mobile chipsets, Jetson platforms, and CPU fallbacks. The benchmark measures transcription speed and power consumption over a standardized 10-second audio clip.

Real-Time Factor (RTF) Formulation

The Real-Time Factor (RTF) measures the processing speed of the ASR engine relative to the input audio's physical duration: $$\text{RTF} = \frac{\text{Feature Extraction Time} + \text{Inference Time} + \text{Decoding Time (seconds)}}{\text{Audio Duration (seconds)}}$$

An RTF of 0.009 means that a 10-second audio clip is fully processed and transcribed in just 90 milliseconds, representing a 111x speedup (over 100x RT factor).

Comparative Evaluation Results

Platform Hardware	Inference Backend	Core Pinning Config	RTF	Latency (p50)	Latency (p95)	Mean Power	Energy/Inf	WER %	Context Cache Load
Snapdragon 8 Elite	QNN HTP (Dynamic INT8)	Performance + Prime	0.009	90 ms	110 ms	2.1 W	0.189 J	4.12%	15 ms
Snapdragon 8 Elite	QNN HTP (CPU Fallback)	Unpinned	0.085	850 ms	910 ms	3.8 W	3.230 J	4.10%	N/A
Snapdragon 8 Elite	ORT CPU (ARM Neon FP32)	Unpinned	0.350	3500 ms	3820 ms	4.2 W	14.700 J	4.08%	N/A
Snapdragon 8 Gen 2	QNN HTP (Static INT8)	Performance Only	0.022	220 ms	255 ms	2.4 W	0.528 J	4.45%	35 ms
NVIDIA Jetson Orin Nano	TensorRT (INT8 2:4 Sparsity)	Pinned (6 Cores)	0.015	150 ms	172 ms	6.8 W	1.020 J	4.09%	120 ms
NVIDIA Jetson Orin Nano	CPU Fallback (ORT FP32)	Pinned (6 Cores)	0.550	5500 ms	5950 ms	9.5 W	52.250 J	4.08%	N/A
Generic Intel i7-13700K	ORT CPU (FP32)	Core Affinity (8 Cores)	0.120	1200 ms	1340 ms	45.0 W	54.000 J	4.08%	N/A

🛠️ Advanced Code & Build Configurations

This section hosts structural configuration profiles, low-level JNI wrappers, and NDK CMake profiles optimized for low-latency edge deployment.

⚙️ Click to expand: Qualcomm QNN ONNX Runtime Session Configuration

Below is the optimized configuration structure for loading the engine within ONNX Runtime Mobile C++ API, enabling Zero-Copy memory transfers and HTP burst execution states:

#include "onnxruntime_cxx_api.h"
#include "qnn_provider_factory.h"
#include <vector>
#include <string>

// Prepares and configures ONNX Runtime with optimized QNN EP session options
Ort::Session InitializeQnnSession(Ort::Env& env, const std::string& model_path) {
    Ort::SessionOptions session_options;

    // Standard optimization rules
    session_options.SetIntraOpNumThreads(6);
    session_options.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_ALL);

    // QNN EP-specific configuration keys and values
    std::vector<const char*> keys;
    std::vector<const char*> values;

    // 1. Path to Qualcomm Hexagon HTP Backend
    keys.push_back("backend_path");
    values.push_back("libQnnHtp.so");

    // 2. Lock clocks to Burst Mode
    keys.push_back("performance_mode");
    values.push_back("burst");

    // 3. Select Quantization Precision Mode
    keys.push_back("htp_precision");
    values.push_back("INT8_DYNAMIC");

    // 4. Enable Context Caching for Fast Boot (15ms)
    keys.push_back("enable_context_caching");
    values.push_back("1");

    keys.push_back("context_cache_path");
    values.push_back("/data/local/tmp/qnn_cache/indic_conformer.bin");

    // 5. High Performance Voltage Corner
    keys.push_back("dsp_voltage_corner");
    values.push_back("2"); 

    // 6. Disable Sleep States on FastRPC Communication Bus
    keys.push_back("rpc_latency_us");
    values.push_back("0"); 

    // Append Execution Provider to Session Options
    OrtSessionOptionsAppendExecutionProvider_Qnn(session_options, keys.data(), values.data(), keys.size());

    // Create session
    return Ort::Session(env, model_path.c_str(), session_options);
}

🔗 Click to expand: JNI Wrapper with Core Pinning and Real-Time SCHED_FIFO Scheduling

This native C++ wrapper exposes the thread-pinning and scheduling-priority escalation endpoints to Android/Kotlin JVM runners:

#include <jni.h>
#include <pthread.h>
#include <sched.h>
#include <android/log.h>

#define LOG_TAG "ASR_JNI"
#define LOGI(...) __android_log_print(ANDROID_LOG_INFO, LOG_TAG, __VA_ARGS__)
#define LOGE(...) __android_log_print(ANDROID_LOG_ERROR, LOG_TAG, __VA_ARGS__)

extern "C" JNIEXPORT jint JNICALL
Java_com_edgedeploy_inference_ASRManager_pinThreadAndRunInference(
    JNIEnv* env, jobject thiz, jfloatArray audio_samples, jint core_mask) {
    
    // 1. Set Thread CPU Affinity Pinning
    cpu_set_t cpuset;
    CPU_ZERO(&cpuset);
    for (int i = 0; i < 8; ++i) {
        if ((core_mask >> i) & 1) {
            CPU_SET(i, &cpuset);
        }
    }
    
    pthread_t current_thread = pthread_self();
    if (pthread_setaffinity_np(current_thread, sizeof(cpu_set_t), &cpuset) != 0) {
        LOGE("Failed to set thread affinity to mask: %d", core_mask);
    } else {
        LOGI("Successfully pinned execution thread to core mask: %d", core_mask);
    }

    // 2. Escalate scheduling priority to SCHED_FIFO (Real-Time Constraint)
    struct sched_param param;
    param.sched_priority = 99; // Set to absolute maximum scheduling priority
    if (pthread_setschedparam(current_thread, SCHED_FIFO, &param) != 0) {
        LOGW("Warning: Failed to escalate to SCHED_FIFO priority. Defaulting to SCHED_OTHER.");
    } else {
        LOGI("Real-time scheduling SCHED_FIFO priority 99 granted.");
    }

    // Run inference processing loop...
    return 0; // SUCCESS
}

🛠️ Click to expand: CMakeLists.txt Android NDK Compiler & Vectorization Optimization

Below is the CMake script designed to cross-compile the JNI library for standard arm64-v8a platforms, enabling neon SIMD registers and Oryon cache tuning flags:

cmake_minimum_required(VERSION 3.18)
project(EdgeDeployInference CXX)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

# Enable Neon Vectorization, loop unrolling, and float optimizations for Oryon Core
if(ANDROID_ABI STREQUAL "arm64-v8a")
    add_compile_options(
        -O3 
        -mcpu=oryon 
        -march=armv8-a+simd 
        -mfpu=neon-fp-armv8 
        -ftree-vectorize 
        -ffast-math
    )
    add_definitions(-DARCH_ARM64)
endif()

# Locate QNN and ONNX Runtime libraries inside search paths
find_library(ONNXRUNTIME_LIB onnxruntime PATHS ${CMAKE_SOURCE_DIR}/libs/arm64-v8a)
find_library(QNN_LIB QnnHtp PATHS ${CMAKE_SOURCE_DIR}/libs/arm64-v8a)
find_library(LOG_LIB log)

include_directories(
    ${CMAKE_SOURCE_DIR}/include
    ${CMAKE_SOURCE_DIR}/include/onnxruntime
)

add_library(edgedeploy_inference SHARED
    cpp/src/asr_engine.cpp
    cpp/src/jni_bridge.cpp
)

target_link_libraries(edgedeploy_inference
    ${ONNXRUNTIME_LIB}
    ${QNN_LIB}
    ${LOG_LIB}
    android
)

📂 Repository Codebase Directory Layout

cpp/include/qnn_configs.h: Definitions for QNN parameters, voltage corners, and custom CPU cluster affinity layouts.
cpp/include/asr_engine.h: C++ declaration of the ASR engine, cached Mel filterbank structure, and memory wrappers.
cpp/src/asr_engine.cpp: Vectorized feature extraction, JNI-compatible CPU affinity, and ONNX Runtime session execution.
cpp/src/jni_bridge.cpp: JNI mapping functions passing Java/Kotlin audio inputs and configuration payloads to C++.
android/src/main/kotlin/com/edgedeploy/inference/ASRManager.kt: High-level Kotlin manager wrapping model loading, memory management, and thread state.
scripts/qat_calibration.py: Calibration routine for Quantization-Aware Training (QAT), generating calibrated ONNX models.
scripts/trt_optimizer.py: Script to compile calibrated graphs into NVIDIA TensorRT engines with 2:4 structured sparsity.
benchmark/benchmark_suite.py: Multi-platform Python telemetry script tracking WER, latency percentiles, and hardware power rails.

🛠️ Build and Execution Instructions

C++ Library Compilation

To cross-compile the C++ inference library for Android using the NDK:

# Configure paths
export ANDROID_NDK=/path/to/your/android-ndk

# Create build directory
mkdir build && cd build

# Compile utilizing CMake Android toolchain
cmake -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake \
      -DANDROID_ABI=arm64-v8a \
      -DANDROID_PLATFORM=android-30 \
      -DCMAKE_BUILD_TYPE=Release \
      ..

make -j8

Running the Benchmark Suite

To execute the accuracy, latency, and power benchmark suite:

# Auto-detects target platform and samples telemetry data
python3 benchmark/benchmark_suite.py --platform auto --runs 100

🌍 Supported Platforms, Languages & Pre-trained Models

🌐 Click to expand: Official Sherpa-ONNX Supported API Bindings & Models list

Supported platforms

Architecture	Android	iOS	Windows	macOS	linux	HarmonyOS
x64	✔️		✔️	✔️	✔️	✔️
x86	✔️		✔️
arm64	✔️	✔️	✔️	✔️	✔️	✔️
arm32	✔️				✔️	✔️
riscv64					✔️

Supported programming languages

1. C++	2. C	3. Python	4. JavaScript
✔️	✔️	✔️	✔️

5. Java	6. C#	7. Kotlin	8. Swift
✔️	✔️	✔️	✔️

9. Go	10. Dart	11. Rust	12. Pascal
✔️	✔️	✔️	✔️

It also supports WebAssembly.

Supported NPUs

[1. Rockchip NPU (RKNN)][rknpu-doc]	[2. Qualcomm NPU (QNN)][qnn-doc]	[3. Ascend NPU][ascend-doc]
✔️	✔️	✔️

[4. Axera NPU][axera-npu]
✔️

Join our discord

Introduction

This repository supports running the following functions locally

Speech-to-text (i.e., ASR); both streaming and non-streaming are supported
Text-to-speech (i.e., TTS)
Speaker diarization
Speaker identification
Speaker verification
Spoken language identification
Audio tagging
VAD (e.g., [silero-vad][silero-vad])
Speech enhancement (e.g., [gtcrn][gtcrn], DPDFNet)
Keyword spotting
Source separation (e.g., [spleeter][spleeter], [UVR][UVR])

on the following platforms and operating systems:

x86, x86_64, 32-bit ARM, 64-bit ARM (arm64, aarch64), RISC-V (riscv64), RK NPU, Ascend NPU
Linux, macOS, Windows, openKylin
Android, WearOS
iOS
HarmonyOS
NodeJS
WebAssembly
[NVIDIA Jetson Orin NX][NVIDIA Jetson Orin NX] (Support running on both CPU and GPU)
[NVIDIA Jetson Nano B01][NVIDIA Jetson Nano B01] (Support running on both CPU and GPU)
[Raspberry Pi][Raspberry Pi]
[RV1126][RV1126]
[LicheePi4A][LicheePi4A]
[VisionFive 2][VisionFive 2]
[旭日X3派][旭日X3派]
[爱芯派][爱芯派]
[RK3588][RK3588]
[SpacemiT-K1][SpacemiT-K1]
[SpacemiT-K3][SpacemiT-K3]
etc

with the following APIs

C++, C, Python, Go, C#
Java, Kotlin, JavaScript
Swift, Rust
Dart, Object Pascal

Links for Huggingface Spaces

You can visit the following Huggingface spaces to try sherpa-onnx without installing anything. All you need is a browser.

Description	URL	中国镜像
Speaker diarization	[Click me][hf-space-speaker-diarization]	[镜像][hf-space-speaker-diarization-cn]
Speech recognition	[Click me][hf-space-asr]	[镜像][hf-space-asr-cn]
Speech recognition with [Whisper][Whisper]	[Click me][hf-space-asr-whisper]	[镜像][hf-space-asr-whisper-cn]
Speech synthesis	[Click me][hf-space-tts]	[镜像][hf-space-tts-cn]
Generate subtitles	[Click me][hf-space-subtitle]	[镜像][hf-space-subtitle-cn]
Audio tagging	[Click me][hf-space-audio-tagging]	[镜像][hf-space-audio-tagging-cn]
Source separation	[Click me][hf-space-source-separation]	[镜像][hf-space-source-separation-cn]
Spoken language identification with [Whisper][Whisper]	[Click me][hf-space-slid-whisper]	[镜像][hf-space-slid-whisper-cn]

We also have spaces built using WebAssembly. They are listed below:

Description	Huggingface space	ModelScope space
Voice activity detection with [silero-vad][silero-vad]	[Click me][wasm-hf-vad]	[地址][wasm-ms-vad]
Real-time speech recognition (Chinese + English) with Zipformer	[Click me][wasm-hf-streaming-asr-zh-en-zipformer]	[地址][wasm-hf-streaming-asr-zh-en-zipformer]
Real-time speech recognition (Chinese + English) with Paraformer	[Click me][wasm-hf-streaming-asr-zh-en-paraformer]	[地址][wasm-ms-streaming-asr-zh-en-paraformer]
Real-time speech recognition (Chinese + English + Cantonese) with [Paraformer-large][Paraformer-large]	[Click me][wasm-hf-streaming-asr-zh-en-yue-paraformer]	[地址][wasm-ms-streaming-asr-zh-en-yue-paraformer]
Real-time speech recognition (English)	[Click me][wasm-hf-streaming-asr-en-zipformer]	[地址][wasm-ms-streaming-asr-en-zipformer]
VAD + speech recognition (Chinese) with Zipformer CTC	[Click me][wasm-hf-vad-asr-zh-zipformer-ctc-07-03]	[地址][wasm-ms-vad-asr-zh-zipformer-ctc-07-03]
VAD + speech recognition (Chinese + English + Korean + Japanese + Cantonese) with [SenseVoice][SenseVoice]	[Click me][wasm-hf-vad-asr-zh-en-ko-ja-yue-sense-voice]	[地址][wasm-ms-vad-asr-zh-en-ko-ja-yue-sense-voice]
VAD + speech recognition (English) with [Whisper][Whisper] tiny.en	[Click me][wasm-hf-vad-asr-en-whisper-tiny-en]	[地址][wasm-ms-vad-asr-en-whisper-tiny-en]
VAD + speech recognition (English) with [Moonshine tiny][Moonshine tiny]	[Click me][wasm-hf-vad-asr-en-moonshine-tiny-en]	[地址][wasm-ms-vad-asr-en-moonshine-tiny-en]
VAD + speech recognition (English) with Zipformer trained with [GigaSpeech][GigaSpeech]	[Click me][wasm-hf-vad-asr-en-zipformer-gigaspeech]	[地址][wasm-ms-vad-asr-en-zipformer-gigaspeech]
VAD + speech recognition (Chinese) with Zipformer trained with [WenetSpeech][WenetSpeech]	[Click me][wasm-hf-vad-asr-zh-zipformer-wenetspeech]	[地址][wasm-ms-vad-asr-zh-zipformer-wenetspeech]
VAD + speech recognition (Japanese) with Zipformer trained with [ReazonSpeech][ReazonSpeech]	[Click me][wasm-hf-vad-asr-ja-zipformer-reazonspeech]	[地址][wasm-ms-vad-asr-ja-zipformer-reazonspeech]
VAD + speech recognition (Thai) with Zipformer trained with [GigaSpeech2][GigaSpeech2]	[Click me][wasm-hf-vad-asr-th-zipformer-gigaspeech2]	[地址][wasm-ms-vad-asr-th-zipformer-gigaspeech2]
VAD + speech recognition (Chinese 多种方言) with a [TeleSpeech-ASR][TeleSpeech-ASR] CTC model	[Click me][wasm-hf-vad-asr-zh-telespeech]	[地址][wasm-ms-vad-asr-zh-telespeech]
VAD + speech recognition (English + Chinese, 及多种中文方言) with Paraformer-large	[Click me][wasm-hf-vad-asr-zh-en-paraformer-large]	[地址][wasm-ms-vad-asr-zh-en-paraformer-large]
VAD + speech recognition (English + Chinese, 及多种中文方言) with Paraformer-small	[Click me][wasm-hf-vad-asr-zh-en-paraformer-small]	[地址][wasm-ms-vad-asr-zh-en-paraformer-small]
VAD + speech recognition (多语种及多种中文方言) with [Dolphin][Dolphin]-base	[Click me][wasm-hf-vad-asr-multi-lang-dolphin-base]	[地址][wasm-ms-vad-asr-multi-lang-dolphin-base]
Speech synthesis (Piper, English)	[Click me][wasm-hf-tts-piper-en]	[地址][wasm-ms-tts-piper-en]
Speech synthesis (Piper, German)	[Click me][wasm-hf-tts-piper-de]	[地址][wasm-ms-tts-piper-de]
Speech synthesis (Matcha, Chinese)	[Click me][wasm-hf-tts-matcha-zh]	[地址][wasm-ms-tts-matcha-zh]
Speech synthesis (Matcha, English)	[Click me][wasm-hf-tts-matcha-en]	[地址][wasm-ms-tts-matcha-en]
Speech synthesis (Matcha, Chinese+English)	[Click me][wasm-hf-tts-matcha-zh-en]	[地址][wasm-ms-tts-matcha-zh-en]
Speaker diarization	[Click me][wasm-hf-speaker-diarization]	[地址][wasm-ms-speaker-diarization]
Voice cloning with ZipVoice (Chinese+English)	[Click me][wasm-hf-voice-cloning-zipvoice]	[地址][wasm-ms-voice-cloning-zipvoice]
Voice cloning with Pocket TTS (English)	[Click me][wasm-hf-voice-cloning-pocket]	[地址][wasm-ms-voice-cloning-pocket]

Links for pre-built Android APKs

You can find pre-built Android APKs for this repository in the following table

Description	URL	中国用户
Speaker diarization	[Address][apk-speaker-diarization]	[点此][apk-speaker-diarization-cn]
Streaming speech recognition	[Address][apk-streaming-asr]	[点此][apk-streaming-asr-cn]
Simulated-streaming speech recognition	[Address][apk-simula-streaming-asr]	[点此][apk-simula-streaming-asr-cn]
Text-to-speech	[Address][apk-tts]	[点此][apk-tts-cn]
Voice activity detection (VAD)	[Address][apk-vad]	[点此][apk-vad-cn]
VAD + non-streaming speech recognition	[Address][apk-vad-asr]	[点此][apk-vad-asr-cn]
Two-pass speech recognition	[Address][apk-2pass]	[点此][apk-2pass-cn]
Audio tagging	[Address][apk-at]	[点此][apk-at-cn]
Audio tagging (WearOS)	[Address][apk-at-wearos]	[点此][apk-at-wearos-cn]
Speaker identification	[Address][apk-sid]	[点此][apk-sid-cn]
Spoken language identification	[Address][apk-slid]	[点此][apk-slid-cn]
Keyword spotting	[Address][apk-kws]	[点此][apk-kws-cn]

Links for pre-built Flutter APPs

Details

Real-time speech recognition

Description	URL	中国用户
Streaming speech recognition	[Address][apk-flutter-streaming-asr]	[点此][apk-flutter-streaming-asr-cn]

Text-to-speech

Description	URL	中国用户
Android (arm64-v8a, armeabi-v7a, x86_64)	[Address][flutter-tts-android]	[点此][flutter-tts-android-cn]
Linux (x64)	[Address][flutter-tts-linux]	[点此][flutter-tts-linux-cn]
macOS (x64)	[Address][flutter-tts-macos-x64]	[点此][flutter-tts-macos-x64-cn]
macOS (arm64)	[Address][flutter-tts-macos-arm64]	[点此][flutter-tts-macos-arm64-cn]
Windows (x64)	[Address][flutter-tts-win-x64]	[点此][flutter-tts-win-x64-cn]

Note: You need to build from source for iOS.

Links for pre-built Lazarus APPs

Details

Generating subtitles

Description	URL	中国用户
Generate subtitles (生成字幕)	[Address][lazarus-subtitle]	[点此][lazarus-subtitle-cn]

Links for pre-trained models

Details

Description	URL
Speech recognition (speech to text, ASR)	[Address][asr-models]
Text-to-speech (TTS)	[Address][tts-models]
VAD	[Address][vad-models]
Keyword spotting	[Address][kws-models]
Audio tagging	[Address][at-models]
Speaker identification (Speaker ID)	[Address][sid-models]
Spoken language identification (Language ID)	See multi-lingual [Whisper][Whisper] ASR models from [Speech recognition][asr-models]
Punctuation	[Address][punct-models]
Speaker segmentation	[Address][speaker-segmentation-models]
Speech enhancement	[Address][speech-enhancement-models]
Source separation	[Address][source-separation-models]

Some pre-trained ASR models (Streaming)

Details

Please see

for more models. The following table lists only SOME of them.

Name	Supported Languages	Description
[sherpa-onnx-streaming-zipformer-bilingual-zh-en-2023-02-20][sherpa-onnx-streaming-zipformer-bilingual-zh-en-2023-02-20]	Chinese, English	See also
[sherpa-onnx-streaming-zipformer-small-bilingual-zh-en-2023-02-16][sherpa-onnx-streaming-zipformer-small-bilingual-zh-en-2023-02-16]	Chinese, English	See also
[sherpa-onnx-streaming-zipformer-zh-14M-2023-02-23][sherpa-onnx-streaming-zipformer-zh-14M-2023-02-23]	Chinese	Suitable for Cortex A7 CPU. See also
[sherpa-onnx-streaming-zipformer-en-20M-2023-02-17][sherpa-onnx-streaming-zipformer-en-20M-2023-02-17]	English	Suitable for Cortex A7 CPU. See also
[sherpa-onnx-streaming-zipformer-korean-2024-06-16][sherpa-onnx-streaming-zipformer-korean-2024-06-16]	Korean	See also
[sherpa-onnx-streaming-zipformer-fr-2023-04-14][sherpa-onnx-streaming-zipformer-fr-2023-04-14]	French	See also

Some pre-trained ASR models (Non-Streaming)

Details

Please see

for more models. The following table lists only SOME of them.

Name	Supported Languages	Description
sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8	English	It is converted from https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2
Whisper tiny.en	English	See also
[Moonshine tiny][Moonshine tiny]	English	See also
sherpa-onnx-zipformer-ctc-zh-int8-2025-07-03	Chinese	A Zipformer CTC model
[sherpa-onnx-sense-voice-zh-en-ja-ko-yue-2024-07-17][sherpa-onnx-sense-voice-zh-en-ja-ko-yue-2024-07-17]	Chinese, Cantonese, English, Korean, Japanese	支持多种中文方言. See also
[sherpa-onnx-paraformer-zh-2024-03-09][sherpa-onnx-paraformer-zh-2024-03-09]	Chinese, English	也支持多种中文方言. See also
[sherpa-onnx-zipformer-ja-reazonspeech-2024-08-01][sherpa-onnx-zipformer-ja-reazonspeech-2024-08-01]	Japanese	See also
[sherpa-onnx-nemo-transducer-giga-am-russian-2024-10-24][sherpa-onnx-nemo-transducer-giga-am-russian-2024-10-24]	Russian	See also
[sherpa-onnx-nemo-ctc-giga-am-russian-2024-10-24][sherpa-onnx-nemo-ctc-giga-am-russian-2024-10-24]	Russian	See also
[sherpa-onnx-zipformer-ru-2024-09-18][sherpa-onnx-zipformer-ru-2024-09-18]	Russian	See also
[sherpa-onnx-zipformer-korean-2024-06-24][sherpa-onnx-zipformer-korean-2024-06-24]	Korean	See also
[sherpa-onnx-zipformer-thai-2024-06-20][sherpa-onnx-zipformer-thai-2024-06-20]	Thai	See also
[sherpa-onnx-telespeech-ctc-int8-zh-2024-06-04][sherpa-onnx-telespeech-ctc-int8-zh-2024-06-04]	Chinese	支持多种方言. See also

Useful links

Documentation: https://k2-fsa.github.io/sherpa/onnx/
Bilibili 演示视频: https://search.bilibili.com/all?keyword=%E6%96%B0%E4%B8%80%E4%BB%A3Kaldi

Name		Name	Last commit message	Last commit date
.github		.github
android		android
benchmark		benchmark
c-api-examples		c-api-examples
cmake		cmake
cpp		cpp
cxx-api-examples		cxx-api-examples
dart-api-examples		dart-api-examples
dotnet-examples		dotnet-examples
ffmpeg-examples		ffmpeg-examples
flutter-examples		flutter-examples
flutter		flutter
go-api-examples		go-api-examples
harmony-os		harmony-os
ios-swift		ios-swift
ios-swiftui		ios-swiftui
java-api-examples		java-api-examples
kotlin-api-examples		kotlin-api-examples
lazarus-examples		lazarus-examples
mfc-examples		mfc-examples
nodejs-addon-examples		nodejs-addon-examples
nodejs-examples		nodejs-examples
pascal-api-examples		pascal-api-examples
python-api-examples		python-api-examples
rust-api-examples		rust-api-examples
scripts		scripts
sherpa-onnx		sherpa-onnx
swift-api-examples		swift-api-examples
tauri-examples		tauri-examples
toolchains		toolchains
wasm		wasm
.clang-format		.clang-format
.clang-tidy		.clang-tidy
.flake8		.flake8
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CMakeLists.txt		CMakeLists.txt
CPPLINT.cfg		CPPLINT.cfg
LICENSE		LICENSE
MANIFEST.in		MANIFEST.in
README.md		README.md
jitpack.yml		jitpack.yml
new-release.sh		new-release.sh
pom.xml		pom.xml
release.sh		release.sh
setup.py		setup.py

Latest commit

History

Folders and files

Repository files navigation

⚡ EdgeDeploy-Inference: Snapdragon & Jetson Edge ASR Architecture & Developer Guide

🏛️ System Architecture Overview

🚫 Why llama.cpp is Unsuitable for Non-Autoregressive ASR

⚡ Snapdragon 8 Elite Hardware Deep-Dive

1. Custom Oryon CPU Cluster & Cache Topology

2. OS Thread Affinity Pinning (sched_setaffinity)

3. ARM Neon Vectorization (q0-q15 Registers)

4. Qualcomm Hexagon NPU & HTP Co-Design

5. Low-Latency Audio Capture Scheduling via SCHED_FIFO

Preemption Sequence Diagram:

Linux Priority Spectrum & Thread Hierarchy:

6. Qualcomm QNN EP Memory Architecture & Voltage Rails

FastRPC Zero-Copy Buffer Mapping:

HTP Voltage Corners and Dynamic Clock & Voltage Scaling (DCVS):

7. Quantization & Calibration Strategy for Code-Mixed Hinglish ASR

Quantization Paradigm Comparison:

Mathematical Impact of Static Quantization on Softmax:

Hybrid Quantization Flow (Selective Precision):

🧮 Mathematical Formulation

1. Cooley-Tukey Radix-2 FFT

2. Mel-Spectrogram Processing Frontend

3. Connectionist Temporal Classification (CTC) Decoding

4. Post-Training Quantization (PTQ) Rounding Noise Propagation

Static Quantization (Fixed Bounds):

Dynamic Quantization (Adaptive Scaling):

📊 Benchmarks & Performance Telemetry

Real-Time Factor (RTF) Formulation

Comparative Evaluation Results

🛠️ Advanced Code & Build Configurations

📂 Repository Codebase Directory Layout

🛠️ Build and Execution Instructions

C++ Library Compilation

Running the Benchmark Suite

🌍 Supported Platforms, Languages & Pre-trained Models

Supported platforms

Supported programming languages

Supported NPUs

Introduction

Links for Huggingface Spaces

Links for pre-built Android APKs

Links for pre-built Flutter APPs

Real-time speech recognition

Text-to-speech

Links for pre-built Lazarus APPs

Generating subtitles

Links for pre-trained models

Some pre-trained ASR models (Streaming)

Some pre-trained ASR models (Non-Streaming)

Useful links

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

🚫 Why `llama.cpp` is Unsuitable for Non-Autoregressive ASR

2. OS Thread Affinity Pinning (`sched_setaffinity`)

3. ARM Neon Vectorization (`q0-q15` Registers)

5. Low-Latency Audio Capture Scheduling via `SCHED_FIFO`