Environment: basic-memory 0.22.1, SQLite 3.43.2
Problem
The FTS index hardcodes tokenize='unicode61 tokenchars 0x2F' with prefix='1,2,3,4' (in basic_memory/models/search.py:91 and the migration alembic/versions/cc7172b46608_update_search_index_schema.py). unicode61 performs no CJK word segmentation, so a continuous run of CJK characters becomes a single token, and only prefixes match.
Repro: a note contains 即适者生存，生存等于正确的道理:
- search-notes "适者生存" → no results (适者生存 is a mid-token substring of 即适者生存, not a prefix).
- search-notes "财务护城河" in a note containing 做好财务护城河 → wrong/no result, same cause.
This makes keyword search effectively unusable for CJK notes.
Why trigram is not sufficient
SQLite's built-in trigram tokenizer only matches queries ≥3 characters. CJK words are frequently 2 characters (止损, 健康, 红线), so switching to trigram would break the most common search pattern. (Verified: trigram returns nothing for 2-char queries.)
Request
Add CJK word segmentation support, e.g.:
- A config option for the FTS tokenizer (so users can choose trigram or a jieba-backed tokenizer), and/or
- Built-in jieba segmentation (pre-segment CJK before indexing and before querying).
Workaround
None clean. CJK users must rely on vector/semantic search for phrase lookup.
Environment: basic-memory 0.22.1, SQLite 3.43.2
Problem
The FTS index hardcodes tokenize='unicode61 tokenchars 0x2F' with prefix='1,2,3,4' (in basic_memory/models/search.py:91 and the migration alembic/versions/cc7172b46608_update_search_index_schema.py). unicode61 performs no CJK word segmentation, so a continuous run of CJK characters becomes a single token, and only prefixes match.
Repro: a note contains 即适者生存，生存等于正确的道理:
This makes keyword search effectively unusable for CJK notes.
Why trigram is not sufficient
SQLite's built-in trigram tokenizer only matches queries ≥3 characters. CJK words are frequently 2 characters (止损, 健康, 红线), so switching to trigram would break the most common search pattern. (Verified: trigram returns nothing for 2-char queries.)
Request
Add CJK word segmentation support, e.g.:
Workaround
None clean. CJK users must rely on vector/semantic search for phrase lookup.