[MOD-17688] Add relabelVector API

[nonirosenfeldredis]

Describe the changes in the pull request

Adds relabelVector(old_label, new_label), which moves the vector(s) stored under a label to a new label without touching the stored vector data or the index structure.

HNSW keys its graph purely on internal ids — links, entry point and levels never mention a label — so a relabel only has to rewrite the two places a label is kept: idToMetaData[id].label (read by getExternalLabel on every query result) and the derived class's label→id lookup. That makes it O(1) per stored vector, with no distance computations and no graph mutation: a cheap alternative to delete-then-add for callers whose external id changed while the vector itself did not.

Implemented for brute force (single/multi), HNSW (single/multi) and tiered HNSW. SVS delegates label management to the external library, so it inherits the interface default that reports VecSimRelabel_Unsupported rather than being forced to implement it.

Semantics. The move is rejected — leaving the index untouched — when the source label is absent, the target label is already taken, or the two labels are equal. Rejecting rather than overwriting keeps the caller in control: overwriting the target would silently drop a vector, and unlike addVector there is no replacement data to justify it. In the tiered index the target must be free in all three homes of a label (flat buffer, pending insert jobs, HNSW), which is what makes a partially applied move impossible.

Each outcome has its own code rather than a shared 0, because a caller acts on them differently — OldLabelMissing and SameLabel mean there is nothing to do, NewLabelTaken is a conflict it may resolve, and Unsupported means it has to fall back to delete + insert:

typedef enum {
    VecSimRelabel_OK = VecSim_OK,
    VecSimRelabel_OldLabelMissing, // `old_label` is not in the index
    VecSimRelabel_NewLabelTaken,   // `new_label` is already in the index
    VecSimRelabel_SameLabel,       // `old_label` and `new_label` are equal
    VecSimRelabel_Unsupported      // this index type does not implement relabeling
} VecSimRelabelCode;

This follows the existing VecSimResolveCode / VecSimDebugCommandCode conventions, including _OK = VecSim_OK as the first member.

The codes are ordered rather than independent: an absent old_label is reported ahead of an occupied new_label, in all three implementations and — for the tiered index — across all three homes of a label. They are not interchangeable, so a caller that resolves NewLabelTaken by freeing the target must never be sent down that path for a move that had nothing to move.

Three details worth a reviewer's attention

1. HNSW assigns idToMetaData[id].label as a field rather than replacing the ElementMetaData struct, whose constructor resets flags to IN_PROCESS. Relabeling an element that is mid-insertion is a reachable state — executeInsertJob populates HNSW before removing the vector from the flat buffer — and struct assignment would reset that flag on an element the ingesting worker is about to unmarkInProcess, leaving it permanently invisible to queries.

2. The tiered index moves all four homes of a label under one flatIndexGuard-exclusive then mainIndexGuard-exclusive section. That is the flat→main order already used by insertVectorToHNSW and topKQueryImp, so it cannot deadlock against them; repair and swap jobs only ever take the main guard. The main guard must be exclusive rather than shared (unlike markDelete, which only touches the atomic flags byte): query threads read getExternalLabel holding only a shared main guard — topKQuery does not take indexDataGuard — and ElementMetaData is #pragma pack(1), so the label store can be unaligned and is not atomic. Downgrading to shared would require unpacking the struct at +7 bytes per element, and would not reduce query blocking anyway, since every read path takes the flat guard first.

3. In the tiered index, each of the three homes is asked to move the label only once it has reported holding it. With both guards held and every other rejection already ruled out — the target is free everywhere, the labels differ — a home that is asked can only answer OK, so a refusal would mean the state changed underneath us; that invariant is asserted (under BUILD_TESTS) rather than folded into the return value, and the outcome is decided by whether the label moved anywhere. The earlier ret |= ... form would have reported OK over a refusal from one home.

Critically, the job map key and each pending job's own label copy must move together: executeInsertJob indexes into HNSW under job->label and then looks the job up with labelToInsertJobs.at(job->label), so a half-applied move either indexes the vector under the stale label or throws std::out_of_range out of a worker thread — which terminates the process rather than failing a command. relabelVectorDuringIngestion covers that path; removing the job->label rewrite makes it abort with exactly that exception.

Which issues this PR fixes

1. MOD-17688

Main objects this PR modified

1. VecSimIndexInterface — new virtual with an "unsupported" default; VecSimIndex_RelabelVector C API
2. BruteForceIndex_Single / BruteForceIndex_Multi — re-key the label lookup, keep the id→label direction in sync via setVectorLabel
3. HNSWIndex — shared implementation over the existing label-lookup primitives, plus a new isLabelExists primitive implemented by both single and multi
4. TieredHNSWIndex — moves both tiers plus the pending-insert-job bookkeeping under one critical section

Testing

14 tests across test_bruteforce.cpp, test_bruteforce_multi.cpp, test_hnsw.cpp, test_hnsw_multi.cpp, test_hnsw_tiered.cpp and test_svs.cpp: happy path, all three rejection cases, marked-deleted elements, multi (all ids under a label move together), the SVS "unsupported" default, tiered flat-only with a pending job drained afterwards, tiered both-tiers, and a 500-vector concurrent test that races relabels against live ingestion by the worker threads. Every rejection now asserts the specific code, so a wrong reason fails rather than passing as a generic 0 — including the case where the source is absent and the target is taken, which is what pins the ordering between the two codes (verified by restoring the old check order: relabelVectorRejects fails).

relabelVectorMulti (tiered) covers the case where a label has several ids in each of its three homes at once — flat buffer, pending insert jobs and HNSW — and asserts on the whole id set of every home, so a move that handles only the first id of a label, or only one home, fails instead of looking right. Verified by patching HNSWIndex::relabelVector to move only ids.front(): that test fails, as does the existing HNSWMultiTest one.

Full unit suite green; checkIntegrity() asserted after every relabel, and per-label distance checks confirm no relabel crossed wires.

Mark if applicable

• This PR introduces API changes
• This PR introduces serialization changes

Serialization is unaffected: saveGraph/restoreGraph already persist the label from idToMetaData and rebuild the label lookup from it, so a relabel round-trips with no format change.

🤖 Generated with Claude Code

---

Note

Medium Risk
New concurrent mutation path on core index label maps, including exclusive locking in tiered HNSW while queries read labels. Incorrect bookkeeping could desync pending insert jobs or hide live HNSW elements.

Overview
Adds VecSimIndex_RelabelVector so callers can move vector(s) from old_label to new_label by rewriting label bookkeeping only — no vector copy, no HNSW graph change. Intended as a cheap alternative to delete-then-add when the external id changes.

Implemented for brute force (single/multi), HNSW (single/multi), and tiered HNSW. SVS keeps the interface default VecSimRelabel_Unsupported. Rejections leave the index untouched: missing source, occupied target, or same labels, with missing source reported first. Distinct VecSimRelabelCode values let callers distinguish no-op, conflict, and fallback-to-delete+insert.

HNSW updates idToMetaData[id].label as a field (not whole-struct assign, which would reset IN_PROCESS flags) and re-keys the label lookup. Tiered HNSW moves the flat buffer, pending insert jobs (map key and job->label), and HNSW under exclusive flatIndexGuard then exclusive main guard, so a label that lives in more than one home during ingest cannot be half-moved.

Covered by unit tests (happy path, rejects, multi-id, marked-deleted, SVS unsupported, tiered both-tiers and concurrent ingest) plus a microsecond-scale relabel benchmark.

^{Reviewed by Cursor Bugbot for commit 1a23d0e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.28%. Comparing base (7556270) to head (1a23d0e).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1017      +/-   ##
==========================================
+ Coverage   97.18%   97.28%   +0.09%     
==========================================
  Files         141      141              
  Lines        8537     8621      +84     
==========================================
+ Hits         8297     8387      +90     
+ Misses        240      234       -6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[nonirosenfeldredis]

Benchmark results

From the bm-basics-fp32-single and bm-basics-fp32-multi runs on this branch (single, multi), comparing the new RelabelLabel benchmark against the AddLabel + DeleteLabel pair it replaces for callers whose external id changed while the vector did not. Same dataset, same harness, same run.

relabeled_ratio=1 on every row, so no timed call was a rejected no-op.

Single (1 vector/label) — arm64 / x86_64

Index	Relabel	AddLabel	DeleteLabel	Add+Delete	Speedup
BF	0.447 / 0.266 µs	2 / 2 µs	1 / 2 µs	3 / 4 µs	~7× / ~15×
HNSW	0.291 / 0.207 µs	5300 / 4950 µs	6470 / 3310 µs	11770 / 8260 µs	~40,000×
Tiered	0.412 / 0.310 µs	3 / 4 µs	39 / 33 µs	42 / 37 µs	~100×

Multi (25 vectors/label) — arm64

Index	Relabel	AddLabel	DeleteLabel	Add+Delete	Speedup
BF	0.248 µs	118 µs	18 µs	136 µs	~550×
HNSW	1.06 µs	61,200 µs	74,000 µs	135,200 µs	~128,000×
Tiered	1.04 µs	18,300 µs	5,390 µs	23,690 µs	~23,000×

Reading

• HNSW is the case that motivates the API. Delete+add there means graph surgery plus a full reinsertion at M=64/efC=512 on a 1M-vector index. Relabel is two hash operations and a size_t store, which is what the 4–5 decade gap reflects: the graph is genuinely never touched.
• Multi scales as designed. HNSW relabel goes 0.291 → 1.06 µs for 25× the vectors per label — sub-linear, because the per-call lock and lookup overhead is fixed and only the idToMetaData[id].label writes multiply. That is the "O(1) per stored vector" claim, measured.
• Tiered costs HNSW plus a small constant (0.412 vs 0.291 µs single; 1.04 vs 1.06 µs multi, i.e. within noise). The two extra lock acquisitions and three existence checks are cheap.
• BF multi beats BF single (0.248 vs 0.447 µs) despite 25× the label writes, because the multi index has 40k labels rather than 1M, so labelToIdsLookup stays in cache. Cache locality dominates the extra writes.

Caveats

• These runs only exercise the HNSW-resident tiered path. The benchmark loads a serialized HNSW index, so the flat buffer is empty and the flat-only path (relabel while ingestion is still pending) is not measured — it is unit-tested but unmeasured.
• BF add/delete are reported in ms to 3 decimals, i.e. ±1 µs granularity on values of 1–4 µs, so the BF speedups are order-of-magnitude only.
• The BF-vs-HNSW relabel ordering in the single case (0.447 vs 0.291 µs) is noise at 1024 fixed iterations and should not be read as "relabel is slower on BF". The HNSW and multi comparisons span 4–5 decades and are well outside noise.

🤖 Generated with Claude Code

[ofiryanai]

Overall looking good, one comment regarding the API

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit aa417ae. Configure here.}