feat(turbo4+acrp): Turbo4 quantized datatype with packed HNSW scoring + adaptive compression/retrieval plane (ADR-296/297)

[ruvnet]

Summary

Two ADRs (both included), implemented incrementally on this branch:

• ADR-296 — Turbo4 datatype: Qdrant-style 4-bit Lloyd-Max codes over a deterministic randomized rotation, scored directly on packed codes inside HNSW. Closes the Turbo4 slice of DbOptions.quantization is silently ignored — never applied by VectorDB (misleading default + docs) #563.
• ADR-297 — Adaptive Compression & Retrieval Plane: one EncodedVector interface for every representation, storage/search precision separation, per-query automatic precision selection, provenance, and a three-policy product surface — gated by an ablation acceptance test.

New crate: crates/ruvector-turboquant

• Deterministic rotation: sign → permutation → block-FWHT rounds, in-crate SplitMix64 (no rand; codes are a persisted format). No zero-padding: any even D decomposes into power-of-two FWHT blocks → codes are exactly D/2 + 8 bytes (7.9× at 1536-D).
• Lloyd-Max 16-level tables with per-vector standardization α = ‖v‖/√D; RaBitQ-style length renormalization (√(D/S) dot correction, exact norms α²D) adopted from a verified deep-research pass over TurboQuant/Qdrant/RaBitQ primary sources — estimator-bias test gates mean signed L2 error < 1 %.
• Three scoring tiers, zero reconstruction: symmetric code×code (graph build), asymmetric int8-query×nibble (traversal), SIMD f32×nibble rescore (final ranking). Measured AVX2 throughput: 24.5 Gelem/s asymmetric at 1536-D (62.6 ns/vector) — ~20× the previous widen-sequence kernel, which stays in the criterion bench as a permanent baseline. Kernels are bit-exact vs scalar oracles (the oracle caught a real sign(−128) wrap during development).
• 1-bit candidate plane (bits1): RaBitQ-style sign codes over the same rotation; query decomposed into 8 bit-planes → traversal scoring is pure AND+POPCNT with ~4× less memory traffic.

Core integration

• QuantizationConfig::Turbo4 { rotation_seed, rescore_multiplier, policy, search_quantization } — applied (legacy variants keep their not-applied warning).
• Turbo4HnswIndex: hnsw_rs over u8 blobs; blob roles structurally disjoint by length, giving symmetric construction and true asymmetric traversal without forking hnsw_rs. Serialization roundtrips (graph rebuilt from blobs; identical rescored distances).
• SearchQuantization::RaBitQ1 cascade (ADR-297 §2): node data [bits1 ‖ turbo4] with the 1-bit plane first (traversal reads only the short cache-friendly prefix); graph construction scores Turbo4 sections; candidates rescore on the shared Turbo4 plane. One rotation per query serves both. The ~5 bits/dim active search plane.
• ruvector-core::encoding: CodecKind + VectorCodec/EncodedQuery traits; Fp32/Fp16 (in-crate RNE binary16)/Int8/Turbo4 codecs; codec_for() registry; Pq/RaBitQ1 kinds reserved.
• Adaptive escalation (SearchPolicy::{Quality,Balanced,MaxCompression}): kept/dropped score-margin triggers widened re-search, stopping on top-k stability; MaxCompression never escalates; adaptive_stats() telemetry.
• Provenance guard (ADR-297 §7): a VectorProvenance record (codec, version, rotation seed, dim, metric) is persisted in the vector store when Turbo4 is active and validated on every reopen — mismatches refuse to open instead of silently serving wrong results.
• Node binding accepts type: "turbo4"; ann_benchmark/memory_benchmark accept turbo4.

Tests (all green: 27 turboquant + 259 core lib, clippy --all-targets clean)

Rotation orthogonality/determinism (incl. non-power-of-two dims); codec roundtrip bounds; kernel bit-exactness + f32-kernel epsilon gates; estimator unbiasedness; recall gates vs f32 HNSW baseline on clustered data (≤3pp incl. measured hnsw_rs graph nondeterminism) + Gaussian worst-case floor; cascade recall within 5pp of direct; adaptive policy behavior; serialization roundtrips (both modes); provenance guard; end-to-end VectorDB insert/search/restart.

Follow-ups (ADR-297 phases D–G)

Drift monitors, memory tiers, topology-aware allocation, NEON/AVX-512-VNNI/WASM kernels, SIFT1M/GIST1M/Deep1M acceptance + the adaptive-vs-uniform ablation in ruvector-sota-bench.

🤖 Generated with claude-flow

https://claude.ai/code/session_01XFWB9PKwsZYk5FbjBRY6mk

[ruvnet]

Ablation results (ADR-297 phase G, first measurement)

turbo4-ablation, 20k × 768-D deterministic clustered corpus, 200 queries, identical HNSW params (m=16, ef_c=200, ef_s=100), brute-force f32 ground truth:

config	recall@1	recall@10	P50 µs	P95 µs	payload B/vec	vs f32
f32 HNSW (baseline)	0.945	0.947	1708	2136	3072	—
turbo4 maxcomp	0.970	0.877	991	1367	392 (7.8×)	−7.0 pp, 1.7× faster
turbo4 balanced	0.975	0.884	2521	2988	392 (7.8×)	−6.4 pp
turbo4 quality	0.990	0.880	3623	4289	392 (7.8×)	−6.8 pp
cascade balanced	0.985	0.879	4118	4674	496 (6.2×)	−6.8 pp
t4 maxcomp + f32 verify	0.950	0.961	1664	2175	392 (7.8×)	+1.4 pp, ≈ latency
cascade mc + f32 verify	0.955	0.962	1202	1453	496 (6.2×)	+1.5 pp, 30 % faster

Findings

1. Candidate generation is not the bottleneck — Turbo4 recall@1 beats f32 (0.99 vs 0.945; escalation searches wider than the baseline's ef), and raising the rescore pool from 2× to 8× left recall@10 unchanged. The true neighbors are in the pool; 4-bit rescoring cannot order ranks 5–10 inside tightly concentrated neighborhoods (quantization noise ≈ inter-rank gaps). Widening traversal cannot fix a precision-bound ranking — measured, not assumed.
2. The ADR-297 §2 verification tier fixes exactly this: re-ranking the top-2k against stored f32 vectors recovers recall@10 to above the f32 baseline at equal-or-better latency. Now wired into VectorDB::search for Turbo4 databases (nearly free — enrichment already fetches those vectors).
3. Best configuration (cascade + verify): better recall (+1.5 pp), better P50/P95 (−30 %), and 6.2× less vector memory than f32 HNSW, simultaneously.

Caveats: synthetic clustered corpus, single machine, index-payload bytes only (graph links/ids excluded — same for all configs). SIFT1M/GIST1M runs remain the ADR acceptance gate. The escalation-margin threshold needs recalibration (fires 100 % on this concentrated workload; the verification tier now covers what escalation was reaching for).

---

Generated by Claude Code

[ruvnet]

CI note: Tests (vector-index) failed on 95634a8 with a single failure out of 995 — ruvector-graph typed_graph::tests::indexed_path_finds_top_result_and_traverses. Investigated: that test touches no code path this PR changes (no VectorDB/quantization/encoding usage), and it passes 9/9 consecutive runs locally on this exact branch. It's a pre-existing intermittent failure (ANN top-result assertion, the same nondeterminism class as hnsw_rs graph construction), unrelated to this diff. Not fixing it here to keep the PR scoped; the next push will re-run the suite.

---

Generated by Claude Code

[ruvnet]

Final review and validation

Reviewed and validated through head 4cf23d3ac5a618c3ef024438c8571cd1ae85b037 (generated platform binaries built from reviewed source head 7aa9ea82cdedf924d32ed4d41406b2910a58d4fa).

Review fixes published

• f890b8a7c: replaced release-only-elided SIMD buffer preconditions with enforced checks, preventing malformed public inputs from reaching unchecked AVX2 loads; added malformed-buffer coverage.
• 7aa9ea82c: added ruvector-turboquant to the crates.io workflow ahead of ruvector-core, and updated release documentation/counts to 27 crates.
• 4cf23d3ac: GitHub Actions rebuilt and committed NAPI binaries for Linux x64/ARM64, macOS x64/ARM64, and Windows x64.

Evidence

• Branch synchronized with main: 0 behind.
• cargo test -p ruvector-turboquant --all-targets: 28 passed.
• cargo test -p ruvector-core --lib: 259 passed, 1 intentional network-test ignore.
• Focused Clippy with -D warnings: TurboQuant, core, and turbo4-ablation clean.
• cargo fmt --all -- --check: clean.
• Release workflow parses; publish order contains 27 crates and puts TurboQuant before core.
• cargo package -p ruvector-turboquant --allow-dirty: package archive verified and compiled.
• Documented 20k×768 ablation executed: direct Turbo4 remains roughly 9 pp below FP32 recall@10 on the synthetic clustered corpus; Turbo4 + FP32 verification restored parity (+0.1 pp in this run). Real SIFT1M/GIST1M/RuVector corpus acceptance remains the explicitly documented ADR follow-up and is not claimed here.
• Remote: all five NAPI platform builds passed; WASM dedup passed; SOTA smoke and BigANN streaming passed. The generated Linux x64 addon loads locally, and its duplicate artifacts are byte-identical.

No unresolved review threads were present. Remaining manually dispatched exact-head GitHub jobs are queued without runner assignment; local equivalents and the relevant completed remote platform gates are green.