fix: tolerate matched docs with missing field payload (Redis 8.8 expiry race)

[vishal-bala]

Summary

Harden RedisVL result parsing so a matched document with a missing field payload is tolerated (skipped) instead of raising or surfacing a partial record. This is client-side future-proofing for a Redis 8.8+ server-side race and the durable fix behind the recent CI flakiness on redis:latest.

Background

Redis 8.8 changed a RediSearch default: the query worker pool now runs a nonzero number of background worker threads. With a background executor active, a document (or an indexed hash field) that expires via TTL/HPEXPIRE at the exact instant a background FT.SEARCH materializes it is returned as a matched id with a nil field array instead of being dropped server-side. redis-py collapses that into a Document carrying only {"id": ..., "payload": None} — no exception — so the malformed record lands in RedisVL, which previously raised KeyError in the vector-normalize branch or leaked a partial record into CacheHit/ChatMessage/RouteMatch. The prior mitigation (--search-workers 0 in CI) only hid this; real users on Redis 8.8 with TTLs are still exposed.

What changed

• Core parser (redisvl/index/index.py): process_results now skips a matched doc whose field payload is missing, detected only on zero-false-positive signals — a vector/range query missing its always-present vector_distance, or a JSON full-object unpack missing its json key. A plain hash FilterQuery/TextQuery doc is passed through as an id-only dict rather than dropped, so legitimate id-only projections and INDEXMISSING results are never over-skipped. process_aggregate_results and the hybrid/SQL paths drop entirely-empty rows.
• Extension guards: the semantic cache, message history, and semantic router independently tolerate an id-only/incomplete record (skipping it) for the paths the core rule intentionally doesn't cover.
• Completeness signal: SearchIndex.query() now returns a SearchResults object — a drop-in list subclass that additionally exposes dropped_count and complete, so callers that need completeness guarantees (audit, eval harnesses, agents) can detect and react to a race-shortened result set. Fully backward compatible: it behaves exactly like a list everywhere else.
• Observability: skips are logged at WARNING level, aggregated once per query.
• Docs: a new "Results and expiring documents" section in docs/concepts/queries.md explains the behavior, the detection boundaries, and how to use results.complete / dropped_count.

Behavior and compatibility

Backward compatible for every caller not hitting the race. Under the race, result counts become best-effort: query() may return fewer than num_results, paginate() fewer than page_size, and CountQuery.total can exceed the number of materialized documents (it reflects the server's match count). Silent skip is the default; no config knob is added.

Testing

Deterministic unit tests simulate the malformed response (a Result with a None fields slot / hand-built Document), covering every raise-site and skip branch, an over-skip guard proving legitimate id-only results survive, the SearchResults signal, and the extension guards. CI stays pinned at --search-workers 0; a live REDIS_SEARCH_WORKERS>0 reproduction is documented as a manual step. make check-types and make format are clean; unit and integration query tests pass.

Follow-ups (out of scope)

A process-level skip metric/callback and semantic-cache race-miss vs. real-miss decomposition, a nightly workers>0 integration test to guard the undocumented server shape, optional router detect-and-retry for the top-1 mis-route case, and raw=True message-history hardening.

---

Note

Medium Risk
Changes the shared query result-processing path used by all indexes and extensions; behavior is backward compatible but result counts and extension routing can differ under TTL-heavy concurrent search.

Overview
Hardens RedisVL against a Redis 8.8+ background-search race where a document expiring during FT.SEARCH can come back as a matched id with an empty field payload (instead of being dropped server-side), which previously caused KeyErrors, partial records, or extension validation failures.

Core parsing (process_results, aggregate/hybrid/SQL paths) now skips those hits when detection is unambiguous (e.g. missing vector_distance on vector queries with scores, missing json on JSON full-object unpack), logs a WARNING, and returns a SearchResults list subclass exposing dropped_count and complete so callers can spot shortened result sets without breaking list usage. Plain hash FilterQuery/TextQuery still passes through id-only rows to avoid dropping legitimate sparse results.

Extensions add defense-in-depth: semantic cache skips invalid CacheHits (no TTL refresh), message history skips bad ChatMessage rows, semantic router drops incomplete aggregate rows. Docs document the behavior and completeness signals; tests cover skip branches, over-skip guards, and extension parsers.

^{Reviewed by Cursor Bugbot for commit c554a07. Bugbot is set up for automated code reviews on this repo. Configure here.}