Skip to content

perf: specialize single-column hash-join key equality check - #24067

Closed
shehab-ali wants to merge 5 commits into
apache:mainfrom
shehab-ali:shehab/shehab/hashjoin-filter-keys-single-column
Closed

perf: specialize single-column hash-join key equality check#24067
shehab-ali wants to merge 5 commits into
apache:mainfrom
shehab-ali:shehab/shehab/hashjoin-filter-keys-single-column

Conversation

@shehab-ali

@shehab-ali shehab-ali commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

Hash-join key-equality verification (equal_rows_arr) checks every candidate row pair produced by a hash-bucket match. The current implementation calls JoinKeyComparator::is_equal per pair, which for each pair:

  1. invokes a boxed DynComparator closure (an un-inlinable indirect call)
  2. computes a full three-way Ordering, only to discard everything except equal / not-equal.

For single-column, high-fanout joins with near-100% match rates (long string keys are the worst case), that per-pair overhead dominates the join and measurably regresses throughput relative to the previous vectorized approach.

Benchmarking against main on the hj SQL benchmark suite (TPC-H SF10, interleaved runs to cancel drift) showed this cost concentrated in single-column-key queries:

  • Q23 (a high-fanout ~28-char string key) ran ~5% slower, with a cluster of other single/low-column joins 6–22% slower.

What changes are included in this PR?

  • Add equal_rows_single_col: a specialized fast path for single-column join keys. It downcasts the key array once and runs a monomorphized == loop over the candidate pairs, eliminating the per-pair boxed-closure dispatch and the Ordering computation.
  • Covered types: integers, Decimal128, dates, timestamps, boolean, and all Utf8/Binary variants (incl. Utf8View/BinaryView).
  • Multi-column keys and unspecialized types fall through unchanged to the existing JoinKeyComparator path. Floats (Float16/32/64) are intentionally excluded so their -0.0/NaN equality semantics remain on the exact same code path as before.
  • Null handling is preserved: NullEqualsNothing -> both-null is unequal; NullEqualsNull -> both-null is equal.

@github-actions github-actions Bot added the physical-plan Changes to the physical-plan crate label Aug 3, 2026
@codecov-commenter

codecov-commenter commented Aug 3, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.61111% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.90%. Comparing base (1955d5a) to head (b2eef82).
⚠️ Report is 63 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/physical-plan/src/joins/utils.rs 98.61% 2 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #24067      +/-   ##
==========================================
+ Coverage   80.75%   80.90%   +0.14%     
==========================================
  Files        1096     1102       +6     
  Lines      373582   376357    +2775     
  Branches   373582   376357    +2775     
==========================================
+ Hits       301686   304482    +2796     
+ Misses      53893    53769     -124     
- Partials    18003    18106     +103     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@adriangbot

Copy link
Copy Markdown

@shehab-ali shehab-ali closed this Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants