Skip to content

Fuse discovery and evaluation in the proxy reduce tactic - #826

Merged
frankmcsherry merged 3 commits into
master-nextfrom
proxy-reduce-fused-only
Aug 11, 2026
Merged

Fuse discovery and evaluation in the proxy reduce tactic#826
frankmcsherry merged 3 commits into
master-nextfrom
proxy-reduce-fused-only

Conversation

@frankmcsherry

Copy link
Copy Markdown
Member

The proxy reduce tactic enumerated each key's interesting times and then walked them a second time to evaluate.
The conventional reduce does neither: it runs one ascending pass and evaluates as it discovers.
It can because discovery never looks backwards — every synthesized time is next_time.join(t) for a t not at or below next_time, so it is strictly greater, and new work only ever lands ahead of the sweep.

Sweep is that pass, cut where the conventional operator calls user logic.
Each next_crossing runs to the next in-interval time needing evaluation and suspends with the buffers positioned to read the accumulations; the caller evaluates and returns the corrections through commit; the next call resumes.
The tactic runs every live key to its next suspension and crosses them together, which is what a batched backend wants, and no key enumerates its times first.

It is written as a state machine rather than one body, so that each clause of formal/Differential/RoundCoverage.lean's round_coverage sits in one place: frontier finds the position, absorb steps the sources and decides whether the time is reached, close joins forward, settle recomputes the meet.

The second commit deletes what the sweep replaced.
discover_times lost its last caller, and KeyView, DiscoverScratch and TimeHistory existed only to serve it.
operators/common.rs goes with them: every live item in it had exactly one caller and all three callers were in int_proxy, so Sweep, tile_descriptions and update_meet move beside the tactic in int_proxy/reduce.rs and bilinear_wave into int_proxy/join.rs.
Net for the branch: -57 lines.

Measurements

tests/int_proxy_bench.rs, proxy tactic against the cursor tactic over identical ChunkSpine storage.
Both sides built in one target directory, three interleaved runs per binary, with the cursor mode carried as a control (it matched to within 1.5% across binaries throughout).

workload cursor before after
churn 50k 50.1ms 30.7ms (0.61x) 28.1ms (0.56x)
churn 4M 4235ms 2614-2801ms (0.62-0.66x) 2419-2434ms (0.57x)
multimoment 10k 19.7ms 23.5ms (1.19x) 21.9ms (1.11x)
multimoment 1M 2293ms 2389ms (1.04x) 2171ms (0.95x)
wide 1M (String values) 2088ms 1909ms (0.91x) 1881ms (0.90x)
propagate (iterate, Product times) 2628us 3200us (1.22x) 3041us (1.16x)

On corgi, scc goes 4.84s to 4.47s with the vec control flat at 3.79-3.82s, so corgi/vec moves 1.27x to 1.17x.
reach is 293ms to 284ms and tour 188ms to 183ms, both near the noise floor.

A warning for anyone benchmarking the second commit on its own

Under the workspace release profile it appears to cost 8% on churn and multimoment.
That is an artifact of codegen-units, which [profile.release] leaves at the default 16 despite lto = true.
Rustc partitions codegen units by module, so folding common.rs into int_proxy/reduce.rs merges two partitions and the optimizer makes different choices.
Setting codegen-units = 1 makes the folded and unfolded arrangements indistinguishable, and both land at the faster number.
Code layout was ruled out separately, by permuting function order within an untouched file and remeasuring: that moves each arrangement by under 1% while the gap between them stays put.

Testing

cargo test -p differential-dataflow and -p interactive in debug, where debug_assert_sorted_bridge and the reduce_with_tactic contract checks are live.
That includes the ten int_proxy tests, which compare the proxy tactic against the mainline cursor reduce over the VecReduceBackend, and the twelve-program corgi-vs-vec differential gate.
Whole-workspace check under -D warnings.

🤖 Generated with Claude Code

frankmcsherry and others added 3 commits August 11, 2026 15:01
The tactic enumerated each key's interesting times and then walked them a second
time to evaluate. The conventional reduce does neither: it runs one ascending
pass and evaluates as it discovers. It can because discovery never looks
backwards — every synthesized time is `next_time.join(t)` for a `t` not at or
below `next_time`, so it is strictly greater, and new work only ever lands ahead
of the sweep.

`Sweep` is that pass, cut where the conventional operator calls user logic. Each
`next_crossing` runs to the next in-interval time needing evaluation and
suspends with the buffers positioned to read the accumulations; the caller
evaluates and returns the corrections through `commit`; the next call resumes.
The tactic runs every live key to its next suspension and crosses them together,
which is what a batched backend wants, and no key enumerates its times first.

Written as a state machine rather than one body: `frontier` finds the position,
`absorb` steps the sources and decides whether the time is reached, `close`
joins forward, `settle` recomputes the meet. One tick reports `Passed`,
`Pended`, `Crossing(t)` or `Done`. That puts each clause of
`RoundCoverage.lean`'s `round_coverage` in one place — the novel witness is the
whole of `absorb`'s return value, and it is also why `close` is split, going
against the novel times always but against the prior times only for a time that
already carries a witness.

`discover_times` and the per-key moment lists go with it; so does the second
ordering of every key's records.

Measured against the two-phase schedule, on the reference backend as fixed by
the commits below it:

  churn        0.64x -> 0.58x of the cursor tactic
  multimoment  1.22x -> 1.09x
  propagate    1.20x -> 1.13x

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`discover_times` lost its last caller when the fused sweep landed.
`KeyView`, `DiscoverScratch` and `TimeHistory` existed only to serve it.
The containment test that checked the sweep against it goes too: it was scaffolding for building the sweep, and the ten `int_proxy` tests and the corgi gate check the same property end to end against the conventional reduce.

`common.rs` said "Types and methods generally useful for differential computation".
Every live item in it had exactly one caller, and all three callers were in `int_proxy`.
Naming it that way claimed a generality it did not have, hid the tactic's size from anyone reading the tactic, and is how four items died unnoticed: a module named for a role nobody audits accumulates things.

So `Sweep`, its `update_meet` helper and `tile_descriptions` move to the tail of `int_proxy/reduce.rs`, and `bilinear_wave` to `int_proxy/join.rs`, in each case beside their only caller.
None of them is `pub` any more; none was ever part of an interface.
The sweep and the tactic that drives it now sit in one file, which is where they belong: the boundary between them is meant to dissolve as more of the work fuses.

A warning for anyone who benchmarks this commit.
Under the workspace release profile it appears to cost 8% on churn and multimoment, and that is an artifact of `codegen-units`, which the profile leaves at the default 16 despite `lto = true`.
Rustc partitions codegen units by module, so folding two modules into one merges two partitions and the optimizer makes different choices; setting `codegen-units = 1` makes this commit and its module-per-item alternative indistinguishable, and both land at the faster number.
Layout was ruled out separately, by permuting function order within an untouched file and remeasuring.

The branch was +301 lines against master-next and is now -67.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The key-slot loop tested its terminating condition with a `let ... else { break }` where a `while let` says the same thing, and the step-live-keys loop ranged over indices into `live` only to index `live`, shadowing the loop variable with the value it read.

`join_key`'s `too_many_arguments` is left alone: it predates this branch.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@frankmcsherry
frankmcsherry merged commit 186263d into master-next Aug 11, 2026
6 checks passed
@frankmcsherry
frankmcsherry deleted the proxy-reduce-fused-only branch August 11, 2026 20:52
frankmcsherry added a commit that referenced this pull request Aug 11, 2026
A DDIR variant column becomes a corgi `Sum`: a tag column plus one lane per
constructor.  Nothing carries the declared universe, so each shape derivation
infers the arity from what it sees -- `infer_shape_cols` from the batch's tags,
`infer_term_shape` from `Inject(tag, _)` as `tag + 1`.  A collection holding only
`Rare(_)` therefore gets one lane and one holding only `Common(_)` gets two, for
the same DDIR type.  Concatenating them into one arrangement panicked corgi while
vec rendered it fine.

frankmcsherry/WIP#10 fixed that where it lives: `gather_lanes` took the output Sum's
arity from `sums[0]` while gathering out tags from every source, dropping lanes the
tags still named.  This bumps the pin onto it and keeps the two programs that found
it as gate coverage -- one on the row-wise fallback, one on the compiled path, since
both derivations under-approximate.

Earlier revisions of this PR reconciled arities DDIR-side in `chunk.rs` at the four
sites that read two independently-inferred columns.  That is all gone: the corgi fix
subsumes it, and dropping the per-merge shape walk is worth ~4% on scc steady-state
(4.54s vs 4.72s, median of 3) now that #824/#826 have taken the baseline down to
where it shows.

Gate 14/14 debug and release; both new programs panic without the pin bump.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168NJWzHwrLGW2RXToGooYP
frankmcsherry added a commit that referenced this pull request Aug 12, 2026
A DDIR variant column becomes a corgi `Sum`: a tag column plus one lane per
constructor.  Nothing carries the declared universe, so each shape derivation
infers the arity from what it sees -- `infer_shape_cols` from the batch's tags,
`infer_term_shape` from `Inject(tag, _)` as `tag + 1`.  A collection holding only
`Rare(_)` therefore gets one lane and one holding only `Common(_)` gets two, for
the same DDIR type.  Concatenating them into one arrangement panicked corgi while
vec rendered it fine.

frankmcsherry/WIP#10 fixed that where it lives: `gather_lanes` took the output Sum's
arity from `sums[0]` while gathering out tags from every source, dropping lanes the
tags still named.  This bumps the pin onto it and keeps the two programs that found
it as gate coverage -- one on the row-wise fallback, one on the compiled path, since
both derivations under-approximate.

Earlier revisions of this PR reconciled arities DDIR-side in `chunk.rs` at the four
sites that read two independently-inferred columns.  That is all gone: the corgi fix
subsumes it, and dropping the per-merge shape walk is worth ~4% on scc steady-state
(4.54s vs 4.72s, median of 3) now that #824/#826 have taken the baseline down to
where it shows.

Gate 14/14 debug and release; both new programs panic without the pin bump.


Claude-Session: https://claude.ai/code/session_0168NJWzHwrLGW2RXToGooYP

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant