Fuse discovery and evaluation in the proxy reduce tactic - #826
Merged
Conversation
The tactic enumerated each key's interesting times and then walked them a second time to evaluate. The conventional reduce does neither: it runs one ascending pass and evaluates as it discovers. It can because discovery never looks backwards — every synthesized time is `next_time.join(t)` for a `t` not at or below `next_time`, so it is strictly greater, and new work only ever lands ahead of the sweep. `Sweep` is that pass, cut where the conventional operator calls user logic. Each `next_crossing` runs to the next in-interval time needing evaluation and suspends with the buffers positioned to read the accumulations; the caller evaluates and returns the corrections through `commit`; the next call resumes. The tactic runs every live key to its next suspension and crosses them together, which is what a batched backend wants, and no key enumerates its times first. Written as a state machine rather than one body: `frontier` finds the position, `absorb` steps the sources and decides whether the time is reached, `close` joins forward, `settle` recomputes the meet. One tick reports `Passed`, `Pended`, `Crossing(t)` or `Done`. That puts each clause of `RoundCoverage.lean`'s `round_coverage` in one place — the novel witness is the whole of `absorb`'s return value, and it is also why `close` is split, going against the novel times always but against the prior times only for a time that already carries a witness. `discover_times` and the per-key moment lists go with it; so does the second ordering of every key's records. Measured against the two-phase schedule, on the reference backend as fixed by the commits below it: churn 0.64x -> 0.58x of the cursor tactic multimoment 1.22x -> 1.09x propagate 1.20x -> 1.13x Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`discover_times` lost its last caller when the fused sweep landed. `KeyView`, `DiscoverScratch` and `TimeHistory` existed only to serve it. The containment test that checked the sweep against it goes too: it was scaffolding for building the sweep, and the ten `int_proxy` tests and the corgi gate check the same property end to end against the conventional reduce. `common.rs` said "Types and methods generally useful for differential computation". Every live item in it had exactly one caller, and all three callers were in `int_proxy`. Naming it that way claimed a generality it did not have, hid the tactic's size from anyone reading the tactic, and is how four items died unnoticed: a module named for a role nobody audits accumulates things. So `Sweep`, its `update_meet` helper and `tile_descriptions` move to the tail of `int_proxy/reduce.rs`, and `bilinear_wave` to `int_proxy/join.rs`, in each case beside their only caller. None of them is `pub` any more; none was ever part of an interface. The sweep and the tactic that drives it now sit in one file, which is where they belong: the boundary between them is meant to dissolve as more of the work fuses. A warning for anyone who benchmarks this commit. Under the workspace release profile it appears to cost 8% on churn and multimoment, and that is an artifact of `codegen-units`, which the profile leaves at the default 16 despite `lto = true`. Rustc partitions codegen units by module, so folding two modules into one merges two partitions and the optimizer makes different choices; setting `codegen-units = 1` makes this commit and its module-per-item alternative indistinguishable, and both land at the faster number. Layout was ruled out separately, by permuting function order within an untouched file and remeasuring. The branch was +301 lines against master-next and is now -67. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The key-slot loop tested its terminating condition with a `let ... else { break }` where a `while let` says the same thing, and the step-live-keys loop ranged over indices into `live` only to index `live`, shadowing the loop variable with the value it read.
`join_key`'s `too_many_arguments` is left alone: it predates this branch.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
frankmcsherry
added a commit
that referenced
this pull request
Aug 11, 2026
A DDIR variant column becomes a corgi `Sum`: a tag column plus one lane per constructor. Nothing carries the declared universe, so each shape derivation infers the arity from what it sees -- `infer_shape_cols` from the batch's tags, `infer_term_shape` from `Inject(tag, _)` as `tag + 1`. A collection holding only `Rare(_)` therefore gets one lane and one holding only `Common(_)` gets two, for the same DDIR type. Concatenating them into one arrangement panicked corgi while vec rendered it fine. frankmcsherry/WIP#10 fixed that where it lives: `gather_lanes` took the output Sum's arity from `sums[0]` while gathering out tags from every source, dropping lanes the tags still named. This bumps the pin onto it and keeps the two programs that found it as gate coverage -- one on the row-wise fallback, one on the compiled path, since both derivations under-approximate. Earlier revisions of this PR reconciled arities DDIR-side in `chunk.rs` at the four sites that read two independently-inferred columns. That is all gone: the corgi fix subsumes it, and dropping the per-merge shape walk is worth ~4% on scc steady-state (4.54s vs 4.72s, median of 3) now that #824/#826 have taken the baseline down to where it shows. Gate 14/14 debug and release; both new programs panic without the pin bump. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0168NJWzHwrLGW2RXToGooYP
frankmcsherry
added a commit
that referenced
this pull request
Aug 12, 2026
A DDIR variant column becomes a corgi `Sum`: a tag column plus one lane per constructor. Nothing carries the declared universe, so each shape derivation infers the arity from what it sees -- `infer_shape_cols` from the batch's tags, `infer_term_shape` from `Inject(tag, _)` as `tag + 1`. A collection holding only `Rare(_)` therefore gets one lane and one holding only `Common(_)` gets two, for the same DDIR type. Concatenating them into one arrangement panicked corgi while vec rendered it fine. frankmcsherry/WIP#10 fixed that where it lives: `gather_lanes` took the output Sum's arity from `sums[0]` while gathering out tags from every source, dropping lanes the tags still named. This bumps the pin onto it and keeps the two programs that found it as gate coverage -- one on the row-wise fallback, one on the compiled path, since both derivations under-approximate. Earlier revisions of this PR reconciled arities DDIR-side in `chunk.rs` at the four sites that read two independently-inferred columns. That is all gone: the corgi fix subsumes it, and dropping the per-merge shape walk is worth ~4% on scc steady-state (4.54s vs 4.72s, median of 3) now that #824/#826 have taken the baseline down to where it shows. Gate 14/14 debug and release; both new programs panic without the pin bump. Claude-Session: https://claude.ai/code/session_0168NJWzHwrLGW2RXToGooYP Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The proxy reduce tactic enumerated each key's interesting times and then walked them a second time to evaluate.
The conventional reduce does neither: it runs one ascending pass and evaluates as it discovers.
It can because discovery never looks backwards — every synthesized time is
next_time.join(t)for atnot at or belownext_time, so it is strictly greater, and new work only ever lands ahead of the sweep.Sweepis that pass, cut where the conventional operator calls user logic.Each
next_crossingruns to the next in-interval time needing evaluation and suspends with the buffers positioned to read the accumulations; the caller evaluates and returns the corrections throughcommit; the next call resumes.The tactic runs every live key to its next suspension and crosses them together, which is what a batched backend wants, and no key enumerates its times first.
It is written as a state machine rather than one body, so that each clause of
formal/Differential/RoundCoverage.lean'sround_coveragesits in one place:frontierfinds the position,absorbsteps the sources and decides whether the time is reached,closejoins forward,settlerecomputes the meet.The second commit deletes what the sweep replaced.
discover_timeslost its last caller, andKeyView,DiscoverScratchandTimeHistoryexisted only to serve it.operators/common.rsgoes with them: every live item in it had exactly one caller and all three callers were inint_proxy, soSweep,tile_descriptionsandupdate_meetmove beside the tactic inint_proxy/reduce.rsandbilinear_waveintoint_proxy/join.rs.Net for the branch: -57 lines.
Measurements
tests/int_proxy_bench.rs, proxy tactic against the cursor tactic over identicalChunkSpinestorage.Both sides built in one target directory, three interleaved runs per binary, with the cursor mode carried as a control (it matched to within 1.5% across binaries throughout).
Producttimes)On corgi,
sccgoes 4.84s to 4.47s with theveccontrol flat at 3.79-3.82s, so corgi/vec moves 1.27x to 1.17x.reachis 293ms to 284ms andtour188ms to 183ms, both near the noise floor.A warning for anyone benchmarking the second commit on its own
Under the workspace release profile it appears to cost 8% on churn and multimoment.
That is an artifact of
codegen-units, which[profile.release]leaves at the default 16 despitelto = true.Rustc partitions codegen units by module, so folding
common.rsintoint_proxy/reduce.rsmerges two partitions and the optimizer makes different choices.Setting
codegen-units = 1makes the folded and unfolded arrangements indistinguishable, and both land at the faster number.Code layout was ruled out separately, by permuting function order within an untouched file and remeasuring: that moves each arrangement by under 1% while the gap between them stays put.
Testing
cargo test -p differential-dataflowand-p interactivein debug, wheredebug_assert_sorted_bridgeand thereduce_with_tacticcontract checks are live.That includes the ten
int_proxytests, which compare the proxy tactic against the mainline cursor reduce over theVecReduceBackend, and the twelve-program corgi-vs-vecdifferential gate.Whole-workspace check under
-D warnings.🤖 Generated with Claude Code