Skip to content

interactive: pin corgi past the gather_lanes arity fix, with gate coverage - #817

Merged
frankmcsherry merged 1 commit into
master-nextfrom
corgi-sum-arity
Aug 12, 2026
Merged

interactive: pin corgi past the gather_lanes arity fix, with gate coverage#817
frankmcsherry merged 1 commit into
master-nextfrom
corgi-sum-arity

Conversation

@frankmcsherry

@frankmcsherry frankmcsherry commented Aug 4, 2026

Copy link
Copy Markdown
Member

A DDIR variant column becomes a corgi Sum: a tag column plus one lane per constructor. Nothing carries the declared universe (Parser.cons holds it and discards it after desugaring), so each shape derivation infers the arity from whatever it sees:

  • infer_shape_cols (row-wise path) — from the batch's tags
  • infer_term_shape (compiled path) — from Inject(tag, _), as tag + 1

A collection holding only Rare(_) therefore gets Sum([Some]) and one holding only Common(_) gets Sum([None, Some])same DDIR type, two arities. Concatenate them into one arrangement and corgi panicked while vec rendered it fine.

frankmcsherry/WIP#10 fixed that where it lives: gather_lanes took the output arity from sums[0] while gathering out-tags from every source, dropping lanes the tags still named, so sum_from_prim indexed past its own (index out of bounds: the len is 1 but the index is 1).

This PR is now just the pin bump plus the two programs that found it — one on the row-wise fallback, one on the compiled path, since both derivations under-approximate and I'd initially claimed only the fallback did.

What changed since the first revision

Earlier revisions reconciled arities DDIR-side in chunk.rs, at the four sites that read two independently-inferred columns together. All of that is gone. The corgi fix subsumes it, and removing the per-merge shape walk is worth ~4% on scc steady-state — 4.54s vs 4.72s (median of 3). That cost was invisible when the baseline was 17.4s; #824 and #826 took it down far enough to show.

Checks

  • Gate 14/14 debug and release; both new programs panic without the pin bump.
  • Full interactive suite green.
  • Rebased on current master-next (no conflicts — nothing upstream touched these files).

🤖 Generated with Claude Code

https://claude.ai/code/session_0168NJWzHwrLGW2RXToGooYP

@frankmcsherry

Copy link
Copy Markdown
Member Author

Follow-up study found the root cause one layer down, and it is smaller than this PR.

The bug is corgi's, and it is one line. engine::gather_lanes takes the output Sum's arity from sums[0], while the gathered tag column names tags from every source:

let arity = sums[0].2.len();   // <- drops lanes the out tags still reference

A shorter source 0 therefore loses lanes the output still names, and sum_from_prim indexes past its own. That contradicts the ⊥ rule the same function already documents for None lanes — the arity simply wasn't reading it. Fixed in frankmcsherry/WIP#10 by taking the max over sources and reading lanes with get.

It subsumes this PR's DDIR-side reconciliation. With wip#10 patched in and all four chunk.rs sites reverted, the gate passes 14/14 in debug and release. So the choice is:

lines layer needs
this PR +74/−7 in DDIR, 4 sites reconciles shapes in DDIR (the P2 smell I flagged) nothing
wip#10 +1 real line in corgi fixes it where it is, for every caller a pin bump

My recommendation: merge wip#10, then reduce this PR to the two test programs plus the pin bump. That is strictly fewer lines and fewer concepts, and it removes a unify_sum_arity walk from merge, which is hot. Perf on the corgi fix is flat (scc 16.92s vs a 17.10s same-build-type control, median of 3).

I have also corrected a wrong claim in the fixture comment I pushed earlier. I wrote that hash was load-bearing — that only the row-wise fallback derives shape from data, so only it could skew. That is false. infer_term_shape gives Inject(tag, _) an arity of tag + 1, so the compiled path under-approximates the variant universe too, just from the term instead of the data. sum_skew_compiled.ddp drops the hash and panics identically. Both programs are now in the PR.

The cause underneath both is still open (charter S18): Parser.cons holds the declared variant universe and discards it after desugaring, so neither shape derivation ever sees the real arity. Carrying it into the plan would make every column agree by construction and need no reconciliation anywhere — but I sized the plumbing at ~45 mechanical touch points (parseVec<Stmt>, 10 call sites; lower_tree, 18; plus the recursive shape functions), for one usize. Worth doing on its own merits, not as a prerequisite for either fix here.

A DDIR variant column becomes a corgi `Sum`: a tag column plus one lane per
constructor.  Nothing carries the declared universe, so each shape derivation
infers the arity from what it sees -- `infer_shape_cols` from the batch's tags,
`infer_term_shape` from `Inject(tag, _)` as `tag + 1`.  A collection holding only
`Rare(_)` therefore gets one lane and one holding only `Common(_)` gets two, for
the same DDIR type.  Concatenating them into one arrangement panicked corgi while
vec rendered it fine.

frankmcsherry/WIP#10 fixed that where it lives: `gather_lanes` took the output Sum's
arity from `sums[0]` while gathering out tags from every source, dropping lanes the
tags still named.  This bumps the pin onto it and keeps the two programs that found
it as gate coverage -- one on the row-wise fallback, one on the compiled path, since
both derivations under-approximate.

Earlier revisions of this PR reconciled arities DDIR-side in `chunk.rs` at the four
sites that read two independently-inferred columns.  That is all gone: the corgi fix
subsumes it, and dropping the per-merge shape walk is worth ~4% on scc steady-state
(4.54s vs 4.72s, median of 3) now that #824/#826 have taken the baseline down to
where it shows.

Gate 14/14 debug and release; both new programs panic without the pin bump.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168NJWzHwrLGW2RXToGooYP
@frankmcsherry frankmcsherry changed the title Widen inferred Sum arities before the corgi chunker compares or gathers interactive: pin corgi past the gather_lanes arity fix, with gate coverage Aug 11, 2026
@frankmcsherry
frankmcsherry merged commit bc3166a into master-next Aug 12, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant