perf: adopt fat-LTO release profile and a release-fast test profile

[MegaRedHand]

🗒️ Description / Motivation

We had no [profile] section at all, so release builds ran on cargo's defaults:
16 codegen units, no LTO. The optimizer never inlined across crate
boundaries, and our hot paths are spread across workspace crates and their
dependencies (SSZ encode/decode, hash_tree_root, fork choice traversal) — the
shape that loses most to per-crate codegen units.

[profile.release]          # ships the binary
opt-level = 3
lto = "fat"
codegen-units = 1

[profile.release-fast]     # runs the tests
inherits = "release"
lto = false
codegen-units = 16
debug = "line-tables-only"
incremental = true

Two deliberate divergences from ethrex,
whose profiles this started as a copy of:

1. lto = "fat", not thin. Measured, see below.
2. release-fast runs the whole test suite, where ethrex applies it only to
   its ef_tests Makefiles and runs everything else on the dev profile. We
   can't: signature verification/aggregation stack-overflows without
   release-grade opt-level, which is why make test used --release to begin
   with.

ethrex's other profiles are not ported: [profile.dev] debug = 2 is already
cargo's default, and release-with-debug / release-with-debug-assertions have
no user here.

What Changed

File	Change
Cargo.toml	added [profile.release] (fat LTO, 1 CGU) and [profile.release-fast]
Makefile	test target: --release → --profile release-fast
CLAUDE.md	documented the test profile and where its artifacts land
.github/PULL_REQUEST_TEMPLATE.md	checklist now names make test

No source changes.

Correctness / Behavior Guarantees

• No behavioral change beyond optimization and code layout.
• debug-assertions / overflow-checks stay at release defaults (off) in both
  profiles, so tests keep the assertion semantics they had under --release.
  release-fast inherits opt-level = 3, the property the stack-overflow
  avoidance depends on.
• The fat-LTO release build was exercised beyond compiling: it ran 8 nodes on the
  eth-4 devnet for over 30 minutes, finalizing normally, heads in sync.

Measured: latency, on an 8-node devnet

Method: images built from the same commit differing only in [profile.release],
then run simultaneously on 4 nodes each so both arms see identical chain
conditions (a sequential before/after would confound the profile with chain
load — an earlier crossover attempt of mine did exactly that and had to be
thrown out). Aggregator node excluded; its extra load isn't what's being
compared. Negative = lower latency.

metric	default	thin	fat	thin vs default	fat vs thin	fat vs default
state transition	8.998 ms	8.961 ms	8.771 ms	−0.41%	−2.12%	−2.52%
block processing	0.404 ms	0.355 ms	0.361 ms	−12.13%	+1.59%	−10.73%
attestation processing	16.06 µs	13.32 µs	13.48 µs	−17.04%	+1.20%	−16.05%
block build	946.9 ms	941.1 ms	931.9 ms	−0.61%	−0.98%	−1.58%
payload aggregation	8.930 ms	8.927 ms	8.822 ms	−0.03%	−1.18%	−1.21%

The two LTO steps buy different things:

• default → thin buys cross-crate inlining: block processing −12%,
  attestation processing −17%. Total state transition barely moves, since those
  sub-millisecond stages are a small slice of it.
• thin → fat buys total state transition (−2.12%) and block build (−0.98%),
  while block processing and attestation processing land slightly the wrong way
  (+1.6%, +1.2%) — both inside noise at 0.2σ and 0.5σ.

On the fat-vs-thin window, every fat node beat every thin node, no overlap:

state transition   thin: 8.997  9.026  9.036  9.018   fat: 8.791  8.822  8.855  8.875
block build        thin: 945.1  946.1  943.9  950.0   fat: 939.0  936.4  938.8  935.3

Confidence differs sharply by leg, so both are worth stating:

leg	separates cleanly	inside noise
thin vs default (30 min)	block processing 3.7σ, attestation ~8σ, block build 5.5σ	state transition 0.9σ
fat vs thin (30 min)	state transition 7.1σ, block build 3.6σ	block processing 0.2σ, attestation 0.5σ, payload agg 1.2σ

fat vs default is composed from the two 30-minute measurements, not measured
directly. A direct 5-minute fat-vs-default window gave larger numbers
(−5.30% / −21.40% / −19.10%) but its baselines ran 1–5% high, so I trust the
composed column and report the short one only as a sanity check.

Heavy crypto specifically

leanVM's XMSS aggregation benchmark at the rev we pin, its profiles replaced by
ours, paired over 5 alternating rounds on a quiesced eth-4 (devnet paused):

profile	proof time	95% CI
default	5.0905 s	±0.0171
thin LTO	5.0642 s	±0.0104

−0.52%, significant (all 5 rounds agree). Small, as expected: that code is
intra-crate, heavily monomorphized numeric work with little cross-crate inlining
for LTO to find. It rules out a regression on the proving path, which is what it
was run for.

Measured: build cost

Whole-workspace binary, clean builds, isolated CARGO_TARGET_DIR, 11-core
M-series (thin measured; the default→LTO delta is the point):

	wall	CPU (user+sys)	binary
default release	315.9 s	1714 s	30.6 MiB
thin LTO, 1 CGU	377.8 s	1516 s	26.6 MiB
Δ	+20%	−12%	−13%

Wall grows because one codegen unit serializes per-crate codegen even as total
CPU drops. Docker builds on the 16-core eth-4 host: fat 284 s / 205 MB against
thin 290 s / 209 MB, i.e. fat costs nothing measurable relative to thin.

Test builds, which is why release-fast exists — rebuild after one real one-file
edit, cargo test --workspace --no-run, both profile dirs pre-warmed:

	wall
--release	88.0 s
--profile release-fast	4.6 s

Without release-fast, this PR would make every test rebuild pay a full LTO
relink. Costs: cold build of that profile dir is 461.9 s and target/release-fast
holds ~5.3 GiB against target/release's 1.6 GiB, so local disk grows for anyone
who both builds --release and runs tests.

CI note: the test job's Swatinem/rust-cache sets CARGO_INCREMENTAL=0,
which overrides incremental = true (verified — no -C incremental= reaches
rustc under that env var, and the action strips incremental artifacts before
caching). In CI the release-fast win is lto = false plus parallel codegen.

Risks / open items

• Fat LTO's peak link memory is untested on 16 GB CI runners. CI never builds
  this profile today (cargo check for lint, release-fast for tests), but the
  Docker publish workflow does.
• Build-time parity between fat and thin is one measurement per arm on one host.
• Branch is still named ...thin-lto from before the fat switch; the profile is
  fat.

Tests Added / Run

No tests added (build-configuration change).

• make fmt — clean
• make lint (cargo clippy --workspace --all-targets -- -D warnings) — clean
• make test under release-fast — all passing: forkchoice spectests 122, stf
  spectests 46, ssz spectests 119, signature spectests 3, plus all unit tests;
  0 failures. Also green under --release before the profile switch.
• Fat release build validated by docker build on eth-4 and by running the
  resulting image on 8 devnet nodes for 30+ minutes.

Related Issues / PRs

• None.

✅ Verification Checklist

• Ran make fmt — clean
• Ran make lint (clippy with -D warnings) — clean
• Ran make test (cargo test --workspace --profile release-fast) — all passing

[github-actions]

🤖 Claude Code Review

This is a small, self-contained build-configuration change. Full review below.

Summary

Cargo.toml adds a [profile.release] section enabling opt-level = 3, lto = "thin", codegen-units = 1. No source code changes.

Review

Correctness: No functional risk — this is a compiler/linker configuration change only. opt-level = 3 is already cargo's release default, so the only substantive changes are enabling thin LTO and collapsing to a single codegen unit. debug-assertions/overflow-checks are untouched, as the PR description notes, so no consensus-critical checks are silently disabled.

Consensus-layer impact: None. No logic in fork choice, state transition, attestation processing, or signature handling is touched.

Performance: Reasonable trade-off and well-justified — cross-crate inlining plausibly helps here since the hot paths (SSZ encode/decode, hash_tree_root, fork choice traversal) do span crate boundaries in this workspace. The PR is honest that:

• Build time increases (~20% wall time in the author's measurement) despite lower total CPU time, because CGU=1 serializes codegen.
• Runtime improvement is not measured — the claim is plausible but unverified. Given this is a "perf:" commit, it would be stronger to have at least one devnet A/B data point (e.g., lean_state_transition_time_seconds) before merging, as the author themselves suggests as an option. This isn't a blocker given the low risk, but worth doing as a fast follow.

CI/build implications to double check (not visible in this diff, so flagging for the reviewer to verify, not a code issue):

• make test runs cargo test --workspace --release, so CI will now pay the full LTO cost on every cold-cache run. Confirm CI timeout budgets have headroom for the ~20% increase, especially combined with Docker image builds (make docker-build uses --profile release too).
• Thin LTO with codegen-units = 1 also increases peak memory during linking, which can matter on constrained CI runners — worth a sanity check on the CI runner's available memory if not already verified.

Rust idiom: Profile section placement and comment are clear and appropriately explain the why (cross-crate inlining rationale) rather than just restating the what. No issues.

Verdict

Low-risk, well-documented change with clear rationale and honest trade-off disclosure. No correctness, security, or consensus-logic concerns. Only suggestion is to follow up with the mentioned devnet A/B to confirm the runtime win materializes, and to confirm CI time/memory budgets tolerate the slower cold build.

---

Automated review by Claude (Anthropic) · sonnet · custom prompt

[greptile-apps]

Greptile Summary

Configures optimized release builds in the workspace root.

• Enables optimization level 3 and Thin LTO.
• Restricts release builds to one codegen unit.
• Documents the intended runtime-performance versus build-time trade-off.

Confidence Score: 5/5

The PR appears safe to merge with no actionable correctness or security issues identified.

The changed Cargo release-profile values are valid, preserve the existing release-mode correctness defaults, and affect optimization and build performance rather than application contracts.

Important Files Changed

Filename	Overview
Cargo.toml	Adds valid workspace-wide Cargo release-profile settings for Thin LTO and single-unit code generation; no correctness issue identified.

_{Reviews (1): Last reviewed commit: "perf: enable thin LTO and single codegen..." | Re-trigger Greptile}

[github-actions]

🤖 Kimi Code Review

Cargo.toml (lines 29-37)

This change is appropriate for a consensus client where runtime performance directly affects network viability. The configuration correctly prioritizes cross-crate inlining for SSZ serialization and fork-choice hot paths.

Observations:

1. Build time trade-off: codegen-units = 1 significantly increases compile times (often 2-3x for large workspaces). For a consensus client, this is usually acceptable for release builds, but ensure CI pipelines account for the longer build duration. Consider documenting this in CONTRIBUTING.md so developers know to expect slower --release builds.

2. Redundancy: opt-level = 3 is the default for release profiles, but keeping it explicit documents the performance priority. This is fine.

3. Thin LTO choice: lto = "thin" strikes a reasonable balance between the performance benefits of full LTO and build time. For a maximum-performance consensus node, you might eventually want lto = "fat" (full LTO), but thin LTO is the correct conservative choice while you measure the impact.

Recommendation:
Consider adding a profile.release-with-debug for production debugging scenarios (e.g., debug = 1 or debug = 2 with line tables only) to make profiling and post-mortem analysis of consensus failures tractable without rebuilding:

[profile.release-with-debug]
inherits = "release"
debug = 1  # Line tables only, minimal binary size impact
strip = false

Verdict: No blocking issues. The change is correct and appropriate for the domain.

---

Automated review by Kimi (Moonshot AI) · kimi-k2.5 · custom prompt

[github-actions]

🤖 Codex Code Review

1. Low: Cargo.toml makes thin LTO plus codegen-units = 1 the default for every release build across the workspace. That is not a protocol-correctness risk, but it is an operational one: both the Docker path (Dockerfile) and Shadow path (Makefile, shadow/build.sh) build in release, so this change will increase build latency and peak memory everywhere, including the cached cargo chef cook step and simulator builds. On constrained CI/builders this can turn into OOMs or materially slower deploy pipelines. I’d want either benchmark/build-resource numbers in the PR description or for this to move to a dedicated opt-in profile instead of globally changing release.

No correctness, safety, or consensus-layer findings in the actual client logic: the PR only changes Cargo release-profile settings and does not touch fork choice, attestation handling, state transition, XMSS, or SSZ code.

I couldn’t run Cargo verification in this environment because rustup attempted to write outside the workspace, so this review is based on static inspection of the diff and build entry points.

---

Automated review by OpenAI Codex · gpt-5.4 · custom prompt

[Sahilgill24]

Hey @MegaRedHand , I think lto set to fat has the best runtime improvement