Skip to content

test(review): add reviewer evals and the 21k expression check - #28

Open
zfarrell wants to merge 1 commit into
mainfrom
feat/reviewer-evals
Open

test(review): add reviewer evals and the 21k expression check#28
zfarrell wants to merge 1 commit into
mainfrom
feat/reviewer-evals

Conversation

@zfarrell

@zfarrell zfarrell commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Adds behavioural evals for the reviewer, plus a static check for the limit that caused today's second outage.

Evals. Eight scenarios in tests/eval/ each become a real pull request in hotdata-dev/pr-review-eval (new, private), with a copy of the reviewer workflow committed into the head branch — so the graded thing is the real file on the real pull_request path, taken from the PR under review rather than from main. That is the only way a prompt change gets pre-merge exposure, since the workflow resolves the prompt from main. Grading reads PR state through the API rather than the action's execution log, which is what keeps the reviewer swappable: reviewer_workflow and reviewer_login are the only things tying it to Claude. Reporting-only, passing on a rate out of repeats.

21k expression check. A block scalar containing an interpolation compiles to one format(...) expression capped at 21,000 characters; over it, Actions refuses to start the workflow — zero jobs, no check, org blocked. Verified against both commits: fails at 24,860 on 1f12c05, passes at 20,545 on 8b04393. Current main has 455 characters of headroom, so it warns from 90%, and separately warns about a long block with no expression yet.

Complementary to #27 rather than overlapping: #27 catches startup failure dynamically for any cause, this warns before you spend the commit. No file overlap.

Two things deferred to avoid colliding with #27, which is open on the same two workflow files: the verdict label in TOOL_USAGE_JQ, and moving the two interpolations in the context step into env: (which would remove the 21k cap from that block entirely and let #26's hardening re-land).

@zfarrell
zfarrell requested a review from a team as a code owner August 5, 2026 18:58
@zfarrell
zfarrell requested review from anoop-narang and removed request for a team August 5, 2026 18:58
@zfarrell
zfarrell force-pushed the feat/reviewer-evals branch from 840fb79 to a6101f4 Compare August 5, 2026 19:19
@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown

Automated review unavailable (Claude step failed). Please review manually.

@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown

Reviewer eval

No results were produced. Every scenario job failed before grading — treat this as
an eval failure, not as a passing reviewer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant