test(review): add reviewer evals and the 21k expression check - #28
Open
zfarrell wants to merge 1 commit into
Open
test(review): add reviewer evals and the 21k expression check#28zfarrell wants to merge 1 commit into
zfarrell wants to merge 1 commit into
Conversation
zfarrell
force-pushed
the
feat/reviewer-evals
branch
from
August 5, 2026 19:19
840fb79 to
a6101f4
Compare
|
Automated review unavailable (Claude step failed). Please review manually. |
Reviewer evalNo results were produced. Every scenario job failed before grading — treat this as |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds behavioural evals for the reviewer, plus a static check for the limit that caused today's second outage.
Evals. Eight scenarios in
tests/eval/each become a real pull request inhotdata-dev/pr-review-eval(new, private), with a copy of the reviewer workflow committed into the head branch — so the graded thing is the real file on the realpull_requestpath, taken from the PR under review rather than frommain. That is the only way a prompt change gets pre-merge exposure, since the workflow resolves the prompt frommain. Grading reads PR state through the API rather than the action's execution log, which is what keeps the reviewer swappable:reviewer_workflowandreviewer_loginare the only things tying it to Claude. Reporting-only, passing on a rate out of repeats.21k expression check. A block scalar containing an interpolation compiles to one
format(...)expression capped at 21,000 characters; over it, Actions refuses to start the workflow — zero jobs, no check, org blocked. Verified against both commits: fails at 24,860 on 1f12c05, passes at 20,545 on 8b04393. Currentmainhas 455 characters of headroom, so it warns from 90%, and separately warns about a long block with no expression yet.Complementary to #27 rather than overlapping: #27 catches startup failure dynamically for any cause, this warns before you spend the commit. No file overlap.
Two things deferred to avoid colliding with #27, which is open on the same two workflow files: the verdict label in
TOOL_USAGE_JQ, and moving the two interpolations in the context step intoenv:(which would remove the 21k cap from that block entirely and let #26's hardening re-land).