Skip to content

feat(review): frontload PR context into the prompt - #22

Merged
zfarrell merged 10 commits into
mainfrom
feat/frontload-review-context
Aug 5, 2026
Merged

feat(review): frontload PR context into the prompt#22
zfarrell merged 10 commits into
mainfrom
feat/frontload-review-context

Conversation

@zfarrell

@zfarrell zfarrell commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

A week of tool-usage artifacts showed the reviewer spending 19.5 Bash calls and 5.2 permission denials per run fetching the diff, CI status, its own last-reviewed SHA, and the PR conversation — none of which it can reach under the allowlist and fetch-depth: 1. All four now come frontloaded, rg is allowed, and the artifact carries closed-vocabulary command labels so the next pass can read artifacts instead of replaying reviews.

Also fixes two things the frontloading needed: actions: read/issues: read on the job (both reads would have 403'd silently) and --allow-escape-sequences on the job-log fetch (without it gh writes zero bytes).

@zfarrell
zfarrell requested a review from a team as a code owner August 4, 2026 22:39
@zfarrell
zfarrell requested review from rohan-hotdata and removed request for a team August 4, 2026 22:39
Comment thread .github/workflows/claude-pr-review.yml Outdated
Comment thread .github/workflows/claude-pr-review.yml Outdated
Comment thread .github/workflows/claude-pr-review.yml Outdated
Comment thread tests/context-step-test.sh
claude[bot]
claude Bot previously approved these changes Aug 4, 2026
@zfarrell

zfarrell commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Codex review pass — 7 findings, all addressed in 6f3eaa4:

Injection (real). A PR body, diff hunk, or CI log containing </pr_context> ended the data block early, putting the rest outside the marked region where it reads as prompt. Both blocks are now sanitised, including the pre-existing <prior_review_comments> which had the same hole. Tested with a body that closes both tags and reopens one.

Artifact boundary. tool_calls[].name and denials[].name were copied straight from the transcript, so the "closed vocabulary" claim was only true of the command labels. Names are now bounded to a registry shape; anything else becomes unknown.

Null as a claim. A files or checks payload missing a field rendered +null/-null and / null — numbers and names the reviewer quotes back. Defaulted, with partial-payload tests.

Test strength. The containment assertion now covers names; the degradation loop asserts each endpoint's specific missing-data sentence rather than just exit 0; the diff assertions check the body, not only the heading.

Separately, running the step against this PR found a bug Codex could not see statically: gh pr diff also refuses a body containing escape sequences, and this PR's own job-log fixtures carry them — so the diff block silently degraded to "Could not read the diff". Both raw-text calls now pass --allow-escape-sequences, and the test stub refuses without it.

Note on validation: the review check here runs the workflow from main (org ruleset), not from this branch — confirmed in the run log at head c3c4137. So this change cannot be exercised by its own PR and goes live org-wide on merge. Pre-merge evidence is 102 local assertions plus dry-runs of the real step against dlthubworker#172, dlthubworker#168, and this PR.

Comment thread .github/workflows/claude-pr-review.yml Outdated
Comment thread tests/context-step-test.sh

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review

Blocking Issues

  • .github/workflows/claude-pr-review.yml:309gh pr diff does not accept --allow-escape-sequences. Verified against the gh on an ubuntu-latest runner: unknown flag: --allow-escape-sequences, exit 1. The flag belongs to gh api and is not persistent, so it does not carry to gh pr diff. With 2>/dev/null swallowing the error, the Full diff block silently degrades to Could not read the diff; run gh pr diff. on every run — the central block this PR adds never renders. The gh api uses at L245 and L285 are correct.

Action Required

  • Replace --allow-escape-sequences on L309 with --color never (or drop it), and confirm the block renders its body rather than the fallback sentence.
  • tests/context-step-test.sh:102 asserts the stub was called with that flag, so the suite is green on the broken invocation and will go red on the fix. Move require_escape_flag off the pr diff branch (keeping it on the two gh api branches) and update the comment at L179.

Comment thread .github/workflows/claude-pr-review.yml Outdated
claude[bot]
claude Bot previously approved these changes Aug 4, 2026

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prior cycle's blocking issue is resolved: fetch_raw tries --allow-escape-sequences and falls back to the bare call, so the raw fetches land on both gh generations. Confirmed on this runner that gh pr diff --help lists no such flag, which is the fallback path. The stub now rejects unknown flags per GH_VERSION and asserts the diff body reaches the context on 2.96 and 2.97 rather than asserting the flag. The compare probe, the quoted-span handling in compound, the prompt's missing-block line, and shell: bash are all in. One non-blocking note inline.

claude[bot]
claude Bot previously approved these changes Aug 4, 2026
@zfarrell

zfarrell commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

Codex branch review — 9 findings, all addressed in 90f372a and d3cffe6.

Delimiter neutralisation was incomplete. strip_block_tags matched four exact lowercase literals, so </pr_context >, </PR_CONTEXT> and < / pr_context foo="1"> all survived — and read as the same delimiter to a model. Now matched by shape via perl (BSD sed has no case-insensitive substitute flag, so a sed version would be GNU-only or twenty character classes). The injection test carries all those variants.

Two reads were rendering failures as claims. A failed /reviews became REVIEW CYCLE: 1 and a failed /pulls/{n}/comments became "No prior review comments." — indistinguishable from the truthful empty case, and on cycle 4 that makes the reviewer follow cycle-1 instructions and re-raise settled findings. Both now disclose themselves in a ## Context warnings block at the top of the context, where truncation can't remove it. The ::warning:: in the job log never reached the model.

threads was unbounded and outside the cap. Confirmed worse than described: 400 inline comments render 1.14 MB on their own. Comment bodies are now capped at 3,000 characters like the conversation block, and the two outputs share a 300 KB budget (100 KB threads + 200 KB context).

A single CI log line could evict the diff. LOG_WINDOW counts lines and a log line has no length limit, so one base64 dump near the first error marker consumed the budget before ## Full diff was written. Log excerpts are now byte-capped at 40 KB, through a cap_file helper that works on files rather than sed | head -c — that pipe is the SIGPIPE shape that already cost this step its error window once.

Test gaps: the truncation test only grew the last block, so "the diff survives" merely confirmed fixture ordering; it now grows an earlier block (a 900 KB single log line) and the threads block. The reviews/comments degradation rows were deliberately vacuous and now assert their disclosure sentences.

On the UTF-16 accounting: I did not verify that mechanism, so I did not size the caps against it — 300 KB total is far under any plausible limit either way, and the largest PR reviewed across the org in a week renders about 150 KB. Verified live against github-workflows#22 (112 KB), dlthubworker#168 (70 KB) and monopoly#1647 (93 KB).

143 assertions passing.

@zfarrell
zfarrell merged commit 036744e into main Aug 5, 2026
2 checks passed
@zfarrell
zfarrell deleted the feat/frontload-review-context branch August 5, 2026 01:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant