fix(review): measure the prompt budget in escaped bytes - #32
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds two changes to #31. The base branch is
fix/bound-review-prompt-size.Why the budget must count escaped bytes
#31 counts the raw bytes of the prompt. But the prompt does not go to the reviewer only as raw bytes.
The
action.ymlfile ofclaude-code-actionsetsALL_INPUTS: toJson(inputs)on the step that it runs. Thus the step carries the prompt two times:PROMPTALL_INPUTSThe escaped copy is always larger. Therefore the escaped copy always reaches the limit first. If you bound the raw string, the string that fails stays unbounded.
Pull request hotdata-dev/monopoly#1670 shows this. Its prompt was 123,401 raw bytes, which is inside the limit. The same prompt was 135,366 bytes after escaping. The review failed two times.
I ran the script of #31 against that same pull request:
The cap of #31 does not operate at all here. The context is near 115 KB, which is less than the budget of 121 KB. The prompt goes through with no change, and
execfails again.You cannot correct for this with a constant factor. Prose costs approximately 1.02x after escaping. A quote-dense JSON diff costs approximately 1.20x. The same 3,000 lines can be on one side of the limit or the other. Only the content decides.
What this changes
escaped_bytesmeasures them withjq -Rs ., which does the same operation astoJson. Against the run that failed,jqwas within 0.5%, and it was high. High is the safe direction.ALL_INPUTS_OTHER_BYTESholds the 38 other inputs thattoJson(inputs)writes beside the prompt. They are 1,356 bytes in the same variable. fix(review): bound the assembled prompt below MAX_ARG_STRLEN #31 does not count them.cap_file_escapedtrims a file until the escaped size fits. It measures, scales, and measures again, because the ratio is a property of the content. It also removes the incomplete last line, so the context does not stop in the middle of a patch line.Why the two diff blocks need one budget
The since-last-review diff is a subset of the full diff. On a pull request with one file, it is almost all of it. With independent caps the two blocks can reach 5,000 lines of nearly the same patch. On PR #1670 this was 42 KB of "since your last review" on top of 66 KB of "full diff".
The budget of #31 keeps that inside the limit. But it does not stop the cost. Every line the copy spends is a line that the rest of the context does not get.
The since-diff keeps its share, because on cycle 2 and later the new changes are the subject of the review. The full diff gives up its lines. Its notice tells the reviewer to run
gh pr difffor the remainder.Result
The same pull request, with this branch:
Tests
The tests of #31 keep their structure. I changed the totals to count escaped bytes and to include the other inputs. Two new cases:
All five suites pass. The workflow YAML does not change.
One point for your decision
The margin is 1,881 bytes, which is 1.4% of the limit. This is a result of the 1 KB of slack in #31, which your comment explains.
The risk:
claude-code-actioncan add a new input. ThenALL_INPUTS_OTHER_BYTESgrows past its 2,000-byte allowance and the failure returns.I did not change your slack, because the small number is your decision and your reason for it is written down. Tell me if you want it larger.