Skip to content

fix(docs-eval): tolerate nested artifact layout in CI executor#1001

Open
alexluong wants to merge 1 commit into
mainfrom
fix/docs-eval-artifact-layout
Open

fix(docs-eval): tolerate nested artifact layout in CI executor#1001
alexluong wants to merge 1 commit into
mainfrom
fix/docs-eval-artifact-layout

Conversation

@alexluong

Copy link
Copy Markdown
Collaborator

The docs-agent-eval job (run 29858282052) failed with no .ts file found even though both scenarios passed scoring: the scenario-02 agent laid out its quickstart as a typescript/ subproject, and execute-ci-artifacts.sh only globs the run-dir root. Agent file layout is nondeterministic, so root-only discovery flakes.

pick_sh/pick_ts now fall back to a recursive search (node_modules pruned, .d.ts skipped, quickstart names preferred, sorted so the pick is deterministic), and install/execute run from the picked file's directory so a nested package.json and .env stash apply. The smoke test writes its TS project under typescript/ to keep the nested path covered, and the agent prompt template now tells quick-path runs to write files directly in the working directory — that lowers drift; the executor change is what removes the failure mode.

Verified by replaying the failed run's downloaded artifact through the executor: unpatched reproduces no .ts file found; patched picks typescript/outpost-quickstart.ts and proceeds to execution.

🤖 Generated with Claude Code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant