feat(agent): resolve visual targets to semantic refs before coordinate fallback

[feiniao87968492]

Summary

• add a read-only resolve_visual_target({ x, y, from_screenshot }) agent tool
• resolve screenshot-selected points to stable semantic accessibility refs across composed ancestors and open shadow DOM
• preserve canonical CSS coordinates for deliberate fallback through the existing coordinate click tool
• mirror Chrome and Firefox behavior, hide WebBrain indicators during hit testing, and classify page-authored results as untrusted

Testing

• node test/run.js — 1622 passed, 0 failed
• npm test — passed before rebasing onto the latest upstream main
• git diff --check — passed
• Playwright fixtures added for nested SVG/button targets, canvas fallback, and open shadow DOM; local execution requires Playwright browser binaries

🤖 Generated with Claude Code

[vercel]

@feiniao87968492 is attempting to deploy a commit to the esokullu's projects Team on Vercel.

A member of the Team first needs to authorize it.

[esokullu]

thank you. here's the inspection results:

It genuinely helps when a visual point lands on an SVG/icon inside a button or inside open shadow DOM. Converting that point into a stable ref_id enables the safer click_ax path. The implementation is also thoughtful about coordinate conversion, indicator hiding, Chrome/Firefox parity, and untrusted page-authored labels.

But exposing it as another LLM tool creates avoidable overhead:

Effect	Impact
Tool schema and prompt guidance	A few hundred additional prompt tokens on eligible Act calls
Tool choice	Another option the model must understand and select correctly
Execution	An extra full model turn: inspect_viewport → resolve_visual_target → click_ax
Fallback cases	Canvas, iframes, labels/siblings, and slight coordinate misses often return the original CSS point, making that extra turn accomplish nothing

The extra inference turn is the real cost—not the schema tokens.

There are also limits hidden by the wording: it does not find the “nearest” target. It only checks the exact hit element and its composed ancestors. It won’t semantically recover many near misses, associated label/input siblings, iframe contents, or canvas controls. The existing coordinate-click path already performs screenshot scaling, form/select handling, label/input redirection, trusted CDP dispatch, and progress verification in agent.js (line 16800).

My recommendation:

• Keep the semantic hit-testing implementation.
• Do not expose resolve_visual_target to the model.
• When the model calls existing click({x,y,from_screenshot:true}), internally:
  1. Convert coordinates once.
  2. Hide indicators.
  3. Attempt the semantic/shadow-DOM resolution.
  4. Bind and click the resolved target through the existing safe path.
  5. Fall back to the existing coordinate click when resolution fails.
• Return diagnostic metadata such as semanticTargetResolved, role, and bounded name.

That gets almost all the benefit with no new tool choice and no additional LLM round trip. If internal integration proves too invasive, second-best is exposing the resolver only dynamically after a visual inspection—not in every Act tool catalog.

The fixtures are good mechanical tests, but I’d also want a small model-level evaluation comparing success rate, latency, tool calls, and tokens across nested SVG, shadow DOM, canvas, iframe, label/input, and dynamic rerender cases.

[feiniao87968492]

Thank you for your guidance!I'll make corrections in this direction