Skip to content

fix(evaluation): preserve trace boundaries in eval sets - #907

Open
linhongyu510 wants to merge 1 commit into
volcengine:mainfrom
linhongyu510:fix/trace-eval-case-boundaries
Open

fix(evaluation): preserve trace boundaries in eval sets#907
linhongyu510 wants to merge 1 commit into
volcengine:mainfrom
linhongyu510:fix/trace-eval-case-boundaries

Conversation

@linhongyu510

Copy link
Copy Markdown

Summary

Fix trace-to-eval-set conversion when a tracing file contains more than one trace.

Previously, conversion state such as conversation, app_name, and user_id was shared across trace groups, while eval_cases.append() ran after the grouping loop. This could collapse multiple traces into one eval case and mix tool calls or metadata across trace boundaries.

This change:

  • creates one isolated eval case per trace
  • sorts spans by start_time before deriving input, output, and tool order
  • resets conversation and metadata state for every trace
  • includes the trace ID in generated eval IDs
  • derives the eval-set timestamp from the earliest generated case

The regression test uses two interleaved, out-of-order traces and verifies that conversations, tool calls, metadata, and timestamps remain isolated.

Validation

  • .venv/bin/python -m pytest tests/test_evaluator.py -q — 3 passed
  • ruff check veadk/evaluation/base_evaluator.py tests/test_evaluator.py
  • ruff format --check veadk/evaluation/base_evaluator.py tests/test_evaluator.py
  • git diff --check

AI assistance

This change was developed with AI assistance. I reviewed the trace grouping logic, rebased it onto the latest main, and ran the checks above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant