Skip to content

feat(nodes): add pluggable routing pipeline - #11200

Open
localai-org-maint-bot wants to merge 1 commit into
masterfrom
bot/issue-10066-routing-pipeline
Open

feat(nodes): add pluggable routing pipeline#11200
localai-org-maint-bot wants to merge 1 commit into
masterfrom
bot/issue-10066-routing-pipeline

Conversation

@localai-org-maint-bot

Copy link
Copy Markdown
Collaborator

Summary

  • route loaded-replica selection through composable candidate filters, named weighted scorers, and a replaceable picker
  • preserve the existing load guard, hot-prefix, cold-placement, least-loaded fallback, and deterministic replica tie-break behavior
  • expose validated per-model scorer_weights through scheduling YAML/JSON, REST persistence, and the database model
  • document the pipeline and its initial prefix_cache scorer

Closes #10066.

Verification

  • go test -count=1 ./core/services/nodes/prefixcache
  • go test -count=1 ./core/services/nodes -ginkgo.focus=ParseSchedulingSeed
  • go test -run "^$" ./core/services/nodes/... ./core/http/endpoints/localai
  • go vet ./core/services/nodes/prefixcache ./core/services/nodes
  • git diff HEAD^ --check

The full nodes integration suite was also attempted: 254 specs passed, while 200 database-backed specs failed at setup because this host has no rootless Docker provider/PostgreSQL test container.

Checklist

  • Tests added for pipeline filtering, weighted-sum scoring, abstaining signals, deterministic/default and custom pickers, scorer validation, and declarative config parsing
  • User-facing distributed-mode documentation updated
  • No coverage baseline or tolerance changes

Route prefix-cache replica selection through composable filters, weighted scorers, and a replaceable picker. Preserve the existing load guard and deterministic selection policy while exposing per-model scorer weights through scheduling configuration.

Assisted-by: Codex:gpt-5 [go-vet]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Distributed routing: pluggable scorer/filter/picker pipeline

1 participant