fix(gallery): persist inference defaults where the loader reads them - #11232
Draft
jimmykarily wants to merge 1 commit into
Draft
fix(gallery): persist inference defaults where the loader reads them#11232jimmykarily wants to merge 1 commit into
jimmykarily wants to merge 1 commit into
Conversation
The recommended sampling parameters for a model family were applied at install and then never took effect. Two things went wrong on the way to disk. They were written as top level keys. ModelConfig embeds PredictionOptions under the "parameters" yaml key, so temperature, top_p, top_k, min_p, repeat_penalty and presence_penalty are only read from there. At the top level they parse without error and are then ignored for the life of the model. They were also merged in after the YAML had already been marshalled. The only re-marshal sat behind the artifact binding, which an entry carrying files: never reaches, so for those entries the defaults were computed and then dropped before anything was written. Neither failure was visible in normal use. ApplyInferenceDefaults runs again at load time and fills the same values from the same table, so the model ends up tuned correctly while the file on disk pins nothing. It surfaces when someone edits one of those values expecting it to win, or when a family is absent from inference_defaults.json and there is nothing to refill from. The empty base spec asserted that the authored parameters block landed verbatim. It now checks the authored keys individually, because the family defaults are merged into that same block. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
localai-org-maint-bot
requested changes
Jul 30, 2026
localai-org-maint-bot
left a comment
Collaborator
There was a problem hiding this comment.
The code change looks focused, but the commit trailer needs to follow this repository's AI-assistance policy before merge. Please replace:
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
with an Assisted-by: trailer in the format documented in .agents/ai-coding-assistants.md, for example:
Assisted-by: Claude:claude-opus-5
Keep your human Signed-off-by: trailer when amending the commit.
I also attempted go test ./core/gallery -run TestGallery -count=1; it is currently blocked before this package compiles by the missing generated pb.DetokenizeRequest / pb.DetokenizeResponse symbols that are also present on current master, so I am not attributing that failure to this PR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The recommended sampling parameters for a model family were applied at install and then never took effect. Two things went wrong on the way to disk.
They were written as top level keys. ModelConfig embeds PredictionOptions under the "parameters" yaml key, so temperature, top_p, top_k, min_p, repeat_penalty and presence_penalty are only read from there. At the top level they parse without error and are then ignored for the life of the model.
They were also merged in after the YAML had already been marshalled. The only re-marshal sat behind the artifact binding, which an entry carrying files: never reaches, so for those entries the defaults were computed and then dropped before anything was written.
Neither failure was visible in normal use. ApplyInferenceDefaults runs again at load time and fills the same values from the same table, so the model ends up tuned correctly while the file on disk pins nothing. It surfaces when someone edits one of those values expecting it to win, or when a family is absent from inference_defaults.json and there is nothing to refill from.
The empty base spec asserted that the authored parameters block landed verbatim. It now checks the authored keys individually, because the family defaults are merged into that same block.
Fixes #11230
Signed commits