Skip to content

fix(gallery): persist inference defaults where the loader reads them - #11232

Draft
jimmykarily wants to merge 1 commit into
mudler:masterfrom
jimmykarily:fix/gallery-inference-defaults-nesting
Draft

fix(gallery): persist inference defaults where the loader reads them#11232
jimmykarily wants to merge 1 commit into
mudler:masterfrom
jimmykarily:fix/gallery-inference-defaults-nesting

Conversation

@jimmykarily

@jimmykarily jimmykarily commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

The recommended sampling parameters for a model family were applied at install and then never took effect. Two things went wrong on the way to disk.

They were written as top level keys. ModelConfig embeds PredictionOptions under the "parameters" yaml key, so temperature, top_p, top_k, min_p, repeat_penalty and presence_penalty are only read from there. At the top level they parse without error and are then ignored for the life of the model.

They were also merged in after the YAML had already been marshalled. The only re-marshal sat behind the artifact binding, which an entry carrying files: never reaches, so for those entries the defaults were computed and then dropped before anything was written.

Neither failure was visible in normal use. ApplyInferenceDefaults runs again at load time and fills the same values from the same table, so the model ends up tuned correctly while the file on disk pins nothing. It surfaces when someone edits one of those values expecting it to win, or when a family is absent from inference_defaults.json and there is nothing to refill from.

The empty base spec asserted that the authored parameters block landed verbatim. It now checks the authored keys individually, because the family defaults are merged into that same block.

Fixes #11230

Signed commits

  • Yes, I signed my commits.
  • Documentation updated (docs/content/) for user-facing changes, or not applicable

The recommended sampling parameters for a model family were applied at
install and then never took effect. Two things went wrong on the way to
disk.

They were written as top level keys. ModelConfig embeds PredictionOptions
under the "parameters" yaml key, so temperature, top_p, top_k, min_p,
repeat_penalty and presence_penalty are only read from there. At the top
level they parse without error and are then ignored for the life of the
model.

They were also merged in after the YAML had already been marshalled. The
only re-marshal sat behind the artifact binding, which an entry carrying
files: never reaches, so for those entries the defaults were computed and
then dropped before anything was written.

Neither failure was visible in normal use. ApplyInferenceDefaults runs
again at load time and fills the same values from the same table, so the
model ends up tuned correctly while the file on disk pins nothing. It
surfaces when someone edits one of those values expecting it to win, or
when a family is absent from inference_defaults.json and there is nothing
to refill from.

The empty base spec asserted that the authored parameters block landed
verbatim. It now checks the authored keys individually, because the family
defaults are merged into that same block.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>

@localai-org-maint-bot localai-org-maint-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code change looks focused, but the commit trailer needs to follow this repository's AI-assistance policy before merge. Please replace:

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

with an Assisted-by: trailer in the format documented in .agents/ai-coding-assistants.md, for example:

Assisted-by: Claude:claude-opus-5

Keep your human Signed-off-by: trailer when amending the commit.

I also attempted go test ./core/gallery -run TestGallery -count=1; it is currently blocked before this package compiles by the missing generated pb.DetokenizeRequest / pb.DetokenizeResponse symbols that are also present on current master, so I am not attributing that failure to this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gallery install writes sampling parameters where the config loader cannot read them

2 participants