Chart: exclude 503 from gateway retries and make the retry policy configurable

[lorenzo-norcini-scale]

What

• The VirtualService retry policy defaults change from retryOn: connect-failure,unavailable,gateway-error to retryOn: connect-failure,unavailable,502,504 (numeric codes map to Envoy retriable-status-codes). 503 is no longer retried.
• attempts, retryOn, and perTryTimeout are now configurable under virtualservice.retries; the previous values were hardcoded.

Why

gateway-error bundles 502/503/504. 503 is the overload and no-healthy-upstream signal, so retrying it multiplies offered load when the service is least able to absorb it; during the 2026-08-13 incident this roughly 4x-ed traffic at saturation (disabling retries live cut envoy rps 68 to 39). 502/504 remain retried since they are usually transient single-upstream failures.

Notes

• perTryTimeout is unset by default: the single catch-all route also carries streaming and long-lived LLM requests, which a per-try timeout would abort and re-send mid-flight.
• Deleting the retries block entirely would restore Istio's implicit default of 2 attempts, so the block always renders.
• Rendered with default and override values to verify output.

🤖 Generated with Claude Code

Greptile Summary

The PR removes HTTP 503 from the gateway's default retry policy and exposes the retry count, retry conditions, and optional per-try timeout as Helm values.

• Defaults retries to connection failures, unavailable gRPC responses, HTTP 502, and HTTP 504.
• Keeps per-try timeout unset for streaming and long-lived requests.
• Adds a production-shaped retry configuration example.

Confidence Score: 4/5

The retry-default change is directionally safe, but explicit zero-attempt configurations must be preserved before merging; the Helm values reference should also document the new settings.

The template correctly removes HTTP 503 from its default retry set, but Helm's empty-value semantics cause an explicit attempts: 0 to render as three attempts, defeating an operator's attempt to disable retries.

Files Needing Attention: charts/model-engine/templates/istio-virtualservice.yaml and charts/model-engine/values_sample.yaml

Important Files Changed

Filename	Overview
charts/model-engine/templates/istio-virtualservice.yaml	Makes the retry policy configurable and excludes 503 by default, but default 3 silently overrides an explicit zero-attempt setting.
charts/model-engine/values_sample.yaml	Documents the new retry settings in the sample values, although the corresponding published Helm values reference was not updated.

Prompt To Fix All With AI

### Issue 1
charts/model-engine/templates/istio-virtualservice.yaml:29
**Zero retry count is overwritten**

When a deployment sets `virtualservice.retries.attempts: 0` to disable retries, Helm treats zero as empty and `default 3` emits three attempts, causing requests to be retried despite the explicit configuration and potentially restoring load amplification.

### Issue 2
charts/model-engine/values_sample.yaml:87-90
**Retry values missing from reference**

The new `virtualservice.retries` settings are absent from the published Helm values reference, so operators cannot discover the supported fields and duration format there and may continue using defaults or provide an incorrectly formatted `perTryTimeout`.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

_{Reviews (1): Last reviewed commit: "fix(chart): stop retrying 503 at the gat..." | Re-trigger Greptile}

Greptile also left 1 inline comment on this PR.

Context used:

• Knowledge Base — Model-engine deployment and startup

[greptile-apps]

Zero retry count is overwritten

When a deployment sets virtualservice.retries.attempts: 0 to disable retries, Helm treats zero as empty and default 3 emits three attempts, causing requests to be retried despite the explicit configuration and potentially restoring load amplification.

Knowledge Base Used: Model-engine deployment and startup

Prompt To Fix With AI

This is a comment left during a code review.
Path: charts/model-engine/templates/istio-virtualservice.yaml
Line: 29

Comment:
**Zero retry count is overwritten**

When a deployment sets `virtualservice.retries.attempts: 0` to disable retries, Helm treats zero as empty and `default 3` emits three attempts, causing requests to be retried despite the explicit configuration and potentially restoring load amplification.

**Knowledge Base Used:** [Model-engine deployment and startup](https://app.greptile.com/scale-ai/-/custom-context/knowledge-base/scaleapi/llm-engine/-/docs/deployment-and-startup.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[lorenzo-norcini-scale]

Superseded by the consolidated incident-hardening PR: see the PR referencing this one.

[lorenzo-norcini-scale]

Superseded by #868.