docs(postgresql): solution for Bus error (exit 135) on hugepage-enabled nodes - #825
Open
SuJinpei wants to merge 3 commits into
Open
docs(postgresql): solution for Bus error (exit 135) on hugepage-enabled nodes#825SuJinpei wants to merge 3 commits into
SuJinpei wants to merge 3 commits into
Conversation
…nabled nodes PostgreSQL dies with SIGBUS during bootstrap on any node where vm.nr_hugepages is non-zero, because kubelet caps hugetlb at 0 on the pod slice of pods that do not request hugepages, and that limit is charged at page-fault time rather than at mmap. Documents the two-level cgroup diagnosis (the in-pod view is unable to show the refusal), the version-dependent fix for both the Zalando and CloudNativePG operators, and the hugepage-request alternative. Reproduced and all four remedies verified on ACP 4.2 / k8s 1.33.7 / cgroup v1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Static review (codex, passes 1+3), an independent domain cold-read, and my own pass-3 self-test of every recommended command. Confirmed findings only: - initdb.options discards 10 explicit fields, not the 5 originally listed; locale, localeProvider, icuLocale, icuRules and builtinLocale were missing, so ICU-configured clusters were the most exposed to silent loss. - shared_memory_size_in_huge_pages does not exist before PostgreSQL 15 (verified: 13/14 return "unrecognized configuration parameter"), yet the version table routed PG13/14 readers straight at it. Added a manual fallback. - The sizing command could not run as written: it requires -D/PGDATA, and no form works pre-bootstrap. It also returns a page COUNT, not a size - the same default shared_buffers reports 75 with 2MiB pages and 1 with 512MiB pages. - Page size is node-dependent. arm64 Kylin v10 defaults to 512MiB, so hugepages-2Mi is the wrong resource name there and the pod stays Pending. - The pod-UID snippet swallowed kubectl failure through a pipe (empty UID, status 0); added pipefail and an emptiness guard. - Scoped the cgroup paths to the systemd cgroup driver and gave the cgroupfs shape; noted that blank jsonpath output is a result, not an error. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…strap
The cold-read flagged that patching the Cluster stops working once the initdb
Job exhausts its backoff. Confirmed, but the suggested remedy (delete the failed
Job) does not work, so this documents what does, from an end-to-end run:
- On v1.29.1-acp.1 the cluster gives no signal at all: status.phase stays
"Setting up primary" while the reconciler hot-loops on "Selected PVC is not
ready yet". Diagnose from the Job and status.danglingPVC instead. v1.29.2+
carries upstream #11035 and surfaces it in the phase; v1.29.1-acp.1 does not
(verified by merge-base against the tags).
- Patching the spec: no effect. Deleting the Job: no effect.
- Deleting the PVC alone is actively harmful - it trips a second Unrecoverable
branch ("no PersistentVolumeClaims exist ... restore from a recent backup").
- What works: delete and recreate the Cluster with the fix already in the spec,
which is safe only because bootstrap never completed. Verified the recreated
cluster reaches Running with huge_pages=off on a hugepage-enabled node.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a Troubleshooting solution for PostgreSQL dying with
Bus error/ exit 135 on nodes where hugepages are enabled.Why
Reported from a customer environment (Harbor's PostgreSQL crash-looping after
vm.nr_hugepageswas set for a business workload). The failure is not PostgreSQL-specific and not NUMA-specific: on any node with a hugepage pool, kubelet writeshugetlb.<size>.limit_in_bytes = 0on the pod slice of every pod that does not request hugepages. Reservation is global sommap(MAP_HUGETLB)succeeds; the limit is charged at page-fault time, where a refusal has no errno and the kernel sends SIGBUS. PostgreSQL's defaulthuge_pages=tryonly falls back whenmmapitself fails, so it dies.The single most useful thing in here: the cgroup counters visible inside the pod cannot show this. The container scope reads
unlimitedwithfailcntstuck at 0 while the parent pod slice refuses every charge. Reading the wrong level is what sent the original investigation down a false path for days.Validation
Reproduced and every remedy run end-to-end on ACP 4.2 / k8s 1.33.7 / containerd 1.7.29 / cgroup v1, on a single-NUMA node with no CPU pinning —
vm.nr_hugepages=512alone is sufficient to break every PostgreSQL bootstrap on the node. Covered both the CloudNativePG operator (v1.29.1-acp.1, PG 18.4) and the Zalando operator (spilo v4.3.0-beta.36, PG 13-17).Measured, not inferred:
limit=0,failcnt0 → 25 across one failinginitdb; container scopeunlimited, 0 → 0huge_pagesGUC alone does not fix a fresh bootstrap (the operator writespostgresql.confafterinitdb) — a cluster with only that setting still crash-loopsinitdb -c huge_pages=offworks on PG 16/17/18 and does not exist on 13/15hugepages-2Miworks and PostgreSQL genuinely consumes the pagesshared_buffersreports75pages on a 2 MiB-page node and1on a 512 MiB-page node — the sizing value is a page count, not a sizeTest environment fully restored afterwards (pool back to 0, kubelet restarted, allocatable back to its original value).
Review
Ran the three-pass protocol before opening this:
$(… | tr)that swallowedkubectlfailure into an empty UID with exit status 0.shared_memory_size_in_huge_pagesdoes not exist before PG 15 while the version table routed PG 13/14 readers straight at it.-D/PGDATA, and has no pre-bootstrap form) and the page-count/page-size unit trap above.The two review modalities caught disjoint defect classes; both independently found the
bootstrap.initdb.optionscaution listed 5 discarded fields when it discards 10.One reviewer recommendation was not taken: for a CloudNativePG bootstrap that has already exhausted its Job backoff, the suggested "delete the failed Job" does not recover the cluster. Testing showed patching does nothing, deleting the Job does nothing, and deleting the dangling PVC is actively harmful (it trips a second unrecoverable branch advising restore-from-backup). The documented recovery is delete-and-recreate the Cluster with the fix in the spec, which is safe only because bootstrap never completed.
Notes for reviewers
docs/zh/is generated by the translation pipeline.id:field, so CI'sadd_id.shassigns one from the H1 title. Please keep the title stable.cgroupfsshape given as an alternative.🤖 Generated with Claude Code