design-proposal: Cozystack as a distribution — semver core, versioned packages, monthly CalVer releases - #46
Conversation
Proposes replacing Cozystack's single build-stamped version with four: a semver core (apiserver, controller, operator, CRDs, delivery contract), independently semver-versioned packages and apps, and a monthly CalVer distribution release defined by a manifest that pins an exact, tested set. The case is made from platform release engineering rather than from ecosystem ambition. Four costs are paid every cycle today, in a repository with no external catalogs at all: a fix cannot reach a released cluster except through backport automation that fails silently on conflict; a month whose substance is three bug fixes still moves 160 chart versions; a breaking change in one app is a platform-level event, which is why the schema debt #6 wants to pay down accumulated in the first place; and release readiness is all-or-nothing, as v1.6.0's four RCs and rc.1 NO-GO showed. None of those require a marketplace. The machinery that fixes them happens to be the machinery an ecosystem needs, which sets build order rather than motivation. Written as a joining piece for work already in flight. It supplies the version axis #43 assumes for catalog tiers and its declared minimum platform version, reuses the cozypkg surface from #12 and the backend from #23, pairs package MAJOR with #6's storage-version conversion to give "breaking" a testable definition, and extends #39's "metadata belongs on the ApplicationDefinition, not in a directory name" one tier up. On repository-as-unit versus package-as-unit: #18's reasoning is adopted unchanged for community repositories, where the tested-together guarantee holds and the author owes Cozystack no compatibility. The first-party archive is treated differently for one reason only -- a repository-level version cannot express a partial upgrade. #12's package-level axis is therefore revived on release-engineering grounds rather than its original ecosystem ones, with its concrete surface reused as written. Grounded in a survey of the current delivery chain: the whole packages/ tree is one OCI artifact selected by one digest and fanned out to 98 PackageSources with no version field anywhere in the data model. Notes that the runtime is already data-driven, that bundles are already metapackages, that resource-policy=keep already gives dpkg-selections semantics, and that four packages already carry per-package pre-upgrade migration hooks the global counter could be generalised onto. Records that core is not low-churn (~620 commits/12mo across internal/, pkg/, api/) and flags the one experiment that decides the implementation: whether ExternalArtifact revision is content-derived or source-derived. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…tion from source
The proposal gated its implementation choice on whether ExternalArtifact
revision is content-derived or source-derived, and proposed a cluster
experiment to settle it. It is settled from source instead, and the
answer enables Option A.
The upstream ArtifactGenerator CRD documents spec.artifacts[].revision
as optional, defaulting to "the digest of the artifact content"
(internal/fluxinstall/manifests/fluxcd.yaml:522-528), and cozystack
never sets it -- reconcileArtifactGenerators builds
OutputArtifact{Name, Copy} and the string Revision does not appear in
packagesource_reconciler.go at all. The earlier reading of the
"@<alias>" pattern as evidence of source-derivation was wrong: it
constrains the field when set.
This is the strongest finding in the proposal, because it inverts the
causality. Generated revisions being content-derived means
helm-controller already skips a package whose content did not change.
The reason every package upgrades on every release is the version stamp
itself: rewriting Chart.yaml from COZYSTACK_VERSION for 160 charts
guarantees no package is ever unchanged. The single global version
number is not just a labelling problem, it is the direct mechanical
cause of full-platform upgrade churn -- so giving charts stable
versions does not merely enable partial upgrades later, it stops
manufacturing the churn that makes them impossible.
One narrower question replaces the old one: whether the generated
tarball is reproducible for identical input content across two source
revisions. If source-watcher preserves unpack-time mtimes, Option A
needs an upstream determinism fix alongside it. That is checkable on
kind with source-controller and source-watcher alone -- no Cozystack
cluster and no dev stand.
Updates the overview, the problem statement, section 4, rollout phase
2, testing, open questions, and appendix A accordingly.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
#18 and #23 were merged on 2026-08-24 and are therefore accepted under the approval process, not drafts. Record that, link the merged documents, and note that #23 restates #18's position on per-package pinning — so section 8 now differs from two accepted documents rather than one draft. Also scope #18's non-goal correctly: per-package pinning is out of scope for its Phase 1, which is a sequencing statement rather than a permanent exclusion, and describe #23's credential threading the way the merged text does — a secretRef on the Flux source a tap already creates, no CRD change. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
…the core boundary The proposal's headline finding was wrong. Nothing rewrites Chart.yaml in the pool: `helm package --version` runs only in the `repo` target that builds _out/repos/, the neighbouring fix-charts target resets the field to 0.0.0, and the pool is a `flux push artifact` over the source tree — so all 164 charts ship `version: 0.0.0` and the version stamp never enters the artifact at all. What actually manufactures churn is a decorative version substring vendored inside each chart's image reference and rewritten at promotion by hack/promote-rewrite-tags.sh. Measured over v1.6.1..v1.6.2: of 164 packages, 129 are untouched, 9 changed chart source, 13 had an image digest move, and 13 moved with a byte-identical digest because only the tag string changed. Kubernetes resolves those references by digest and never reads the tag. So the churn is neither total nor unavoidable, and dropping the vendored tag becomes rollout phase 1 — cheap, independent of every design commitment here, and measurable on the next patch release. The tarball determinism question is answered rather than deferred: flux 2.8.6 normalises archive entries to epoch mtime and zero uid/gid, and building one tree twice is byte-identical. The residual is source-watcher's own re-tar, which is a code read rather than a cluster experiment. Section 1 now draws the core boundary the way Kubernetes draws its own — core is the mechanism, a package is anything that runs a workload, storage and networking included — and classifies the eleven charts that build from the root Go context against it. Two consequences are recorded: the delivery boundary is in the wrong place today, since the apiserver, controller and lineage webhook travel through the pool as ordinary packages and extracting them is the largest unscoped piece of work here; and because the Go tree stays one module by choice, a package's version cannot be derived from its image digest, which is an independent reason the digest must move to the manifest. Section 5 follows from that: PATCH and MINOR are derived from a package's own directory, MAJOR is declared, and CI checks the declaration is honest — reusing cmd/api-gate, which already computes that verdict for the API surface. Section 8 is reframed as inheritance rather than divergence. #18 scopes its exclusion of per-package pinning to Phase 1, so this is the Phase 2 that sentence leaves room for, and nothing here asks #18 or #23 to be amended. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The strongest objection to per-package versioning is that testing cost explodes into an N-by-M matrix, and #43 names that cost as its own driver. Answer it where the answer already is: the release manifest is a tested-together set by definition, e2e certifies the pinned combination, and per-package versions give the manifest something to pin rather than multiplying what must be run. A package that ships between trains is bounded by a declared requiresCore range exercised at its endpoints — Kubernetes' version skew policy, one tier down — instead of by re-testing everything. Both precedents carry a piece that is missing here, so the section states three preconditions rather than claiming the model is free. cozystack#3276 already builds the right harness: install previous stable, seed workloads with canary data, upgrade, verify survival, data integrity, all-HelmReleases-Ready, PVs Bound and the migration stamp advanced. It is opt-in by label and gates nothing, which is right for a lane nothing depends on and wrong for the lane that certifies a release. Recorded as a prerequisite and added to the rollout, since everything past the manifest needs to be able to test an upgrade of one. The earlier text called this lane "existing", which it is not yet. A declared range that no job exercises is documentation. Kubernetes publishes its skew policy and also runs skew jobs; the equivalent here is two endpoint runs per out-of-train package, not a matrix. The third precondition is the one with no precedent-supplied answer, and it was already an open question here: package-to-package edges are invisible. DependsOn is a list of names with no version constraint — 52 edges that order installation and say nothing about compatibility — and the larger coupling is undeclared entirely, with 35 charts reaching into each other's live state across 267 lookup sites, several reading another package's CRDs. Such a break changes no values schema and involves no core version, so neither gate sees it; the lookup returns empty and the template renders something else. The open question is promoted to a prerequisite and the guarantee is described as covering the edges the manifest can see. The acceptance test gains its missing half: #3276 asserts that every HelmRelease reconciled, and a partial upgrade needs the complement — that those outside the manifest diff did not. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
…r measurement The churn figures in section 4 were derived from `git diff` and predicted 35 of 164 packages moving on v1.6.1 -> v1.6.2. The prediction was run: a three-node Talos stand, cozy-installer 1.6.1 with the isp-full variant covering 157 of the 164 package directories as 207 ExternalArtifacts, a snapshot of every artifact digest and HelmRelease revision, helm upgrade to 1.6.2, and the same snapshot once the platform re-converged at 95/95 Ready. Result, with every moved artifact attributed and none left over: 134 held, 18 moved for the tag string alone, 25 for a cozy-lib fan-out, 18 for a genuine image rebuild, 12 for changed chart source. 23 of 95 HelmReleases took a new revision and 43 of 164 pods were replaced. Two things the diff could not show. The cosmetic case restarted the data plane: Cilium, LINSTOR, MetalLB and objectstorage-controller are all tag-only movers, and they restarted 3, 8, 4 and 1 pods; the metallb pods came back running the same two digests packages/system/metallb/values.yaml carries at both tags. And the ArtifactGenerator copies library charts into each consuming package's artifact — 36 artifacts vendor cozy-lib — so one edit to _barman.tpl moved 25 otherwise-untouched packages. A package directory is not self-contained, which is why the diff-based prediction missed them. That second cause is not removable the way the tag is: the library really is part of the rendered chart. It means a package's version is a function of its directory and its vendored libraries, and it makes cozy-lib an unversioned part of 36 packages' interface — the undeclared-interface problem Testing raises for cross-package lookup, showing up inside the first-party archive, and the tractable half of it since the edge is visible in the copy operations. The run also settles the last open question by observation rather than by reading upstream code: 134 artifacts held a byte-identical digest across two different pool revisions, which is only possible if source-watcher's re-tar normalises entry metadata as the flux CLI's does. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
… is now the bigger cause "Drop the vendored tag and the churn goes away" was a claim, so it was run. Both release trees were reassembled the way the ArtifactGenerator does — package directory plus the library charts copied into it — and content-hashed twice: as shipped, and with the Cozystack version removed from every image reference. Unmodified, the model reproduces the cluster exactly at 73 moved and 134 held with the same split by cause, which is what licenses trusting it on the change it cannot observe. With the version gone from the refs: 59 moved, 148 held. The fourteen that stop are precisely the data plane — Cilium in all five variants, LINSTOR, MetalLB, Multus, kubeovn-plunger, Kamaji, linstor-gui, objectstorage-controller and seaweedfs-system — the same packages whose 16 pod restarts were measured, so the fix removes all of them. Four tag-only movers survive, and instructively: clickhouse, http-cache, mariadb and extra/seaweedfs stop moving for the tag and keep moving because they vendor cozy-lib. That reorders the work. Of the 59 survivors, 29 are library fan-out against 18 genuine rebuilds and 12 changed sources, so the library question stops being a footnote and becomes the largest single cause of churn left; the open question is relabelled accordingly. Fixing both takes the release from 73 moved artifacts to 30, each for a reason inside its own package. One implementation note the experiment produced: a first transform matching only `repo:tag@sha256:` recovered 4 of 18, because two common shapes put the tag elsewhere — a bare `tag: v1.6.1` with the digest in a sibling key, and `tag: v1.6.1@sha256:…` as a YAML value. A digest-only change has to cover every shape hack/lib/image-refs.sh enumerates, which is also the only place they should be enumerated. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Andrei Kvapil (kvaps)
left a comment
Andrei Kvapil (kvaps)
left a comment
There was a problem hiding this comment.
Reviewed in full. The release-engineering framing is right, and the §4 artifact-revision finding is the strongest thing in the document — that the build-time Chart.yaml rewrite is the mechanical cause of full-platform upgrade churn, not merely a labelling problem, reframes the discussion. Several parts below should proceed regardless of what happens to the rest.
On §8 — repository-as-unit vs package-as-unit. This is where I disagree, and the argument is not "it was already decided".
#18 and #23 both merged on 24 August, and cozystack/cozystack#3940 implements them (+5857/−18 across 48 files: cozypkg tap/untap/search/validate/init/push, the Tap resource, the materializer, the publication gate). The table in "Scope and related proposals" still lists both as Draft — worth refreshing, since it changes what §8 argues against.
The substantive objection, though, is inside this proposal. Open questions says:
Inter-package compatibility. Debian works because packages have declared ABIs. Helm charts have none — the real interface is the values schema plus what one chart
lookups about another's live state.
That is not a loose end, it is load-bearing. Package-as-unit requires those contracts: if packages version and upgrade independently, something must state what one package promises another. The proposal correctly observes there is nothing to state it with, and names live violations — Harbor's storage-class preservation, SeaweedFS fullname adoption.
Repository-as-unit does not need them. The guarantee comes from the set being tested together, which is what a Cozystack release actually sells. That is #18's thesis, and it is why the unit is thematic rather than individual.
The four costs in "Why this is a release-engineering proposal" are real and I don't want them waved away. They are also addressed by splitting the tree into a few officially maintained repositories rather than one — core (Go only), a system layer, and app layers above it, each independently versioned and declaring a minimum version of the layer beneath:
- a Postgres fix ships as a release of the apps repository — no backport into the platform
- upgrading it does not touch the virtualization or system layers
- a breaking app change is a major of that repository, not of the platform
- a blocker in one layer does not hold the others
Coarser granularity, all four costs gone, no ABI required. I'd like §8 re-argued against that shape rather than against today's monorepo.
Worth splitting out now, independent of the above:
- §9's consolidating pass on
ApplicationDefinition. Four proposals are extending onev1alpha1object from four directions and nobody owns its coherence — the most urgent item in the document. - The per-package migration framework in
cozy-lib. 53 migrations in a single cluster-admin Job is worth fixing on its own merits. - Appendix B — eight defects, each cheap and unrelated. File them.
- Cadence/CalVer is a separate axis. One note if it proceeds: twelve trains a year while e2e still fails on upstream dependencies will mean skipped months more often than shipped ones.
Two smaller things:
- Goals says a security fix can ship "without a platform release", but §4 selects Option A, which republishes the whole artifact. Components aren't re-reconciled — that's the real win — but the publication itself is still a platform release. Worth restating the goal as no re-reconciliation.
- With 164 charts, who picks the version on each PR? The schema-diff gate classifies a floor, it doesn't assign a number. That's per-PR reviewer load and it isn't costed anywhere.
…a cluster The counterfactual had been modelled by content-hashing both trees. It has now been run: both release trees rebuilt with the Cozystack version removed from every image reference across all five ref shapes, pushed as two pool artifacts, and the same stand pointed at the first, settled, then pointed at the second. The model was exactly right. 59 artifacts moved against 73 as shipped, 148 held against 134, and the 14 that stopped are the 14 predicted with nothing newly moving. The residual splits as predicted too: 25 library fan-out, 18 image rebuilds, 12 changed sources, 4 tag-only survivors that move for cozy-lib rather than for the tag. The part worth having run: the data-plane restarts go to zero. Cilium 3 -> 0, LINSTOR 8 -> 0, MetalLB 4 -> 0, Multus 3 -> 0, objectstorage-controller 1 -> 0, and total pod churn halves from 43 of 164 to 21. HelmReleases taking a new revision fall from 23 to 16. One Kamaji pod still cycles, which its artifact no longer explains — a reschedule, not a delivery event. Rollout phase 1 and appendix A now carry the measured figures rather than the modelled ones. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
…e tree
Each of these was carrying a placeholder answer. Three turn out to be much
smaller than the document claimed and two needed a different question.
Extracting core needs no new machinery. The installer chart renders three
files; everything else in the bootstrap lane is installed by the operator at
startup from embedded manifests, twice already — crdinstall for the CRDs and
fluxinstall for Flux. Moving the apiserver, controller and lineage webhook out
of the pool is a third Install() beside those two, not a new delivery path, and
ordering is already handled. The real work is handing ownership over from the
platform Package without recreating the workloads, plus one coupling that was
unstated: cozystack-controller rolls the apiserver DaemonSet on a config-hash,
so if the apiserver moves and the controller does not, that loop crosses the
tier boundary.
cozy-lib has an answer, and it is not the one offered before. Helm requires a
library chart to sit in the consumer's charts/, so referencing it across
artifact boundaries is not available at all. The defect is that the copy always
takes the head of the pool: Library is {Name, Path} with no version, the same
shape defect as DependsOn, while apps/tenant already declares the dependency
with version "*". Adding version to Library and resolving cozy-lib at it is the
fix, and it makes the library the natural pilot for the versioned pool — one
library, 36 consumers, visible edges, and 29 of the 59 moves that survive the
tag fix. Added to the rollout as its own phase.
Testing precondition 3 was overstated. 267 lookup sites across 35 charts is the
raw count and most are the self-referential idiom; charts reading another
package's CRDs number eleven, and the groups are enumerable. Eleven is a list
somebody can read in an afternoon, so the ask is now concrete: declare those
eleven and fail CI on a new one, which is #39's capability field doing the job
it was proposed for. The precondition and #39 are the same work.
The cadence section asked for two or three cycles of evidence before trusting a
monthly train. The evidence is in the tags: the last four minors landed 27, 34
and 30 days apart, so monthly is already the observed cadence and writing it
down ratifies rather than changes. The support window is likewise already
chosen — backport.yaml enumerates release-X.Y branches and targets current plus
previous — so the recommendation is to ratify two minor lines rather than
invent a number that would need the bot changed first.
The ApplicationDefinition consolidation is not about field names, which do not
collide. It is about shape: #6's versions[] decides whether every other field
lives at the top level or inside a version, so it is one decision, it is #6's,
and the rest sequence behind it.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The previous commit said everything in the bootstrap lane past the operator Deployment is installed by the operator at startup. True on the Helm path, silent about the other one. The installer chart carries no crds/ directory and no CRD hook; its only pre-install hook is the Job that labels cozy-system. It passes --install-crds=true and --install-flux=true to the operator, both of which default to false in the binary, so on the Helm path the operator does install them from embedded manifests. On the kubectl apply path the same manifests ship as a release asset: make manifests concatenates internal/crdinstall/manifests/*.yaml into _out/assets/cozystack-crds.yaml and upload-assets.sh puts it in the release beside the rendered operator. One source, two deliveries. That adds a requirement to the core extraction which the two existing installs answer implicitly: anything moved into the bootstrap lane has to be present on both paths, or the Helm install and the kubectl apply install stop containing the same platform. The apiserver would need a release-asset entry, not only an Install(). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
…it as its own phase Versioning Library is the structural fix for the fan-out. Counting what is actually in the library turns up a cheaper one that, for this release, would have been enough on its own. cozy-lib is 728 lines and 43 helpers, and the consumer counts are very uneven. Two helpers carry 48 of roughly 55 relationships: the tenant RBAC subject model at 27 consumers and the resource-preset model at 21, both genuinely cross-cutting and both belonging there. Against that, _tls.tpl has no caller at all and says so in its own comment, 14 of the 15 helpers in _cozyconfig.tpl are exercised only by the library's test chart (their apparent in-library callers are Usage: lines in doc comments), _strings.tpl is three lines for one consumer, and rbac.subjectsForTenant has no caller of either kind. _barman.tpl explains the whole event. Its two consumers are apps/postgres and system/keycloak, and keycloak is there because it runs a CNPG Postgres of its own, so the library is being used as a private channel between two packages rather than as a shared abstraction. That is the package-to-package edge Testing worries about, wearing a library's clothes and so invisible to anything looking for one. And it was the only change to cozy-lib in this release: 25 packages moved because a 24-line template with two consumers landed in a chart 36 packages embed. So the two fixes compose. Versioning decides when a consumer takes a change, shrinking decides how often there is one. The four candidates for eviction are 265 of the 728 lines and they change for reasons that concern one or two packages. Added to the rollout ahead of versioning the library, on the same grounds as the tag fix: hygiene, measurable, no design commitment. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
… the split proposal Review asks for section 8 to be re-argued against a layered multi-repository split, on the premise that the document argues only against today's monorepo. The premise is half right and the earlier draft of this commit got the other half wrong by claiming the split has no written form. It does: #43 is the repository split, open since July, moving the application catalog out of the core repository into separately hosted, separately tested catalog repositories with three tiers assigned by explicit criteria. So section 8 now names #43 as the third position on its axis rather than implying the axis has two, and states the relationship plainly: a split decides where a package lives, this decides what its version means, and #43's own tier 1 and tier 2 catalogs need a version axis or each invents a local one. The scope table row for #43 said "component structure", which is what let it be read as something other than the split; it now leads with what it is. The increment the review actually adds over #43 is cutting the system layer out too. That part is unwritten, it is recorded as such in Alternatives, and it is not opposed: the version axis applies to it unchanged, requiresCore becoming "requires the system layer at or above" and the manifest pinning layer versions. One observation carried for whoever writes it. Coarser granularity does not remove the need for declared contracts, and the archive has a live example: _barman.tpl is an undeclared edge between apps/postgres and system/keycloak, keycloak being a consumer because it runs a CNPG Postgres of its own. Any split putting those two in different repositories makes that edge cross-repository rather than eliminating it. Three other things the review caught. The security-fix goal was wrong as written: under Option A the pool is republished, so the publication is still platform-wide and only the re-reconciliation is avoided. cozystack#3940 implements #18 and #23 and touches internal/operator/packagesource_reconciler.go, the same file the versioned pool changes, so it is in the scope table as a coordination point; its Tap is an aggregated-apiserver resource rather than a CRD, which puts the marketplace surface in core by this document's own boundary. And the cadence section now answers the e2e objection head on. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
…Source, and coarser breaks MAJOR Open question left standing: how do inter-package version dependencies work, and would versioning by group instead of by package avoid needing them. Both halves are now measured, and they point at a unit the document was not using. Group versioning does buy something. An edge needs a declared contract only when its two ends can move independently, so grouping absorbs most of them: at directory granularity 247 of the 251 dependsOn edges are intra-group and only 4 cross. But it costs the whole partial-upgrade goal. Every release touches every group, the last four transitions moved 6, 4, 5 and 6 of six, so "upgrade what changed" becomes "upgrade everything" again. The tree already carries a better unit. 31 of the 99 PackageSources span more than one top-level directory, 23 of them apps plus system, because an application's chart lives in apps/<x> while the ApplicationDefinition that serves it lives in system/<x>-rd and neither installs without the other. All 251 dependsOn edges are source-to-source. That is a Debian source package producing several binaries at one version, already implemented. Section 3's manifest example was already keyed on source names before the prose caught up. Measured on v1.6.1 to v1.6.2 at that granularity: 49 of 99 sources hold today, 57 after the tag fix, and 83 after the tag fix and the library shrink. Against 0 of 5 for directory groups. So the unit keeps nearly all of the benefit at 99 objects rather than 164, and both things the proposal needs, a version and a dependency constraint, land on objects that already exist. Section 5 now carries the decisive objection to any coarser unit, which is not about convenience. Gate 2 verifies a declared MAJOR by diffing a generated OpenAPI schema; a group has no single surface to diff, so a group's MAJOR cannot be checked at all. #6's conversions are written between two storage versions of one app's API and have nowhere to attach on a group. And a group MAJOR tells an operator that something broke rather than what, dragging every unbroken package with it, so the signal decays from "you must act" to "somebody must act, possibly not you". Grouping stays a support and presentation concern, which is what section is for. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
Leading with the bad news: the finding you called strongest turned out to be wrong. Nothing rewrites So I measured it instead of guessing from a diff. Three node stand, 1.6.1 with isp-full, 207 artifacts, snapshot of every artifact digest, upgrade to 1.6.2, snapshot again. 134 held. Of the 73 that moved, 18 moved because a decorative version substring inside an image reference gets rewritten at promotion while the digest stays identical, and 25 because the ArtifactGenerator copies Contracts: you are right, and I moved it from open question to precondition. Also narrowed it. 267 lookup sites is the raw count and most are the self-referential idiom, charts reading another package's CRDs are eleven and they are listed by name in Testing now. The ask is concrete, declare those eleven and fail CI on a new one, which is #39's capability field doing what it was proposed for. Security fix goal, correct catch, fixed. Under Option A the publication is platform wide and only the re-reconciliation is avoided. Who picks the version per PR: nobody. Patch and minor derive from the package's own directory, only major is declared, and the schema gate checks the declaration rather than producing it. ApplicationDefinition, agree it is the most urgent item. They do not collide on names though, they collide on shape: On the repository split, §8 was arguing against the wrong thing and I have fixed it, but not in the direction the review asks. The split already has a written proposal and it is #43, open since July: application catalog out of the core repository, separately hosted and separately tested, three tiers by explicit criteria. This document was never opposed to it and says so twice, the scope table calls #43 the stronger near-term driver and Alternatives records splitting first as compatible. What it did wrong was file #43 under "component structure", which let the split read as unwritten. Fixed, §8 now names it as the third position on the axis. Relationship as I see it: a split decides where a package lives, this decides what its version means. #43's tier 1 and tier 2 catalogs need a version axis or each of them invents a local one, which is the failure mode both of us want to avoid. The part of your shape that is genuinely beyond #43 is cutting the system layer out as well. That one is not written up, and I am not going to argue against a reconstruction of it. Write it and I will align. I expect little disagreement: One data point for whoever writes it. Cadence: the train adds no test that is not already run, and monthly is already what happens, last three minors landed 27, 34 and 30 days apart. It makes skipping visible where slipping was not, which is the point. A red upgrade lane is a reason to land cozystack/cozystack#3276, not a reason to avoid the calendar. #3940 is in the scope table now. Its tap materializer touches |
2 participants
Cozystack has one version number and everything wears it. Whole
packages/tree is pushed as single OCI artifact, one digest inpackages/core/installer/values.yamlselects it, and oneOCIRepositoryfans that digest out to 98PackageSourceobjects. Postgres chart fix and change to aggregated apiserver are same event as far as delivery concerned.This proposal replaces that with four version streams the way a distribution arranges its own: core on semver, packages and apps each on their own semver, and distribution on calver released monthly. Release stops being "state of a git tree at a tag" and becomes a manifest pinning exact set of component versions, tested together and supported together.
What changed since the first draft
First draft named the per-release rewrite of every
Chart.yamlas mechanical cause of full-platform upgrade churn. That was wrong. Nothing rewritesChart.yamlin the pool:helm package --versionruns only in therepotarget that builds_out/repos/, neighbouringfix-chartstarget resets the field back to0.0.0, and the pool isflux push artifactover the source tree. All 164 charts shipversion: 0.0.0. The placeholder comment describes a build step that does not exist.So the cause was measured on a cluster instead of guessed from a diff. Three node stand, 1.6.1 with
isp-full(207 artifacts over 157 of the 164 package directories), snapshot of every artifact digest and helmrelease revision, upgrade to 1.6.2, same snapshot again at 95/95 Ready. Every moved artifact attributed, nothing left over:v1.6.1->v1.6.2, 207 artifactscozy-libfan-out23 of 95 helmreleases took new revision and 43 of 164 pods were replaced.
"Every release moves every chart" was never true, 134 of 207 held. Delivery layer already skips what does not change, generated artifact revisions are content-derived and cozystack never sets
spec.artifacts[].revision. Question was only what makes things change.43 of the 73 that moved have nothing to do with the package. 18 moved because decorative version substring inside image reference gets rewritten at promotion while digest stays identical, and kubernetes resolves those refs by digest, it never reads the tag. Another 25 moved because
ArtifactGeneratorcopieslibrary/cozy-lib/**into each consuming package artifact and one library template changed. Only 30 moved for reason inside the package itself.Cosmetic case restarted the data plane. cilium, linstor, metallb, multus and objectstorage-controller are all tag-only movers and they restarted 3, 8, 4, 3 and 1 pods. metallb pods came back running exact digests that
packages/system/metallb/values.yamlcarries at both tags.The fix was tested, not asserted
Both release trees rebuilt with version removed from every image reference, pushed as two pool artifacts, same stand pointed at one and then the other.
Exactly 14 artifacts stopped moving, exactly the 14 predicted, nothing newly moving, and cilium, linstor, metallb, multus and objectstorage-controller restarts went to zero.
That makes rollout phase 1 a change with known effect and no design commitment attached: pin first-party images by digest alone, delete
hack/promote-rewrite-tags.sh, pass--reproducibleto the pool push. Worth doing whatever happens to the rest of this.It also reorders what is left. Of the 59 survivors 29 are library fan-out against 18 real rebuilds and 12 changed sources, so
cozy-libis now the largest single cause of churn. That one is not cosmetic and cannot be deleted, and referencing it instead of embedding is not available either, helm requires a library chart to sit in the consumercharts/. The defect is that the copy always takes head of the pool.Libraryin the data model is{Name, Path}with no version, same shape defect asDependsOn, whilepackages/apps/tenant/Chart.yamlalready declares the dependency withversion: "*". Adding a version toLibraryand resolvingcozy-libat it is the fix, consumer then moves when it takes a new library and not when the library changes. It also makes the library the natural pilot for the versioned pool, one library and 36 consumers is much smaller than versioning all 164 packages and exercises the same mechanism end to end.Counting what is in the library turns up a cheaper fix alongside that one. cozy-lib is 728 lines and 43 helpers, and two of them carry 48 of roughly 55 consumer relationships:
rbac.subjectsForTenantAndAccessLevelat 27 andresources.defaultingSanitizeat 21, both genuinely cross-cutting. Against that,_tls.tplhas no caller at all and says so in its own comment, 14 of the 15 helpers in_cozyconfig.tplare exercised only by the library test chart,_strings.tplis three lines for one consumer, andrbac.subjectsForTenanthas no caller of either kind._barman.tplhas two consumers,apps/postgresandsystem/keycloak, and keycloak is there because it runs a CNPG postgres of its own, so the library is a private channel between two packages rather than a shared abstraction.That last one is the whole event. Only change to cozy-lib in this release was one added file,
_barman.tpl, 24 lines and two consumers. Twenty-five packages moved because of it. Had it lived inapps/postgresthe library would have moved nothing, so shrinking the library is its own rollout phase ahead of versioning it: hygiene, no design commitment, and it removes the class of change that caused the fan-out.Where core boundary falls
Tier table now says what core means in the tree instead of naming three binaries. Core is the mechanism: apiserver, controllers over cozystack own API, tenant machinery, package and application system, CRDs and delivery contract. Package is anything that runs a workload, storage and networking included. Same line kubernetes draws for itself.
Two consequences recorded because they are load-bearing and were not stated before.
crds/dir, it passes--install-crds=trueand--install-flux=trueto the operator (both default false in the binary), so on the helm path the operator installs CRDs and flux itself at startup from embedded manifests and then creates the platform PackageSource. On the kubectl apply path the same manifests ship as a release asset,make manifestsconcatenatesinternal/crdinstall/manifests/*.yamlinto_out/assets/cozystack-crds.yaml. One source, two deliveries. Movingcozystack-api,cozystack-controllerandlineage-controller-webhookinto that lane is a third embedded-manifestInstall()beside two that already work, ordering is already handled. Real work is handing ownership over from the platform Package without recreating the workloads, and anything moved there has to appear on both paths or the two installs stop containing the same platform. One coupling to resolve in the same change: cozystack-controller rolls the apiserver daemonset on a config-hash, so if apiserver moves and the controller stays a package that loop crosses the tier boundary.COPY api pkg cmd internal, so any change underinternal/moves every one of their digests. Splitting the module is not proposed. Consequence is that package version cannot be derived from its image digest, which is separate reason the digest belongs in the manifest and not in the chart.Testing is the manifest, not a matrix
Obvious objection to per-package versioning is that testing explodes into N packages by M core versions, and #43 names that cost as its own driver. Manifest is the answer and the reason it exists: it is a tested-together set, e2e certifies the pinned combination, and per-package versions give it something to pin instead of multiplying what must be run. Package shipping between trains is bounded by declared
requiresCorerange exercised at its endpoints, which is kubernetes version skew policy one tier down.Three preconditions, stated instead of assumed:
DependsOnis a list of names with no version constraint, 52 edges that order installation and say nothing about compatibility. Larger coupling is not declared at all, though smaller than the raw count suggests: of 267lookupsites across 35 charts most are the self-referential idiom, and charts reading another package CRDs number eleven (apps/kubernetes,apps/kubernetes-nodes,apps/tenant,apps/vm-disk,apps/vm-instance,core/platform,system/backupstrategy-controller,system/harbor,system/keycloak,system/monitoring,system/seaweedfs-db). Such a break changes no values schema and involves no core version, so neither CI gate sees it. Eleven is a list somebody can read in an afternoon, so the ask is concrete: declare those eleven and fail CI on a new one, which is design-proposals: fold extra into apps — tenant modules as apps with declarative capabilities #39 capability field doing the job it was proposed for.Relationship to work in flight
#18 and #23 are accepted (merged 2026-08-24) and both scope per-package pinning out of Phase 1. This is written as the Phase 2 that sentence leaves room for, not as competing model. It asks nothing of either document, meta-index, tap flow and repository-level tags are unaffected, and
TapIndexcache is the natural reader for release manifest.#43 defines component structure, this supplies version axis it assumes including declared minimum platform version it wants checked at
PackageSourcereconciliation. #43 has more urgent driver and should not wait on this. #6 versions app API surface while this versions the package, pairing them is what gives "breaking" a testable definition. #39 removes directory-as-metadata one tier down, same argument same object. cozystack#3448 is prerequisite for external tiers and proof that runtime is already data-driven. #21 is affected, per-package pool changes what offline bundle contains and manifest is the mirror list, flagged to coordinate rather than decided here.What needs deciding
.github/workflows/backport.yamlenumerates realrelease-X.Ybranches and targets current plus previous, so the de-facto window is already two minor lines. Open part is whether to ratify that or widen it, and widening means changing the bot first.Libraryis the mechanism, the policy is open: consumers pin and bump individually, which isolates acozy-libfix but means 36 deliberate bumps to roll one out, or platform bumps them together each train.ApplicationDefinitionis extended by four proposals at once (design-proposals: fold extra into apps — tenant modules as apps with declarative capabilities #39, proposal: ApplicationDefinition multi-version conversion #6, #3448 and this one) while docs(out-of-tree-app-catalogs): propose splitting managed apps out of the core repository #43 needs that shape stable across repository boundaries. They do not collide on names, they collide on shape: proposal: ApplicationDefinition multi-version conversion #6versions[]decides whether every other field lives at top level or inside a version. One decision, it is proposal: ApplicationDefinition multi-version conversion #6, and the rest sequence behind it.Two questions from first draft are closed now. Generated artifact revisions are content-derived, settled from source. And tarball is reproducible: 134 artifacts held byte-identical digest across two different pool revisions, only possible if source-watcher re-tar normalises entry metadata the way flux CLI does.
On sequencing, phase 1 is the tag fix and carries no design commitment. Cadence comes next and is mostly ratification, last four minors landed 27, 34 and 30 days apart so monthly is already what happens. Extracting core comes before any versioning change, a tier boundary delivery does not honour cannot carry a version. Calver rename is last.
Appendix A carries measured facts, Appendix B lists eight unrelated defects found while surveying, they are cheap to fix independently.