feat(cleanup): notify and delete idle workspaces on coder.ddev.com#175
Draft
rfay wants to merge 7 commits into
Draft
feat(cleanup): notify and delete idle workspaces on coder.ddev.com#175rfay wants to merge 7 commits into
rfay wants to merge 7 commits into
Conversation
|
Stopped workspaces keep their multi-GB dind-cache Docker volume until deleted, and with no lifecycle policy they've accumulated to ~405GB across 105 volumes on a 512GB disk. Adds a daily janitor that emails owners at 7 days idle and deletes at 7 more days idle unless the workspace is used again, plus the GitHub Actions workflow to run it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
PROD_CODER_SESSION_TOKEN needs the owner role (this deployment has no Premium license, so there's no narrower built-in role that can list every user's workspaces and delete ones it doesn't own). Document the dedicated-service-account provisioning steps in the operations guide and cross-reference them from the workflow header.
rfay
force-pushed
the
20260702_delete_old_workspaces
branch
from
07d2363 to
cecce9d
Compare
…imer instead of GitHub Actions Avoids minting an owner-role Coder token that has to leave production and live in GitHub Secrets/1Password-for-CI. The janitor now runs directly on each Coder server (matching the existing coder-discord-relay pattern), with state kept in a local file instead of being committed back to git.
…istent coder@ddev.com
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/datais at 82% (394GiB/512GiB, 92GiB free); ~405GB across 105*-dind-cacheDocker volumes is the actual disk hog — stopping a workspace doesn't free its volume, onlycoder deletedoes.last_used_at, so there's no lifecycle policy in practice today.scripts/workspace-lifecycle-cleanup.sh: emails the owner (via Mailgun) at 7 days idle, deletes the workspace 7 more days later if still untouched, and clears the pending deletion if the owner comes back. State (who's been notified, when) is tracked in a local JSON file on the server.scripts/workspace-lifecycle-cleanup.service+.timer, matching the existingcoder-discord-relaypattern. staging-coder.ddev.com doesn't get this: its workspaces are almost entirely owned byci-bot(already excluded), it isn't sending real notices to anyone, and a dry-run against production itself (./scripts/workspace-lifecycle-cleanup.sh, no--force) already covers testing the policy without deploying it twice.docs/admin/operations-guide.md, new "Step 14: Set Up Workspace Lifecycle Cleanup" indocs/admin/server-setup.mdwith the prod install steps, entry added toCLAUDE.md.Why on-host, not CI
The script has to list every user's workspaces and delete workspaces it doesn't own, which on this (non-Premium) deployment requires the
ownerrole — there's no narrower built-in role that covers both. An earlier version of this PR ran the job from GitHub Actions, which meant putting thatowner-role token in GitHub Secrets/1Password-for-CI so it could reach production over the network — a real blast-radius increase (a leaked Actions secret would mean full site-admin on prod).Running it as a systemd timer on the Coder server itself avoids that: the token stays on the box (in
/etc/workspace-lifecycle-cleanup.env, mode 600), never travels through GitHub, and state lives in a local file instead of being committed back to git on every run.coderpicks upCODER_URL/CODER_SESSION_TOKENstraight from the environment, so nocoder loginsession is needed either.Before this can actually send/delete anything
workspace-janitorservice account + owner-role token on coder.ddev.com (seedocs/admin/server-setup.md→ Step 14) and put it in/etc/workspace-lifecycle-cleanup.envcoder.ddev.comsending domain is already verified; API key confirmed working via a real test send (1Password shared "DDEV" vault → "Mailgun" item →coder.ddev.com sending api keyfield)MAILGUN_FROMfixed tosupport@ddev.com(the original default,coder@ddev.com, doesn't exist)send_notice_email()in the script before it goes out to ~90 peopleworkspace-lifecycle-cleanup.timeron coder.ddev.com per Step 14Test plan
bash -nsyntax check andshellcheckclean on the scriptcoder/curlbinariescoder.ddev.comdomainsudo -u rfay ... workspace-lifecycle-cleanup, no--force) on coder.ddev.com once the env file is populated--forcerun via the timer (will only send notices — nothing can delete until 7 days later since state starts empty)🤖 Generated with Claude Code