Skip to content

docs: correct config precedence and the .secrets/ file format - #258

Merged
dimitri-yatsenko merged 1 commit into
datajoint:mainfrom
gtouloumes:docs/secrets-precedence
Aug 14, 2026
Merged

docs: correct config precedence and the .secrets/ file format#258
dimitri-yatsenko merged 1 commit into
datajoint:mainfrom
gtouloumes:docs/secrets-precedence

Conversation

@gtouloumes

Copy link
Copy Markdown
Collaborator

What

Corrects two configuration errors in the docs: the precedence order between
datajoint.json and .secrets/, which four pages had inverted, and the .secrets/
credential format, which the docs recommended in a shape DataJoint never reads.

Why

Both were found by following the documented setup and watching it fail, then checking the
loader in datajoint-python 2.3 rather than trusting any prose.

Precedence between datajoint.json and .secrets/ was inverted

Four pages said the secrets directory outranks the config file. It is the reverse: two
separate mechanisms in settings.py make the file win.

  • _update_from_flat_dict skips a file value when its environment variable is set, logging
    "Skipping {key} from file (env var {var} takes precedence)".
  • _load_secrets assigns only when the target is still unset (and self.database.user is None), and for stores if attr not in self.stores[store_name] — carrying the comment
    "Only set if not already present (config / env vars win)".

Confirmed by running the loader against fixtures with the same key in all three sources:

Setup Result
datajoint.json user vs .secrets/database.user the file value
both, plus DJ_USER the env value
user absent from the file the .secrets/ value

So the order is: programmatic, environment variables, datajoint.json, .secrets/,
defaults. This matters in the common local-development case — a stale user left in a
committed datajoint.json silently overrides the one in .secrets/, which the old
ordering said should win.

manage-secrets.md and specs/object-store-configuration.md already had it right.

.secrets/datajoint.json is never read

manage-secrets.md recommended it as "Option 1: Secrets Directory (Recommended for
Development)", and three pages repeated the shape. The secrets directory is read file by
file: only database.user, database.password, and stores.<name>.<attr> are recognized.
A JSON file there contributes nothing — dj.config.database.user and .password both come
back None — so the recommended setup left credentials unset and produced the very
access-denied error it was meant to prevent. Nothing warns; the file is simply ignored.

migrate-to-v20.md was therefore incorrect, instructing migrators to put database.host into
the secrets directory, which does not read that key at all.

Changes

Precedence lists correctedreference/configuration.md,
how-to/configure-storage.md, how-to/deploy-production.md, and
how-to/configure-database.md, now all consistent with manage-secrets.md.

.secrets/ format corrected to one plain-text file per setting:

  • manage-secrets.md — Option 1, the directory-structure tree, the
    Security-Best-Practices block, both configuration templates, and the leaked-credential
    section, whose rotation step and git filter-branch example both named the file that
    cannot hold credentials.
  • configure-database.md — the secrets-directory section, plus a pointer to the
    supported key names.
  • migrate-to-v20.md — credential files written per key; database.host moved into the
    datajoint.json example where it is actually read.

Left as-is: the Multi-Environment pattern in manage-secrets.md keeps its
.secrets/datajoint.{env}.json files. That snippet reads them itself and assigns through
dj.config[...], so it works regardless of the loader — now stated explicitly so it does
not read as contradicting the corrected format.

Notes

mkdocs build is clean — no content warnings, and no new link notices on the six edited
pages.

The module docstring in datajoint-python's settings.py states the inverted precedence
and contradicts its own implementation twenty lines below it. That is the likely origin of
the error in these pages and wants a separate upstream fix.

Two defects, both verified against datajoint-python 2.3 source and by
running the loader against fixtures.

**Precedence was inverted in four pages.** `datajoint.json` outranks
`.secrets/`, not the other way round. Two mechanisms in settings.py
enforce it: `_update_from_flat_dict` skips a file value when its env var
is set ("env var takes precedence"), and `_load_secrets` assigns only
when the target is still unset, with the comment "Only set if not already
present (config / env vars win)". Confirmed empirically — with the same
key in all three sources, `DJ_USER` wins, then datajoint.json, and
`.secrets/` supplies a value only when nothing else did.

The actual order is: programmatic, environment variables,
datajoint.json, .secrets/, defaults. Fixed in reference/configuration.md,
how-to/configure-storage.md, how-to/deploy-production.md, and
how-to/configure-database.md. manage-secrets.md and
specs/object-store-configuration.md already had it right.

Note the module docstring in datajoint-python's settings.py states the
inverted order and contradicts its own implementation twenty lines below;
that is the likely source of the error here and needs an upstream fix.

**`.secrets/datajoint.json` is never read.** The secrets directory is
read file by file: only `database.user`, `database.password`, and
`stores.<name>.<attr>` are recognized. A JSON file there contributes
nothing, so the documented "recommended for development" setup left
credentials unset and produced the same access-denied error the format
was meant to avoid. Replaced with per-key files in manage-secrets.md
(Option 1, the directory tree, the best-practices block, both templates,
and the leaked-credential rotation and history-scrubbing steps),
configure-database.md, and migrate-to-v20.md — which additionally told
migrators to put database.host in the secrets directory, a key that
directory does not read; the host moves to datajoint.json.

The Multi-Environment pattern keeps its JSON files: that snippet loads
them itself and assigns through dj.config, which works regardless.

@dimitri-yatsenko dimitri-yatsenko left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified both fixes against datajoint-python master, not just the prose:

  • Precedence_load_secrets assigns only when the target is still unset (... and self.database.user is None; stores use if attr not in ...), so datajoint.json and env win over .secrets/. Order programmatic > env > file > secrets > defaults is correct; the four inverted pages needed this.
  • .secrets/ format — the loader reads database.user, database.password, and stores.<name>.<attr> file-by-file; a .secrets/datajoint.json (2-part name) matches nothing and is silently ignored. Your 'never read' finding is exactly right.

Corrections are applied consistently across the six pages. Nice, well-sourced work, @gtouloumes. Good catch on the settings.py module docstring stating the inverted order — that's the upstream origin; I'll open a datajoint-python fix so the code and docs agree.

@dimitri-yatsenko
dimitri-yatsenko merged commit 42370a9 into datajoint:main Aug 14, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants