docs: correct config precedence and the .secrets/ file format - #258
Merged
dimitri-yatsenko merged 1 commit intoAug 14, 2026
Merged
Conversation
Two defects, both verified against datajoint-python 2.3 source and by
running the loader against fixtures.
**Precedence was inverted in four pages.** `datajoint.json` outranks
`.secrets/`, not the other way round. Two mechanisms in settings.py
enforce it: `_update_from_flat_dict` skips a file value when its env var
is set ("env var takes precedence"), and `_load_secrets` assigns only
when the target is still unset, with the comment "Only set if not already
present (config / env vars win)". Confirmed empirically — with the same
key in all three sources, `DJ_USER` wins, then datajoint.json, and
`.secrets/` supplies a value only when nothing else did.
The actual order is: programmatic, environment variables,
datajoint.json, .secrets/, defaults. Fixed in reference/configuration.md,
how-to/configure-storage.md, how-to/deploy-production.md, and
how-to/configure-database.md. manage-secrets.md and
specs/object-store-configuration.md already had it right.
Note the module docstring in datajoint-python's settings.py states the
inverted order and contradicts its own implementation twenty lines below;
that is the likely source of the error here and needs an upstream fix.
**`.secrets/datajoint.json` is never read.** The secrets directory is
read file by file: only `database.user`, `database.password`, and
`stores.<name>.<attr>` are recognized. A JSON file there contributes
nothing, so the documented "recommended for development" setup left
credentials unset and produced the same access-denied error the format
was meant to avoid. Replaced with per-key files in manage-secrets.md
(Option 1, the directory tree, the best-practices block, both templates,
and the leaked-credential rotation and history-scrubbing steps),
configure-database.md, and migrate-to-v20.md — which additionally told
migrators to put database.host in the secrets directory, a key that
directory does not read; the host moves to datajoint.json.
The Multi-Environment pattern keeps its JSON files: that snippet loads
them itself and assigns through dj.config, which works regardless.
dimitri-yatsenko
approved these changes
Aug 14, 2026
dimitri-yatsenko
left a comment
Member
There was a problem hiding this comment.
Verified both fixes against datajoint-python master, not just the prose:
- Precedence —
_load_secretsassigns only when the target is still unset (... and self.database.user is None; stores useif attr not in ...), sodatajoint.jsonand env win over.secrets/. Order programmatic > env > file > secrets > defaults is correct; the four inverted pages needed this. .secrets/format — the loader readsdatabase.user,database.password, andstores.<name>.<attr>file-by-file; a.secrets/datajoint.json(2-part name) matches nothing and is silently ignored. Your 'never read' finding is exactly right.
Corrections are applied consistently across the six pages. Nice, well-sourced work, @gtouloumes. Good catch on the settings.py module docstring stating the inverted order — that's the upstream origin; I'll open a datajoint-python fix so the code and docs agree.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Corrects two configuration errors in the docs: the precedence order between
datajoint.jsonand.secrets/, which four pages had inverted, and the.secrets/credential format, which the docs recommended in a shape DataJoint never reads.
Why
Both were found by following the documented setup and watching it fail, then checking the
loader in
datajoint-python2.3 rather than trusting any prose.Precedence between
datajoint.jsonand.secrets/was invertedFour pages said the secrets directory outranks the config file. It is the reverse: two
separate mechanisms in
settings.pymake the file win._update_from_flat_dictskips a file value when its environment variable is set, logging"Skipping {key} from file (env var {var} takes precedence)".
_load_secretsassigns only when the target is still unset (and self.database.user is None), and for storesif attr not in self.stores[store_name]— carrying the comment"Only set if not already present (config / env vars win)".
Confirmed by running the loader against fixtures with the same key in all three sources:
datajoint.jsonuser vs.secrets/database.userDJ_USER.secrets/valueSo the order is: programmatic, environment variables,
datajoint.json,.secrets/,defaults. This matters in the common local-development case — a stale
userleft in acommitted
datajoint.jsonsilently overrides the one in.secrets/, which the oldordering said should win.
manage-secrets.mdandspecs/object-store-configuration.mdalready had it right..secrets/datajoint.jsonis never readmanage-secrets.mdrecommended it as "Option 1: Secrets Directory (Recommended forDevelopment)", and three pages repeated the shape. The secrets directory is read file by
file: only
database.user,database.password, andstores.<name>.<attr>are recognized.A JSON file there contributes nothing —
dj.config.database.userand.passwordboth comeback
None— so the recommended setup left credentials unset and produced the veryaccess-denied error it was meant to prevent. Nothing warns; the file is simply ignored.
migrate-to-v20.mdwas therefore incorrect, instructing migrators to putdatabase.hostintothe secrets directory, which does not read that key at all.
Changes
Precedence lists corrected —
reference/configuration.md,how-to/configure-storage.md,how-to/deploy-production.md, andhow-to/configure-database.md, now all consistent withmanage-secrets.md..secrets/format corrected to one plain-text file per setting:manage-secrets.md— Option 1, the directory-structure tree, theSecurity-Best-Practices block, both configuration templates, and the leaked-credential
section, whose rotation step and
git filter-branchexample both named the file thatcannot hold credentials.
configure-database.md— the secrets-directory section, plus a pointer to thesupported key names.
migrate-to-v20.md— credential files written per key;database.hostmoved into thedatajoint.jsonexample where it is actually read.Left as-is: the Multi-Environment pattern in
manage-secrets.mdkeeps its.secrets/datajoint.{env}.jsonfiles. That snippet reads them itself and assigns throughdj.config[...], so it works regardless of the loader — now stated explicitly so it doesnot read as contradicting the corrected format.
Notes
mkdocs buildis clean — no content warnings, and no new link notices on the six editedpages.
The module docstring in
datajoint-python'ssettings.pystates the inverted precedenceand contradicts its own implementation twenty lines below it. That is the likely origin of
the error in these pages and wants a separate upstream fix.