IMPR: allow s3 stores without static credentials, matching gcs/azure

[ttngu207]

Improvement Request

Problem

An s3 store cannot be configured without embedding a static access_key / secret_key, even when the process already has a perfectly good AWS identity.

s3 is the only protocol in storage.py that mandates credentials:

# settings.py — DataJointConfig.get_store_spec
required_keys = {
    "file":  ("protocol", "location"),
    "s3":    ("protocol", "endpoint", "bucket", "access_key", "secret_key", "location"),
    "gcs":   ("protocol", "bucket", "location"),
    "azure": ("protocol", "container", "location"),
}

gcs and azure require none, and _create_filesystem reads theirs tolerantly — token=self.spec.get("token"), account_key=self.spec.get("account_key") — so they already fall through to Application Default Credentials and DefaultAzureCredential. Only s3 uses hard subscripts:

return fsspec.filesystem(
    "s3",
    key=self.spec["access_key"],
    secret=self.spec["secret_key"],
    ...
)

The result is that every environment where AWS credentials are ambient — EC2 instance profile, EKS service-account role, ECS task role, or a workstation with AWS_PROFILE / SSO — has to mint and distribute long-lived access keys purely to satisfy DataJoint, when boto3 would have resolved an identity by itself.

Requirements

Make access_key and secret_key optional for protocol: s3, matching gcs and azure:

1. settings.py — drop both from required_keys["s3"].
2. storage.py::_validate_spec — drop both from required.
3. storage.py::_create_filesystem — self.spec["access_key"] → self.spec.get("access_key") (same for secret_key). This one is easy to miss: relaxing only the two validators leaves a subscript that raises KeyError after validation passes.

When both are absent, pass nothing and let the underlying chain resolve. When both are present, behaviour is exactly as today — this is backward compatible.

Two details worth deciding explicitly:

• Reject an empty string rather than forwarding it. s3fs filters out None kwargs before calling botocore, so None is genuinely equivalent to absence. "" is not: it survives that filter, and botocore treats any non-None pair as explicit credentials, producing a confusing InvalidAccessKeyId at first object access instead of a clear config error. Since the current check is not self.spec.get(k), empty is already treated as missing — worth keeping that behaviour explicit once the keys become optional.
• The two validators disagree about None today. settings.py tests presence (k not in spec) while storage.py tests truthiness (not self.spec.get(k)), so "bucket": null passes the first and fails the second. Harmless at the moment, but the same divergence would make a partially-applied version of this change behave inconsistently.

No new configuration key is proposed — see below.

Justification

• Consistency. This makes s3 behave like the two object-store backends that already work this way. It is removing an outlier, not adding a mode.
• Credentials that refresh. An instance-profile or web-identity chain re-resolves for the life of the process. Static keys in a store spec do not, which is what makes long-running jobs the awkward case today.
• Fewer long-lived secrets. Environments that have an ambient identity currently have to create static keys anyway, purely to populate a config field — the opposite of what their security posture is usually aiming for.

Alternative Considerations

• Inject temporary STS credentials into the store spec. Works briefly, then expires. The spec is typically materialised once (env var, config file) and never refreshed, so a long job fails partway with ExpiredToken. Letting botocore own the chain avoids the problem rather than shortening it.
• Add an explicit opt-in field (e.g. credentials: "auto"). Rejected on compatibility grounds: allowed_keys is a strict whitelist and unknown keys raise, so any added field breaks every client older than the change. Absence is already the signal s3fs itself uses — core.py comments the key is None or secret is None branch as "automatic credentials".
• anon=True. Different meaning: unsigned/public-bucket access, not "resolve an identity normally".
• Workaround today: none within DataJoint. Callers must create static keys, or monkey-patch _create_filesystem.

Related Errors

With an ambient identity available but no static keys configured:

DataJointError: config.stores["<name>"] is missing: access_key, secret_key

and, if only the settings.py list is relaxed:

DataJointError: Missing S3 configuration: access_key, secret_key

Verified against s3fs 2024.6.1 / botocore 1.38.13: s3fs drops None kwargs (core.py, init_kwargs comprehension) and botocore falls through to the default chain unless both credentials are non-None (session.py::create_client), raising PartialCredentialsError when exactly one is supplied.

[dimitri-yatsenko]

Triaged — accepted for v2.3.3. Verified the three touch points against master:

• settings.py:478 — required_keys["s3"] lists access_key, secret_key; the check at settings.py:543 tests presence (k not in spec).
• storage.py:330-331 — required = ["endpoint", "bucket", "access_key", "secret_key"] with a truthiness check (not self.spec.get(k)).
• storage.py:368-369 — hard subscripts self.spec["access_key"] / self.spec["secret_key"], while gcs/azure at storage.py:376-385 already read theirs with .get(). Confirmed: relaxing only the two validators leaves this subscript to raise KeyError after validation passes.

The request is sound and low-risk — it removes an outlier so s3 matches gcs/azure, and the ambient-identity chains (instance profile, IRSA, ECS task role, SSO) are the ones that most need it. Both-present keeps today's behavior exactly, so it's backward compatible.

Two implementation notes for whoever picks this up:

1. Empty string. Once the keys leave both required lists there is no longer a truthiness guard on them, so _create_filesystem should coerce falsy to None explicitly — key=self.spec.get("access_key") or None (same for secret) — rather than .get(...) alone. That preserves the """ means absent" behavior the issue calls out, since s3fs forwards a non-None "" and botocore then treats it as an explicit (invalid) credential.
2. Exactly one of the two set. botocore raises PartialCredentialsError at first access when only one is supplied. Worth a fail-fast both-or-neither check in _validate_spec that raises a clear DataJointError instead of a late, cryptic one. Small addition; flagging it as an explicit decision rather than leaving it implicit.

The broader validator divergence (settings.py presence vs storage.py truthiness, e.g. "bucket": null) is real but out of scope here — this change removes it for the two credential keys and doesn't make the rest worse.

Please add a test with an ambient-identity spec (no access_key/secret_key) asserting _create_filesystem forwards neither kwarg, plus one asserting both-present is unchanged.