Skip to content

Honor SOURCE_DATE_EPOCH as the clock source for generated timestamps #9

Description

@imnasnainaec

Problem

sil-lift currently generates no timestamps of its own. dateCreated/dateModified are read from the source and written back verbatim; the only other root attributes are version (fixed at 0.13) and producer (caller-supplied). Serialization is therefore fully deterministic today — the same input plus the same in-memory edits always produce the same bytes, which is what makes the byte-exact fidelity contract testable.

Any feature that generates a timestamp from the wall clock breaks that. A dateModified stamped with datetime.now(...) means two runs of the same script over the same input produce different bytes, which:

  • makes byte-exact round-trip tests of the stamping feature itself unwritable without monkeypatching the clock;
  • defeats diff-based CI gates that assert an export is unchanged;
  • makes container/Action runs non-reproducible, which matters because sil-lift ships both a Dockerfile and a GitHub Action (action.yml) explicitly aimed at pipeline use.

Proposed solution

Honor SOURCE_DATE_EPOCH as the default clock source anywhere the library generates a timestamp, with explicit arguments always taking precedence.

Precedence: an explicit when= argument > SOURCE_DATE_EPOCH > wall clock.

def _default_now() -> datetime:
    """The clock for generated timestamps: SOURCE_DATE_EPOCH if usable, else now."""
    raw = os.environ.get("SOURCE_DATE_EPOCH")
    if raw is not None:
        try:
            return datetime.fromtimestamp(int(raw.strip()), tz=timezone.utc)
        except (ValueError, OverflowError, OSError):
            pass  # a malformed value is not honored; fall through to the wall clock
    return datetime.now(timezone.utc).replace(microsecond=0)

Two properties make this a clean fit:

  • SOURCE_DATE_EPOCH is integer POSIX seconds, so the result is naturally seconds-precision — no truncation needed, and no fractional-second output to worry about.
  • _fmt_date in src/sil_lift/_writer.py already rewrites +00:00 to Z, so an aware UTC datetime serializes as YYYY-MM-DDTHH:MM:SSZ. That is the shape real FieldWorks exports use: across the seven FLEx 8.3–9.0 exports in The Combine's Backend.Tests/Assets, all 70,636 dateCreated/dateModified literals are exactly that 20-character form, with no bare dates, numeric offsets, or fractional seconds.

The variable is already set by Debian, Nix, Buildah, setuptools, and Sphinx among others, so pipelines that set it for other tools get sil-lift determinism for free rather than needing sil-lift-specific configuration.

Dependency

This is only observable once something in sil-lift generates a timestamp; nothing does today. It is a prerequisite or companion to any timestamp-stamping API rather than an independently visible change — and arguably a prerequisite rather than a follow-up, since it is what makes byte-exact tests of such a feature possible without clock monkeypatching.

Open questions

  • Malformed values: the spec permits a consumer to either error or ignore. The sketch above ignores and falls through, which keeps a library from raising on an environment variable the caller may not control — but it means a typo silently reintroduces nondeterminism. An alternative is to ignore it but surface a Problem-style warning, which fits the existing validation vocabulary.
  • Scope of "generated timestamp": dateModified only, or dateCreated too? Both are generated in a from-scratch export, so both presumably.
  • Forcing the wall clock: with the env var set, is passing when=datetime.now(timezone.utc) explicitly the sanctioned escape hatch, or is a dedicated override wanted?
  • CLI surface: whether the validate/export commands should also accept an explicit --now value, which would document the behavior in --help where an environment variable is invisible.
  • Documentation home: the reproducibility guarantee belongs next to the fidelity contract in docs/en/fidelity.md, since that is where readers go for byte-exactness claims.

Alternatives considered

  • A sil-lift-specific variable (e.g. SIL_LIFT_NOW). Rejected: it duplicates an established cross-ecosystem standard, and a pipeline already pinning SOURCE_DATE_EPOCH would have to learn a second name for the same concept.
  • An injectable clock parameter and nothing else. A when= parameter should exist regardless, but it does not help when the timestamp is generated inside a call the pipeline does not control — the CLI subcommands and the Docker/Action entrypoint being the concrete cases here.
  • Making generated timestamps opt-in only, and declaring reproducibility the caller's problem. Viable, but it pushes every pipeline that wants both stamping and determinism into monkeypatching or post-processing, and it leaves the library's own test suite without a supported way to exercise stamping byte-exactly.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions