Skip to content
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Agent notes for sil-lift

Read and follow [CONTRIBUTING.md](CONTRIBUTING.md) in full — its byte-exact
Read and follow [CONTRIBUTING.md](CONTRIBUTING.md) in full — its byte-identical
corpus rules and fidelity contract are essential to avoid silently breaking this
repo.

Expand Down
18 changes: 9 additions & 9 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ releases may contain breaking changes.
- Project scaffolding: package skeleton, vendored LIFT 0.13 RELAX NG schema,
test corpus with provenance, corpus-prep and large-file-generator tooling.
- Full object model: all 35 LIFT 0.13 elements as typed dataclasses;
`sil_lift.load()` / `Lexicon.load()` full-document reader with per-node
`Extras` residue capture; LIFT-version guard.
`sil_lift.load()` / `Lexicon.load()` full-document reader that keeps LIFT
residue per node in `Extras`; LIFT-version guard.
- `Lexicon.save()` writer with byte-fidelity passthrough — unchanged
documents and untouched entries are written byte-identically; touched entries
re-serialize canonically with all out-of-schema content preserved. Fidelity
Expand Down Expand Up @@ -63,11 +63,11 @@ releases may contain breaking changes.
members and is capped (entry count and a 10 GiB uncompressed total) against
zip bombs.
- Validation: `validate_file()` / `iter_problems()` /
`Lexicon.iter_problems()` returning an addressable `Problem` stream
(file/entry/line). RELAX NG layer with two documented deviations from raw
libxml2 (href masking with `uri-not-rfc` warnings; tag-grouped validation to
sidestep libxml2's interleave limitation); vendored ranges schema over
companions; semantic checks: duplicate-guid (entries, and
`Lexicon.iter_problems()` returning a `Problem` stream, each carrying the
file, entry, and line it concerns. RELAX NG layer with two documented
deviations from raw libxml2 (href masking with `uri-not-rfc` warnings;
tag-grouped validation to sidestep libxml2's interleave limitation); vendored
ranges schema over companions; semantic checks: duplicate-guid (entries, and
ranges/range-elements within their own document — matching the C#
`Validator`'s document-wide guid scan), dangling-ref, range-parent,
undefined-range-value (every grammatical-info and range-keyed trait
Expand All @@ -84,8 +84,8 @@ releases may contain breaking changes.
available up front) and `open_writer()` (header \+ one canonical chunk per
entry; byte-identical to `canonical_document` output by construction, and
optionally writing a `.lift-ranges` companion via `ranges=`), both
over the same `Entry` types as full-document mode and O(one entry) in
memory (verified on a ~340 MB generated file).
over the same `Entry` types as full-document mode, holding one entry in
memory at a time (verified on a ~340 MB generated file).
- The `sil-lift` CLI (stdlib-only, installed with the package):
`validate` / `stats` / `sort` / `check-media`, plus `export` — one row per
leaf sense (subsenses flattened) to CSV/TSV, streaming; analysis languages
Expand Down
4 changes: 2 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,14 +35,14 @@ Releases are cut by publishing a GitHub Release with a `vX.Y.Z` SemVer tag
matching `__version__`. A release-triggered workflow builds the distribution,
verifies that the tag and version match, and publishes to PyPI.

## The test corpus is byte-exact — never hand-edit it
## The test corpus is byte-identical — never hand-edit it

The fidelity tests assert that saving writes back the **exact bytes** of
`tests/corpus/` fixtures. Consequences:

- Never reformat, re-indent, or "fix" anything under `src/sil_lift/schemas/`,
`tests/corpus/`, or `tests/tools/xslt/`. Even a trailing-newline tweak breaks
the suite. `.gitattributes` and `.editorconfig` carry carve-outs so git and
the suite. `.gitattributes` and `.editorconfig` list exceptions so git and
editors leave these files alone — don't remove them.
- Adding a fixture requires an entry in `tests/corpus/PROVENANCE.md`: source
URL, commit SHA, fetch date, license. Hand-authored fixtures (e.g. under
Expand Down
6 changes: 3 additions & 3 deletions docs/en/csharp-differences.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,20 +14,20 @@ sil-lift is loosely analogous to SIL's C# LIFT tooling — chiefly `SIL.Lift` in

## API shape

`SIL.Lift`'s parser is callback-driven (`ILexiconMerger`): it pushes parse events at a consumer. sil-lift instead returns a plain object graph — typed dataclasses for every LIFT element — because Python scripters want objects, not callbacks. `SIL.DictionaryServices` does layer a `LexEntry`/`LexSense` object model over `SIL.Lift`, but as an application model it represents only the constructs those apps use — so re-serializing through it can't preserve out-of-model content the way sil-lift's residue capture and byte fidelity do (see below). The streaming API yields the _same_ `Entry` type, so there is no capability-reduced twin model.
`SIL.Lift`'s parser is callback-driven (`ILexiconMerger`): it pushes parse events at a consumer. sil-lift instead returns a plain object graph — typed dataclasses for every LIFT element — because Python scripters want objects, not callbacks. `SIL.DictionaryServices` does layer a `LexEntry`/`LexSense` object model over `SIL.Lift`, but as an application model it represents only the constructs those apps use — so re-serializing through it can't preserve out-of-model content the way sil-lift's LIFT residue handling and byte fidelity do (see below). The streaming API yields the _same_ `Entry` type, so there is no second, pared-down model to learn.

## Round-trip fidelity

The strongest deliberate difference. Saving with `SIL.Lift` re-serializes the whole document. sil-lift guarantees:

- an unchanged document saves **byte-identically**, and
- untouched entries keep their exact source bytes even when other entries change (Chorus-grade byte chunking, applied automatically).
- untouched entries keep their exact source bytes even when other entries change — the same per-entry byte chunking Chorus uses, applied automatically.

See [Fidelity guarantees](fidelity.md).

## Validation

The C# `Validator` runs one RELAX NG pass and reports the first errors as strings. sil-lift reports a structured, entry/line-addressed `Problem` stream, and its schema layer knowingly diverges in three places:
The C# `Validator` runs one RELAX NG pass and reports the first errors as strings. sil-lift reports a structured `Problem` stream, each carrying the file, entry, and line it concerns, and its schema layer knowingly diverges in three places:

- **Invalid URIs are warnings, not errors.** The C# RELAX NG engine never enforced the `anyURI` datatype, so FieldWorks (FLEx) has been writing `file://C:/...` hrefs into real lexicons for years. Rejecting those files would flag virtually every FLEx export.
- **Schematron rules are enforced** (as semantic checks): duplicate form languages and similar co-constraints in the LIFT grammar were silently ignored by both C# and raw lxml validation.
Expand Down
11 changes: 7 additions & 4 deletions docs/en/fidelity.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# Fidelity guarantees

LIFT is an _interchange_ format: the cardinal rule is **never drop what you do not understand**. `sil-lift`'s contract, verified by the test suite on every run (corpus files plus property-based generation):
LIFT is an _interchange_ format, so the first rule is **never drop what you do not understand**. `sil-lift`'s contract, verified by the test suite on every run (corpus files plus property-based generation):

## Reading

Any well-formed LIFT 0.13 document loads — schema-invalid content included. Whatever the model does not define is carried in the nearest node's opaque `Extras` bucket: unknown attributes and elements, XML comments and processing instructions, stray text, and malformed typed attributes (a bad date stays as the original string in `Extras`; the typed field is `None`).
Any well-formed LIFT 0.13 document loads — schema-invalid content included. Whatever the model does not define is carried in the nearest node's opaque `Extras` container as _LIFT residue_ — FieldWorks' name for the same idea, which it stores in a `LiftResidue` field: unknown attributes and elements, XML comments and processing instructions, stray text, and malformed typed attributes (a bad date stays as the original string in `Extras`; the typed field is `None`).

## Saving an unchanged document

Expand All @@ -14,7 +14,7 @@ Exceptions (the writer falls back to full canonical serialization, which is sema

- the source encoding is not ASCII-compatible (not UTF-8/US-ASCII), or
- the source contains a DOCTYPE, or
- the byte scanner and the parser disagree about the document's top-level structure — for instance an out-of-spec second `<header>`, which the parser keeps only once (the scanner is deliberately distrustful: any doubt means capturing no source bytes at all), or
- the byte scanner and the parser disagree about the document's top-level structure — for instance an out-of-spec second `<header>`, which the parser keeps only once (the scanner is deliberately conservative: on any doubt it captures no source bytes at all), or
- the source was built in memory rather than loaded from a file.

## Saving an edited document
Expand All @@ -23,8 +23,11 @@ Exceptions (the writer falls back to full canonical serialization, which is sema
- **Touched entries are re-serialized canonically and completely**: UTF-8, 2-space indentation _outside_ mixed content (whitespace inside `<text>` and `<span>` is never altered), a documented child grouping per element (e.g. entry: lexical-unit, citation, pronunciations, variants, senses, notes, relations, etymologies, annotations, traits, fields), fixed attribute order, dates in ISO-8601 (`Z` for UTC). All residue is re-emitted; its position is restored to the original child index, clamped to the new child list (an approximation — exact byte positions are only guaranteed for untouched entries).
- Adding, removing, or reordering entries re-serializes the document structure but still emits every unchanged entry's bytes verbatim.

!!! note "&quot;Canonical&quot; here is not related to any other Canonical XML"
Canonical form on this page means `sil-lift`'s own documented shape, described in a bullet above. It is unrelated to W3C's Canonical XML (C14N) process. It is unrelated to `SIL.Core`'s `CanonicalXmlSettings` class.

## Known approximations (touched nodes only)

- Comments _inside_ a `<text>` run are preserved but hoisted next to the run, not at their exact character offset.
- Comments _inside_ a `<text>` run are preserved but moved next to the run, not kept at their exact character offset.
- Cross-type child order within an edited element is normalized to the canonical grouping (the LIFT schema's `interleave` makes this order semantically insignificant).
- A multitext element that is present but carries nothing — no forms, no residue, e.g. `<definition></definition>` — is not re-emitted. The model represents these fields as an always-present `Multitext` (`lexical-unit`, `citation`, `definition`, a relation's `usage`, and `label` / `abbrev` / `description` on url-refs, ranges, range-elements and the header), so an empty one is indistinguishable from an absent one after parsing. Nothing semantic is lost.
6 changes: 3 additions & 3 deletions docs/en/guides/bulk-edit-glosses.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,12 +53,12 @@ A few things worth noting:
- It compares serialized content, so assigning a field the value it already had isn't reported.
- It reports content changes only; `lex.added_entries()` and `lex.removed_entries()` cover entries that appeared or disappeared since loading.
- It returns the entries themselves, unaffected by `id` being duplicated or absent (which LIFT allows).
- As a count, it is meaningful only where there is something to compare against. When the passthrough layer declines to byte-scan the source — an encoding that is not ASCII-compatible, or a scanner/parser disagreement — there is no baseline, and `changed_entries()` reports _every_ entry. That is the honest answer for a write guard, since `save()` re-serializes the whole file in that case, but it means the count is the size of the lexicon rather than the size of the edit.
- As a count, it is meaningful only where there is something to compare against. When the byte scanner declines to read the source — an encoding that is not ASCII-compatible, or a scanner/parser disagreement — there is no baseline, and `changed_entries()` reports _every_ entry. That is the honest answer for a write guard, since `save()` re-serializes the whole file in that case, but it means the count is the size of the lexicon rather than the size of the edit.
- `lex.changes()` reports whether the document changed _at all_. It covers not just the entries, but also the header, the root element, and every `.lift-ranges` companion.
- It is falsy only when `save()` would reproduce the source bytes, which makes `if not lex.changes(): ...` the right way to skip an unnecessary write. The guarantee runs one way: it never reports "nothing to write" for a document that would be rewritten, while a change that forces a full re-serialization can land back on the original bytes and still be reported.
- It compares content, not destination, so guard only an in-place save with it: `lex.save(some_other_dir / "dictionary.lift")` writes the document and its companions to a location that has nothing in it yet, whether or not anything changed.
- It is a guard, not a speed-up — answering it digests every entry, which is the same work `save()` does to decide passthrough, so what you skip is the write itself (an untouched mtime, no spurious diff), not the effort of deciding.
- Validating in memory (`lex.iter_problems()`) serializes the edited state first, so it correctly reflects the edit before anything is written to disk. Aborting on any `"error"`-level `Problem` — warnings are left for the caller to judge — means a bad edit never reaches `save()`.
- It is a guard, not a speed-up — answering it digests every entry, which is the same work `save()` does to decide which source bytes it can reuse, so what you skip is the write itself (an unchanged file-modification time, no spurious diff), not the effort of deciding.
- Validating in memory (`lex.iter_problems()`) serializes the edited state first, so it correctly reflects the edit before anything is written to disk. Aborting on any `"error"`-level `Problem` — warnings are left for the caller to decide about — means a bad edit never reaches `save()`.

Glosses aren't the only thing worth touching this way. The same `Multitext` mapping surface applies to definitions and every other multilingual field on an entry or sense:

Expand Down
6 changes: 3 additions & 3 deletions docs/en/guides/cli.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# The command line

Installing the package (`pip install sil-lift`) also installs the `sil-lift` command — a supported LiftTools-style tool that ships with the package (and, for `validate`, a worked example of the library API).
Installing the package (`pip install sil-lift`) also installs the `sil-lift` command — a supported tool in the spirit of LiftTools that ships with the package (and, for `validate`, a worked example of the library API).

```
sil-lift validate PATH [--format {text,json}] [--strict] [--no-check-media] [--require-ids]
all problems, entry/line-addressed; exit 1 on errors
all problems, with file/entry/line; exit 1 on errors
sil-lift stats PATH [--format {text,json}]
entry/sense/language counts (streaming; any size)
sil-lift sort PATH [-o OUT] canonically sorted, diff-ready copy (default: in place)
Expand All @@ -13,7 +13,7 @@ sil-lift export PATH [-o OUT] [--langs L] [--tsv]
one row per leaf sense (subsenses flattened) to CSV/TSV (streaming)
```

`--format json` writes a single JSON object to stdout (and nothing else) for CI/automation consumption; see the schema in the example below. `--strict` treats warnings as errors, exiting 1 if any are found — use it to gate a build on a clean bill of health rather than errors alone. `--no-check-media` skips the filesystem media-presence check (suppressing `missing-media` findings), which is useful when validating a freshly generated export whose audio/photo files live elsewhere and aren't colocated on disk. `--require-ids` additionally fails (a `missing-id` error) on any entry lacking a `guid` or sense lacking an `id` — stricter than LIFT, for workflows that re-import by a stable id. Passing `-` as the path reads the document from stdin (a piped document has no folder, so its companion `.lift-ranges` and media are not resolved). `stats` likewise takes `--format json`, emitting the counts as a single JSON object.
`--format json` writes a single JSON object to stdout (and nothing else) for CI/automation consumption; see the schema in the example below. `--strict` treats warnings as errors, exiting 1 if any are found — use it to gate a build on no warnings at all rather than on errors alone. `--no-check-media` skips the filesystem media-presence check (suppressing `missing-media` findings), which is useful when validating a freshly generated export whose audio/photo files live elsewhere rather than in the same folder. `--require-ids` additionally fails (a `missing-id` error) on any entry lacking a `guid` or sense lacking an `id` — stricter than LIFT, for workflows that re-import by a stable id. Passing `-` as the path reads the document from stdin (a piped document has no folder, so its companion `.lift-ranges` and media are not resolved). `stats` likewise takes `--format json`, emitting the counts as a single JSON object.

!!! note
`validate`'s exit codes and `--format json` schema are a supported automation interface: both are covered by tests and change only under SemVer.
Expand Down
2 changes: 1 addition & 1 deletion docs/en/guides/folder-media.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# The LIFT folder: ranges and media

A LIFT lexicon is usually a _folder_: the `.lift` file, one or more `.lift-ranges` companions, and `audio/` / `pictures/` media.
A LIFT lexicon is usually a _folder_: the `.lift` file, one or more `.lift-ranges` companions (sidecar files), and `audio/` / `pictures/` media.

## Ranges

Expand Down
2 changes: 1 addition & 1 deletion docs/en/guides/large-files.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,5 +23,5 @@ with sil_lift.open_reader("big.lift") as reader, sil_lift.open_writer(
Notes:

- The writer's output is exactly what the full-document canonical serializer would produce for the same content — the two modes never drift apart.
- Streaming mode has no byte-passthrough layer: output is always canonical. Root-level residue — comments between entries and out-of-schema attributes on `<lift>` — is not carried; entries and the header are complete, residue included.
- Streaming mode reuses no source bytes: output is always canonical. Root-level LIFT residue — comments between entries and out-of-schema attributes on `<lift>` — is not carried; entries and the header are complete, residue included.
- If the body of an `open_writer` block raises, the file is left visibly unterminated (no closing `</lift>`) — a half-written lexicon must not look complete.
2 changes: 1 addition & 1 deletion docs/en/guides/lift-export-interop.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ sil-lift validate export.lift --strict --no-check-media --format json
```

- `--strict` makes warnings (not just errors) fail the run.
- `--no-check-media` skips the filesystem media-presence check, whose `missing-media` findings are noise when the audio/photo files aren't colocated with the `.lift` in CI.
- `--no-check-media` skips the filesystem media-presence check, whose `missing-media` findings are noise when the audio/photo files are not in the same folder as the `.lift` in CI.
- `--format json` prints a single JSON object (`{"problems": [...], "summary": {...}}`) instead of human text; its exit codes and schema are a supported, SemVer-covered interface (see [the command line guide](cli.md)).
- `--require-ids` additionally errors on entries missing a `guid` or senses missing an `id` — useful when a later re-import must update rather than duplicate.

Expand Down
Loading