Skip to content

Compare range-element parents under NFC - #28

Draft
imnasnainaec wants to merge 3 commits into
mainfrom
range-parent-nfc
Draft

Compare range-element parents under NFC#28
imnasnainaec wants to merge 3 commits into
mainfrom
range-parent-nfc

Conversation

@imnasnainaec

@imnasnainaec imnasnainaec commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Fixes #14. Fixes #27.

The range-parent check compared a range-element's parent against its siblings'
ids as exact strings. FLEx writes those two attributes with different Unicode
normalization — the grammatical-info and lexical-relation range-element ids
bypass its normalizing helper and come out NFD, while the parent attribute on the
same element goes through it and comes out NFC — so a sound parent link in a real
export was reported as an error. #14 reports this from a Sena export, #27 from
Sango; they are the same defect, and #27 adds the exporter-side mechanism.

Commits

1. Compare a range-element's parent against sibling ids under NFC. The nfc()
helper now covers the range-integrity check as well as undefined-range-value,
which already used it. Messages still quote the original spelling. Sango's two
range-parent errors were both this asymmetry, not dangling references.

2. Say which FLEx writes skip normalization, not which file. Three prose sites
and the changelog said FLEx writes the .lift in NFC and the .lift-ranges in NFD.
The corpus does not support that — in sango.lift-ranges every label, abbrev,
description and every parent is NFC, and so are the semantic-domain-ddp4,
translation-type, and usage-type ids. It is specific writes that skip the
normalizer, not the file.

3. Warn when a name resolves only after NFC normalization. Normalizing both
sides would otherwise make the encoding split invisible, and sil-lift never rewrites
the ids (fidelity), so nothing else would tell the user their data will not resolve
in a consumer that compares raw strings. New normalization-mismatch warning,
emitted once per range-element id however many references differ, covering both
parent links and range values, addressed to the file that defines the id, and
naming both spellings by code point ('Compléments' vs 'Compl\xe9ments')
since they render identically.

On the corpus: 6 warnings in Sango (2 parent links + 80 grammatical-info values
resolving to 6 ids), 0 in AllFLExFields, 0 everywhere else.

Fixtures and tests

negative/nfd-range-ids.lift + .lift-ranges, hand-authored per #14: NFD ids, NFC
parents and grammatical-info value, plus one parent that dangles in every
normalization to prove the check still fires. The test asserts one error, one
deduplicated warning, and that both files save byte-identically — normalization
belongs to the comparison, not to what gets written.

The dangling-ref sweep from #14

Checked: all 7 dangling-ref findings in the corpus (negative/dangling-ref.lift,
spec-examples/0.13/{fields any order,hugal-ideal,lexical-relations}.lift) are ASCII
and none resolve under NFC. Entry ids and relation/@ref are left comparing exactly;
duplicate-guid compares guids, which are ASCII by construction. No change made
without a case to justify it.

Verification

ruff check, ruff format --check, mypy, and the full suite (540 passed).

Recommendations for the upstream FieldWorks exporter — including a separate, more
serious escaping gap on the morph-type id — are in a comment on #27.


This change is Reviewable

imnasnainaec and others added 2 commits August 12, 2026 16:21
FLEx normalizes strings to NFC on export, but a few writes emit the NFD
it holds in memory -- among them the grammatical-info and lexical-relation
range-element ids, while the parent attribute on those same elements is
normalized. Comparing the two as exact strings reported a sound parent
link as dangling: both range-parent errors in the sango fixture are that
encoding difference, not defects.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The validation notes described the .lift as NFC and the .lift-ranges as
NFD. It is narrower than that: the normalizer is applied per write, so a
range-element id can be NFD beside a label, a parent, and a .lift value
that are all NFC.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
@imnasnainaec imnasnainaec self-assigned this Aug 12, 2026
Normalizing both sides of the range comparisons hid a real property of
FLEx output: an id and the names referring to it can differ in encoding,
and any consumer comparing raw strings -- a Send/Receive merge, say --
fails to resolve them. Report that as a warning, once per range-element
id however many references differ (6 rather than 82 in the sango
fixture), naming both spellings by code point because they render
identically, and addressing it to the file the id lives in.

The new negative pair carries the shape end to end: NFD ids, NFC parents
and grammatical-info value, one parent that dangles in every
normalization, and bytes that must survive the round trip unchanged --
normalization belongs to the comparison, not to what gets written.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant