Compare range-element parents under NFC - #28
Draft
imnasnainaec wants to merge 3 commits into
Draft
Conversation
FLEx normalizes strings to NFC on export, but a few writes emit the NFD it holds in memory -- among them the grammatical-info and lexical-relation range-element ids, while the parent attribute on those same elements is normalized. Comparing the two as exact strings reported a sound parent link as dangling: both range-parent errors in the sango fixture are that encoding difference, not defects. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The validation notes described the .lift as NFC and the .lift-ranges as NFD. It is narrower than that: the normalizer is applied per write, so a range-element id can be NFD beside a label, a parent, and a .lift value that are all NFC. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Normalizing both sides of the range comparisons hid a real property of FLEx output: an id and the names referring to it can differ in encoding, and any consumer comparing raw strings -- a Send/Receive merge, say -- fails to resolve them. Report that as a warning, once per range-element id however many references differ (6 rather than 82 in the sango fixture), naming both spellings by code point because they render identically, and addressing it to the file the id lives in. The new negative pair carries the shape end to end: NFD ids, NFC parents and grammatical-info value, one parent that dangles in every normalization, and bytes that must survive the round trip unchanged -- normalization belongs to the comparison, not to what gets written. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #14. Fixes #27.
The
range-parentcheck compared arange-element'sparentagainst its siblings'ids as exact strings. FLEx writes those two attributes with different Unicodenormalization — the
grammatical-infoandlexical-relationrange-elementidsbypass its normalizing helper and come out NFD, while the
parentattribute on thesame element goes through it and comes out NFC — so a sound parent link in a real
export was reported as an error. #14 reports this from a Sena export, #27 from
Sango; they are the same defect, and #27 adds the exporter-side mechanism.
Commits
1. Compare a range-element's parent against sibling ids under NFC. The
nfc()helper now covers the range-integrity check as well as
undefined-range-value,which already used it. Messages still quote the original spelling. Sango's two
range-parenterrors were both this asymmetry, not dangling references.2. Say which FLEx writes skip normalization, not which file. Three prose sites
and the changelog said FLEx writes the
.liftin NFC and the.lift-rangesin NFD.The corpus does not support that — in
sango.lift-rangesevery label, abbrev,description and every
parentis NFC, and so are thesemantic-domain-ddp4,translation-type, andusage-typeids. It is specific writes that skip thenormalizer, not the file.
3. Warn when a name resolves only after NFC normalization. Normalizing both
sides would otherwise make the encoding split invisible, and sil-lift never rewrites
the ids (fidelity), so nothing else would tell the user their data will not resolve
in a consumer that compares raw strings. New
normalization-mismatchwarning,emitted once per range-element id however many references differ, covering both
parentlinks and range values, addressed to the file that defines the id, andnaming both spellings by code point (
'Compléments'vs'Compl\xe9ments')since they render identically.
On the corpus: 6 warnings in Sango (2 parent links + 80
grammatical-infovaluesresolving to 6 ids), 0 in AllFLExFields, 0 everywhere else.
Fixtures and tests
negative/nfd-range-ids.lift+.lift-ranges, hand-authored per #14: NFD ids, NFCparents andgrammatical-infovalue, plus one parent that dangles in everynormalization to prove the check still fires. The test asserts one error, one
deduplicated warning, and that both files save byte-identically — normalization
belongs to the comparison, not to what gets written.
The
dangling-refsweep from #14Checked: all 7
dangling-reffindings in the corpus (negative/dangling-ref.lift,spec-examples/0.13/{fields any order,hugal-ideal,lexical-relations}.lift) are ASCIIand none resolve under NFC. Entry ids and
relation/@refare left comparing exactly;duplicate-guidcompares guids, which are ASCII by construction. No change madewithout a case to justify it.
Verification
ruff check,ruff format --check,mypy, and the full suite (540 passed).Recommendations for the upstream FieldWorks exporter — including a separate, more
serious escaping gap on the
morph-typeid — are in a comment on #27.This change is