[NEP-001 Feature] byte-identical port np.save/np.load and variants in NumPy 2.4.2#617
Merged
Conversation
Nucs
changed the title
…tibility
Completely rewrites the .npy/.npz file I/O implementation to match NumPy's
_format_impl.py exactly. This is a breaking change that drops backwards
compatibility with old NumSharp-specific behaviors in favor of full NumPy
interoperability.
New Implementation (NumSharp.IO namespace):
- NpyFormat.cs: Core binary format implementation
- Supports all three format versions: 1.0, 2.0, and 3.0
- Correct 64-byte header alignment for memory-mapping
- Proper handling of magic string \x93NUMPY
- Little-endian header length encoding
- Python dict literal format for headers
- C-order and F-order data layout support
- Byte-order conversion for cross-platform files
- All NumSharp dtypes: bool, byte, int16/32/64, uint16/32/64, float32/64
- NpzFile.cs: Lazy-loading NPZ archive container
- Dictionary-like interface (IReadOnlyDictionary<string, NDArray>)
- Lazy loading - arrays loaded on first access
- Both key formats work: "arr_0" and "arr_0.npy"
- Proper IDisposable pattern for resource cleanup
- Zip64 support for large archives
API Changes (np.* functions):
- np.save(file, arr) - Save single array to .npy
- np.load(file) - Auto-detect and load .npy or .npz
- np.load_npy(file) - Explicit .npy loading (returns NDArray)
- np.load_npz(file) - Explicit .npz loading (returns NpzFile)
- np.savez(file, arrays) - Save multiple arrays uncompressed
- np.savez_compressed(file, arrays) - Save multiple arrays compressed
- Stream and byte[] overloads for all operations
- Named arrays: savez(file, new Dictionary<string, NDArray>{...})
Key Technical Details:
- Magic: \x93NUMPY (0x93 followed by "NUMPY")
- Version 1.0: 2-byte header length (max 65KB), latin1 encoding
- Version 2.0: 4-byte header length (max 4GB), latin1 encoding
- Version 3.0: 4-byte header length, UTF-8 encoding
- Header aligned to 64 bytes for mmap compatibility
- Auto-version selection: tries 1.0 first, upgrades if needed
- Proper stream position handling for multi-array files
Breaking Changes:
- Removed: np.Save(), np.Load<T>(), np.Save_Npz(), np.Load_Npz<T>()
- Removed: NpzDictionary<T> class (replaced by NpzFile)
- Changed: np.load() returns object (NDArray or NpzFile)
- Removed: LoadMatrix(), LoadJagged() methods
Tests updated to use new API and verify NumPy format compliance.
Adds tests that load actual .npy/.npz files created by Python NumPy to verify the implementation correctly handles real-world NumPy files. Test files created by NumPy (in test_compat/): - int32_1d.npy: 1D int32 array - float64_2d.npy: 2D float64 array (3x4) - bool.npy: Boolean array - scalar.npy: True NumPy scalar (shape=(), ndim=0) - empty.npy: Empty float64 array - fortran.npy: Fortran-order (column-major) array - multi.npz: Uncompressed NPZ with multiple arrays - compressed.npz: Compressed NPZ archive All 9 tests pass, verifying: - Correct magic string parsing (\x93NUMPY) - Version 1.0 header format - dtype conversion (int32, float64, bool, int64) - Shape parsing including scalars and empty arrays - Fortran-order data layout conversion - NPZ archive loading (compressed and uncompressed) - Named array access in NPZ files Also verified that NumPy can load files created following our format by manually constructing a .npy file and loading it with np.load().
Adds tests verifying correct handling of .npy format versions: Version 2.0 (large headers): - 4-byte header length field correctly parsed - Successfully read 92KB header with 4000+ structured fields - Fails on dtype parsing (structured arrays unsupported), not format Version 3.0 (UTF-8 encoding): - UTF-8 header decoding works correctly - Unicode field names decoded: 'données', '数据', 'данные', 'δεδομένα' - Fails on dtype parsing, not format Also verifies: - Version 1.0 used by default for simple arrays - 64-byte header alignment (data starts at offset 128) Moved test_compat files to test project directory for correct discovery.
Bug fix: - WriteNonContiguousData was not accounting for Shape.offset, causing sliced arrays with non-zero offset (e.g., arr[1:3, 1:4]) to save incorrect data - The coordinate-based iteration started from offset 0 instead of the slice's actual offset into the underlying storage buffer Test fixes: - NumpyCompatibilityTests: Fixed GetDouble/GetInt32 calls for 2D arrays (use coordinate indexing, not flat indexing) - NumpyCompatibilityTests: Use GetInt64 for fortran.npy since NumPy defaults to int64 New tests (NpyFormatEdgeCaseTests.cs): - Sliced arrays with offset - Column slices (non-contiguous) - Step slices (e.g., arr[::2]) - Transposed arrays - Broadcast views - Special float values (NaN, Inf, -Inf) - Header format compliance (key sorting, trailing comma, 64-byte alignment) - True scalars (ndim=0, shape=()) - Various multi-dimensional shapes - Empty arrays (1D and multi-dimensional) - Multiple arrays in single stream - Error handling (invalid magic, truncated file, unsupported Decimal type) - All supported dtypes round-trip Battle-tested edge cases verified working: - 1M element arrays (8MB) save/load in ~27ms - Compression ratio 274x for zeros array - NPZ byte array API - Lazy loading with caching
Changes to support long[] shape/strides in longindexing branch: NpyFormat.cs: - HeaderData.Shape: int[] -> long[] (matches NumPy's 64-bit indexing) - ParseHeader: Use long.TryParse for shape values - ReadArrayData: Use long[] for reversedShape in F-order handling - WriteContiguousData: Use long for totalBytes, remaining, offset - WriteNonContiguousData: Use long[] for shape, strides, coords - FormatHeaderDict: Add long[] support for shape serialization - Chunked I/O casts to int are safe (chunk size limited to 16MB) Test files: - Change int[] to long[] for shape comparisons - Cast size calculations to int where needed for .NET Array methods - Work around GetInt32 indexing issues via flatten
The growth axis padding code (which reserves space for in-place header modification when appending data) only handled int[] shapes. After the long indexing migration, shapes are long[] which caused the padding to be silently skipped. This fix handles both int[] and long[] shape types, ensuring the growth axis padding is applied correctly for arrays with any size dimensions.
The npsave rewrite was authored 2026-03-28 against a NumSharp that has since moved 810 commits. The rebase itself was clean for all four rewrite sources (NpyFormat.cs, NpzFile.cs, np.save.cs, np.load.cs are byte-identical to the pre-rebase tree); this commit carries only the adaptations that 810 commits of API drift made necessary. Core (2 errors): - np.save.cs: `arr == null` no longer yields bool — master's comparison operators return NDArray<bool> element-wise. Switched both guards to the house `is null` pattern, which bypasses operator overloading. Tests (framework drift): - master migrated TUnit -> MSTest v3. The rewrite's three added suites were still TUnit-only (they already used MSTest assertions, so only attributes moved): [Test] -> [TestMethod], [Test]+[Arguments(x)] -> [DataTestMethod]+ [DataRow(x)], [Before(Test)]/[After(Test)] -> [TestInitialize]/[TestCleanup], added [TestClass], dropped `using TUnit.Core`. - The two conflicted suites (np.load.Test.cs, np.save_load.Test.cs) resolved to the branch's content — a strict superset — ported to MSTest. Master's old-API tests (np.Save/np.Load<T>/np.Save_Npz/np.Load_Npz<T>) are dropped because the rewrite deletes that API; their coverage is carried by the renamed equivalents (e.g. SaveAndLoadWithNpyFileExt -> Save_AddsNpyExtension). Tests (dtype drift): - NpyFormatEdgeCaseTests.Save_Load_SlicedArray_WithOffset asserted via GetAtIndex<int>, but np.arange is Int64 on master -> Debug.Fail. Read back at <long>. The Shape.offset behaviour under test was already correct; only the accessor was stale. Callers master added since the fork (OrderSupport.OpenBugs.Tests.cs): - np.Save((Array)f, stream) -> np.save(stream, f); the old overload is gone. - np.load(stream) now returns object, so the three sites that touch .shape/ .Shape/[i,j] use np.load_npy(stream) (the typed accessor). - Dropped [OpenBugs] from NpLoad_NumPyFortranOrderTrue_DoesNotThrow: master documented "np.load.cs:322 throws on fortran_order: True" and the rewrite fixes it — the test is green, so per convention the attribute goes. - Repointed two stale [OpenBugs] comments from np.save.cs:172 to NpyFormat.HeaderDataFromArray, which still hardcodes fortranOrder=false. F-order WRITE remains unimplemented and correctly stays [OpenBugs]. Verified: NumSharp.Core builds net8.0 + net10.0; the rewrite's five suites are 80/80; full CI-filtered suite 11076 passed / 0 failed; OrderSupport OpenBugs went 13 -> 12 failing (the one fix above), nothing regressed. Pre-rebase state preserved at tag backup/npsave-pre-rebase.
…o NumPy 2.4.2 Closes the np.save/load rewrite (docs/plans/gh-issue-npy-npz-rewrite.md, issue #597). The old implementation was a years-old quick solution: System.Array-based, v1.0-only, 16-byte aligned, IndexOf/Substring header parsing, and it threw on fortran_order, big-endian, and any format version above 1.0. It is replaced wholesale by a port of NumPy 2.4.2's numpy/lib/_format_impl.py, structured function-for-function so the two can be diffed. The contract is stronger than "NumPy can read it": np.save output is BYTE-FOR-BYTE what NumPy's own np.save writes for the same array. 157 committed cases prove it. Core (src/NumSharp.Core/IO/) NpyFormat.cs the format: magic/version, dtype<->descr, header write/read, array write/read. Format versions 1.0/2.0/3.0 (2- vs 4-byte header length, latin-1 vs UTF-8), 64-byte ARRAY_ALIGN (was 16 — the data is now mmap-ready and SIMD-aligned), fortran_order written via the transpose and read via reshape-reversed+transpose, big-endian byte-swapped to native per COMPONENT (>c16 swaps in 8s, not 16s; >U1 in 4s), 256KB chunked I/O with a short-read loop (ZipExtFile/network streams). PyLiteral.cs a real Python-literal parser standing in for ast.literal_eval, replacing IndexOf/Substring/regex. Key order, whitespace, quoting, trailing commas and the Python 2 `3L` suffix (v<=2.0 only, as NumPy gates it) all parse. Deliberately NOT more permissive than literal_eval: bare nan/inf are Names, not literals, and are rejected the same way. NpzFile.cs replaces NpzDictionary<T> and its 6-interface generic constraint. Lazy + cached, "w" and "w.npy" both resolve, .Files strips ".npy", npz.f.weights dot access (NumPy's BagObj), IDisposable. Bugs fixed along the way - `|u1` was mapped to SByte, and `|i1` was WRITTEN for Byte — both simply wrong. - Padding: NumPy emits a FULL 64 bytes when the header is already aligned, never 0. The prior art's `if (padlen == ARRAY_ALIGN) padlen = 0` diverges. - Magic \x93 is byte 147; the old reader compared BinaryReader.ReadChar() to '?' (63) and only worked by accident. - latin-1 encoding must THROW to select v3.0; Encoding.Latin1 silently substitutes '?'. API (NumPy signatures) np.save(file, arr, allow_pickle=true) · np.savez / np.savez_compressed (positional arr_N + named, with NumPy's collision check) · np.load(file, mmap_mode, allow_pickle, fix_imports, encoding, max_header_size). np.load returns object — NDArray for .npy, NpzFile for .npz — mirroring NumPy's content-dependent return (it dispatches on magic bytes, not extension); np.load_npy / np.load_npz are the typed forms and each reports a directed error if handed the other kind. Save_Npz/Load_Npz/LoadMatrix/LoadJagged/Load<T>/NpzDictionary<T> are gone. Security (both were missing entirely) max_header_size (default 10000) bounds hostile header parsing; allow_pickle=false is the default and lifts the guard only when the caller declares the file trusted. Dtypes — all 15 handled or explicitly rejected Boolean/SByte/Byte -> |b1/|i1/|u1 (single-byte types take `|`, never `<`); Int16..UInt64, Half(<f2), Single, Double direct; Complex -> <c16 (<c8 widens on read); Char <-> <U1 (2-byte UTF-16 <-> 4-byte UCS-4, non-BMP rejected — it needs a surrogate pair and would corrupt the file); Decimal throws naming the fix. Object/structured/subarray/datetime64/ timedelta64/|S/<U(n>1)/|V/<f16/<c32 parse then fail with a precise message. mmap_mode validates its value then throws NotImplementedException, with a step-by-step implementation sketch in CheckMmapMode (the blocker is UnmanagedMemoryBlock assuming it owns its allocation). Gates test/oracle/gen_npy_oracle.py generates IO/corpus/npy_oracle.zip: 217 cases of REAL NumPy 2.4.2 output — every dtype x {0-d, empty, 1-D, 2-D, 3-D, unit, empty-2d/3d} x {C, F, strided, reversed, offset, broadcast, transposed} x versions x endianness, plus 32 malformed/unsupported files. NpyOracleTests replays it under [NpyOracle] with no Python: read / byte-exact write / verbatim error / round-trip / npz / live-view write (views built in NumSharp, not rebuilt from bytes) / corpus non-vacuity. verify_npy_interop.py + interop_write.cs are the reverse manual gate — real NumPy reads 28 NumSharp-written files. This is the only way to prove .npz output, whose ZIP framing legitimately differs from Python's zipfile and cannot be byte-compared. Two corpus expectations were wrong and got corrected against NumPy, not worked around: np.load reports the pickle message for a corrupt magic (only read_array/load_npy reports the magic error), and asfortranarray promotes 0-d to shape (1,) (documented ndim>=1). 4 fortran_order [OpenBugs] repros in OrderSupport now pass and lost the marker. 11,380 tests green on net8.0 and net10.0.
…load
Adversarial sweep of the NEP-01 port against real NumPy 2.4.2 across header grammar,
dtype descriptors, value bit-patterns, npz semantics and hostile input. 62/67 exotic
header+dtype cases already agreed; the 5 that do not are the documented unsupported
dtypes. Three real defects fell out, each now pinned in the corpus.
1. `(1)` was parsed as a 1-tuple (PyLiteral.ParseTuple)
In Python it is the COMMA that makes a tuple, not the parentheses: `(1)` is the int 1
in grouping parens, `(1,)` is a 1-tuple. So NumPy rejects `'shape': (1)` with
"shape is not valid: 1" while NumSharp happily loaded it as shape (1,) — silently
accepting a file NumPy refuses. ParseTuple now tracks sawComma and collapses a
comma-less single element back to the value. `[1]` is unaffected: brackets always
build a list.
2. Duplicate npz member names resolved to the WRONG entry (NpzFile)
A zip may legally hold two entries with the same name and the runtimes disagree on the
winner: .NET's ZipArchive.GetEntry returns the FIRST, Python's zipfile builds a
name->info dict while scanning so the LAST wins. Verified against zipfile: z['dup'] is
[2], NumSharp returned [1]. The key map now holds ZipArchiveEntry references assigned
in scan order (last wins), and the cache is keyed by entry identity so duplicates
cannot alias. .files still lists both, as NumPy's does.
3. A hostile header-length field was handed straight to the allocator (NpyFormat.ReadBytes)
`new byte[count]` where count comes from the FILE: a 28-byte file claiming a 1 GB header
made NumSharp allocate 1,000,266,632 bytes to then reject it — ~35,000,000x amplification.
NumPy shrugs the same file off in ~2 KB because fp.read(n) only allocates what it
returns. ReadBytes now grows as it reads, so the cost is bounded by BUFFER_SIZE instead
of by the attacker's number: 1,000,266,632 -> 519,128 bytes (~1900x less). This also
IMPROVES parity — the >int.MaxValue special case is gone, so a 4 GB claim now produces
NumPy's verbatim "EOF: reading array header, expected 4294967280 bytes got 16".
Also: '<U0'/'|S0'/'|V0' are VALID zero-width NumPy dtypes, so reporting them as "not a
valid dtype descriptor" was untrue — they are unsupported, which is a different thing.
And PyLiteral no longer accepts bare nan/inf: they are Names, not literals, so
ast.literal_eval rejects them and accepting them made the parser more permissive than the
thing it stands in for (it was also asymmetric — `-inf` never parsed).
Corpus 217 -> 281 cases; the gate grew a `header` kind and an allocation test:
* value fidelity (32 cases): NaN payloads incl. sNaN and custom payloads, +/-0.0,
subnormals, integer extremes at every width, complex NaN/inf per component, and the
big-endian conversion paths where a byte copy cannot save you — '>c8' must swap in 4s
AND widen, '>c16' swaps in 8s (per component, NOT 16s), '>b1'/'>i1' must no-op.
* BMP seams for Char incl. U+0000/U+FFFF, plus astral + lone-surrogate rejection.
* `kind: "header"` (29 cases) reaches two branches NO real array can. The growth padding
is normally INVISIBLE — shrink the body 5 chars and the alignment padding grows 5, so
the file is unchanged — meaning a wrong growth axis (shape[0] vs shape[-1]) passes
every ordinary test. It only shows on shapes that tip the header across a 64-byte
bucket, and those need 10^17 elements. Same for the v1.0->v2.0 auto-selection boundary
at 21,817 dims. Both verified byte-exact against _write_array_header.
* HostileHeaderLength_DoesNotAllocateTheClaim guards #3; the error-message cases would
stay green while allocating 1 GB, so the resource behaviour needs its own test.
Two generator bugs fixed while investigating, both of which would have masked real
divergence: ns_bytes for '<U' used "".join(tolist()), but NumPy's <U treats NUL as string
padding so a U+0000 element reads back as '' and joining DROPPED it (the expectation came
out an element short of the file); and the complex64-widening check compared dtypes before
normalizing byte order, so '>c8' missed it.
11,382 tests green on net8.0 and net10.0; NumPy still reads all 28 interop files.
…-test both
Re-examined the two fixes from 2da32a48. Both were directionally right and both were
incomplete — the npz one was still wrong on a case it never tested, and the allocation one
still reserved a fixed 256 KB where NumPy uses ~2 KB.
1. npz name map: one interleaved loop is not NumPy's two passes (NpzFile)
The previous fix made lookups last-wins, which fixed duplicate names but silently kept a
second, subtler divergence. NumPy builds the map in TWO passes over ALL entries:
self._files = dict(zip(self.files, _files)) # pass 1: stripped -> entry
self._files.update(zip(_files, _files)) # pass 2: full -> entry
The passes are load-bearing. An archive may hold BOTH 'a' and 'a.npy', which both strip to
the key 'a'; pass 2 then re-points 'a' at the entry literally NAMED 'a'. A single loop
assigns both mappings per entry and resolves 'a' to whichever came last instead. Probed
against zipfile: for entries ['a', 'a.npy'] NumPy gives entry 0, the single loop gave 1.
Now a faithful two-pass port. 7 archive shapes x 22 lookups verified against real NumPy:
duplicates, stripped/full collisions in both orders, both at once, a triple, and ''/'.npy'.
2. Hostile header length: bounded is not the same as small (NpyFormat.ReadBytes)
The previous fix stopped allocating the claim but still reserved a BufferSize chunk, so a
28-byte file cost 519,128 bytes to reject vs NumPy's ~2,147. The claim never needed to size
anything: a seekable stream states exactly how many bytes remain, which is a hard upper
bound on what any claim can deliver, and a non-seekable one can start small and double as
bytes actually arrive. 1,000,266,632 -> 519,128 -> 1,160 bytes, now BELOW NumPy's ~2,147.
A well-formed 118-byte header still reads in one exactly-sized allocation (no chunking, no
MemoryStream, no final copy), so the fast path got cheaper too, and all three hostile claims
(4 GB / 1 GB / v1.0's 65535) report NumPy's message verbatim.
Both fixes are mutation-tested — a test that cannot fail proves nothing:
* Collapsing the two passes back into one loop -> Npz_ReadsNumPyArchives fails on
npz_stripped_and_full and npz_names_interleaved, naming the mechanism.
* Restoring `new byte[count]` -> HostileHeaderLength fails reporting the exact
1,000,001,264-byte allocation.
The allocation guard now covers all three hostile claims and its bound is 64 KB (was a
placeholder 100 MB, which the mutation would have squeaked under had the claim been smaller).
Corpus 281 -> 286 cases (npz 9 -> 14). npz cases gained `suffix_alias`, since in an ambiguous
archive npz["k"] and npz["k.npy"] resolve to DIFFERENT entries on purpose — asserting they
alias is exactly wrong there. Expectations for those cases are taken from real NumPy at
generation time rather than hand-written, so the oracle cannot drift from what zipfile does.
11,382 tests green on net8.0 and net10.0; NumPy still reads all 28 interop files.
…ering
Answering "can we do a 5 GB file of bytes?" by trying it. .npy: yes, first time — 5 GiB
round-tripped clean, every 32-bit boundary intact, NumPy read it, and the header came out
byte-identical to NumPy's own. .npz: NumSharp WROTE a valid 5 GiB Zip64 archive that NumPy
read perfectly, and then could not read its own file.
IOException: Stream was too long.
NpzFile buffered each member into a MemoryStream so the magic could be sniffed by seeking
(NumPy does `bytes.read(6); bytes.seek(0)`, which Python's ZipExtFile supports and .NET's
forward-only entry stream does not). MemoryStream caps at 2 GB, so any member past that was
unreadable — a write/read asymmetry, the worst kind: you can produce data you cannot consume.
It also made every ordinary load pay twice, buffer plus the array built from it.
NpyFormat.ReadArray is strictly forward-only, so no buffer is needed at all: sniff the 6 magic
bytes on one entry.Open() and stream the reader from a fresh one (entries may be opened any
number of times in Read mode; the sniff only ever inflates 6 bytes). Now:
* 5 GiB npz member loads in 1.8s where it previously threw.
* A 64 MB member costs 280 KB of managed memory instead of ~64 MB (0.004x, was ~2x).
Everything else at 5 GiB was already correct and is now pinned rather than assumed: every
(int) cast in the IO layer is bounded by BUFFER_SIZE, array data is unmanaged, and both paths
stream in 256 KB chunks — so no managed buffer scales with the array. Zip64 comes from .NET's
ZipArchive on demand (NumPy forces force_zip64=True for the same reason, gh-10776).
Tests
NpyLargeFileTests:
* Npz_StreamsMembers_RatherThanBufferingThem — the CI-cheap guard for the 5 GiB bug: a
16 MB member whose managed allocation must not track its size. Mutation-tested — restoring
the buffering fails it at 4.01x the member.
* Npy_5GiB_RoundTrips / Npz_5GiBMember_RoundTrips — [HighMemory][LongIndexing], excluded
from CI (~8 GB RAM + ~6 GB disk), probing 0 / int.MaxValue+-1 / 3e9 / uint.MaxValue+-1 /
last element. The npz one asserts the archive really exceeds 4 GiB, or it would not be
exercising Zip64 at all.
Also hardened both allocation assertions after ONE unreproducible net8.0 failure (9 clean runs
after it, name not captured). GC.GetTotalAllocatedBytes is process-wide, so a background thread
allocating inside the window inflates a single reading and a real regression is indistinguishable
from a neighbour's garbage. Noise can only ever ADD, so AllocationTests.MinAllocated takes the
min of 5 rounds after a warm-up — the repo's own best-of-rounds benchmark rule. Both mutations
are still caught with the change, so the tests kept their teeth. 5x full net8.0 suite clean.
Known ceilings, now documented: the byte[]-returning conveniences (np.save(arr), np.savez(...)
-> byte[], NpzFile.GetRawBytes) stop at 2 GB. That is a byte[] limit, not a format one — use the
path/stream overloads above 2 GB.
11,383 tests green on net8.0 and net10.0; NumPy still reads all 28 interop files.
Surfaced by cherry-picking the rewrite onto this branch: the prior art's Load_TruncatedFile_ThrowsEndOfStreamException was the single failure out of 11,100, and it was the test that was wrong, not the code. NumPy draws a deliberate line here, verified against 2.4.2: np.load(b'\x93NUMPY') -> ValueError: EOF: reading magic string, expected 8 bytes got 6 np.load(b'') -> EOFError: No data left in file A truncated-but-present file is corrupt data (ValueError -> FormatException); EOFError is reserved for "there is nothing here at all". The prior art mapped both onto EndOfStreamException, which throws that distinction away — "this file is corrupt" and "the stream is finished" are different answers and a caller may well want to treat them differently (the second is normal when draining a multi-array stream, the first never is). Renamed to Load_TruncatedFile_ThrowsFormatException, now asserting NumPy's verbatim message, and added Load_EmptyFile_ThrowsEndOfStreamException to pin the other side of the line. 11,090 tests green on net8.0 and net10.0 on this branch.
Merge-readiness review of this branch. The rewrite itself was clean, but the check turned up
three things that had to be fixed before this could honestly be called ready.
1. 13 VACUOUS tests (NumpyCompatibilityTests, NpyFormatVersionTests)
Both resolve "test_compat" relative to the working directory and then SKIP on absence:
if (!TestFilesExist()) return; // NumpyCompatibilityTests
if (!File.Exists(path)) { Console.WriteLine(...); return; } // NpyFormatVersionTests
Nothing in the csproj ever copied test_compat/ to the output directory. They were green here
only because of a stale bin/ artifact dated 2026-03-28; on a clean checkout — i.e. CI — all 13
passed while touching no fixture at all. Proved it by deleting the fixtures: still 13/13 green.
A test that cannot fail is worse than no test, because it reports coverage that does not exist.
Fixed by copying test_compat\* in the csproj AND replacing both silent skips with loud
assertions naming the resolved path. Re-proved by deleting the fixtures again: now 13/13 RED.
With the fixtures actually loading, all 13 pass against the rewrite — so they are now real,
independent corroboration on hand-made NumPy files (v2.0, v3.0, fortran, scalar, empty, bool,
multi, compressed), alongside the 286-case generated corpus.
2. Duplicate fixtures at the repo root
The prior art committed test_compat/ twice: once at the root and once under
test/NumSharp.UnitTest/. The root copy is referenced by nothing and is not the one the csproj
copies. Removed (11 files).
3. Doc drift in .claude/CLAUDE.md
Said 217 cases in two places and 281 in two others; the corpus is 286. The 217s came from the
cherry-pick conflict resolution, the 281s predate the npz name-map cases. All four corrected,
and the IO test block now documents NpyLargeFileTests/AllocationTests plus a do-not-drop note
on the test_compat copy rule, since that rule is the only thing standing between these tests
and silent vacuity again.
State: worktree-npsave = master + 13 commits (8 prior-art + 5 mine); master is an ancestor, so
merging is a fast-forward with no conflicts possible. 11,090 tests green on net8.0 and net10.0,
286-case NumPy oracle green, NumPy reads all 28 interop files, and the 70 OpenBugs failures are
pre-existing repros in unrelated areas (no IO among them).
…ay guide The NEP-01 rewrite on this branch added np.savez, np.savez_compressed, np.load_npy and np.load_npz and changed np.load to return `object` (NDArray for .npy, NpzFile for .npz), but the user-facing NDArray guide still showed only the old two-line np.save/np.load example and a two-row persistence table. Bring both in line with the shipped API. Saving, Loading, and Interop: - Expand the example to the single-array .npy path (np.save + typed np.load_npy), the multi-array .npz path (savez positional arr_0/arr_1 and IDictionary keys; savez_compressed), and NpzFile use (using-dispose, ["w"]/"w.npy" keys, npz.f.w dot access, .Files stripped). - Add a paragraph: version 1.0/2.0/3.0 + C/F-order + big-endian coverage, the 64-byte-aligned mmap-ready byte-identical-to-NumPy writer, why np.load returns object (magic-byte dispatch like NumPy), and the allow_pickle=false default. Persistence & Buffers table: add savez, savez_compressed, load_npy, load_npz rows; note the Stream/byte[] overloads and that load returns object. Doc-only; no code change. Full dtype map + unsupported-type list lives in compliance.md (updated earlier on this branch). Both case-variant tracked paths of this file (NDArray.md and ndarray.md - a pre-existing case collision) are updated together so the tree stays consistent on case-sensitive checkouts.
…x 4 stale ones) Turn every code block in docs/website-src/docs/ndarray.md into a runnable MSTest so the guide cannot silently drift from the library. NDArrayDocExamplesTests has 38 tests, one per documented snippet: creation, frombuffer (byte/segment/memory/span/ native/endian), core properties, indexing/slicing/fancy/boolean, views-vs-copies, every operator (arith/bitwise/shift/compare/logical + the np.* forms C# lacks), dtype conversion + scalar casts, 0-d scalars, the four element read/write paths, foreach/flat iteration, reshape/transpose patterns, generic NDArray<T>, the NEW .npy/.npz I/O (save/load_npy, savez positional+dict, savez_compressed, NpzFile using/["w"]/"w.npy"/f.w/Files, np.load->object, tofile/fromfile), .NET-array interop, sameness helpers, and the troubleshooting snippets. Expected values were captured by running each snippet against this branch. Writing the tests surfaced four examples that had gone stale (the rebase onto master made arange int64) or were never valid C#; fixed in the doc AND pinned by the tests: - arange defaults to int64 now, not int32. Core Properties claimed dtype==typeof(int)/ Int32; Reading & Writing used <int> strict accessors (GetValue<int>/GetAtIndex<int>) that Debug.Assert-abort on an int64 array. Updated both to int64 (<long>, 99L) and noted the typed accessors must match the dtype exactly (the indexer, item<T> and the explicit cast convert; the strict getters do not). - np.arange(12).reshape(3,4).@base is NOT null - reshape returns a view (like NumPy). The comment claimed '// null (arange owns its data)'. - arr.@base != null does not return a bool: NDArray's ==/!= are element-wise and return an array. Documented the C# 'is not null' idiom in Views and the sameness table. - a[...] = a + 1 is not valid C# ('...' is not an expression). Corrected to a["..."]. Doc-only + test-only; no library change. The doc is tracked under two case-variant paths (NDArray.md / ndarray.md); both are updated together.
…ntics (+ fix 5 more) A second pass over docs/website-src/docs/ndarray.md: the first pass tested example VALUES; this one tests the view/copy/read-only SEMANTICS the guide's tables assert (the doc's most-emphasized topic). 11 new tests (49 total), and running them surfaced five more claims that were stale or never true - fixed in the doc and pinned: - Boolean masking a[mask] returns a WRITEABLE independent copy (IsWriteable=true; writing it leaves the parent untouched), matching NumPy's advanced-index result. The doc's view/copy callout called it a 'read-only copy'. - AsGeneric<T>() returns NULL on a dtype mismatch (the impl is 'return null', like C# 'as') - it does not throw. Fixed in the Generic table and the API-reference table. - MakeGeneric<T>() THROWS ArgumentException on a dtype mismatch (the doc was silent; it is the throwing one, not AsGeneric). Documented in both tables. - The troubleshooting header named ReadOnlyArrayException, which does not exist in the codebase. Writing a broadcast/read-only view throws 'NumSharpException: assignment destination is read-only' (NumPy's message). Header and the test now use the real type. - Another invalid-C# [...]: b.copy()[...] = value -> b.copy()["..."] = value. New tests, all from observed behaviour: frombuffer byte[]=view / big-endian=copy / length-not-multiple throws; plain-slice writeable view; fancy + boolean-mask writeable independent copies; the copy-semantics-at-a-glance table (T/ravel/reshape views, flatten/copy copies); generic mismatch (AsGeneric null, MakeGeneric throws) + storage sharing; integer-index dimension reduction (1-D->0-d, 2-D->1-D, 3-D->2-D); broadcast read-only; and every compound-assignment operator (+= -= *= %= &= |= ^= <<= >>= /=, with the int/int->double promotion). Doc-only + test-only; no library change. NOTE: the project .claude/CLAUDE.md still lists boolean masking as 'read-only' in its Indexing section - the same stale claim, left untouched here (out of scope for the website doc). Both case-variant paths of the doc file are updated together.
Nucs
changed the title
Nucs
changed the title
Nucs
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Complete rewrite of NumSharp's
.npy/.npzfile I/O as a port of NumPy 2.4.2'snumpy/lib/_format_impl.py(NEP‑01), structured function‑for‑function so the two can be diffed. The old implementation (a quick solution from years ago) only understood format v1.0, rejected Fortran‑order and big‑endian files, used the wrong (16‑byte) data alignment, mismapped|u1, and exposed aSave_Npz/Load_Npz<T>/NpzDictionaryAPI that matched neither NumPy nor .NET conventions.The new writer is byte‑for‑byte identical to NumPy's own
np.save— not merely readable by it — and this is a committed, tested contract.Closes #597.
What changed
New public API (NumPy‑shaped):
np.save(file, arr, allow_pickle=true)string/Stream/byte[]overloadsnp.savez(file, …)arr_0,arr_1, …) andIDictionary<string,NDArray>keysnp.savez_compressed(file, …)np.load(file, mmap_mode, allow_pickle=false, …, max_header_size=10000)object—NDArrayfor.npy,NpzFilefor.npz, dispatched on magic bytesnp.load_npy(...)/np.load_npz(...)NpyFormat_format_impl.pyport)NpzFile"w"/"w.npy"keys;.Filesstripped;npz.f.weightsdot access (NumPy'sBagObj);IDisposablePyLiteralast.literal_evalFormat coverage: versions 1.0 / 2.0 / 3.0 (2‑ vs 4‑byte header length; latin‑1 vs UTF‑8, auto‑selecting the oldest that fits) · 64‑byte
ARRAY_ALIGNso data is mmap‑ready ·fortran_orderboth ways (write via transpose; read via reshape‑reversed + transpose) · big‑endian byte‑swapped to native on read (per component:>c16in 8s,>U1in 4s) ·max_header_sizeDoS guard (default 10000, lifted byallow_pickle) · verbatim NumPy error messages.Dtype map:
|b1/|i1/|u1|, never<(fixes old|u1→sbyte bug)<i2…<u8,<f2,<f4,<f8<c16<c8widens toSystem.Numerics.Complexon read<U1NotSupportedException(no NumPy dtype)Breaking changes
Save_Npz/Load_Npz<T>removednp.savez/np.load_npz(no generic type param)NpzDictionary<T>removedNpzFile(lazy, cached, disposable)np.loadnow returnsobjectnp.load_npy/np.load_npznp.loadallow_pickledefaults tofalseKnown intentional divergences from NumPy
Differential‑verified; everything else agrees byte‑for‑byte. These are the only files NumPy loads that NumSharp refuses, plus a few representational deltas:
datetime64/timedelta64, byte‑strings\|S,<Un≠1, void\|V,float128<f16,complex256<c32— including the zero‑width<U0/\|S0/\|V0, which are valid NumPy dtypes.float64first).Char↔<U1:U+0000round‑trips as a NUL where NumPy's<Ureports''(bytes are identical —<Utreats NUL as string padding). Non‑BMP code points and unpaired surrogates are rejected rather than corrupted.complex64widens toComplex(float64 pair) on read; writes always emit<c16.shape: (True,)/ overflowing dims via a laterTypeError/OverflowError; both refuse, the text differs.np.loadreturnsobject— C# has no union type to express "ndarrayorNpzFile";load_npy/load_npzare the typed forms.mmap_modevalidates its value then throwsNotImplementedException..npzZIP framing legitimately differs from Python'szipfile, so it can't be byte‑compared — verified instead by reverse interop (real NumPy reading NumSharp's output).byte[]‑returning convenience overloads cap at 2 GB (np.save(arr),np.savez(...),NpzFile.GetRawBytes) — abyte[]limit, not a format one. The path/stream overloads round‑trip > 4 GiB (verified at 5 GiB, NumPy reads the result).Testing
test/oracle/gen_npy_oracle.py→IO/corpus/npy_oracle.zip— 286 committed cases of real NumPy 2.4.2 output (every dtype × {0‑d, empty, 1‑D/2‑D/3‑D, unit} × {C, F, strided, reversed, offset, broadcast, transposed} × versions {1.0, 2.0, 3.0} × {little, big, native} endian, plus NaN/sNaN payloads, subnormals, signed zero, integer extremes, BMP seams, and 42 malformed/hostile files with verbatim‑matched messages).NpyOracleTests([NpyOracle]) — replays the corpus: read / byte‑exact write / header‑only write / verbatim error / round‑trip / npz / live‑view write / hostile‑allocation. No Python at test time.NpyFormatVersionTests/NumpyCompatibilityTests— hand‑made NumPy fixtures (test_compat/: v2.0, v3.0, Fortran, scalar, empty, bool, multi, compressed).NpyFormatEdgeCaseTests,NpyLargeFileTests(2/4 GiB walls; 5 GiB pair is[HighMemory], excluded from CI),AllocationTests(hostile header allocates ~1.2 KB, not the claimed size).test/oracle/verify_npy_interop.py— manual reverse‑interop gate (NumSharp writes 28 files, real NumPy reads them) — the only way to prove.npzoutput.All IO/save/load tests pass in Release (
dotnet test --filter TestCategory=NpyOracleand the wider IO suite: 0 failures).Note for reviewers / merge
The branch is based on an older master point and is currently 27 commits behind
origin/master. A test‑merge shows all code merges cleanly; the only conflict is.claude/CLAUDE.md(both master and this branch append doc sections). Recommend a rebase/merge of master (resolving that one docs file) before final merge.