Skip to content

Upload exactly the server-validated package - #1806

Open
PaulHax wants to merge 1 commit into
mainfrom
fm-stack-00-upload-package-authority
Open

Upload exactly the server-validated package#1806
PaulHax wants to merge 1 commit into
mainfrom
fm-stack-00-upload-package-authority

Conversation

@PaulHax

@PaulHax PaulHax commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

First of a four-PR stack. Foundation only: no attachment semantics or frame metadata yet.

Summary

Make server validation authoritative for the browser upload package.

image

Upload validated one interpretation of the selected files, then rebuilt the package in the browser
with separate rules. The two could disagree: an accepted file could be dropped, an ignored file
could still upload, and ordinary and multicamera imports kept different side files.

validate_files now returns semantic roles (media, annotations, dataset configuration) plus
ignored with a per-file reason. The browser uploads the selection minus the ignored names — one
derivation, no second classifier, same contract for ordinary and multicamera camera-folder uploads.
Files that upload today still upload; a skipped file is now listed with its reason instead of
vanishing.

The per-role slots stay; their contents and meaning change. processImport is deleted and
suggestUploadSlots replaces it as a convenience split that is never trusted — slots are
re-validated at Start upload, and the server decides media type, folder name, fan-out, and what
uploads. A rejected selection now yields a row in an error state rather than a discarded pick,
since the slot editor is the only place to fix it without reopening the OS picker.

Four intentional behavior changes ride along:

  • The media file dialog offers every side file a slot can hold. Its filter was built from the
    annotation extensions alone, so .yml, .yaml, .json, and .csv were selectable but .txt
    was not — the browser silently withheld a file the server would have classified, which is the
    drift this PR exists to remove. The filter is now the union of the annotation and metadata
    attachment extensions.
  • Response shape. The flat media/annotations lists are replaced by roles + ignored. Both
    in-tree consumers (browser upload, girder-worker zip path) are updated here; external callers of
    this endpoint must adopt the new shape.
  • Stricter annotation multiplicity. Five selections main accepted and then mishandled via
    silent last-write-wins or an ambiguous config pick are now rejected explicitly: two annotation
    JSONs, CSV + JSON, YAML + JSON, more than one YAML, two config JSONs. Everything else validates as
    before.
  • Row retirement. UploadGirder emitted a row index while Upload expected the row object, so
    splice(-1, 1) retired the last queued row instead of the finished one. A pre-existing main
    bug, fixed here because per-row error state makes retiring the wrong row destructive.

The zip path is unchanged: validation gates the whole archive and every extracted file uploads. Only
interactive upload drops ignored files — the same files main already dropped, silently.

Reviewer focus

  • Server and browser classification cannot drift after validation, and the file dialog no longer
    pre-empts it by withholding a selectable side file.
  • Ignored files stay visible below the pickers and never enter the upload request.
  • Accepted packages behave as on main; the five new rejections are the intended change.
  • Multicamera flattening preserves every server-approved file, including the annotation and config
    side files main's media-only path dropped.
  • When a camera-folder file and a separately picked file share a name, the picked file uploads and
    the displaced folder copy is reported as ignored. Re-picking the folder's own file displaces
    nothing.
  • An edited slot changes what uploads: the package is rebuilt and re-validated at Start upload.
  • An error row stays correctable — slots editable, media slot unfiltered (the row's type is only the
    import menu opened until the server confirms it), never uploads into an unnamed folder.

Manual test

Uses okeanos-upload-authority-ignore from okeanos-frame-metadata-fixtures.zip, attached to this
PR — CC0 NOAA Okeanos Explorer media with synthetic metadata and annotations. Select all files in
the directory.

okeanos-frame-metadata-fixtures.zip

  1. Select all 17 files; confirm the 16 JPEGs validate as image-sequence media.
  2. Confirm notes.txt is listed under "Ignored (not uploaded)", below the pickers, with the
    unsupported-side-file reason.
  3. Create and open the dataset: 16 frames, and notes.txt uploaded as neither media, annotation,
    nor dataset config.
  4. Repeat through the folder/multicamera surface — identical ignored result and media set.

Classify each selected file into a semantic upload role on the server, then
upload exactly what that classification accepts and report the rest, instead
of re-deriving the set in the browser.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant