feat(pose): SLEAP and DeepLabCut become installable plugins - #148
Merged
Conversation
Core runs ONNX; a plugin turns one vendor's folder into it. That division is the point of this change, and it is what lets each vendor's framework stay out of GLIDER's dependency tree -- TensorFlow is a large thing to hand a lab that tracks with YOLO and has never opened SLEAP. A new `glider.pose` entry-point group and a POSE_CONVERTERS registry, shaped exactly like the driver, device and node registries already there. A converter answers three questions about a folder from *paths alone* -- is it mine, is what I made from it still current, please convert it -- because it is asked on every model selection, including for folders belonging to other vendors, and importing TensorFlow to decide that would cost seconds on the common path where the answer is no. glider-sleap is the existing core converter moved out, unchanged in behaviour: point at the folder SLEAP wrote and it still converts on first selection. `pip install glider-sleap` replaces the `glider[sleap]` extra, which is gone. It still does not depend on sleap. The dependency floors moved with it and are not cosmetic -- tensorflow-cpu>=2.20 because earlier releases publish no 3.13 wheels and would have capped the plugin below GLIDER's own Python range, and tf2onnx>=1.17 because 1.16 cannot convert the Keras 3 model any usable TensorFlow loads a checkpoint into. glider-dlc is new, and carries no dependencies at all. DeepLabCut is needed to read a snapshot -- a checkpoint is a bare PyTorch state_dict, so rebuilding the network needs DLC's own model classes -- but it brings 1.3 GB with it. So the first conversion builds a private Python 3.12 environment under ~/.glider/envs with uv and converts in there. A new optional `preflight` hook puts that size in the dialog before anything is downloaded; it is deliberately not on the converter Protocol, so adding it cannot make an existing converter stop satisfying the check. GLIDER_DLC_ENV points at an existing DeepLabCut venv and skips it all. Two things in the DLC converter are worth knowing about: The output stride is measured from the network, not read from its config. It is the backbone's stride divided by whatever the head's deconvolutions undo, and a ResNet-50 and an HRNet-w32 trained on the same data differ by a factor of four. Measured from *two* input sizes, because every DLC head adds one cell past the edge (out = in/stride + 1) -- a single ratio reads a stride-8 ResNet as 7.53 at 128 px and 7.88 at 512 px. The snapshot is looked for beside its own config rather than anywhere under the selected folder. A project with two training runs has two of each, and pairing one run's config with another's weights is not an error that surfaces: load_state_dict accepts it whenever the shapes match, and the model then answers with a network nobody trained for this. The search is also depth-bounded, because a DLC project holds labeled-data with thousands of frames and this runs on every model selection. Nothing in either sidecar is defaulted. A wrong stride or normalisation does not fail -- it shifts every keypoint by a constant and still draws a plausible skeleton, which is the failure worth refusing outright. Verified against DeepLabCut 3.0.1 (torch 2.13, py3.12) for both backbone families: ResNet-50 at stride 8 and HRNet-w32 at stride 2, each matching DLC's own inference to within 3.1e-06 relative, exporting with dynamic height and width, and correctly reading a retrained snapshot as stale. SLEAP conversion was verified earlier against SLEAP's own minimal_robot.UNet.single_instance fixture to 7e-7. Also here: - Release workflows for glider-maimu, glider-sleap and glider-dlc, using Trusted Publishing like glider-harp already does. Each asserts the built wheel's entry-point group and name, which is the one failure that is otherwise invisible: a plugin registered under the wrong name loads without error and does nothing, which is how glider-maimu shipped once. Three near-identical files on purpose -- PyPI matches its publisher against the workflow requesting the token and does not support reusable workflows (warehouse#11096). - glider-maimu added to the bundled plugin catalogue. It was installable but absent from it, so it never appeared in the Plugins window at all.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four installable plugins, and pose conversion becomes an extension point.
Core runs ONNX. A plugin turns one vendor's folder into it. That division is
the change; everything else follows from it. It is what lets each vendor's
framework stay out of GLIDER's dependency tree — TensorFlow is a large thing to
hand a lab that tracks with YOLO and has never opened SLEAP.
glider-harpharp-protocolglider-maimubleakglider-sleaptensorflow-cpu,tf2onnxglider-dlcThe extension point
A new
glider.poseentry-point group and aPOSE_CONVERTERSregistry, shapedexactly like the driver, device and node registries already there.
A converter answers three questions about a folder — is it mine, is what I made
from it still current, please convert it — and answers the first two from
paths alone. That constraint is the load-bearing one:
claims()is asked onevery model selection, including for folders belonging to other vendors, and
importing TensorFlow to decide would cost seconds on the common path where the
answer is no. There is a test for it in each plugin.
Core needed no decoder changes. The sidecar already parameterises both
decoders, and YOLO stays in core.
glider-sleap
The existing core converter, moved out, unchanged in behaviour: point at the
folder SLEAP wrote and it still converts on first selection and runs.
pip install glider-sleapreplaces theglider[sleap]extra, which is gone.It still does not depend on
sleap, and does not need to — a SLEAP model is anordinary Keras checkpoint.
The dependency floors moved with it and are not cosmetic:
tensorflow-cpu>=2.20— earlier releases publish no cp313 wheels, so a lowerfloor would silently cap this plugin below GLIDER's own Python range. (I had
it capped at
<3.13first, on a stale belief that TF has no 3.13 wheels atall. It does, from 2.20.)
tf2onnx>=1.17— 1.16 cannot convert a Keras 3 model, which is what anyTensorFlow new enough to be worth installing loads a checkpoint into.
Verified end to end on tensorflow-cpu 2.21 + tf2onnx 1.17.
glider-dlc
New, and carries no dependencies at all. DeepLabCut is needed to read a
snapshot — a checkpoint is a bare PyTorch
state_dict, so rebuilding thenetwork needs DLC's own model classes — but it brings about 1.3 GB with it.
So instead of carrying that, the first conversion builds a private Python 3.12
environment under
~/.glider/envs/deeplabcutwithuvand converts in there.GLIDER's own environment is untouched.
GLIDER_DLC_ENVpoints at an existingDeepLabCut venv and skips the download entirely.
A new optional
preflighthook puts the size in the confirmation dialog beforeanything downloads. It is deliberately not on the converter
Protocol, soadding it cannot make an existing converter stop satisfying the check.
Two things in the converter worth a reviewer's attention:
The output stride is measured from the network, not read from its config.
It is the backbone's stride divided by whatever the head's deconvolutions undo,
and a ResNet-50 and an HRNet-w32 trained on the same data differ by a factor of
four. It is measured from two input sizes, because every DLC head adds one
cell past the edge (
out = in/stride + 1) — a single ratio reads a stride-8ResNet as 7.53 at 128 px and 7.88 at 512 px. The difference cancels the
constant and lands on 8 exactly.
The snapshot is looked for beside its own config, not anywhere under the
selected folder. A project with two training runs has two of each, and pairing
one run's config with another's weights is not an error that surfaces:
load_state_dictaccepts it whenever the shapes happen to match, and the modelthen answers with a network nobody trained for this. The search is also
depth-bounded — a DLC project holds
labeled-datawith thousands of frames init, and this runs on every model selection.
Nothing in either sidecar is defaulted. A wrong stride or normalisation does not
fail; it shifts every keypoint by a constant and still draws a plausible
skeleton, which is the failure worth refusing outright.
Verified against DeepLabCut 3.0.1
torch 2.13, Python 3.12, both backbone families:
Relative, not absolute: an untrained HRNet emits values around 1e7, where an
absolute tolerance says nothing about whether the graph matches.
PyPI
glider-maimu,glider-sleapandglider-dlceach get a release workflowusing Trusted Publishing, exactly as
glider-harpalready does — no API tokenexists anywhere. All three names are available on PyPI; all three build and pass
twine checklocally.Each release asserts the built wheel's entry-point group and name. That is
the one failure that is otherwise invisible: a plugin registered under the wrong
name loads without error and does nothing, which is how
glider-maimushippedonce. The assertion was tested against a deliberately-corrupted wheel and
catches exactly that shape.
Three near-identical workflow files on purpose. PyPI matches its trusted
publisher against the workflow that requests the token and does not support
reusable workflows (warehouse#11096),
so factoring the shared body into a
workflow_callwould stop publishingworking at all.
Before the first release, three pending publishers are needed
At https://pypi.org/manage/account/publishing/ → Add a new pending
publisher, once per plugin:
glider-maimuLaingLabgliderrelease-glider-maimu.ymlpypiglider-sleapLaingLabgliderrelease-glider-sleap.ymlpypiglider-dlcLaingLabgliderrelease-glider-dlc.ymlpypiThen, after merging:
Each workflow also has a
workflow_dispatch→ TestPyPI option for a dry runfirst. A tag that disagrees with the declared version fails before the build,
because PyPI never lets a version be re-uploaded.
Also
glider-maimuadded to the bundled plugin catalogue. It was installable butabsent from it, so it never appeared in the Plugins window at all — which is
what you hit earlier.
Scope
Single animal, as agreed. Multi-animal SLEAP models (top-down, bottom-up) and
multi-animal DLC heads (
paf,identity) are reported by name rather thanconverted wrongly. DeepLabCut 2.x is TensorFlow rather than PyTorch and is not
converted automatically; a 2.x folder is recognised and says so rather than
reporting a missing
pytorch_config.yaml.Testing
4438 passed, 4 skipped, 0 failures across core and all four plugins. Ruff
and black clean.
The plugin suites run without the frameworks they convert for — that is the
point of the plugins, so it is also the point of the tests.
glider-dlcfakesevery
uvcall and checks the decision-making around them: when to build, whennot to, what to do with a half-built environment, and what a person is told when
it fails. The one test that performs a real DLC conversion is marked
slowandskips wherever DeepLabCut is not importable, including CI.
CI installs all four plugins.
glider-sleapgoes in with--no-deps— its oneTensorFlow test skips without it, and downloading TF on nine matrix jobs to skip
one test is not a trade worth making.
One PR, not two
I offered to split this into extension-point-plus-SLEAP and then DLC. I have
kept it as one: the DLC half is what motivates
preflight, and reconstructinga green intermediate state I never actually ran the suite against would be
inventing a commit rather than splitting one. Happy to split if you would
rather review it in halves.