Skip to content

MAVLink: refine reconnect STATUSTEXT (per-peer tracking, snapshot rate limit) - #11731

Open
xznhj8129 wants to merge 2 commits into
iNavFlight:maintenance-10.xfrom
xznhj8129:mav/reconnect-refine
Open

MAVLink: refine reconnect STATUSTEXT (per-peer tracking, snapshot rate limit)#11731
xznhj8129 wants to merge 2 commits into
iNavFlight:maintenance-10.xfrom
xznhj8129:mav/reconnect-refine

Conversation

@xznhj8129

Copy link
Copy Markdown
Contributor

Follow-up to the reconnect STATUSTEXT handling merged in #11715, addressing review feedback on the reconnect-detection design.

The original commit tracked heartbeat presence with a single timestamp per physical port and sent an arming-reason snapshot on every judged reconnect. Review raised two soundness issues with that:

  • a steady peer on a port masks a newly joining peer behind the same port (their heartbeats keep the single timestamp fresh), and a peer failing over to another port is never noticed;
  • MAVLink leaves heartbeat rate channel-dependent, so a peer heartbeating slower than the 5 s gap threshold is classified as reconnecting on every beat, resending the arming notice each time.

Changes

  • Per-peer tracking. Heartbeat presence now lives in the route-table entry (sysid/compid), not per port. A steady peer no longer masks a new one; a peer moving to another port registers as a reconnect there. If the route table is full the new peer degrades gracefully to the pre-existing broadcast-only behaviour.
  • Snapshot rate limit. A 10 s per-port floor on arming snapshots, so a slow-heartbeat peer can't elicit a resend on every beat.
  • Dropped the enable-transition snapshot. It fired into a port nobody had connected to yet (boot) and duplicated the first-heartbeat snapshot on shared-port enable. Reconnect is now driven purely by the heartbeat signal; the enable transition still resets per-port one-shot state via the port reset path.

Testing

  • SITL builds clean with warnings-as-errors; mavlink_unittest 38/38 at this level (no new unit tests — behaviour is timing/multi-peer and covered by the live rig below).
  • Live SITL + pymavlink: existing reconnect regression 6/6, plus new cases — a 6.5 s-period peer is capped to 3 arming sends across 5 beats (was 5), and a second sysid joining a busy port gets its own snapshot while the steady peer produces none.

Built on the mav/04-streams-protocol content already in maintenance-10.x via #11715, so this shows a single commit.

@qodo-code-review

Copy link
Copy Markdown
Contributor

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

@sensei-hacker

Copy link
Copy Markdown
Member

Went through this one closely since it's specifically about reliability of the arming-state notification — a couple of things I noticed that both seem to reproduce the same class of issue this PR is fixing, wanted to check my understanding:

Route-table-full peers get no notification at all, not "broadcast-only" fallback
When mavlinkFindRoute() returns NULL (table full — MAVLINK_MAX_ROUTES with no eviction, only cleared by a full mavlinkRuntimeFreePorts() reinit), the whole reconnect-detection block is skipped for that peer, so mavlinkPortReconnected() never fires for it. The comment describes this as degrading "to the pre-existing broadcast-only behavior," but mavlinkSendArmingStatusText() (the broadcast path) only fires on disableFlags changing — if arming is already disabled and stays disabled, a peer joining after the table is full gets no notice at all, not a broadcast one. Is the intent for the route table to have eviction (e.g. LRU by last-heartbeat) so it can't permanently fill with stale peers, or is capacity expected to never realistically be hit in practice?

The new per-port snapshot floor can suppress a second peer's first-ever snapshot
mavlinkSendArmingStatusTextToPort()'s new 10s floor is keyed by mavPortStates[portIndex].lastArmingSnapshotMs — per port, not per peer. If peer A's heartbeat correctly triggers a snapshot on a shared port, and peer B (different sysid/compid) sends its own first-ever heartbeat on that same port within the next 10s, peer B's reconnect is correctly detected via its own route entry, but the resulting snapshot silently no-ops because of the floor peer A just reset — and there's no retry. That's the exact multi-peer-per-port scenario this PR's per-peer route-table change was written to fix, just reappearing at the delivery stage instead of the detection stage. Would scoping the floor to (port, sysid) — or storing it in mavlinkRouteEntry_t instead of mavlinkPortRuntime_t — avoid that, or is a shared per-port floor intentional here?

Removing the enable-transition mavlinkPortReconnected() call
resetMAVLinkPortRuntimeState() still resets the port's own one-shot/text state, but it doesn't touch route-table entries, and nothing else does either outside a full mavlinkRuntimeFreePorts() reinit. So if a shared port is disabled and quickly re-enabled, and the same peer's heartbeat resumes within the 5s gap window, its route entry's lastHeartbeatMs survives untouched — no firstHeartbeat/heartbeatGap/portChanged trips, so no fresh snapshot goes out despite the port having just been fully reset. Was dropping this call intentional on the assumption route-based detection would always catch it, or would it be worth invalidating the relevant route entry's timestamp on port disable to keep that guarantee?

Not trying to block on all of these — mostly want to understand whether the route-table-full and shared-floor cases are known/acceptable tradeoffs before flagging them as gaps. The per-peer keying itself (the core fix for the steady-peer-masking-a-new-one bug) looks correct and clean, and the millis()-wraparound-safe comparisons and the != 0 guard on the floor check are both handled right.

Review feedback on the reconnect STATUSTEXT commit:

- Track heartbeat presence per peer in the route table instead of one
  timestamp per port, so a steady peer cannot mask a newly joining one
  behind the same port, and a peer failing over to another port
  registers as a reconnect there. If the route table is full the new
  peer degrades to broadcast-only behavior.
- Rate-limit arming snapshots per port so a peer heartbeating slower
  than the gap threshold cannot elicit a resend on every beat.
- Drop the enable-transition snapshot: it fired into a port nobody had
  connected to yet (boot) and duplicated the first-heartbeat snapshot on
  shared-port enable. Reconnect is now driven purely by the heartbeat
  signal; the enable transition still resets per-port one-shot state via
  the port reset path.
…rdown

- Scope the arming-snapshot floor to the peer (route entry) instead of the
  port. Two peers sharing a port each get their own allowance, so a second
  peer's first-ever snapshot is no longer swallowed by the first peer's
  window. mavlinkPortReconnected() becomes mavlinkPeerReconnected() and
  takes the route it fired for.
- Clear heartbeat and snapshot state for peers last heard on a port when
  that port is torn down. The port's one-shot state is reset there, so a
  peer resuming inside the gap window must not skip its fresh snapshot.
- Say plainly what happens when the route table is full: that peer gets no
  snapshot at all. The broadcast path is edge-triggered on the arming flags
  changing, so calling it a "broadcast-only" fallback was wrong.
@xznhj8129
xznhj8129 force-pushed the mav/reconnect-refine branch from 9a4d419 to 179b548 Compare August 21, 2026 19:46
@xznhj8129

Copy link
Copy Markdown
Contributor Author

All three are fair. Two were real, one was a bad comment on my part.

Shared floor — you're right, and right that it's the same bug at the delivery stage. Moved lastArmingSnapshotMs into mavlinkRouteEntry_t; mavlinkPortReconnected() is now mavlinkPeerReconnected(portIndex, route).

Caught live on SITL, same rig and eeprom, only the firmware differing: peer A connects, peer B heartbeats 4s later on the same port. Pre-fix, B gets nothing (after B joined=0); post-fix it gets its snapshot. The existing rig had B joining at t=12, deliberately past the floor — which is why it passed on the broken code.

Enable-transition removal — agreed, nothing invalidated route state on teardown. Added mavlinkForgetHeartbeatsForPort(), called from freeMAVLinkTelemetryPortByIndex(), clearing heartbeat and snapshot state for peers last heard on that port.

Worth noting the reachable scope is narrow: determinePortSharing() returns NOT_SHARED for a MAVLink-only UART, so enabled is always true and teardown never runs. On a shared port with telemetry_switch = OFF the gate is ARMING_FLAG(ARMED), and while armed disableFlags == 0, so there's nothing to snapshot. That leaves shared port + telemetry_switch = ON as the only configuration this affects. Fixed anyway — it costs nothing — but I didn't build a rig for it. The path is straight-line and single-threaded, and the SITL/UART difference (TCP drops the client, a real radio keeps transmitting into a dead port) is a hardware question a rig wouldn't settle.

Route table full — comment was simply wrong; corrected to say plainly that such a peer gets no snapshot at all, since the broadcast path is edge-triggered on the flags changing.

Not adding eviction. LRU by last heartbeat is the obvious key and it's wrong here: routes are learned from any message, so a peer that sends data but never heartbeats sits at lastHeartbeatMs == 0 and looks stalest — eviction would drop an actively-routing peer. Doing it right needs a separate last-seen timestamp on all 32 entries, and I can't construct 32 distinct (sysid,compid) pairs on real hardware.

@github-actions

Copy link
Copy Markdown

Test firmware build ready — commit 179b548

Download firmware for PR #11731

244 targets built. Find your board's .hex file by name on that page (e.g. MATEKF405SE.hex). Files are individually downloadable — no GitHub login required.

Development build for testing only. Use Full Chip Erase when flashing.

@github-actions

Copy link
Copy Markdown

RAM / Flash usage vs. base branch — commit 179b548

Target Flash Δ RAM Δ
MATEKF405 ⚠️ +592 B (+0.09%) +264 B (+0.18%)
MATEKF722 ±0 B (±0.00%) ±0 B (±0.00%)
MATEKF765 ⚠️ +400 B (+0.06%) +272 B (+0.18%)
MATEKH743 ⚠️ +616 B (+0.09%) +256 B (+0.17%)

See RAM/flash optimization guide for techniques to reduce usage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants