fix(replay): Don't let a wedged video encoder freeze the app

[romtsn]

📜 Description

Fixes an ANR where Android apps freeze on foreground/background transitions until the system kills the process.

A reporter pulled the full deobfuscated 131-thread dump from Play Console (the Sentry ANR event only carries main, which is why the lock holder was invisible), giving us the exact chain:

main ──waits──▶ lifecycleLock ──held by──▶ timer thread
                                              │ waits
                                              ▼
                                          encoderLock ──held by──▶ SentryReplayIntegration-0
                                                                       │ parked forever
                                                                       ▼
                                                          MediaCodec.dequeueOutputBuffer

Two cooperating defects, so two bounds — both are needed:

1. SimpleVideoEncoder.drainCodec() had a while (true) with no exit on the endOfStream = true path. Some hardware encoders never emit BUFFER_FLAG_END_OF_STREAM after signalEndOfInputStream(), so the loop spun forever while holding encoderLock. It now gives up after 10 consecutive no-progress iterations (~1s at the existing 100ms poll) and drops the remaining frames.
2. ReplayCache.close() used an unbounded encoderLock.acquire(). It runs inline on the caller's thread from BaseCaptureStrategy.stop(), which ReplayIntegration.stop() invokes while holding lifecycleLock — that nesting is what promotes a stuck encoder into a frozen main thread. It now waits at most 2s and skips the release otherwise.

The loop bound alone isn't sufficient: the native frames show MediaCodec::dequeueOutputBuffer → AMessage::postAndAwaitResponse → ALooper::awaitResponse with MediaCodec_looper idle. TIMEOUT_USEC is handed to the codec looper, which is supposed to reply once it elapses — a dead looper never replies, and awaitResponse has no deadline of its own. So a single call can never return, and only bounding the lock wait keeps the lifecycle path free.

On the timeout path the (already-dead) codec is not released, leaking a native handle. That's the deliberate trade: leaking a handle in a process whose encoder is wedged beats freezing the app. isClosed.set(true) still runs on every path, since persistSegmentValues gates on it.

The other four encoderLock.acquire() sites in ReplayCache are unchanged — they all run on the replay worker, the thread that would be stuck, so a deadline there buys nothing.

Adds AutoClosableReentrantLock.tryAcquire(timeout, unit) (@ApiStatus.Internal, additive) for the second bound. Returns the token on success and null on timeout, so callers must branch explicitly rather than silently no-op via ?.use {}.

Net effect: a wedged codec degrades to "replay stops working for this process" instead of "the app freezes."

Fixes getsentry/sentry-dart#3556
Fixes #5870

💚 How did you test it?

ReplayShadowMediaCodec gained two hooks, since it ignores timeoutUs and always eventually produces EOS: neverSignalEos and blockOnDequeue. New tests cover both bounds in ReplayCacheTest, plus 4 for tryAcquire in AutoClosableReentrantLockTest.

Both ReplayCacheTest cases were mutation-checked — reverting each fix in turn makes the matching test fail with its assertion message rather than hang. Full :sentry-android-replay:testReleaseUnitTest passes; apiDump adds exactly the one tryAcquire line.

No device repro is available (Redmi Note 14 Pro 5G / Pro+ 5G, Android 16, MediaTek), so the shadow-based tests are the verification of record.

📝 Checklist

• I reviewed the submitted code.
• I added tests to verify the changes.
• No new PII added or SDK only sends newly added PII if sendDefaultPii is enabled.
• I updated the docs if needed.
• No breaking change or entry added to the changelog.

🤖 Generated with Claude Code

[sentry]

📲 Install Builds

Android

🔗 App Name	App ID	Version	Configuration
SDK Size	io.sentry.tests.size	8.52.0 (1)	release

⚙️ sentry-android Build Distribution Settings

[github-actions]

Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	292.27 ms	374.31 ms	82.04 ms
Size	0 B	0 B	0 B

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
6b019b7	319.84 ms	333.15 ms	13.31 ms
d15471f	322.58 ms	396.08 ms	73.50 ms
b750b96	421.25 ms	444.09 ms	22.84 ms
d217708	411.22 ms	430.86 ms	19.63 ms
604a261	380.65 ms	451.27 ms	70.62 ms
5b1a06b	352.27 ms	413.70 ms	61.43 ms
fcec2f2	357.47 ms	447.32 ms	89.85 ms
7414e9b	370.39 ms	422.18 ms	51.79 ms
fcec2f2	311.35 ms	384.94 ms	73.59 ms
f6cdbf0	314.19 ms	357.59 ms	43.40 ms

App size

Revision	Plain	With Sentry	Diff
6b019b7	0 B	0 B	0 B
d15471f	1.58 MiB	2.13 MiB	559.54 KiB
b750b96	1.58 MiB	2.10 MiB	533.20 KiB
d217708	1.58 MiB	2.10 MiB	532.97 KiB
604a261	1.58 MiB	2.10 MiB	533.42 KiB
5b1a06b	0 B	0 B	0 B
fcec2f2	1.58 MiB	2.12 MiB	551.50 KiB
7414e9b	0 B	0 B	0 B
fcec2f2	1.58 MiB	2.12 MiB	551.51 KiB
f6cdbf0	0 B	0 B	0 B

Previous results on branch: rz/fix/replay-encoder-wedge-anr

Startup times

Revision	Plain	With Sentry	Diff
5c3296f	373.74 ms	470.27 ms	96.53 ms
22606d4	321.23 ms	374.37 ms	53.13 ms

App size

Revision	Plain	With Sentry	Diff
5c3296f	0 B	0 B	0 B
22606d4	0 B	0 B	0 B

[runningcode]

looks good, i left a question for a discussion!

[0xadam-brown]

A few quick comments for your consideration (no blockers); otherwise lgtm

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 228b6e9. Configure here.}