Dictation: migrate on-device runtime to Foundry Local streaming ASR#326678
Merged
Conversation
Migrate chat dictation from the transformers.js/onnxruntime-node pipeline (Whisper + hand-rolled Nemotron RNN-T decoder) to Microsoft's Foundry Local streaming ASR runtime, matching the engine the GitHub Copilot app ships. The default model is NVIDIA's nemotron-speech-streaming-en-0.6b; Foundry Local handles decode, VAD and endpointing natively for lower latency and higher accuracy. - Rewrite node LocalTranscriptionService on foundry-local-sdk: async manager, model download/load progress, live streaming session; buffer PCM until the session opens; accumulate cumulative transcript from per-segment is_final results. - Delete nemotronTranscriber.ts (native decode now lives in Foundry). - Swap chat.speechToText.model enum to the Foundry model alias. - Update ILocalTranscriptionService docs and the renderer proxy comments; keep the platform gate (targets already match Foundry's prebuilds). - Bundle foundry-local-sdk native addon + core libs in gulpfile/darwin/linux packaging in place of onnxruntime-node; drop @huggingface/transformers, onnxruntime-node and the sharp stub. Draft: cross-platform native bundling and the on-device runtime path need validation on supported machines/CI. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d527ed3c-29a9-435f-8772-c8117077d394
Contributor
There was a problem hiding this comment.
Pull request overview
Migrates on-device dictation from the custom ONNX/Whisper pipeline to Foundry Local streaming ASR.
Changes:
- Rewrites transcription around Foundry Local streaming sessions.
- Replaces legacy model options and dependencies.
- Updates native packaging for supported platforms.
Show a summary per file
| File | Description |
|---|---|
src/vs/workbench/services/localTranscription/electron-browser/localTranscriptionService.ts |
Updates runtime documentation. |
src/vs/workbench/contrib/chat/browser/chat.shared.contribution.ts |
Changes the configured model. |
src/vs/platform/localTranscription/node/nemotronTranscriber.ts |
Removes the custom decoder. |
src/vs/platform/localTranscription/node/localTranscriptionService.ts |
Implements Foundry streaming ASR. |
src/vs/platform/localTranscription/common/localTranscription.ts |
Updates service contracts. |
package.json |
Replaces runtime dependencies. |
package-lock.json |
Updates locked dependencies. |
build/npm/stubs/sharp/package.json |
Removes the Sharp stub package. |
build/npm/stubs/sharp/index.js |
Removes the Sharp stub implementation. |
build/linux/dependencies-generator.ts |
Adds Foundry native libraries. |
build/gulpfile.vscode.ts |
Packages Foundry native assets. |
build/darwin/verify-macho.ts |
Excludes Foundry ARM64 binaries from universal checks. |
build/darwin/create-universal-app.ts |
Handles Foundry files in universal builds. |
Review details
- Files reviewed: 12/13 changed files
- Comments generated: 6
- Review effort level: Medium
meganrogge
marked this pull request as draft
July 20, 2026 17:17
- Replace blind-append with a per-segment TranscriptAccumulator (keys finals
by start:end time, replaces refinements, joins distinct segments by start
time), fixing duplicated words ("my my my") and restoring monotonic
growth needed for interim shimmer.
- Emit cumulative text (accumulated finals + interim partial) so the renderer
shimmer and finalized-text contract keep working.
- Fetch target-RID Foundry Local core libs at package time via os-spoof so
cross-arch product builds bundle the correct native libraries.
- Add config migration for legacy Whisper/Nemotron model IDs to the Foundry
alias.
- Drop the unused http.proxy plumbing (Foundry Local manages its own
downloads) and document the limitation.
- Propagate stream/append runtime errors to the renderer (Error status +
rejected pushAudio) and drain buffered audio on stop without racing.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d527ed3c-29a9-435f-8772-c8117077d394
Foundry Local emits non-final (interim) transcription results as deltas of the in-progress segment — each carries only the newly recognized text with its own spacing, not the cumulative partial. The service was replacing the partial with each interim result, which dropped earlier partial words and made the visible transcript replace/rewrite the text already streamed in (and re-shimmer settled text). Concatenate interim deltas verbatim (mirroring the GitHub Copilot app's appendVoiceTranscriptChunk) so the cumulative transcript grows monotonically: earlier text stays put and the renderer settles/de-shimmers the unchanged prefix, leaving only the in-progress tail shimmering. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d527ed3c-29a9-435f-8772-c8117077d394
The interim shimmer split relied purely on diffing consecutive interim transcripts: a word only stopped shimmering once a later interim confirmed it had stopped changing. After the user goes silent no further interim arrives, so the final endpointed words kept shimmering until the whole session was stopped. Thread the actually-finalized transcript (Foundry's endpointed segments) from the utility-process service through to the renderer and force-settle it: the shimmer now covers only the in-progress interim tail, and finalized text — including the last segment during a trailing silence — stops shimmering as soon as it is endpointed. Mirrors the GitHub Copilot app's solid-final / shimmering-partial split. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d527ed3c-29a9-435f-8772-c8117077d394
Foundry holds the last spoken segment as an interim result until more audio or an explicit stop arrives, so on a trailing silence it never emits a final for it and the last words shimmered indefinitely. Add an idle-settle timer: when transcript updates stop arriving for a short window (the user has paused), stop shimmering the current tail (keeping the in-progress placeholder styling). A later interim/final update re-applies the shimmer to any new tail, and stop/cancel still finalize as before. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d527ed3c-29a9-435f-8772-c8117077d394
meganrogge
marked this pull request as ready for review
July 20, 2026 17:55
meganrogge
enabled auto-merge (squash)
July 20, 2026 18:02
pwang347
approved these changes
Jul 20, 2026
connor4312
approved these changes
Jul 20, 2026
aeschli
added a commit
that referenced
this pull request
Jul 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Vastly improves latency and accuracy of dictation
Screen.Recording.2026-07-20.at.1.53.50.PM.mov
What & why
The GitHub Copilot app's on-device dictation feels snappier and more accurate than VS Code's. It runs through Microsoft Foundry Local's native streaming ASR runtime (onnxruntime + onnxruntime-genai), which does decode, VAD and endpointing in native code, using NVIDIA's
nemotron-speech-streaming-en-0.6bstreaming RNN-T model.This draft replaces VS Code's dictation pipeline (transformers.js Whisper + a hand-rolled Nemotron RNN-T decoder on
onnxruntime-node) with the same Foundry Local engine for parity.Changes
platform/localTranscription/node/localTranscriptionService.ts): drivesfoundry-local-sdk— asyncFoundryLocalManager, model download/load progress, a live streaming session. Buffers captured PCM until the session opens (so first-use download doesn't drop leading audio), then streams raw PCM16; accumulates a cumulative transcript from per-segmentis_finalresults and emits interim/final transcripts.nemotronTranscriber.ts(native decode now lives in Foundry).chat.speechToText.modelenum → Foundry model alias (nemotron-speech-streaming-en-0.6b, default).SUPPORTED_TARGETSalready match Foundry's prebuild targets exactly: darwin-arm64, linux-x64/arm64, win32-x64/arm64).foundry-local-sdknative addon + core libs (gulpfile ASAR-unpack + per-target prebuild filter, darwin macho/universal, linux dep scanner) in place ofonnxruntime-node. Dropped@huggingface/transformers,onnxruntime-node, and thesharpstub.