fix: bound docs manifest fan-out to stop long hangs on redirect-miss docs requests

[LadyBluenotes]

Problem

Any docs request for a path missing from the lightweight path manifest (typo, dead link, aggregation page, or literally any nonexistent slug) falls through to buildDocsManifest, which fetches every markdown file in the library's docs tree one at a time to read redirect frontmatter. For tanstack/query that's 494 sequential GitHub round-trips with no concurrency and no per-file fault tolerance.

Reproduced against production: a docs path with no .md sibling hangs 45-90s+ with zero bytes returned, both with and without Accept: text/markdown. A single recoverable GitHub error (rate limit, 5xx, transient network failure) on any one file also rejected the entire manifest build, which could cascade into a real 404 for completely unrelated docs pages.

Fix

• Bound the per-file fetch fan-out to concurrency 6 (matches RAW_FETCH_CONCURRENCY used elsewhere in this codebase) instead of fully sequential.
• One file's recoverable fetch error now skips that file's redirect entries instead of failing the whole manifest build.

Not in scope

There's a second, separate slow path (fetchApiContentsRemoteFromContentsApi, a sequential per-directory GitHub Contents API fallback) that this PR does not touch. Flagging for a follow-up rather than guessing at a fix without production visibility into how often it fires.

Summary by CodeRabbit

• New Features

  • Improved docs redirect manifest generation with bounded concurrency for faster builds.
  • Added fault-tolerant per-file redirect extraction that flattens valid redirect entries into the final manifest.

• Bug Fixes

  • Recoverable document fetch issues (such as rate limiting) no longer fail the overall manifest build.
  • Canonical docs paths are still recorded even when a file’s content can’t be retrieved.

• Tests

  • Added tests covering concurrency limits, stable output ordering, and recoverable vs non-recoverable fetch error handling.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: a6e3131b-5712-4ef3-ae25-c384fafdf853

📥 Commits

Reviewing files that changed from the base of the PR and between a497420 and ed986b5.

📒 Files selected for processing (1)

• src/utils/docs.functions.ts

🚧 Files skipped from review as they are similar to previous changes (1)

• src/utils/docs.functions.ts

---

📝 Walkthrough

Walkthrough

This PR adds a concurrency-limited docs redirect collector, refactors manifest building to use it, switches several docs server helpers to dynamic imports, and adds tests for concurrency and recoverable error handling.

Changes

Docs manifest concurrency

Layer / File(s)	Summary
Shared concurrency and loader helpers src/utils/docs.functions.ts	DocsTreeNode is exported, mapWithConcurrency is added, and repo/file helper lookups move behind dynamic imports.
Per-file redirect collection and manifest build src/utils/docs.functions.ts	collectRedirectEntriesForFile centralizes canonical-path reporting, markdown fetches, recoverable error handling, and redirect normalization; buildDocsManifest now gathers redirects concurrently from per-file results.
Docs server-module loading updates src/utils/docs.functions.ts	Docs manifest/path handlers and related fetch routes dynamically load server helpers and GitHub cache helpers with Promise.all where needed.
Concurrency and fault-tolerance tests tests/docs-manifest-concurrency.test.ts	New tests cover mapWithConcurrency ordering and concurrency limits, plus collectRedirectEntriesForFile recoverable-error and mixed-success behavior.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant buildDocsManifest
  participant mapWithConcurrency
  participant collectRedirectEntriesForFile
  participant fetchFile

  buildDocsManifest->>mapWithConcurrency: markdown files, concurrency limit
  mapWithConcurrency->>collectRedirectEntriesForFile: node, fetchFile, onCanonicalPath
  collectRedirectEntriesForFile->>fetchFile: fetch markdown content
  fetchFile-->>collectRedirectEntriesForFile: content or GitHubContentError
  collectRedirectEntriesForFile-->>mapWithConcurrency: redirect entries or empty list
  mapWithConcurrency-->>buildDocsManifest: flattened redirect entries

Loading

Suggested reviewers: TkDodo

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly reflects the main change: limiting docs manifest fan-out to prevent hangs on missing-docs requests.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix-agent-404

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

tests/docs-manifest-concurrency.test.ts (1)

58-58: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Remove stray console.log debug statements.

These top-level console.log calls at Line 58 and Line 132 are debug artifacts unrelated to node:test assertions; the test runner already reports pass/fail per test.

🧹 Remove debug logs

-console.log('docs manifest concurrency tests passed')

-console.log('buildDocsManifest per-file fault tolerance tests passed')

Also applies to: 132-132

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/docs-manifest-concurrency.test.ts` at line 58, Remove the stray
top-level debug logging from the test module: the `console.log` calls in
`tests/docs-manifest-concurrency.test.ts` are unrelated to `node:test`
assertions and should be deleted. Update the test file around the affected
statements so only the actual test cases and assertions remain, without any
extra logging in the module scope.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@tests/docs-manifest-concurrency.test.ts`:
- Line 58: Remove the stray top-level debug logging from the test module: the
`console.log` calls in `tests/docs-manifest-concurrency.test.ts` are unrelated
to `node:test` assertions and should be deleted. Update the test file around the
affected statements so only the actual test cases and assertions remain, without
any extra logging in the module scope.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: a84bdd7e-3a39-4016-a8a2-36da1218abdb

📥 Commits

Reviewing files that changed from the base of the PR and between d17bbc7 and a497420.

📒 Files selected for processing (2)

• src/utils/docs.functions.ts
• tests/docs-manifest-concurrency.test.ts

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	tanstack-com	ed986b5	Commit Preview URL Branch Preview URL	Jul 02 2026, 07:44 PM