feat: hard peak memory limit for codeseek init (observed ~10GiB RSS / multi-GB IO on medium monorepo)

[aider4ryder]

Summary

codeseek init on a medium-large Rust monorepo can spike to ~10 GiB RSS (here ~28–30% of a 32 GiB host) within seconds, with multi-GB sequential disk reads. This stalls interactive shells and co-resident processes (IO + memory pressure). There is currently no config knob for a hard peak-memory / RSS ceiling.

Please add a hard memory limit (config + optional CLI) that caps peak process RSS / working set during init / reindex / search, and fails closed or degrades (spill / smaller batch / abort with a clear error) instead of unbounded growth.

Environment

Item	Value
codeseek	0.1.30 (/usr/local/bin/codeseek, binary size ~304 MiB)
Host RAM	~32 GiB (MemTotal ≈ 32555664 kB)
OS	Linux amd64
Command	codeseek init
Project	Rust workspace monorepo (~484–591 MiB on disk; not a multi-GB source tree)
Index dir	~/.codeseek/projects/<hash>/ for that project
Notable index artifacts	graph.bin ~1.6 GiB, lancedb ~262 MiB, tantivy_bm25 ~4.7 MiB
Config	~/.codeseek/config.json — embedding dims 4096, hybrid BM25+vector top_k=300, reranker enabled; no memory fields exist in config schema (Config in rust-core/src/config.rs only has embedding / index / hooks)

Observed spike (live sampling every 5s)

Process: codeseek init (PID 3615423), cwd = large Rust monorepo, parent = codeseek serve --mcp.

utc (approx)	VmRSS	% of 32 GiB host
T+0 (07:50:09Z)	910 MiB	2.9%
T+6s	~5.4–5.6 GiB	~17% (alert threshold 5% crossed)
T+12s	~4.8 GiB	~15%
T+18s	~8.8 GiB	~28%
shortly after	VmHWM ≈ 10.1 GiB, VmPeak ≈ 12.5 GiB	~31% / virtual ~12 GiB

At alert time (2026-07-16T07:50:15Z):

ALERT — codeseek PID 3615423 RSS 5426.18MB (17.067% > 5%)
cmd=/usr/local/bin/codeseek init

/proc/<pid>/io while still running (order of magnitude):

rchar ≈ 3.3e9
read_bytes ≈ 3.3e9   # ~3.1–3.3 GiB read from disk in <1 minute
write_bytes ≈ 0

Concurrently, many idle codeseek serve --mcp workers sit at ~9–14 MiB RSS each (fine). The spike is specific to init / reindex of a project whose on-disk index already has a 1.6 GiB graph.bin.

Earlier the same host, other codeseek init runs peaked around 1.0–1.2 GiB for the same project class — so behavior is not always 10 GiB, but unbounded growth is possible and already hits multi‑GB.

Why this hurts

1. Memory: 5–10 GiB for one init competes with IDEs, agents, browsers, local LLM/proxy stacks on a 32 GiB workstation.
2. IO: multi‑GB read_bytes in a short window causes system-wide stalls (user reports interactive shells freezing briefly).
3. Multiplicity: MCP installs leave many codeseek serve --mcp processes; each can spawn init (e.g. hooks / first-use / refresh), so peaks can stack if two inits overlap.
4. No operator control: config has knobs for retrieval quality (bm25_top_k, dims, reranker) but nothing for resource ceilings.

Expected / feature request

Hard limit (primary)

Add something like:

{
  "resources": {
    "max_rss_mb": 2048,
    "on_exceed": "abort"
  }
}

or CLI:

codeseek init --max-rss-mb 2048

Semantics (suggested):

• Measure process RSS (or cgroup memory.current if available) on a timer during init / heavy graph load / embedding batching.
• On exceed: abort with non-zero exit + clear message (preferred for hard limit), or configurable spill / reduce concurrency / stream graph sections.
• Document that the limit is a hard safety rail, not a soft target.

Nice-to-haves

• Bound peak during graph.bin load (mmap + fault-in of 1.6 GiB should not imply 10 GiB process growth — please investigate temporary buffers / full graph materialization / cloning).
• Cap parallel workers / batch sizes for parse + embed under memory pressure.
• Related: daemon / unified resource management discussions may complement this; a per-process hard RSS cap is still needed for CLI init when no daemon is used.

Minimal repro shape

1. Index a large Rust monorepo with existing large graph.bin (multi‑hundred MB to multi‑GB).
2. Run codeseek init (or trigger via MCP codeseek_init / serve path).
3. Watch RSS every 1–5s (ps, /proc/<pid>/status VmRSS/VmHWM).
4. Observe multi‑GB climb without any user-configured memory bound.

Request

1. Config + CLI hard peak memory limit for init (and ideally other heavy ops).
2. Investigation of why RSS grows to ~10 GiB when source tree is ~0.5 GiB and graph.bin is ~1.6 GiB (suggests temporary over-allocation or full in-memory expansion beyond the on-disk graph).
3. Docs: recommended limits for agent/MCP hosts (e.g. 1–2 GiB for laptop, higher for servers).

Happy to attach more /proc samples or re-run with RUST_LOG if you specify flags.