Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ A monorepo task runner (like Nx/Turbo) with intelligent caching and dependency r
- `crates/vt_graph` β€” Task dependency graph construction and config loading
- `crates/vt_plan` β€” Execution planning (resolves env vars, working dirs, commands)
- `crates/vt_workspace` β€” Workspace detection and package dependency graph
- `crates/vt_fs_fingerprint` β€” Filesystem fingerprinting for task caching (pre-run snapshot, traced-access judgment, cache-entry validation)
- `crates/fspy*` β€” File system access tracing (9 crates: supervisor, preload libs, platform backends)
- `crates/pty_terminal*` β€” Cross-platform headless terminal emulator (3 crates)
- `crates/vt_path` β€” Type-safe absolute/relative path system
Expand Down
26 changes: 23 additions & 3 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,7 @@ tui-term = "0.3.1"
twox-hash = "2.1.1"
uuid = "1.18.1"
vec1 = "1.12.1"
vt_fs_fingerprint = { path = "crates/vt_fs_fingerprint" }
vt_glob = { path = "crates/vt_glob" }
vt_graph_ser = { path = "crates/vt_graph_ser" }
vt_path = { path = "crates/vt_path" }
Expand Down
4 changes: 1 addition & 3 deletions crates/vt/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,6 @@ once_cell = { workspace = true }
owo-colors = { workspace = true }
petgraph = { workspace = true }
pty_terminal_test_client = { workspace = true }
rayon = { workspace = true }
rusqlite = { workspace = true, features = ["bundled"] }
rustc-hash = { workspace = true }
serde = { workspace = true, features = ["derive", "rc"] }
Expand All @@ -43,9 +42,9 @@ tokio = { workspace = true, features = [
] }
tokio-util = { workspace = true }
tracing = { workspace = true }
twox-hash = { workspace = true }
materialized_artifact = { workspace = true }
uuid = { workspace = true, features = ["v4"] }
vt_fs_fingerprint = { workspace = true }
vt_glob = { workspace = true }
vt_path = { workspace = true }
vt_select = { workspace = true }
Expand All @@ -55,7 +54,6 @@ vt_ipc_shared = { workspace = true }
vt_plan = { workspace = true }
vt_server = { workspace = true }
vt_workspace = { workspace = true }
wax = { workspace = true }
zstd = { workspace = true }

# Artifact build-deps must be unconditional: cargo's resolver panics when
Expand Down
13 changes: 7 additions & 6 deletions crates/vt/docs/task-cache.md
Original file line number Diff line number Diff line change
Expand Up @@ -567,18 +567,19 @@ Each `&&` separated command is cached independently. If only terser config chang
### Core Cache Components

```
crates/vt_fs_fingerprint/src/ # The run's filesystem story (own crate)
β”œβ”€β”€ task_run.rs # TaskFs (pre_run/post_run), Conclusion
β”œβ”€β”€ fingerprint.rs # InputFingerprints, PathFingerprint, InputChange
β”œβ”€β”€ tracked_accesses.rs # fspy access normalization
β”œβ”€β”€ glob.rs # Glob walking + input hashing
└── hash.rs # Content hashing

crates/vt/src/session/
β”œβ”€β”€ cache/
β”‚ β”œβ”€β”€ mod.rs # ExecutionCache, CacheEntryKey/Value, FingerprintMismatch
β”‚ └── display.rs # Cache status display formatting
β”œβ”€β”€ execute/
β”‚ β”œβ”€β”€ mod.rs # execute_spawn, SpawnOutcome
β”‚ β”œβ”€β”€ task_fs/ # The run's filesystem story
β”‚ β”‚ β”œβ”€β”€ task_run.rs # TaskFs (pre_run/post_run), Conclusion
β”‚ β”‚ β”œβ”€β”€ fingerprint.rs # InputFingerprints, PathFingerprint, InputChange
β”‚ β”‚ β”œβ”€β”€ tracked_accesses.rs # fspy access normalization
β”‚ β”‚ β”œβ”€β”€ glob.rs # Glob walking + input hashing
β”‚ β”‚ └── hash.rs # Content hashing
β”‚ β”œβ”€β”€ cache_update.rs # Post-run cache update decision
β”‚ β”œβ”€β”€ post_run.rs # TrackedEnvFingerprints (tracked env validation)
β”‚ └── spawn.rs # spawn_with_tracking, fspy integration
Expand Down
4 changes: 2 additions & 2 deletions crates/vt/src/session/cache/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ pub use display::{
use rusqlite::{Connection, OptionalExtension as _};
use serde::{Deserialize, Serialize};
use tokio::sync::Mutex;
pub use vt_fs_fingerprint::InputChangeKind;
use vt_fs_fingerprint::{InputChange, InputFingerprints};
use vt_graph::config::ResolvedGlobConfig;
use vt_path::{AbsolutePath, RelativePathBuf};
use vt_plan::cache_metadata::{CacheMetadata, ExecutionCacheKey, SpawnFingerprint};
Expand All @@ -25,11 +27,9 @@ use wincode::{
io::{Reader, Writer},
};

pub use super::execute::task_fs::InputChangeKind;
use super::execute::{
pipe::StdOutput,
post_run::{PostRunMismatch, TrackedEnvFingerprints, TrackedEnvQuery},
task_fs::{InputChange, InputFingerprints},
};

const TASK_CACHE_PREALLOCATION_SIZE_LIMIT: usize = 256 * 1024 * 1024;
Expand Down
2 changes: 1 addition & 1 deletion crates/vt/src/session/execute/cache_update.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
use std::{collections::BTreeMap, sync::Arc, time::Duration};

use rustc_hash::FxHashSet;
use vt_fs_fingerprint::{Conclusion, PostRunError};
use vt_path::{AbsolutePath, RelativePathBuf};
use vt_plan::cache_metadata::{CacheMetadata, EnvValueHash};
use vt_server::Reports;
Expand All @@ -13,7 +14,6 @@ use super::{
CacheState,
post_run::{TrackedEnvFingerprints, TrackedEnvQuery},
spawn::ChildOutcome,
task_fs::{Conclusion, PostRunError},
};
use crate::session::{
cache::{CacheEntryValue, ExecutionCache, archive},
Expand Down
3 changes: 1 addition & 2 deletions crates/vt/src/session/execute/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@ pub mod pipe;
pub mod post_run;
mod scheduler;
pub mod spawn;
pub mod task_fs;
#[cfg(windows)]
mod win_job;

Expand All @@ -15,6 +14,7 @@ use std::{

use futures_util::future::LocalBoxFuture;
use tokio_util::sync::CancellationToken;
use vt_fs_fingerprint::TaskFs;
use vt_ipc_shared::NODE_CLIENT_PATH_ENV_NAME;
use vt_path::AbsolutePath;
use vt_plan::{SpawnExecution, cache_metadata::CacheMetadata};
Expand All @@ -23,7 +23,6 @@ use vt_server::{Recorder, Reports, ServerHandle, StopAccepting, serve};
use self::{
pipe::{PipeSinks, StdOutput, pipe_stdio},
spawn::{ChildHandle, ChildOutcome, SpawnStdio, spawn},
task_fs::TaskFs,
};
use super::{
cache::{CacheEntryValue, CacheMiss, ExecutionCache, archive},
Expand Down
2 changes: 1 addition & 1 deletion crates/vt/src/session/execute/post_run.rs
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
//! Post-run environment fingerprinting: env values and bulk env queries
//! observed by runner-aware tools during execution, validated again at cache
//! lookup. The filesystem half of post-run fingerprinting lives in
//! [`super::task_fs`].
//! [`vt_fs_fingerprint`].

use std::{collections::BTreeMap, ffi::OsStr, sync::Arc};

Expand Down
2 changes: 1 addition & 1 deletion crates/vt/src/session/execute/spawn.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
//!
//! [`spawn`] does one thing: hand back the child's stdio pipes plus a
//! cancellation-aware `wait` future. Draining the pipes is [`super::pipe`]'s
//! job; the raw path accesses are judged by [`super::task_fs`] during the
//! job; the raw path accesses are judged by [`vt_fs_fingerprint`] during the
//! cache update.

use std::{ffi::OsStr, io, process::Stdio};
Expand Down
36 changes: 36 additions & 0 deletions crates/vt_fs_fingerprint/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
[package]
name = "vt_fs_fingerprint"
version = "0.0.0"
authors.workspace = true
edition.workspace = true
license.workspace = true
publish = false
rust-version.workspace = true

[dependencies]
anyhow = { workspace = true }
fspy_shared = { workspace = true }
rayon = { workspace = true }
rustc-hash = { workspace = true }
serde = { workspace = true, features = ["derive"] }
thiserror = { workspace = true }
tracing = { workspace = true }
twox-hash = { workspace = true }
vt_glob = { workspace = true }
vt_graph = { workspace = true }
vt_path = { workspace = true }
vt_str = { workspace = true }
wax = { workspace = true }
wincode = { workspace = true, features = ["derive"] }

[target.'cfg(unix)'.dependencies]
nix = { workspace = true }

[dev-dependencies]
tempfile = { workspace = true }

[lints]
workspace = true

[lib]
doctest = false
41 changes: 41 additions & 0 deletions crates/vt_fs_fingerprint/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# vt_fs_fingerprint

The filesystem story of one task run, extracted from the execution engine so
it can be reasoned about and tested on its own: what a run read, what it
produced, and whether a previous run's cached result still matches the
filesystem. Everything else about caching β€” storage, archiving, env tracking,
process spawning β€” stays with the engine; this crate is the part that decides
what filesystem facts _mean_.

The API is one typestate with a call per stage:

- **`TaskFs::pre_run`** β€” before the task executes: validate its io
configuration, capture the state of its listed inputs, and (when a previous
run's fingerprints were fetched) report the first input that changed since
that run.
- **`TaskFs::post_run`** β€” after it finished: judge the traced file accesses
and either declare caching unsound (`Conclusion::InputModified` β€” the task
wrote a path it also read) or return everything the cache should remember
(`Conclusion::Cacheable`: the run's `InputFingerprints` and its outputs).

`InputFingerprints` is the opaque record linking runs together: `post_run`
produces it, the caller stores it in the cache entry, and a later run's
`pre_run` checks the filesystem against it.

Why a separate crate: the policy for a path that a task both reads and writes
is unsettled and expected to be rewritten several times. Here it can be driven
directly with synthetic traces and temp directories, instead of only being
observable by running a whole task through the engine.

Design notes worth knowing:

- The pre-run snapshot is taken before the task runs as a **soundness**
requirement, not a convenience: a task that modifies one of its own listed
inputs without tracing to catch it must perpetually miss. A snapshot taken
after the run would capture the post-run state and turn that safe perpetual
miss into a false cache hit.
- `pre_run` is handed the previous fingerprints (fetch-first) rather than
exposing a separate comparison method, so comparing after the run β€” or
against the wrong run's record β€” is unrepresentable.
- "No filesystem change" from `pre_run` is not yet a cache hit: the engine
still validates tracked environment state, which lives outside this crate.
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
use std::{hash::Hasher as _, io};

/// Hash content using 8 KiB buffered `xxHash3_64`.
pub(super) fn hash_content(mut stream: impl io::Read) -> io::Result<u64> {
pub fn hash_content(mut stream: impl io::Read) -> io::Result<u64> {
let mut hasher = twox_hash::XxHash3_64::default();
let mut buf = [0u8; 8192];
loop {
Expand Down
Loading