diff --git a/README.md b/README.md index bf9bc21f..e4fb50dd 100644 --- a/README.md +++ b/README.md @@ -93,13 +93,21 @@ make install # ~/Applications/disktree.app, and ~/.local/bin/disktree make uninstall ``` -To see everything, give disktree **Full Disk Access** in System Settings › -Privacy & Security (the panel offers a button when it is missing), then +To see privacy-protected files, give disktree **Full Disk Access** in System +Settings › Privacy & Security (the panel offers a button when it is missing), then reopen it. Without it macOS hides Mail, Messages, Safari, other apps' data and the Trash, and disktree counts them as unreadable. Started from a terminal, it is the terminal that needs the access. macOS also asks once each for Desktop, Documents and Downloads. +Full Disk Access does not override filesystem ownership and permissions. +Administrator-owned service stores can still be unreadable. Their sizes are +**Unavailable**, or **measured · incomplete** when only part was readable; +neither means zero bytes or that the service is absent. Include hidden entries, +scan without a depth limit, and use the directory chooser to scan a listed +store directly, or rescan after read access has been made available outside +disktree. The app does not request elevation or install a privileged helper. + What is different from Linux: - **Free space** is what `df` reports. Finder's figure is larger: it counts @@ -112,6 +120,51 @@ What is different from Linux: - **Cloud-only folders** (iCloud Drive, Dropbox and the like, evicted to the server) are not opened, so a scan never downloads them. +#### FSEvents history and Spotlight indexes + +Whole-disk scans show **FSEvents history** (`.fseventsd`) and **Spotlight index** +(`.Spotlight-V100`) in the treemap and in **macOS service stores** in the panel, +including an availability label when there is no measurable tile. On a startup +disk these private stores live in `/System/Volumes/Data`; external volumes +have their own stores. A home-directory scan does not measure them. + +The figures are allocated file bytes (`st_blocks × 512`), unless apparent size +is selected, and are already part of the scan total. They are not additional +APFS volume, container or snapshot figures. Hardlinks are counted once across +the tree; shared APFS clone blocks and snapshots mean measured bytes are not +a promise of space freed. The rest of the second Data mount stays excluded, +so firmlinked user files are not counted again. + +FSEvents stores persistent filesystem change history that backup and sync +apps can use. At an administrator's explicit discretion, Apple's supported, +root-only [`FSEventsPurgeEventsForDeviceUpToEventId` API](https://developer.apple.com/documentation/coreservices/1447985-fseventspurgeeventsfordeviceupto) +can shorten old history. It destroys history for **every consumer on that +volume**, and those consumers may need a full rescan. Apple's +[FSEvents guide](https://developer.apple.com/library/archive/documentation/Darwin/Conceptual/FSEvents_ProgGuide/UsingtheFSEventsFramework/UsingtheFSEventsFramework.html) +advises against routine purging and requires an explicit administrator request +and confirmation. disktree only explains this option; it does not call it or +suggest raw deletion of `.fseventsd`. + +An administrator implementing a chosen retention cutoff would resolve the +volume's current `st_dev`, obtain an event ID with +`FSEventsGetLastEventIdForDeviceBeforeTime`, and only then use the purge API. +The installed macOS SDK's `FSEvents.h` documents this function's time argument +as **seconds since January 1, 1970**, despite its `CFAbsoluteTime` type. The +archived guide describes the usual 2001 epoch instead; check the target SDK +contract rather than applying that conversion. No retention cutoff or space +saving is guaranteed by disktree. + +Spotlight's index supports file and content search. Keep indexing for that +functionality, exclude selected folders through +[Spotlight Search Privacy](https://support.apple.com/guide/mac-help/prevent-spotlight-searches-in-files-mchlp2811/mac), +or have an administrator choose whether a volume should be indexed. The +installed `mdutil(1)` manual describes `-s` as a read-only status query, +`-i on|off` as the indexing choice, and `-X` as removing the index directory: +**`-X` does not itself disable indexing**. An erased index may be rebuilt when +indexing is enabled, consuming space and work again; disabling indexing reduces +search coverage. A failed status query is unknown, not evidence of disabled +indexing. disktree does not run these commands or alter indexing settings. + To sign and notarize a build for others, with a Developer ID certificate in the keychain and credentials saved by `xcrun notarytool store-credentials`: @@ -451,8 +504,9 @@ Click `/` (or any directory above the scanned root) in the trail, press `g`, run `disktree --disk`, or use the launcher's *Scan the whole disk* action. `g` and `--disk` scan the disk your home directory lives on — `/` on Omarchy and on macOS. On macOS the Data volume's second mount, -`/System/Volumes/Data`, is skipped: it is `/Users`, `/Applications` and the -rest again under other names. +`/System/Volumes/Data`, exposes only its private `.fseventsd` and +`.Spotlight-V100` stores: `/Users`, `/Applications` and the rest would be +the same files again under other names. Widening is memoized: the tree already measured is handed to the wider walk and reused where it is reached, so going from `~` to `/` reads only what is diff --git a/crates/disktree-app/src/palette.rs b/crates/disktree-app/src/palette.rs index 1f5ad7e7..92bdd970 100644 --- a/crates/disktree-app/src/palette.rs +++ b/crates/disktree-app/src/palette.rs @@ -29,6 +29,8 @@ const fn hue(category: Category) -> (f32, f32) { Category::Git => (0.955, 1.0), Category::Media => (0.745, 1.0), Category::Cache => (0.125, 0.95), + Category::FilesystemEvents => (0.31, 0.6), + Category::Spotlight => (0.855, 0.6), Category::Documents => (0.6, 0.18), Category::Other => (0.6, 0.08), } diff --git a/crates/disktree-app/src/state.rs b/crates/disktree-app/src/state.rs index efb9730a..dd23cb36 100644 --- a/crates/disktree-app/src/state.rs +++ b/crates/disktree-app/src/state.rs @@ -948,7 +948,13 @@ impl Disktree { cx.notify(); return; } - if !path.is_absolute() { + // SSH roots use the Unix server's namespace, even on Windows. + let absolute = if self.volumes_source.is_some() { + path.to_str().is_some_and(|path| path.starts_with('/')) + } else { + path.is_absolute() + }; + if !absolute { self.picker_error = Some("Enter an absolute directory path.".into()); cx.notify(); diff --git a/crates/disktree-app/src/tests.rs b/crates/disktree-app/src/tests.rs index 0520e4cb..3ddd4e77 100644 --- a/crates/disktree-app/src/tests.rs +++ b/crates/disktree-app/src/tests.rs @@ -71,6 +71,76 @@ fn draw(cx: &mut Window) { }); } +#[cfg(target_os = "macos")] +#[gpui_kit::test] +fn private_service_stores_draw_unavailable_and_partial_figures( + cx: &mut TestAppContext, +) { + use disktree_core::macos_services::{Usage, usage}; + use disktree_core::tree::{Metric, Node, NodeKind, aggregate}; + cx.update(gpui_omarchy::init); + let temp = fixture(); + let (view, cx) = view_over(temp.path(), cx); + update(&view, cx, |app, cx| { + // Replace the fixture's tree without starting a real system scan. + let mut history = Node::directory(".fseventsd"); + history.read_error = true; + let mut index = Node::directory(".Spotlight-V100"); + index + .children + .push(Node::entry("store", NodeKind::File, 8192)); + index.read_error = true; + let mut data = Node::directory("Data"); + data.children = vec![history, index]; + let mut volumes = Node::directory("Volumes"); + volumes.children.push(data); + let mut system = Node::directory("System"); + system.children.push(volumes); + let mut root = Node::directory("/"); + root.children.push(system); + aggregate(&mut root, Metric::Bytes); + disktree_core::classify::classify(&mut root); + app.tree = Some(std::sync::Arc::new(root)); + app.root_path = PathBuf::from("/"); + app.selected = None; + app.hovered = None; + app.options = ScanOptions::default(); + app.progress.errors = 2; + cx.notify(); + }); + draw(cx); + assert!(cx.debug_bounds("macos-services").is_some()); + read(&view, cx, |app| { + let tree = app.tree().expect("fixture tree"); + let data = Path::new(disktree_core::space::MACOS_DATA_VOLUME); + assert_eq!( + usage(&app.root_path, tree, &app.options, &data.join(".fseventsd")), + Usage::Unavailable + ); + assert_eq!( + usage( + &app.root_path, + tree, + &app.options, + &data.join(".Spotlight-V100") + ), + Usage::Incomplete(8192) + ); + }); + update(&view, cx, |app, cx| { + app.selected = app.tree().and_then(|tree| { + tree.find(".Spotlight-V100").map(|(crumbs, _)| crumbs) + }); + cx.notify(); + }); + draw(cx); + press(cx, "space"); + assert!(read(&view, cx, |app| app.plan().targets.is_empty())); + assert!(read(&view, cx, |app| !app.plan().blocked.is_empty())); + press(cx, "c"); + assert_eq!(read(&view, cx, |app| app.screen), Screen::Review); +} + fn press(cx: &mut Window, keys: &str) { cx.simulate_keystrokes(keys); draw(cx); @@ -451,6 +521,14 @@ fn remote_root_acceptance_never_stats_the_path_locally( }); draw(cx); update(&view, cx, |app, cx| { + let before = app.root_path.clone(); + for path in ["relative", r"C:\local-only", r"\local-only"] { + app.choose_source_root(PathBuf::from(path), cx); + assert!(app.picker_error.is_some()); + assert_eq!(app.root_path, before); + assert!(app.remote.is_none()); + assert!(!app.marks.is_empty()); + } app.choose_volume(cx); assert_eq!(app.remote.as_deref(), Some("-test-fixture")); assert_eq!(app.root_path, PathBuf::from("/only-on-the-remote-host")); @@ -1031,7 +1109,10 @@ fn enter_in_the_focused_volume_picker_scans_the_selected_root( assert!(!read(&view, cx, Disktree::can_start_over)); press(cx, "enter"); assert!(!read(&view, cx, |app| app.volumes_open)); - assert_eq!(read(&view, cx, |app| app.root_path.clone()), target); + assert_eq!( + read(&view, cx, |app| app.root_path.clone()), + dunce::canonicalize(&target).unwrap() + ); finish_scan(&view, cx); assert!(read(&view, cx, |app| app.tree().is_some())); } diff --git a/crates/disktree-app/src/views.rs b/crates/disktree-app/src/views.rs index 32aedf5f..b5b62252 100644 --- a/crates/disktree-app/src/views.rs +++ b/crates/disktree-app/src/views.rs @@ -1490,7 +1490,8 @@ fn side_panel( .overflow_y_scroll() .child(worth_section(app, theme, cx)) .child(rule()) - .child(marked_section(app, theme, cx)), + .child(marked_section(app, theme, cx)) + .children(macos_services_section(app, theme, cx)), ) .children(notice_line(app, theme, cx)) .children(privacy_line(app, theme, cx)) @@ -1568,10 +1569,26 @@ fn selection_section( })), ); - let (number, unit) = match app.options.metric { + let (mut number, mut unit) = match app.options.metric { Metric::Bytes => widgets::split_size(&human_bytes(node.bytes)), Metric::Files => (widgets::human_count(node.files), "files".into()), }; + if disktree_core::macos_services::is_service_name(&node.name) + && let (Some(tree), Some(path)) = (app.tree(), path.as_deref()) + { + use disktree_core::macos_services::{Usage, usage}; + match usage(&app.root_path, tree, &app.options, path) { + Usage::Unavailable | Usage::OutsideScan => { + number = "Unavailable".into(); + unit.clear(); + } + Usage::Incomplete(_) => { + number = format!("≥ {number}"); + unit.push_str(" measured"); + } + Usage::Measured(_) => {} + } + } let share = node.bytes as f32 / root_value.max(1) as f32; let measure = div() .flex() @@ -1735,6 +1752,140 @@ fn selection_section( .child(grid) .children(badges) .child(actions) + .children(match node.category { + Category::FilesystemEvents => Some(service_guidance( + disktree_core::macos_services::Service::Fsevents, + theme, + cx, + )), + Category::Spotlight => Some(service_guidance( + disktree_core::macos_services::Service::Spotlight, + theme, + cx, + )), + _ => None, + }) +} + +fn service_guidance( + service: disktree_core::macos_services::Service, + theme: &Theme, + cx: &Context<'_, Disktree>, +) -> Div { + div() + .flex() + .flex_col() + .gap(space::SM) + .text_size(text::CAPTION) + .text_color(theme.secondary) + .child(service.guidance()) + .child( + button( + ElementId::Name( + format!("service-docs-{}", service.name()).into(), + ), + "Apple's guidance", + ButtonVariant::Outline, + cx, + ) + .tab_stop(false) + .on_click(cx.listener(move |this, _, window, cx| { + cx.open_url(service.documentation()); + window.focus(&this.focus, cx); + })), + ) +} + +/// Private stores remain visible here even when unknown sizes have no tile. +fn macos_services_section( + app: &Disktree, + theme: &Theme, + cx: &Context<'_, Disktree>, +) -> Option { + use disktree_core::macos_services::{Service, Usage, stores_volume, usage}; + if !cfg!(target_os = "macos") { + return None; + } + let volume = stores_volume(&app.root_path)?; + let tree = app.tree()?; + let rule = || div().h(px(1.)).bg(theme.divider()); + let mut section = div() + .id("macos-services") + .debug_selector(|| "macos-services".into()) + .flex() + .flex_col() + .gap(space::SM) + .child(rule()) + .child(widgets::eyebrow("macOS service stores", cx)); + let mut incomplete = false; + for service in Service::ALL { + let path = volume.join(service.name()); + let state = usage(&app.root_path, tree, &app.options, &path); + let measured = match state { + Usage::Measured(bytes) => human_bytes(bytes), + Usage::Incomplete(bytes) => { + incomplete = true; + format!("{} measured · incomplete", human_bytes(bytes)) + } + Usage::Unavailable => { + incomplete = true; + "Unavailable".into() + } + Usage::OutsideScan => { + incomplete = true; + "Outside scan".into() + } + }; + section = section.child( + div() + .flex() + .flex_col() + .gap(space::XS) + .child(widgets::figure( + service.label(), + measured, + theme.bright, + cx, + )) + .child( + div() + .text_size(text::CAPTION) + .text_color(theme.secondary) + .child(path.display().to_string()), + ) + .child(service_guidance(service, theme, cx)), + ); + } + if incomplete { + section = section.child( + div() + .text_size(text::CAPTION) + .text_color(theme.warning) + .child( + "Unknown contents are not zero. Include hidden entries \ + and remove any scan depth limit. Full Disk Access can \ + resolve privacy restrictions; filesystem permissions \ + may still require administrator read access. Scan the \ + listed folder with the directory chooser, or rescan \ + after read access is available. disktree does not \ + request elevation.", + ), + ); + } + Some( + section.child( + div() + .text_size(text::CAPTION) + .text_color(theme.secondary) + .child(if app.options.apparent_size { + "Apparent file sizes, already included in the scan. Switch \ + to disk usage for allocated bytes." + } else { + "Allocated file bytes, already included in the scan. APFS \ + shared blocks and snapshots can change actual space freed." + }), + ), + ) } /// The biggest things that could plausibly go, with their total. @@ -2050,7 +2201,8 @@ fn privacy_line( .child(div().text_color(color).child(format!( "macOS kept {} {noun} unreadable. Give disktree Full Disk \ Access, or your terminal if you started it there, then \ - reopen it.", + reopen it. Filesystem permissions can still restrict \ + administrator-owned service stores.", widgets::human_count(errors) ))) .child( diff --git a/crates/disktree-core/src/classify.rs b/crates/disktree-core/src/classify.rs index bfe24a7f..44b96eb5 100644 --- a/crates/disktree-core/src/classify.rs +++ b/crates/disktree-core/src/classify.rs @@ -36,6 +36,10 @@ pub enum Category { Documents, /// Caches and other regenerable state. Cache, + /// macOS's persistent record of filesystem changes. + FilesystemEvents, + /// macOS's on-volume search database. + Spotlight, /// Nothing recognisable. #[default] Other, @@ -43,7 +47,7 @@ pub enum Category { impl Category { /// The categories the legend lists, in its order. - pub const LEGEND: [Self; 8] = [ + pub const LEGEND: [Self; 10] = [ Self::Code, Self::AgentScratch, Self::Toolchain, @@ -52,6 +56,8 @@ impl Category { Self::Media, Self::Documents, Self::Cache, + Self::FilesystemEvents, + Self::Spotlight, ]; pub const fn label(self) -> &'static str { @@ -64,6 +70,8 @@ impl Category { Self::Media => "Media", Self::Documents => "Documents", Self::Cache => "Cache", + Self::FilesystemEvents => "FSEvents history", + Self::Spotlight => "Spotlight index", Self::Other => "Other", } } @@ -161,6 +169,8 @@ pub fn category_of_name(name: &str) -> Option { // Windows: iCloud for Windows. | "iclouddrive" => Category::Synced, ".git" => Category::Git, + ".fseventsd" => Category::FilesystemEvents, + ".spotlight-v100" => Category::Spotlight, "pictures" | "photos" | "music" | "videos" | "movies" | "steam" | "steamlibrary" | "steamapps" | "emulation" | "models" | ".ollama" | ".lmstudio" | "games" | "wineprefix" => { @@ -227,6 +237,12 @@ pub fn reclaim_of( /// Top-down: a node's own name wins, otherwise it inherits. Reclaimable /// space is inherited too, so everything under a cache is hatched. pub fn classify(root: &mut Node) { + if let Some(category) = category_of_name(&root.name) + && matches!(category, Category::FilesystemEvents | Category::Spotlight) + { + classify_below(root, category, None, 0); + return; + } root.category = Category::Other; root.reclaim = None; for index in 0..root.children.len() { @@ -300,7 +316,11 @@ fn kind_of( reclaim: Option, ) -> (Category, Option) { let child = &siblings[index]; - if !child.is_dir() { + // These stores belong to macOS even when an internal folder is named + // `cache` or `tmp`; raw deletion is not an appropriate cleanup method. + if !child.is_dir() + || matches!(category, Category::FilesystemEvents | Category::Spotlight) + { return (category, reclaim); } let has_sibling = @@ -308,8 +328,14 @@ fn kind_of( let child_category = category_of_name(&child.name) .or_else(|| is_git_store(child).then_some(Category::Git)) .unwrap_or(category); - let child_reclaim = - reclaim.or_else(|| reclaim_of(&child.name, category, has_sibling)); + let child_reclaim = if matches!( + child_category, + Category::FilesystemEvents | Category::Spotlight + ) { + None + } else { + reclaim.or_else(|| reclaim_of(&child.name, category, has_sibling)) + }; (child_category, child_reclaim) } @@ -324,6 +350,12 @@ fn dominant_child_category(node: &Node) -> Option { .filter(|child| child.is_dir()) .find_map(|child| { category_of_name(&child.name) + .filter(|category| { + !matches!( + category, + Category::FilesystemEvents | Category::Spotlight + ) + }) .or_else(|| is_git_store(child).then_some(Category::Git)) }) { @@ -607,6 +639,30 @@ mod tests { assert_eq!(category_of_name("OneDriveSetup"), None); } + #[test] + fn macos_stores_are_named_without_suggesting_raw_deletion() { + for (name, category) in [ + (".fseventsd", Category::FilesystemEvents), + (".Spotlight-V100", Category::Spotlight), + ] { + let mut store = + dir(name, vec![dir("cache", vec![file("blob", 123)])]); + aggregate(&mut store, Metric::Bytes); + classify(&mut store); + assert_eq!(store.category, category); + assert_eq!(store.children[0].category, category); + assert_eq!(store.children[0].children[0].category, category); + assert_eq!(store.children[0].reclaim, None); + assert_eq!(store.bytes, 123); + let mut volume = + dir("volume", vec![store, dir("ordinary", vec![])]); + classify(&mut volume); + assert_eq!(volume.children[0].category, category); + assert_eq!(volume.children[1].category, Category::Other); + assert_eq!(volume.children[0].reclaim, None); + } + } + #[test] fn the_legend_lists_every_named_category_once() { let mut seen = std::collections::HashSet::new(); diff --git a/crates/disktree-core/src/lib.rs b/crates/disktree-core/src/lib.rs index 0f58c7ba..0f37e4a5 100644 --- a/crates/disktree-core/src/lib.rs +++ b/crates/disktree-core/src/lib.rs @@ -14,6 +14,7 @@ pub mod identity; pub mod insights; #[cfg(target_os = "macos")] mod macos; +pub mod macos_services; #[cfg(windows)] mod mft; pub mod payload; diff --git a/crates/disktree-core/src/macos_services.rs b/crates/disktree-core/src/macos_services.rs new file mode 100644 index 00000000..b276df4c --- /dev/null +++ b/crates/disktree-core/src/macos_services.rs @@ -0,0 +1,207 @@ +//! Read-only visibility into macOS's private, on-volume service stores. +//! +//! These figures come from the same tree as the treemap, not another walk or +//! a subtraction from APFS capacity. They therefore share hardlink accounting +//! and never add volume or snapshot usage to file totals. + +use std::path::{Path, PathBuf}; + +use crate::scan::ScanOptions; +use crate::tree::Node; + +/// A service store whose name alone does not explain its contents. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Service { + Fsevents, + Spotlight, +} + +impl Service { + pub const ALL: [Self; 2] = [Self::Fsevents, Self::Spotlight]; + + pub const fn name(self) -> &'static str { + match self { + Self::Fsevents => ".fseventsd", + Self::Spotlight => ".Spotlight-V100", + } + } + + pub const fn label(self) -> &'static str { + match self { + Self::Fsevents => "FSEvents history", + Self::Spotlight => "Spotlight index", + } + } + + pub const fn guidance(self) -> &'static str { + match self { + Self::Fsevents => { + "Filesystem change history used by backup and sync apps. \ + At an administrator's explicit request, Apple's root-only \ + purge API can shorten old history. Other consumers may need \ + a full rescan. disktree does not purge it." + } + Self::Spotlight => { + "Search metadata for this volume. Keep indexing for file \ + search, or choose folders in Spotlight's Search Privacy \ + settings. Turning indexing off reduces search coverage; \ + removing an index alone can cause it to be rebuilt. \ + disktree does not change indexing." + } + } + } + + pub const fn documentation(self) -> &'static str { + match self { + Self::Fsevents => { + "https://developer.apple.com/library/archive/documentation/\ + Darwin/Conceptual/FSEvents_ProgGuide/UsingtheFSEventsFramework/\ + UsingtheFSEventsFramework.html" + } + Self::Spotlight => { + "https://support.apple.com/guide/mac-help/\ + prevent-spotlight-searches-in-files-mchlp2811/mac" + } + } + } +} + +pub fn is_service_name(name: &str) -> bool { + Service::ALL + .iter() + .any(|service| name.eq_ignore_ascii_case(service.name())) +} + +/// Unknown contents never acquire a zero-byte measurement. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Usage { + Measured(u64), + /// Bytes successfully measured, with some contents still unknown. + Incomplete(u64), + Unavailable, + OutsideScan, +} + +/// Service-store location for whole startup-disk and external-volume scans. +/// A home scan has no service-store figures; it is not evidence of absence. +pub fn stores_volume(root: &Path) -> Option { + let data = Path::new(crate::space::MACOS_DATA_VOLUME); + if data.starts_with(root) || root == data { + return Some(data.to_path_buf()); + } + if let Ok(below) = root.strip_prefix("/Volumes") { + return below + .components() + .next() + .map(|name| Path::new("/Volumes").join(name.as_os_str())); + } + if root + .file_name() + .is_some_and(|name| is_service_name(&name.to_string_lossy())) + { + return root.parent().map(Path::to_path_buf); + } + None +} + +/// Look up an actual path in the scan; no I/O and no invented totals. +pub fn usage( + root_path: &Path, + tree: &Node, + options: &ScanOptions, + path: &Path, +) -> Usage { + let Ok(below) = path.strip_prefix(root_path) else { + return Usage::OutsideScan; + }; + let mut node = tree; + for part in below.components() { + let Some(child) = node.child_named(&part.as_os_str().to_string_lossy()) + else { + // Missing from a walk can mean failed metadata, filtering, or + // no directory. It cannot establish a service's installation. + return Usage::Unavailable; + }; + node = child; + } + if !node.is_dir() { + return Usage::Unavailable; + } + if node.read_error || !options.include_hidden || options.max_depth.is_some() + { + return if node.bytes == 0 { + Usage::Unavailable + } else { + Usage::Incomplete(node.bytes) + }; + } + Usage::Measured(node.bytes) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::tree::{Metric, NodeKind, aggregate}; + + #[test] + fn unavailable_is_not_zero_and_descendant_failures_are_incomplete() { + let path = Path::new("/volume/.Spotlight-V100"); + let mut tree = Node::directory("volume"); + let options = ScanOptions::default(); + assert_eq!( + usage(Path::new("/volume"), &tree, &options, path), + Usage::Unavailable + ); + let mut store = Node::directory(".Spotlight-V100"); + store.read_error = true; + tree.children.push(store); + aggregate(&mut tree, Metric::Bytes); + assert_eq!( + usage(Path::new("/volume"), &tree, &options, path), + Usage::Unavailable + ); + let store = &mut tree.children[0]; + store + .children + .push(Node::entry("index", NodeKind::File, 4096)); + let mut locked = Node::directory("locked"); + locked.read_error = true; + store.read_error = false; + store.children.push(locked); + aggregate(&mut tree, Metric::Bytes); + assert_eq!( + usage(Path::new("/volume"), &tree, &options, path), + Usage::Incomplete(4096) + ); + } + + #[test] + fn readable_empty_store_is_measured_but_filtered_contents_are_unknown() { + let tree = Node::directory(".fseventsd"); + let path = Path::new("/volume/.fseventsd"); + let mut options = ScanOptions::default(); + assert_eq!(usage(path, &tree, &options, path), Usage::Measured(0)); + options.include_hidden = false; + assert_eq!(usage(path, &tree, &options, path), Usage::Unavailable); + options.include_hidden = true; + options.max_depth = Some(0); + assert_eq!(usage(path, &tree, &options, path), Usage::Unavailable); + assert_eq!( + usage(Path::new("/Users/me"), &tree, &options, path), + Usage::OutsideScan + ); + } + + #[test] + fn private_stores_belong_to_the_data_volume_and_external_volumes() { + assert_eq!( + stores_volume(Path::new("/")), + Some(PathBuf::from(crate::space::MACOS_DATA_VOLUME)) + ); + assert_eq!( + stores_volume(Path::new("/Volumes/Backup")), + Some(PathBuf::from("/Volumes/Backup")) + ); + assert_eq!(stores_volume(Path::new("/Users/me")), None); + } +} diff --git a/crates/disktree-core/src/remote.rs b/crates/disktree-core/src/remote.rs index 27da007f..7718c4be 100644 --- a/crates/disktree-core/src/remote.rs +++ b/crates/disktree-core/src/remote.rs @@ -577,7 +577,7 @@ impl RemoteScan { Self::spawn_with_archive(command, host, request, archive) } - #[cfg(test)] + #[cfg(all(test, unix))] fn spawn_command( command: Command, host: &str, @@ -744,6 +744,7 @@ mod tests { } } + #[cfg(unix)] fn wait(scan: &RemoteJob) -> io::Result { wait_for(scan, std::time::Duration::from_secs(10)) } @@ -1209,18 +1210,25 @@ mod tests { .unwrap(); assert_eq!(snapshot.tree.bytes, if dedup { 10 } else { 15 }); assert_eq!(snapshot.tree.files, 3); - let preview = shared.lock().unwrap().preview.clone().unwrap(); - let nested = preview - .entries - .iter() - .find(|entry| entry.name == "nested") - .unwrap(); - assert_eq!(nested.bytes, 5); - assert_eq!(nested.files, 1); - assert_eq!( - preview.entries.iter().map(|e| e.bytes).sum::(), - snapshot.tree.bytes - ); + // Only the Unix streaming walker emits live-tree controls; + // the Windows fallback sends the completed tree without them. + #[cfg(unix)] + { + let preview = shared.lock().unwrap().preview.clone().unwrap(); + let nested = preview + .entries + .iter() + .find(|entry| entry.name == "nested") + .unwrap(); + assert_eq!(nested.bytes, 5); + assert_eq!(nested.files, 1); + assert_eq!( + preview.entries.iter().map(|e| e.bytes).sum::(), + snapshot.tree.bytes + ); + } + #[cfg(not(unix))] + assert!(shared.lock().unwrap().preview.is_none()); } } diff --git a/crates/disktree-core/src/removal.rs b/crates/disktree-core/src/removal.rs index 60277529..3f806f8d 100644 --- a/crates/disktree-core/src/removal.rs +++ b/crates/disktree-core/src/removal.rs @@ -426,6 +426,16 @@ fn refuse( if key == root_key { return Some("the scanned root cannot be removed".into()); } + if path.components().any(|part| { + crate::macos_services::is_service_name( + &part.as_os_str().to_string_lossy(), + ) + }) { + return Some( + "macOS manages this history or index; use Apple's service tools" + .into(), + ); + } if cfg!(windows) && let Some(name) = path.components().find_map(|part| match part { Component::Normal(name) => { @@ -1877,6 +1887,24 @@ mod tests { ); } + #[test] + fn macos_service_stores_and_their_contents_are_never_raw_deleted() { + for volume in ["/", "/System/Volumes/Data", "/Volumes/External"] { + for store in [".fseventsd", ".Spotlight-V100", ".SPOTLIGHT-V100"] { + let path = Path::new(volume).join(store); + for path in [path.clone(), path.join("index/blob")] { + let reason = + refuse(&path, Path::new(volume), None, &[], &[]) + .expect("service-owned contents must be kept"); + assert!( + reason.contains("Apple's service tools"), + "{reason}" + ); + } + } + } + } + #[test] fn paths_outside_the_root_are_refused() { let temp = tree(); diff --git a/crates/disktree-core/src/scan.rs b/crates/disktree-core/src/scan.rs index 5ffe81a0..3caa70ec 100644 --- a/crates/disktree-core/src/scan.rs +++ b/crates/disktree-core/src/scan.rs @@ -247,6 +247,8 @@ impl ScanHandle { root_device: Mutex::new(None), foreign_mounts: OnceLock::new(), never_scanned: OnceLock::new(), + data_services: OnceLock::new(), + data_device: OnceLock::new(), visited_dirs: Mutex::new(FxHashSet::default()), root: Mutex::new(None), volume: OnceLock::new(), @@ -297,6 +299,8 @@ pub fn scan(root: &Path, options: ScanOptions) -> io::Result { root_device: Mutex::new(None), foreign_mounts: OnceLock::new(), never_scanned: OnceLock::new(), + data_services: OnceLock::new(), + data_device: OnceLock::new(), visited_dirs: Mutex::new(FxHashSet::default()), root: Mutex::new(None), volume: OnceLock::new(), @@ -320,6 +324,9 @@ struct WalkContext { /// Directories no scan enters, on any volume setting: see /// [`crate::space::never_scanned`]. never_scanned: OnceLock>, + /// A second Data mount exposes private stores, never firmlink aliases. + data_services: OnceLock, + data_device: OnceLock>, /// Directories already entered, so followed symlinks cannot loop. visited_dirs: Mutex>, /// Set by the root's `complete`, read after the scope joins. @@ -355,7 +362,7 @@ impl WalkContext { Ok(listing) => listing, Err(error) => { self.progress.record_error(&entry.entry_path(dir), &error); - return Classified::Skipped; + return Classified::Unreadable; } }; @@ -380,7 +387,7 @@ impl WalkContext { Ok(facts) => self.leaf(entry.take_name(), kind, &facts), Err(error) => { self.progress.record_error(&entry.entry_path(dir), &error); - Classified::Skipped + Classified::Unreadable } } } @@ -397,6 +404,14 @@ impl WalkContext { { return Classified::Skipped; } + if self.data_services.get() == Some(&path) { + // `/` presents the system volume beside Data. This one mount + // exception is constrained by the whitelist in `walk`. + return Classified::Subdirectory { + path, + name: entry.take_name(), + }; + } if self.options.one_filesystem && let Some(foreign) = self.foreign_mounts.get() { @@ -419,8 +434,8 @@ impl WalkContext { return Classified::Skipped; } // Memoized: the subtree a narrower scan already measured is taken - // whole, before any volume rule, since it was measured under the same - // rules. + // whole after mount boundaries, since a cached subtree cannot make + // crossing a mount allowed. if let Some(known) = &self.known && known.path == path { @@ -429,7 +444,18 @@ impl WalkContext { return Classified::Entry(tree); } if self.options.one_filesystem && self.foreign_mounts.get().is_none() { - match (directory(), self.root_device(&path)) { + let device = if let Some(data) = self + .data_services + .get() + .filter(|data| path.starts_with(data)) + { + *self.data_device.get_or_init(|| { + fs::metadata(data).map(|meta| device_of(&meta)).ok() + }) + } else { + self.root_device(&path) + }; + match (directory(), device) { (Ok(directory), Some(root_device)) if directory.device != root_device => { @@ -437,7 +463,7 @@ impl WalkContext { } (Err(error), _) => { self.progress.record_error(&path, error); - return Classified::Skipped; + return Classified::Unreadable; } _ => {} } @@ -464,7 +490,7 @@ impl WalkContext { } Err(error) => { self.progress.record_error(path, &error); - Classified::Skipped + Classified::Unreadable } }; } @@ -475,6 +501,7 @@ impl WalkContext { // A dangling link is normal, not an error worth counting. if error.kind() != io::ErrorKind::NotFound { self.progress.record_error(path, &error); + return Classified::Unreadable; } return Classified::Skipped; } @@ -775,8 +802,10 @@ enum Classified { Subdirectory { path: PathBuf, name: Box }, /// A leaf that contributes size. Entry(Node), - /// Filtered out, unreadable, or a symlink we chose not to follow. + /// Filtered out, or a symlink we chose not to follow. Skipped, + /// Collection failed, rather than an intentional filter. + Unreadable, } /// One directory being walked, plus the counter that decides when it is done. @@ -883,6 +912,9 @@ fn scan_on_pool( } let resolved = root.canonicalize().ok(); let canonical = resolved.as_deref().unwrap_or(root); + if let Some(data) = crate::space::data_services_only(root, canonical) { + let _ = context.data_services.set(data); + } let mut never = crate::space::never_scanned(root, canonical); // Hidden-entry and depth filters can leave an alias as the only route // to files. Keep every mount view when either filter is active. @@ -1110,6 +1142,13 @@ fn walk(dir: &Arc, context: &WalkContext) -> Vec> { } match entry { Ok(mut entry) => { + if context.data_services.get() == Some(&dir.path) + && !crate::macos_services::is_service_name( + entry.name(), + ) + { + continue; + } match context.classify(&dir.path, &mut entry) { Classified::Subdirectory { path, name } => { tally.dirs += 1; @@ -1125,6 +1164,9 @@ fn walk(dir: &Arc, context: &WalkContext) -> Vec> { leaves.push(node); } Classified::Skipped => {} + Classified::Unreadable => { + dir.read_error.store(true, Ordering::Relaxed); + } } if tally.files + tally.dirs >= TALLY_EVERY { tally.flush(&context.progress, &dir.path); @@ -1132,6 +1174,7 @@ fn walk(dir: &Arc, context: &WalkContext) -> Vec> { } Err(error) => { context.progress.record_error(&dir.path, &error); + dir.read_error.store(true, Ordering::Relaxed); } } } @@ -1413,6 +1456,51 @@ mod tests { }) } + #[cfg(unix)] + #[test] + fn a_second_data_mount_counts_private_stores_without_its_user_aliases() { + let temp = TempDir::new().expect("tempdir"); + let root = temp.path(); + let ordinary = write(root, "Users/me/document", 100); + let data = root.join("Data"); + fs::create_dir_all(data.join("Users/me")).expect("mkdir"); + fs::hard_link(&ordinary, data.join("Users/me/document")) + .expect("an alias for the same file"); + let history = write(root, "Data/.fseventsd/events", 1234); + let index = write(root, "Data/.Spotlight-V100/index", 4567); + let duplicate = data.join(".Spotlight-V100/another-name"); + fs::hard_link(&index, &duplicate).expect("hardlink inside a store"); + write(root, "Data/not-a-private-store/blob", 9999); + let context = Arc::new(WalkContext { + known: None, + options: ScanOptions::default(), + progress: Arc::new(ScanProgress::default()), + root_device: Mutex::new(None), + // The Data mount is an explicit boundary on modern macOS. Only + // its whitelisted private stores may cross it. + foreign_mounts: OnceLock::from(FxHashSet::from_iter( + [data.clone()], + )), + never_scanned: OnceLock::new(), + data_services: OnceLock::from(data), + data_device: OnceLock::new(), + visited_dirs: Mutex::new(FxHashSet::default()), + root: Mutex::new(None), + volume: OnceLock::new(), + }); + let tree = scan_blocking(root, &context).expect("scan"); + let size = |path: &Path| { + allocated_bytes(&fs::metadata(path).expect("metadata")) + .expect("Unix allocation") + }; + assert_eq!(tree.bytes, size(&ordinary) + size(&history) + size(&index)); + let data = child(&tree, "Data"); + assert_eq!(data.children.len(), 2); + assert_eq!(child(data, ".fseventsd").bytes, size(&history)); + assert_eq!(child(data, ".Spotlight-V100").bytes, size(&index)); + assert_eq!(context.progress.snapshot().errors, 0); + } + #[test] fn fixed_worker_pools_change_size_without_reinitializing_rayon() { let cpus = thread::available_parallelism().map_or(1, usize::from); @@ -1476,6 +1564,8 @@ mod tests { root_device: Mutex::new(None), foreign_mounts: OnceLock::new(), never_scanned: OnceLock::new(), + data_services: OnceLock::new(), + data_device: OnceLock::new(), visited_dirs: Mutex::new(FxHashSet::default()), root: Mutex::new(None), volume: OnceLock::new(), @@ -1756,6 +1846,8 @@ mod tests { root_device: Mutex::new(None), foreign_mounts: OnceLock::new(), never_scanned: OnceLock::new(), + data_services: OnceLock::new(), + data_device: OnceLock::new(), visited_dirs: Mutex::new(FxHashSet::default()), root: Mutex::new(None), volume: OnceLock::new(), @@ -1763,12 +1855,22 @@ mod tests { let tree = scan_blocking(root, &context).expect("scan still succeeds"); let _ = fs::set_permissions(&locked, fs::Permissions::from_mode(0o700)); - assert_eq!(tree.bytes, 11); + assert_eq!(tree.bytes, if readable.is_err() { 11 } else { 33 }); let node = child(&tree, "locked"); - if matches!(readable, Ok(false)) { + if readable.is_err() { assert!(node.read_error); + assert!(tree.read_error, "unknown contents affect ancestor totals"); assert!(progress.snapshot().errors >= 1); assert!(!progress.snapshot().messages.is_empty()); + assert_eq!( + crate::macos_services::usage( + root, + &tree, + &context.options, + &locked + ), + crate::macos_services::Usage::Unavailable + ); } } diff --git a/crates/disktree-core/src/space.rs b/crates/disktree-core/src/space.rs index 2434c6cc..2d727a1b 100644 --- a/crates/disktree-core/src/space.rs +++ b/crates/disktree-core/src/space.rs @@ -658,23 +658,35 @@ pub const MACOS_DATA_VOLUME: &str = "/System/Volumes/Data"; /// Directories under `root` a scan must never enter, whatever the volume /// rules say, given where `root` really is (`canonical`). /// -/// On macOS: the Data volume's second mount, which is every firmlinked -/// directory again under another name, so walking it counts the disk twice; -/// and `/Network`, an automount point that would reach for a server. The +/// On macOS: `/Network`, an automount point that would reach for a server. +/// The Data volume is restricted separately to its private service stores, +/// so firmlinked directories are not visited again. The /// paths are returned in `root`'s own spelling, because that is how the walk /// names what it finds. pub fn never_scanned(root: &Path, canonical: &Path) -> Vec { if !cfg!(target_os = "macos") { return Vec::new(); } - [MACOS_DATA_VOLUME, "/Network"] - .iter() + std::iter::once("/Network") .filter_map(|skip| Path::new(skip).strip_prefix(canonical).ok()) .filter(|below| !below.as_os_str().is_empty()) .map(|below| root.join(below)) .collect() } +/// A Data mount below the root must expose only private service stores. +/// Scanning Data directly still walks the entire volume once. +pub fn data_services_only(root: &Path, canonical: &Path) -> Option { + if !cfg!(target_os = "macos") { + return None; + } + Path::new(MACOS_DATA_VOLUME) + .strip_prefix(canonical) + .ok() + .filter(|below| !below.as_os_str().is_empty()) + .map(|below| root.join(below)) +} + /// The top of the disk `path` lives on: its mount point, from `statfs`. /// /// The Data volume answers `/System/Volumes/Data`, but the disk a user @@ -1371,7 +1383,7 @@ map auto_home on /System/Volumes/Data/home (autofs, automounted, nobrowse) } #[test] - fn only_macos_skips_the_data_volume_and_in_the_roots_spelling() { + fn macos_restricts_data_aliases_but_direct_data_scans_are_unrestricted() { let skipped = never_scanned(Path::new("/"), Path::new("/")); let below_system = never_scanned(Path::new("/System/"), Path::new("/System")); @@ -1382,19 +1394,34 @@ map auto_home on /System/Volumes/Data/home (autofs, automounted, nobrowse) Path::new(MACOS_DATA_VOLUME), ); if cfg!(target_os = "macos") { + assert_eq!(skipped, vec![PathBuf::from("/Network")]); + assert!(below_system.is_empty()); assert_eq!( - skipped, - vec![PathBuf::from(MACOS_DATA_VOLUME), "/Network".into()] - ); - assert_eq!( - below_system, - vec![PathBuf::from("/System/Volumes/Data")] + data_services_only( + Path::new("/spelling"), + Path::new("/System") + ), + Some(PathBuf::from("/spelling/Volumes/Data")) ); } else { assert!(skipped.is_empty() && below_system.is_empty()); } assert!(inside.is_empty()); assert!(itself.is_empty(), "asking for it by name scans it"); + assert_eq!( + data_services_only( + Path::new(MACOS_DATA_VOLUME), + Path::new(MACOS_DATA_VOLUME) + ), + None + ); + assert_eq!( + data_services_only( + Path::new("/Users/tobi"), + Path::new("/Users/tobi") + ), + None + ); } #[test] diff --git a/crates/disktree-core/src/tree.rs b/crates/disktree-core/src/tree.rs index 4bbeda44..6a0359a4 100644 --- a/crates/disktree-core/src/tree.rs +++ b/crates/disktree-core/src/tree.rs @@ -84,7 +84,7 @@ pub struct Node { /// `(device, inode)` for files, used to de-duplicate hardlinks. No file /// has inode zero, which lets `None` cost nothing. pub inode: Option<(u64, NonZeroU64)>, - /// The directory could not be read; its contents are unknown. + /// Some entries here or below could not be read; totals are incomplete. pub read_error: bool, /// Newest write time at or beneath this node, in Unix seconds; `0` when /// unknown. Derived for directories by [`aggregate`]. @@ -388,6 +388,7 @@ fn aggregate_at( } } for child in &node.children { + node.read_error |= child.read_error; modified = modified.max(child.modified); // Saturating: a corrupt volume's file table can claim any size. bytes = child.bytes.saturating_add(bytes);