Repository navigation
feat: v2.2 Workflow Objets (v2.2.2) - #165
Merged
Merged
Conversation
…141 #142 #143 #144) - wails-bindings.md: GetFileState(backendID, localPath) → L/P/U/S/C/E/X - wails-events.md: sync:conflict + sync:file-state-changed specs - CHANGELOG.md: BREAKING changes (Rename/Copy interface), new types, new events Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…t queue (#136 #139 #140 #141 #142 #143 #144) P1 (#136) — per-file FileState cache in Engine: - types: FileState L/P/U/S/C/E/X constants - Engine.fileStates sync.Map + Get/SetFileState + makeStateUpdater - Dispatcher.stateUpdater callback (P→U→S/E transitions in execute()) - Reconciler.stateUpdater (L-state for local-only files) + conflictEmitter - App.GetFileState() Wails binding P2a (#139) — ActionRename with server-side rename: - plugins.ErrNotSupported sentinel - StorageBackend.Rename() interface method - Watcher: rename-pair detection with 150ms timer (Rename+Create → FileEventRenamed) - Dispatcher: ActionRename + fallback Upload+Delete on ErrNotSupported - Engine.handleLocalEvent: FileEventRenamed case - Plugins: Rename() in webdav (MOVE), moosefs (native), local (os.Rename), mock - Proto: rpc Rename(MoveRequest) returns (MoveResponse) + regenerated pb.go - gRPC: server.Rename + client.Rename; mapBackendError/mapGRPCError handle ErrNotSupported P2b (#142) — Atomic unpin guard: - App.PinFile: upload first if state != S before dehydrating CF placeholder - Dispatcher: CFSyncStateCloudOnly = 0 constant P2c (#143) — Pin intent queue: - CFManager.intentions sync.Map + QueuePinIntent + ConsumeIntent - Engine.SetIntentCallbacks: consumeIntent + pinFile injected from App - makeStateUpdater: consumes intent when U→S transition completes - App.PinFile: queues intent if state == U or P (active transfer) - App.StartSync: wires intent callbacks into engine P3b (#141) — Guard L-state delete: - Engine.handleLocalEvent FileEventDeleted: skips ActionDelete for L-state files P4 (#144) — Conflict event: - Reconciler emits sync:conflict via conflictEmitter (backendID, path, mods, resolution) - StorageBackend.Copy() interface method + ActionCopy in Dispatcher P2a also adds: - ActionCopy type + Dispatcher execute case (with ErrNotSupported fallback) - plugins.Copy() in webdav (HTTP COPY RFC 4918), moosefs (ErrNotSupported), local (io.Copy) - Proto: rpc Copy(CopyRequest) returns (CopyResponse) + pb.go regenerated - SDK echo plugin: Rename() + Copy() (returns ErrNotSupported as example) Test fixes: Rename/Copy stubs added to all test mocks (17 packages pass) Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
- types/ghostdrive.ts : ConflictEvent interface + WailsEventMap['sync:conflict'] - components/ui/Toast.tsx : composant accessible (role=alert, aria-live=polite), auto-dismiss 5s, AlertTriangle amber, dismiss card/bouton × - hooks/useSyncStatus.ts : listener sync:conflict avec debounce 500ms/path, max 3 toasts individuels, regroupement ">3 conflits" en summary toast - App.tsx : overlay fixed bas-droit, rendu conditionnel des ConflictToast - 13 tests vitest (rendu, dismiss, auto-dismiss, cleanup timer) 42/42 tests pass — closes #144 (frontend) Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
#141 #142 #143 #144) - internal/sync/filestate_test.go: GetFileState/SetFileState concurrent, U→S→E transitions, L-guard (FileStateLocal skips remote delete), makeStateUpdater events + intent callback - internal/sync/dispatcher_test.go: ActionRename (native + ErrNotSupported fallback), ActionCopy (server-side + fallback), StateUpdater P→U→S/E lifecycle, nil-guard - internal/sync/watcher_rename_test.go: rename-pair detection → FileEventRenamed with OldPath, no-Create timeout → FileEventDeleted, cross-dir rename - internal/cfapi/manager_intent_test.go: QueuePinIntent (basic, overwrite, multi-path), ConsumeIntent (no-op, clears entry), concurrent thread-safety - plugins/webdav/rename_copy_test.go: Rename via MOVE (OK, notFound, notConnected), Copy via HTTP COPY (OK, notFound, overwrite, preservesSource, notConnected) - plugins/local/rename_copy_test.go: Rename via os.Rename (OK, notFound, subdir), Copy via io.Copy (OK, notFound, createsParentDirs, overwrite) - plugins/moosefs/copy_errnot_test.go: Copy returns ErrNotSupported (connected/disconnected/ cancelledCtx), Rename delegates to Move - tests/integration/workflow_v22_test.go: full dispatch workflows — ActionRename (native + fallback), ActionDelete L-guard, ActionCopy (server-side + fallback), ErrNotSupported detectability cross-package - frontend/src/hooks/useSyncStatus.test.ts: sync:conflict → toast, debounce 500ms, collapse >3, reset counter on dismiss-all - tests/procedures/v2.2-workflow-objets.md: 8 QA manual scenarios Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…opy, UploadFile 🔴 CRITIQUE — watcher.go: suppression race condition pendingRenames (AfterFunc goroutine vs boucle select) via canal expiredRenames (cap 16) — cleanup délégué au goroutine principal. 🟠 MAJEUR — dispatcher.go ActionCopy: ajout SetSyncState(CFSyncStateSynced) après copy réussie — badge Explorateur ✓✓ au lieu de ☁️ (identique à ActionDownload). 🟠 MAJEUR — engine.go: UploadFile(ctx, localPath) dispatch ActionUpload ciblé (1 fichier) — app.go PinFile remplace ForceSync() (réconciliation complète) par UploadFile() (#142). Optionnel — engine.go makeStateUpdater: guard prev==state évite flood events identiques. Optionnel — dispatcher.go ActionRename fallback: état E si delete échoue (doublon remote). Optionnel — frontend/src/App.tsx: DEFAULT_CONFIG version '' (était '0.6.0' hardcodé). Tests: 17/17 packages ok -race 0 races. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
- Ajout entrée CHANGELOG v2.2.0 avec breaking changes (StorageBackend Rename/Copy), Added (7 issues #136-#144), Fixed (3 bugs), Tests (+67) - Mise à jour versions : config.json + frontend/package.json → 2.2.0 - Nouvelle matrice états/actions : états L/C/X, actions Rename/Copy natives, Guard L-state - docs/plugin-development.md : documentation Rename()/Copy() et ErrNotSupported sentinel - README.md : v2.2 marquée livrée avec highlights Co-Authored-By: Claude Haiku 4.5 <[email protected]>
Vitest's toHaveLength matcher only accepts 1 argument; the message string accepted by Jest causes a tsc error in Vitest's type definitions. Removed the second argument on 3 assertions (lines 165, 183, 228). 51/51 tests pass. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Ajoute `isCompatibleBinary()` qui détecte le token OS dans le nom du fichier (ex: `-linux-`, `-windows-`) en se basant sur le pattern `-v<version>-<os>-<arch>.ghdp`. Tout binaire taggé avec un OS différent de `runtime.GOOS` est ignoré silencieusement avec un log WARN, et n'apparaît ni en « loaded » ni en « failed » dans `GetLoadedPlugins()`. Fichiers modifiés : - plugins/loader/grpc_loader.go : +runtime import, pluginOSRe, isCompatibleBinary, filtre dans Scan() - plugins/loader/export_test.go : expose IsCompatibleBinaryForTest - plugins/loader/grpc_loader_test.go : 5 nouveaux tests (unit + integration) Fixes #145 Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Les 3 chemins d'activation (Startup, SetBackendEnabled, UpdateBackend) émettent désormais `logger.Error(...)` avant de retourner l'erreur, ce qui garantit que les échecs apparaissent dans le log fichier et dans l'onglet Logs de l'UI — pas seulement dans l'événement Wails. Fichiers modifiés : - internal/app/app.go : logger.Error aux 3 sites d'activation - internal/app/app_test.go : 2 nouveaux tests + import bytes/logger Fixes #146 Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…ns .ghdp" This reverts commit 87ca2ae.
…kend Quand l'application démarre avec zéro backend activé, MountUnified n'est jamais appelé (Startup skip la section si len(unifiedBackends)==0). SetBackendEnabled appelait ensuite UpdateBackends qui échouait immédiatement avec "unified drive not mounted", bloquant définitivement l'activation manuelle. Fix : dans le chemin d'activation de SetBackendEnabled, on vérifie si le drive unifié est monté via GetUnifiedStatus() : - drive monté → UpdateBackends (chemin existant) - drive absent → MountUnified (premier backend activé manuellement) Test ajouté : TestSetBackendEnabled_FirstActivation_NoBrackendAtStartup — vérifie que l'erreur n'est plus "unified drive not mounted" et que le rollback est propre en cas d'échec de montage (NullDrive Linux CI). Fixes #147 Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Deux root causes : 1. watcher.go émettait des chemins ABSOLUS (filepath.ToSlash(fsEvent.Name)). handleLocalEvent les traitait comme relatifs en préfixant e.remotePath / e.localDir → chemins distants doublés (/remote/tmp/xxx/new.txt) → backend.Rename échouait → la réconciliation ré-téléchargeait l'ancien nom → 2 fichiers coexistaient localement. Fix : filepath.Rel(w.dir, fsEvent.Name) avant ToSlash → chemins relatifs. 2. handleRemoteEvent n'avait aucun case pour FileEventRenamed → retour nil silencieux. Les backends émettaient FileEventCreated pour le nouveau nom → ActionDownload créait un nouveau fichier sans renommer l'existant. Fix : case FileEventRenamed → os.Rename local (fallback : download si l'ancien fichier local est absent). Tests ajoutés : - TestWatcherEmitsRelativePaths : vérifie que les chemins watcher sont relatifs - TestEngine_HandleLocalEvent_Rename_PropagatesRemote : rename local → backend - TestEngine_HandleRemoteEvent_Rename_RenamesLocally : rename distant → local - TestEngine_HandleRemoteEvent_Rename_FallbackDownload : fallback si absent 73/73 PASS internal/sync — 0 régression sur ./... Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…erts Root causes introduced in bc86602 (#136 fileStates cache): 1. State inheritance bug (engine.go handleLocalEvent FileEventRenamed): SetFileState(newPath, oldState) propagated FileStateLocal to the renamed file's new path. The #141 guard then skipped ActionDelete for that file, causing deleted files to reappear after the next reconciliation. Fix: block inheritance when oldState == FileStateLocal. 2. Reconciler S→L race (engine.go makeStateUpdater): The reconciler could race with a concurrent upload and call updateState(FileStateLocal) on an already-Synced file (because its remote snapshot predated the upload completion). This again triggered the #141 delete-skip guard for a legitimately synced file. Fix: guard in makeStateUpdater prevents S→L downgrades. 3. Stale Create debounce timer (watcher.go): When a file was created and renamed within the 500ms debounce window, the original Create timer for the old path was not cancelled. It fired 500ms later emitting a stale FileEventCreated for a non-existent file, causing a spurious ActionUpload failure. Fix: stop and delete pending[path] when a Rename event is detected. Tests added (3): - TestEngine_HandleLocalEvent_Rename_DoesNotInheritLocalState - TestEngineMakeStateUpdater_NoDowngradeSyncedToLocal - TestWatcherCreateRename_NoStaleCreateEvent Results: 76/76 PASS internal/sync, 18/18 packages PASS, 0 regressions. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…eedback, Explorer refresh) Three sync remote→local bugs fixed: 1. **Delete propagation**: `handleRemoteEvent FileEventDeleted` previously emitted only a "V1 policy — manual review" warning with no `os.Remove`. Local file now deleted when the remote file is removed. 2. **Rename feedback loop**: `handleRemoteEvent FileEventRenamed` called `os.Rename` on the local filesystem, triggering fsnotify events that `handleLocalEvent` re-processed as local renames → spurious `ActionRename` to the backend. Both the old and new paths are now suppressed before `os.Rename` so the feedback loop is broken. Also: when Watch() emits Delete+Create pairs instead of FileEventRenamed, the Delete is now properly applied (fix 1 above), preventing both names from coexisting locally. 3. **Explorer refresh**: Added `LocalRefresher` interface (`NotifyLocalChange(path string)`) satisfied by `*cfapi.CFManager` via `ghd_notify_dir_change` (new C shim calling `SHChangeNotify(SHCNE_UPDATEDIR, SHCNF_PATH|SHCNF_FLUSH, dir, NULL)`). Called after ActionDownload and after each remote-triggered local filesystem change so Explorer refreshes without F5. Implementation: - `suppressedPaths gosync.Map` on Engine + `suppressLocalEvent` / `isLocalEventSuppressed` (TTL = debounceDuration + 1s slack). Checked at top of `handleLocalEvent` for main path and OldPath. - `cfapi/notify_windows.go` (Windows impl) + `cfapi/notify_stub.go` (non-Windows no-op) - `cgo_cfapi_windows.c/.h`: new `ghd_notify_dir_change(LPCWSTR dir)` function - 5 new tests: Delete_RemovesLocally, Delete_SuppressesLocalFeedback, Rename_SuppressesBothPaths, Delete_NonExistentLocal, SuppressExpiry 18/18 packages PASS — 0 regressions. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Script bash exécutable depuis WSL qui valide les 5 opérations CRUD (New/Copy/Rename/Open/Delete) dans les deux sens : LOCAL→GhD: et GhD:→LOCAL. Remplace la validation manuelle Windows avec timeout 10s et analyse logs CF API. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…ate (#151) In v2.2 (bc86602 Phase 4), dispatcher.go calls CfSetInSyncState after ActionDownload to badge newly-downloaded files ✓✓. Download() writes files via os.Rename(tmp, dest) — the result is a regular NTFS file, never a CF placeholder. CfSetInSyncState on a non-placeholder writes invalid CF reparse-point data, corrupting the parent directory's CF state and causing 0x80070781 (ERROR_INVALID_REPARSE_DATA) + 0x8007017c (ERROR_CLOUD_FILE_ INVALID_REQUEST) on subsequent user operations (New File, Rename from local). Fix: add PlaceholderMaker interface to dispatcher.go; call ConvertToPlaceholder (→ ghd_convert_to_placeholder / CF_CONVERT_FLAG_MARK_IN_SYNC) before SetSyncState in ActionDownload and ActionCopy. Idempotent: if the file is already a CF placeholder the call succeeds as a no-op. Files: - internal/cfapi/provider.go — SyncProvider.ConvertToPlaceholder (Windows CGO) - internal/cfapi/provider_stub.go — no-op stub for non-Windows - internal/cfapi/manager.go — CFManager.ConvertToPlaceholder - internal/sync/dispatcher.go — PlaceholderMaker interface + wiring in ActionDownload and ActionCopy - internal/sync/dispatcher_test.go — 3 new tests: TestDispatcher_ActionDownload_ConvertBeforeSetSync TestDispatcher_ActionCopy_ConvertBeforeSetSync TestDispatcher_ActionDownload_NoPlaceholderMaker_NoPanic 18/18 PASS — 0 regressions Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
If CfConvertToPlaceholder fails (handle invalid, file locked, etc.) the previous code still called CfSetInSyncState on the non-placeholder file. That writes invalid CF reparse-point data into the NTFS file, corrupting the parent directory's CF state → 0x80070781 on subsequent user ops. Introduce a placeholderOK bool in ActionDownload and ActionCopy: if ConvertToPlaceholder returns an error, emit sync:warn and skip SetSyncState entirely. Skipping the badge update is safe; calling CfSetInSyncState on a non-placeholder is the root cause of the regression. Closes #152 Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…#153 #154 #155) Three path-handling bugs in engine.go fixed in one commit as they all touch the same region of handleLocalEvent / handleRemoteEvent: #153 — Windows backslash in fsnotify evt.Path (handleLocalEvent) fsnotify on Windows returns paths like "subdir\file.txt" for files in subdirectories. path.Join (Unix semantics) treats "\" as a filename character, producing "/remote/subdir\file.txt" → backend.Delete/Move targets the wrong remote path → 404 → remote preserved → file revives. Fix: filepath.ToSlash(evt.Path) before path.Join. #154 — Double filepath.Dir in NotifyLocalChange calls (handleRemoteEvent) handleRemoteEvent was passing filepath.Dir(localPath) to NotifyLocalChange, which then applied filepath.Dir again → grandparent directory notified instead of the file's parent. Fix: pass localPath (file path) so NotifyLocalChange computes the parent. #155 — Leading "/" in remote Watch() evt.Path (handleRemoteEvent) Remote backends (WebDAV) may emit paths like "/file.txt". On Windows filepath.Join("C:\SyncRoot", "/file.txt") = "C:\file.txt" (absolute second component wins, drops sync root prefix) → os.Remove/Rename targets a wrong path outside the sync root → no-op → file not deleted. Fix: strings.TrimLeft(filepath.FromSlash(evt.Path), "/\") before Join. Added logger.Error/Info for os.Remove and os.Rename outcomes. Closes #153 Closes #154 Closes #155 Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
SHChangeNotify(SHCNE_UPDATEDIR, SHCNF_PATH, dir, NULL) on Windows requires the directory path to end with a backslash. Without it the shell silently ignores the notification and Explorer never refreshes. filepath.Dir() does not append a trailing separator. Add it explicitly in NotifyLocalChange before converting to wide string for CGO. Closes #154 Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…155) Add 4 tests covering the post-151 regressions: TestDispatcher_ActionDownload_ConvertBeforeSetSync (#152) TestDispatcher_ActionCopy_ConvertBeforeSetSync (#152) Verify placeholderOK guard: ConvertToPlaceholder error must prevent SetSyncState from being called (already added in bfada8e, now with correct error-checking semantics). TestEngine_HandleLocalEvent_BackslashPath_Delete (#153) TestEngine_HandleLocalEvent_BackslashPath_Rename (#153) Windows-only (t.Skip on Linux): verify that backslash-separated fsnotify paths are converted to forward slashes before building the remote path. On Linux filepath.ToSlash is a no-op for backslashes (valid filename char), so fsnotify never produces such paths. TestEngine_HandleRemoteEvent_LeadingSlash_Delete (#155) TestEngine_HandleRemoteEvent_LeadingSlash_Rename (#155) Cross-platform: verify that leading "/" in remote Watch() evt.Path is stripped before filepath.Join so os.Remove/Rename targets the correct local path inside the sync root. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…ries (#156) CF_CONVERT_FLAG_ENABLE_ON_DEMAND_POPULATION is not idempotent for directories: calling ghd_convert_dir_to_placeholder on a directory that is already a CF placeholder returns 0x8007017c (ERROR_CLOUD_FILE_INVALID_REQUEST). This is expected and benign on every FETCH_PLACEHOLDERS pass after the first conversion. The previous code logged this as an error on every FETCH_PLACEHOLDERS round-trip, flooding logs with misleading "CfConvertToPlaceholder isDir=true: HRESULT 0x8007017c" messages that suggest CF corruption when no corruption is occurring. Add hrAlreadyPlaceholder = 0x8007017c constant and detect it in the directory conversion branch of createPlaceholdersWithFlags. Treat it as a silent no-op: the directory is already properly set up as a CF placeholder with on-demand population — nothing to do. Unlike MARK_IN_SYNC (for files, truly idempotent), ENABLE_ON_DEMAND_POPULATION deliberately fails on already-converted directories to prevent accidentally clearing the on-demand population state. Note: this does NOT fix the 0x80070781 (ERROR_INVALID_REPARSE_DATA) on New File — that regression has a separate root cause not visible from these logs. The dir 0x8007017c errors are benign; no SetSyncState is called on these directories and the #152 placeholderOK guard does not apply to this code path. Closes #156 Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
#157) ghd_cf_ack_placeholders was using CF_OPERATION_TRANSFER_PLACEHOLDERS_FLAG_NONE (0x0) in the CfExecute ack. FLAG_NONE leaves ENABLE_ON_DEMAND_POPULATION active on the directory → CF considers population incomplete → re-fires FETCH_PLACEHOLDERS on every directory access → 1550+ calls in minutes → concurrent CfCreatePlaceholders calls → CF state corruption → 0x80070781 on New File. Fix: use CF_OPERATION_TRANSFER_PLACEHOLDERS_FLAG_DISABLE_ON_DEMAND_POPULATION (0x2). Marks the directory as fully populated after each callback run. Subdirectories with ENABLE_ON_DEMAND_POPULATION still receive their own FETCH_PLACEHOLDERS on first access; the ack marks each one as done in turn. Ongoing remote changes continue to be delivered by the sync engine (Watch/reconciliation). 3 spec tests added (TestRegression157_*) — 18/18 packages PASS race-detector ✓. Closes #157. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…157 #158) Regression #158: after #157 (ada6568), bidirectional sync completely dead. Root cause: CfExecute with DISABLE_ON_DEMAND_POPULATION modifies the directory CF reparse point, interfering with ReadDirectoryChangesW (fsnotify) and/or blocking CF file operations in the sync root. Fix: revert to FLAG_NONE in ghd_cf_ack_placeholders (no reparse point change) and add a 30s cooldown deduplication guard in ghdOnFetchPlaceholders (provider.go): - First FETCH_PLACEHOLDERS per path: run OnFetchPlaceholders normally + store ts - Subsequent calls within 30s: ack immediately (S_OK, FLAG_NONE), skip backend.List() This stops the callback flood (#157 root cause) without touching CF directory state. Added SyncProvider.populatedDirs (sync.Map) for per-path cooldown tracking. Updated spec tests: replaced #157 DISABLE_ON_DEMAND assertions with #158 FLAG_NONE assertions + 5 new cooldown boundary tests. 18/18 packages PASS, race detector clean. Closes #158. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…branch (#159) 0x80070781 persists on New File even with cooldown (#158): corruption happens during the first FETCH_PLACEHOLDERS pass before the cooldown is active. Root cause: in createPlaceholdersWithFlags, the hrAlreadyExists branch called CfConvertToPlaceholder for all entries including files. When the file is already a CF placeholder (from a prior FETCH_PLACEHOLDERS pass or ActionDownload), calling CfConvertToPlaceholder rewrites its Extended Attributes and can corrupt the parent directory's CF index → 0x80070781 on all subsequent CF operations in that dir. Fix (Option B): skip CfConvertToPlaceholder for files in the hrAlreadyExists branch. Files present when FETCH_PLACEHOLDERS fires are already CF placeholders or NTFS files handled by Watch() + ActionUpload. Directories still need conversion for ENABLE_ON_DEMAND_POPULATION (#156 guard still applies). This reverts the 69628d7 file conversion intentionally: the badge ✓✓ trade-off is worth avoiding CF index corruption that blocks Explorer file creation. Updated spec helpers (convertOnAlreadyExistsSpec, convertPathSpec, dirConvertResultSpec), 4 new TestRegression159_* tests, 6 updated existing tests. 18/18 packages PASS, race detector clean. Closes #159. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…x (#v2.2-bugA) Multiple simultaneous CGO callbacks for the same directory bypassed the 30s cooldown (all saw cache-miss before any stored the timestamp), causing concurrent backend.List() + CfCreatePlaceholders calls logged as "0s ago". Add fetchMu sync.Map of *sync.Mutex keyed by localPath. LoadOrStore ensures only one goroutine executes OnFetchPlaceholders at a time; concurrent ones block then hit the cooldown guard. Also add SetCompletionCallbacks to inject OnDeleteCompletion/OnRenameCompletion after Connect (used by bugD fix). Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…F ops to backend (#v2.2-bugD) CFCallbacks only wired OnFetchData, OnCancelFetch, and OnFetchPlaceholders. Delete and rename on GhD: (CF sync root) fired ghdOnDeleteCompletion / ghdOnRenameCompletion in C but the Go callbacks were nil — the operations were silently dropped and never propagated to the remote backend. Add CompletionCallbacks struct and CFManager.SetCompletionCallbacks() to inject these handlers post-Start. Internally delegates to SyncProvider.SetCompletionCallbacks which updates p.callbacks under the existing mutex. Called from app.go after the sync engine is started. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…ents (#v2.2-bugD) When the user deletes or renames a file in GhD: (the CF sync root, not WinFsp), Windows fires NOTIFY_DELETE_COMPLETION and NOTIFY_RENAME_COMPLETION callbacks. These were unhandled — the remote backend never received the operation and the old path survived on the server, reappearing on the next reconciliation. Add Engine.HandleCFDelete and Engine.HandleCFRename which dispatch ActionDelete and ActionRename respectively to the backend. Both respect the existing file-state guard (#141) and transfer state across rename (#139/#149). Wired from app.go via cfManager.SetCompletionCallbacks after the engine starts. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…art (#v2.2-bugD) After starting the sync engine, inject OnDeleteCompletion and OnRenameCompletion into the CF provider via cfManager.SetCompletionCallbacks. These closures call engine.HandleCFDelete and engine.HandleCFRename respectively, propagating CF API delete/rename notifications from GhD: to the remote backend. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…ws (#v2.2-bugC) strings.HasPrefix is case-sensitive. CF API NormalizedPath may return a drive letter with different casing than the sync root registered path (e.g. "c:\" vs "C:\"). A mismatch caused localToRemote to return the raw Windows absolute path as the remote path, making backend.ReadAt fail and FETCH_DATA transfer 0 bytes (copy/open of a remote placeholder showed 0-byte files). Fix: normalise both paths to lowercase + forward-slash before prefix comparison, then strip using the original byte count (preserves original casing for the relative remote path sent to the backend). Also strip trailing slash from root before comparison to avoid double-slash edge cases. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
cross-compile with mingw failed: undefined reference to __imp_CLSIDFromString. CLSIDFromString (used in cgo_cfapi_windows.c) lives in ole32.dll — adding -lole32 to the LDFLAGS resolves the missing symbol. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…hat #158 sync breakage was Bug C/D, not the flag #158 reverted DISABLE_ON_DEMAND_POPULATION assuming the flag's reparse-point modification killed bidirectional sync. However, Bugs C and D were also present at the time of #158: - Bug C: localToRemote() case mismatch → uploads silently sent to wrong path - Bug D: OnDeleteCompletion/OnRenameCompletion never wired → deletes/renames dropped Now that C and D are fixed (c5875ec, 857f42d, 1005670, bdf0a43) and the per-directory mutex dedup is in place (1377bf5), re-applying the flag tests the hypothesis: if sync remains alive, #158 was caused by Bug C/D, not DISABLE_ON_DEMAND_POPULATION. If sync breaks again, the flag is genuinely at fault and must be reverted. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…sport (#160) Root cause of #160: ReadChunk's frame loop treated the ANTOAN_NOP (cmd=0) keepalive as a fatal unknown opcode, aborting every read a chunk server interleaved a keepalive into — exactly what the reference Python POC skips but the Go port lost. isStaleConnErr then classified that fatal error as a "stale connection" and retried, masking the bug behind a retry that could never converge (the fresh connection gets a NOP too, deterministically). Phase 1 (root cause, csclient.go): - ReadChunk: skip ANTOAN_NOP frames, bounded by maxConsecutiveNOPs so a CS emitting nothing else cannot hang the caller (CA1, CA2, CA3) - errUnexpectedCmd removed from isStaleConnErr; detection is now typed (io.EOF, syscall.ECONNRESET/ECONNABORTED, net.Error) instead of the previous substring matching, which silently absorbed non-retryable errors too (CA4, CA7) - rewrote the file-header and errUnexpectedCmd comments that documented the disproven "half-closed socket" diagnosis Phase 2 (transport hardening, csclient.go/client.go/ecclient.go): - doCSRead: retry policy shared by Client.Read and readEC4At — attempt 0 may reuse a pooled connection, every retry forces a fresh dial (CA5), non-retryable errors fail fast without consuming budget (CA7), retryable ones back off exponentially under a hard cumulative budget kept well below the CF API's own Open() retry interval (CA6) - DialCS: dial timeout + TCP keepalive via net.Dialer - ReadChunk: single deadline covering the whole call; WriteChunk: idle deadline reset before every frame instead, so a large upload isn't falsely timed out by a global deadline (CA9, CA12) - csPool: idle connections are timestamped and evicted past maxIdleConnAge instead of being served stale (CA8); newCSPoolWithMaxAge added for tests go test ./plugins/... -race -cover: all green, mfsclient 74.3% (CA13), webdav/local untouched and green (CA11), WriteChunk suite green (CA12). Co-Authored-By: Claude Sonnet 5 <[email protected]>
…162) D7 — a download interrupted partway left a partial file that was then served as if complete for up to cacheTTL: isCacheFresh only checked size > 0, and the failed Download() never cleaned up its output. - ensureDownloaded/downloadToCache now write to a private temp file and only rename it onto the final cache path on success, so a partial or failed download can never occupy the cache path (task 13, task 14). - isCacheFresh now compares the cached file's size against the remote size (via backend.Stat, reusing the Getattr metadata cache) instead of just checking size > 0; a mismatch invalidates and re-downloads, self-healing any already-truncated entry on next access (task 15). - downloadCoordinator deduplicates concurrent Open() calls on the same cache path so only one download is ever in flight per file (task 16). - Downloads are retried up to 3 times with capped exponential backoff instead of relying on the FUSE client to re-issue Open() in a cascade (task 17). - Each attempt is now bounded by a context timeout instead of context.Background(); sized well above WebDAV's existing 30s http.Client.Timeout so it never cuts off transfers earlier than WebDAV's own limit would (task 18). Fix lives entirely in ensureDownloaded, which is backend-agnostic (StorageBackend interface) — covers WebDAV, Local and MooseFS without touching any plugin. WebDAV/Local test suites green (no regression). Ref: _work/reports/plan-20260812-102022.md, tasks 13-18.
…cial backoff (#160) Revises the Phase 1+2 fix per the planner's revision 2, which corrects two things the first pass got wrong relative to the official MooseFS client (moosefs/mfsclient/readdata.c) used as the reference implementation: - ReadChunk previously used one static deadline covering the whole call. The official client instead uses an *inactivity* timeout — rearmed on every frame received (CHUNKSERVER_ACTIVITY_TIMEOUT, readdata.c:78/:1395) — so a read that keeps progressing is never killed, only a connection that goes silent. ReadChunk and WriteChunk now share one csInactivityTimeout (5s), both rearmed per-frame; the flood guard (maxConsecutiveNOPs) remains the only bound on a CS that never goes silent but never stops sending keepalives either. - A retry now invalidates the cached chunk location and re-queries the master (Client.locateChunk) before contacting a chunk server again, mirroring chunksdatacache_invalidate (readdata.c:1989) — the previous version kept re-attacking the same cached, possibly-faulty CS. doCSRead takes a locateFunc so both Client.Read and readEC4At (now taking nodeID) re-resolve on attempt > 0 instead of reusing the first ChunkInfo. Also per revision 2: - ANTOAN_NOP frames with a non-zero payload are now a hard error (readdata.c:1683) instead of being silently skipped like a real keepalive. - The unexpected-opcode error now includes the peer address (RemoteAddr), matching readdata.c:1694, to keep any residual desync diagnosable. - Retry backoff now follows the official schedule (readdata.c:1994) via a new internal/backoff package: 3 immediate attempts, then a 1ms+(n-3)*300ms ramp capped at 10s — exposed as backoff.MooseFS.Delay(n), a top-level package (not under plugins/moosefs/internal) so dev-backend can reuse it for issue #162 in internal/placeholder. - ReadAt now checks ctx.Err() before starting a read (it previously never checked at all); Download's existing between-block check is now backed by the inactivity timeout for the "stuck mid-block" case. Scope: plugins/moosefs/ only (mfsclient package + moosefs.go) + the new shared internal/backoff package. internal/cfapi and internal/placeholder are out of scope for this commit (Phase 3, dev-backend). go build ./plugins/... and go build ./... (module-wide, cgofuse's unrelated missing-header failure aside) are both green. go vet currently fails only inside csclient_test.go/ecclient_test.go — expected: those files are mid-rewrite by test-writer for revision 2 (readEC4At gained a nodeID parameter; csReadChunkDeadline was renamed to csInactivityTimeout). Co-Authored-By: Claude Sonnet 5 <[email protected]>
… cache integrity) Phase 4 (plan revision 2, _work/reports/plan-20260812-102022.md) — tests derived from docs/diagrams/moosefs-ec4-read-statemachine.md invariants, covering both dev-plugin's mfsclient protocol fix (#160) and dev-backend's placeholder cache-integrity fix (#162). plugins/moosefs/internal/mfsclient (csclient_test.go, ecclient_test.go): - TestReadChunk_NOPskip / NOPBadLength / NOPFlood / UnknownCmd (CA1-CA4) - TestReadEC4Cmd0_NOPIgnored + TestReadEC4Cmd0_TrueStaleness_TriggersRetry, replacing the old TestReadEC4StaleConnection_Cmd0 which encoded the erroneous "NOP = stale connection" diagnosis (CA1, CA5) - TestRetry_ForcesFreshDial / InvalidatesChunkLocation / BackoffBounded / NonRetryableFailsFast (CA6-CA8) - TestReadChunk_InactivityTimeout (progressing read never interrupted vs. silent connection bounded) / TestCSPool_MaxIdleAge (CA9, CA10) - makeECClient upgraded to back tests with a stable in-memory master so the new locateChunk-based retry path (#160 D3bis) works in EC4-level tests internal/placeholder (download_integrity_test.go, new): - TestEnsureDownloaded_PartialRemovedOnError (CA11) - TestEnsureDownloaded_RedownloadsTruncatedCache — end-to-end complement to dev-backend's TestIsCacheFresh_SizeMismatch_ReturnsFalse, proving isCacheFresh/remoteSize/downloadToCache are wired together correctly (CA12) - TestEnsureDownloaded_ConcurrentOpensSingleDownload (CA13) - metacache_vfs_test.go: added ReadAt/ChunkSize/Rename/Copy stubs to the shared mockStorageBackend (interface had grown without every mock being updated — pre-existing gap, fixed incidentally to unblock compilation) tests/procedures/ec4-chunk-read.md: - Scenario 6 (#160 perf) and Scenario 7 (#162 integrity after induced failure) with read-only MooseFS access reminder Non-regression: full mfsclient suite (74 tests) green, -race, 21.6s. internal/placeholder verified via GOOS=windows go vet (Windows-only build tag, cannot execute on this host). Co-Authored-By: Claude Sonnet 5 <[email protected]>
QA (qa-20260812-112858.md) found a preexisting bug that the new strict isCacheFresh size check (CA12) turned blocking: Download() writes every 64 KiB chunk read verbatim, but a real MooseFS chunk server zero-pads a read within a chunk boundary — the last block of a file whose size is not a multiple of chunkSize therefore always arrives padded to a full chunkSize. The local cache file ended up larger than the remote size on essentially every file, so the strict size-equality freshness check could never pass: every Open() redownloaded the whole file, silently defeating #160. Truncate the block to the file's real remaining length before writing it, right after the read. offset/done then advance by the real byte count, so the loop's own EOF guard still terminates cleanly on the next iteration. Also fixes the structural test blind spot QA flagged: the in-memory fake chunk server (moosefs_test.go) always returned exactly the stored data length, never padding like the real protocol does, so no existing test could have caught this. Added an opt-in padReadsToRequestedSize flag on integFakeCSServer (default false — every other test keeps its current exact-length semantics) and TestDownload_TruncatesFinalPaddedBlock, which fails without the fix (verified by reverting moosefs.go locally) and passes with it. go test ./plugins/... -race -cover: all green, mfsclient 74.3%, moosefs 75.3%, webdav/local unaffected. Co-Authored-By: Claude Sonnet 5 <[email protected]>
Fix protocole MooseFS EC4 (keepalive ANTOAN_NOP, timeout inactivite, invalidation localisation, backoff officiel) et perte de donnees silencieuse du cache placeholder (ecriture atomique, validation stricte taille distante, deduplication, relances bornees). Z-segment bugfix, milestone v2.2 actif. QUALIF v2.2.1 : build frais valide (72.5% cov, -race clean), scenarios manuels 6/7 (tests/procedures/ec4-chunk-read.md) valides par QA. Co-Authored-By: Claude Sonnet 5 <[email protected]>
Bugfix #160 (protocole MooseFS EC4) + #162 (perte de donnees silencieuse cache placeholder). APPROVED code-reviewer, VALIDATED QA (suite auto + scenarios manuels 6/7 sur build QUALIF v2.2.1 frais). Version 2.2.1. Co-Authored-By: Claude Sonnet 5 <[email protected]>
…ocation + read granularity) Blocking prerequisite (plan task 2, _work/reports/plan-20260812-155238.md): without a measured "before" baseline, CA7/CA8 (throughput/mass-copy gains) are not demonstrable. Adds readChunkCount (master, CLTOMA_FUSE_READ_CHUNK) and csReadCount (chunk server, CLTOCS_READ) atomic counters to the existing in-memory fake server, and TestDownload_NetworkRoundtripBaseline, which downloads a reference file (2,109,497 B — deliberately not a chunkSize multiple, single MooseFS chunk) and asserts the current 1:1 ratio: one locateChunk call and one ReadChunk call per 64 KiB block (33 blocks -> 33 + 33 calls). This is the exact mechanism the plan's arithmetic proof (from the issue's production logs) attributes the latency plateau to. Measured now, pre-fix: - file=2,109,497 B (33 blocks of 65,536 B) - locateChunk_calls=33, ReadChunk_calls=33 (1:1 with block count) - elapsed=~43ms on this loopback fake server — NOT representative of real network latency; the authoritative duration/throughput baseline remains the issue's own production logs (~4.03-4.25 ms/roundtrip, ~8 MB/s, P2 S14 T2.MP4 and P2 S15 T1.MP4), cited in the plan and repeated in the dev-plugin handoff for #163. The assertions are intentionally pre-fix expectations: they will be updated (not deleted) once the Phase 1 location cache and Phase 2 read-granularity change land, so the improvement is proven by a diff in this same test rather than assumed. Co-Authored-By: Claude Sonnet 5 <[email protected]>
…ularity) Phase 4 (_work/reports/plan-20260812-155238.md) — tests for B1 (chunk location cache) and B2 (larger multi-shard reads), plus non-regression of #160/#162 at the new granularity and a performance measurement procedure. plugins/moosefs/internal/mfsclient (new: chunklocation_cache_test.go): - TestChunkLocationCache_HitAvoidsMasterRoundtrip (CA1) - TestChunkLocationCache_InvalidatedOnReadError (CA2) - TestChunkLocationCache_VersionMismatchRefetches (CA3) - TestChunkLocationCache_Eviction + _LRUOrderPreservedOnHit (CA6) — white-box against chunkLocationCache directly (fast, precise, uses the real chunkLocationCacheMaxEntries constant) rather than thousands of network round-trips through Client.Read() ecclient_test.go: - TestReadEC4At_MultiMiBRead — large single-shard read == concatenation of 64 KiB reads, byte for byte (CA4) - TestReadEC4At_ShardBoundarySplit — a read straddling two shards is split and reassembled correctly (CA4) csclient_test.go: - TestReadChunk_MultiFrameCRC — CRC checked on every frame of a multi-frame response, corrupted frame detected even after valid ones (CA5) - TestReadChunk_NOPSkip_LargeMultiFrame — #160 NOP-skip replayed at B2's larger multi-frame granularity (task 19 non-regression) moosefs_test.go: - TestDownload_LastBlockTruncation — table-driven complement to TestDownload_TruncatesFinalPaddedBlock across chunkSize-relative boundary sizes, granularity-agnostic (references the chunkSize constant, never a literal) (CA11) tests/procedures/: - mass-copy-performance-163.md (new) — reproducible before/after throughput protocol (single file + full folder copy), mandatory measurement table, read-only MooseFS access reminder (CA7, CA8) - ec4-chunk-read.md — cross-reference to the new procedure Non-regression: full mfsclient suite 83/83 PASS, -race, 25.5s (includes existing #160/#162 tests + all new #163 tests). moosefs package: 47/48 PASS; the one failure (TestWatch_adaptiveReset) is a pre-existing order-dependent flake in unrelated Watch/polling code (passes 3/3 in isolation, unrelated to #163's read path) — flagged in the handoff, not fixed here (out of scope). Written in parallel with dev-plugin actively landing B1 (chunklocationcache.go) and B2 (ecclient.go shard-splitting) — most tests found the target behaviour already implemented and passed on first run; TestChunkLocationCache_Eviction's LRU math was corrected once against the real cache implementation. Co-Authored-By: Claude Sonnet 5 <[email protected]>
… (B2) (#163) Issue #163 blamed the lack of inter-file pipelining for slow mass copies. The plan's arithmetic proof of the issue's own logs shows otherwise: two files of very different sizes both plateau at ~4.1 ms per network roundtrip (~8 MB/s), which is a latency ceiling driven by roundtrip COUNT, not bandwidth — pipelining files that are each individually this slow would not fix it. Root cause is two compounding inefficiencies in plugins/moosefs, both intra-file: B1 — no chunk-location cache. Client.Read called locateChunk (a master CLTOMA_FUSE_READ_CHUNK roundtrip) before every chunk-server read, regardless of size. A 64 MiB MooseFS chunk read in 64 KiB blocks meant requesting the SAME location up to 1024 times. Worse, locateChunk held Client.mu — the single master connection's mutex — serializing every other master operation (GetAttr, Lookup, ReadDir) across the whole plugin, not just within one file. #160 already ported the *invalidation* half of the official client's chunksdatacache model (a retry re-queries the master instead of reusing a stale location) without the cache that invalidation was meant to protect; this is the other half. Adds chunklocationcache.go: a bounded (8192 entries), TTL-aware (30s) LRU cache indexed by (nodeID, chunkIndex), guarded by its own mutex (never Client.mu, so a cache hit never contends with other master traffic). Client.locateChunk(nodeID, index, forceRefresh) checks the cache first unless forceRefresh (used exclusively by the #160 D3bis retry path, which now also invalidates the entry before re-querying — the crucial wiring this task required: without it, a cache would have silently defeated #160's "never retry against the same bad location" guarantee). No separate chunk-version validation before use: the wire protocol already carries the cached Version in every CLTOCS_READ, so a stale entry surfaces as a chunk-server status error, which the existing #160 error-driven invalidation path already handles. B2 — 64 KiB read granularity despite the protocol already supporting far larger single requests. ReadChunk already collects multiple CSTOCL_READ_DATA frames (each CRC-checked) per CLTOCS_READ; only readEC4At's documented precondition (size <= one 64 KiB block) forced the caller to chop every read that small. readEC4At now accepts up to a full shard per call and splits a request across shard boundaries as needed (one CS request per shard crossed — a CLTOCS_READ cannot itself read past the shard/physical chunk it names). moosefs.go's shared Upload/Download block size goes from 64 KiB to 4 MiB (divides a full chunk's 16 MiB shard evenly). The #160 zero-pad truncation in Download (4e0cbd3) is untouched and remains size-agnostic, so it keeps working unmodified at the new granularity. Incidentally also resolves the cfapi ReadAt/ChunkSize() 64 MiB-vs-64 KiB precondition mismatch the plan flagged as latent (not exercised by #163's own FUSE path) — full cfapi verification stays with #161. Measured (plugins/moosefs, in-memory fake server, call counts are exact and reproducible; wall-clock is not representative of real network latency — see the issue's own production logs for that, cited in the plan and the dev-plugin handoff for #163): before: 2,109,497 B file, 33 x 64 KiB blocks -> 33 locateChunk calls, 33 ReadChunk calls (1:1 with block count) after: 13,360,689 B file (3 x 4 MiB + partial), still 1 MooseFS chunk -> 1 locateChunk call (cache hit for blocks 2-4), 4 ReadChunk calls (one per Download iteration, as expected — B1 collapses the master roundtrips specifically, B2 raises the bytes moved per chunk-server roundtrip) go test ./plugins/... -race -cover -count=1: all green (mfsclient 74.9%, moosefs 75.5%, webdav/local unaffected). One unrelated, pre-existing flake observed under full-suite -race load (TestWatch_adaptiveReset, Watch/ polling code untouched by this change) — passes 3/3 in isolation, not a regression from this commit. Co-Authored-By: Claude Sonnet 5 <[email protected]>
…ejection (#163) code-reviewer (code-reviewer-20260813-093106.md) found a MAJOR: the chunklocationcache.go design comment claimed a stale cached location "naturally surfaces as a non-OK server status ... already routed through doCSRead's re-locate-on-retry path" — but doCSRead only retries (and thus only forces a relocate) for isStaleConnErr-classified errors (typed connection failures), and a non-OK CSTOCL_READ_STATUS was, since #160, a deliberately non-retryable fmt.Errorf (CRC/status errors must fail fast). #163 added a second, unrelated meaning to that same error class (a stale cached placement) without changing its classification: a chunk relocated or rewritten during the cache's 30s TTL window would make a read fail outright instead of self-correcting via a fresh master query, exactly as the comment implied it would. Fail-safe (no wrong data, CA3 holds in the strict sense) but not the self-healing robustness claimed. - csclient.go: new errServerStatus sentinel, wrapped by ReadChunk's non-OK CSTOCL_READ_STATUS branch. doCSRead grants it exactly ONE forced chunk-location refresh (locationRefreshUsed flag) before treating it as terminal — never the full isStaleConnErr retry/backoff budget, so a persistent non-location status rejection still fails fast on its second occurrence rather than burning through maxCSAttempts pointlessly. - chunklocationcache.go: design comment corrected to describe the actual (bounded, error-triggered) mechanism instead of an inaccurate "already handled by the existing retry path" claim. - chunklocation_cache_test.go: two new tests exercising the REAL scenario end to end through Client.Read() — TestChunkLocationCache_Version- MismatchRefetches only ever exercised a connection-error trigger, as its own comment admitted. TestChunkLocationCache_ServerStatusRejection- ForcesRelocate (a rejecting CS -> one relocate -> a working CS succeeds, exactly 2 master roundtrips) and TestChunkLocationCache_ServerStatus- RejectionIsBoundedNotLooped (both locations reject -> fails with errServerStatus after exactly 2 master roundtrips, never more). Also code-reviewer MINOR-2: ecclient.go readEC4At's multi-segment loop (#163 B2) resolved every segment's shard geometry from the ORIGINAL info parameter, ignoring a fresher ChunkInfo obtained by an earlier segment's retry. Unreachable before #163 (a read never crossed a shard boundary); reachable now that one call can span multiple shards. Fixed by tracking currentInfo across loop iterations, updated by the locate closure whenever a segment's retry re-queries the master, so later segments start from whatever the most recent segment actually learned. go test ./plugins/... -race -cover -count=1: all green (mfsclient 75.3%, moosefs 75.5%). gofmt -l flags csclient.go, confirmed pre-existing at de4dbc6 (before this commit) — not introduced here, left as-is per code-reviewer's MINOR-1 disposition (separate cleanup commit). Co-Authored-By: Claude Sonnet 5 <[email protected]>
…ard-aligned file (#163) QA (qa-20260813-101152.md) found a QUALIF-blocking regression on v2.2.2: 8/8 real large files failed to download, all with the identical signature "shardIdx 4 out of range" at chunkOffset == exactly 4*shardSize. Worse than the performance problem #163 set out to fix — these same files downloaded fine (slowly) before this branch. Root cause: locateEC4Shard only treated chunkDataSize == 0 as EOF. shardSize is rounded UP (divCeil(chunkDataSize, 4) then aligned to a 64 KiB block), so for a chunk whose real data is not an exact multiple of 4*shardSize — the last MooseFS chunk of essentially every real file — 4*shardSize overshoots chunkDataSize. readEC4At's segmented read loop (#163 B2) is bounded by the caller's requested size, not by chunkDataSize, and Download() always requests a fixed block size without knowing in advance how much of the current chunk is real (exactly like the #160 whole-chunk zero-pad case) — so the loop kept walking curOffset forward until it landed precisely on 4*shardSize, one past the 4 valid shards (0-3), and locateEC4Shard computed shardIdx=4 and failed hard instead of recognizing EOF. Before #163 (64 KiB requests, one CLTOCS_READ at a time, no internal segment loop) this path could not be reached the same way. B2's segmented loop is what turned a previously-unreachable arithmetic edge into a guaranteed crash on any non-shard-aligned file. Fix (ecclient.go, locateEC4Shard): check chunkOffset >= chunkDataSize right after computing chunkDataSize, before any shardSize-based division, so EOF is detected at the chunk's real length regardless of the shardSize rounding margin. Test gap this exposed: TestReadEC4At_MultiMiBRead and TestReadEC4At_ShardBoundarySplit both use an info.Length that IS an exact multiple of 4*shardSize, so neither exercised a partial last chunk. TestReadEC4At_PartialLastChunkEOF reproduces the exact QA failure (verified failing pre-fix with the identical "shardIdx 4 out of range" message, verified passing post-fix) with a chunkDataSize deliberately not a multiple of 4*shardSize. go test ./plugins/... -race -cover -count=1: all green (mfsclient 75.5%, moosefs 75.5%). Co-Authored-By: Claude Sonnet 5 <[email protected]>
…up, #163) code-reviewer (code-reviewer-20260813-103203.md, APPROVED) independently wrote and ran a 7-case white-box boundary sweep against locateEC4Shard while verifying 91844b5 (1-byte chunk, empty chunk, exact-4*mfsBlockSize boundary with no rounding margin, etc.) — all passed, then discarded the test. Non-blocking recommendation: pin at least the tiny-final-chunk case permanently, since this same function has now produced two successive edge-case regressions in a row (the server-status-staleness MAJOR in 77ffe04's predecessor, and the QUALIF-blocking crash fixed in 91844b5). TestLocateEC4Shard_BoundaryCases reproduces all 7 of the reviewer's cases as a permanent, table-driven test — cheap insurance against a future change to locateEC4Shard silently breaking one of these magnitudes again without any test catching it. go test ./plugins/... -race -cover -count=1: all green (mfsclient 75.5%, moosefs 75.5%). Co-Authored-By: Claude Sonnet 5 <[email protected]>
Fix performance copie de masse MooseFS : cache de localisation de chunk (LRU+TTL 30s) + granularite de lecture 64 KiB -> 4 MiB (alignee sur shards 16 MiB). Regression critique (locateEC4Shard EOF check manquant, fichiers non-alignes illisibles) trouvee et corrigee avant merge. Z-segment bugfix, milestone v2.2 actif. QUALIF v2.2.2 : build frais valide (72.5% cov, -race clean), mesure perf reelle validee par QA (fichier unique x2.63, copie masse x1.5, integrite complete). Co-Authored-By: Claude Sonnet 5 <[email protected]>
Bugfix #163 : performance copie de masse MooseFS. Cache de localisation de chunk + granularite de lecture 4 MiB (vs 64 KiB). Regression critique (locateEC4Shard) trouvee et corrigee avant merge. APPROVED code-reviewer, VALIDATED QA (suite auto + mesure perf reelle sur build QUALIF v2.2.2). Version 2.2.2. Co-Authored-By: Claude Sonnet 5 <[email protected]>
…ble + versioning docs Migre les contextes partages legacy (agents/context/*.md sans compagnon) vers le pattern *.template.md, met a jour la table "Agents Disponibles" et la section Versioning dans CLAUDE.md, et resynchronise init-project.md lui-meme. Le schema infrastructure (binaires GitHub Releases) et stack.plugins (StorageBackend Go) restent inchanges — pas d'equivalent dans le nouveau modele environments[] generique du template. Co-Authored-By: Claude Sonnet 5 <[email protected]>
… block, mockup/tests indexes Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
4 tasks
QUALIF installait protoc-gen-go-grpc@latest (v1.6.2, requiert Go >= 1.25) alors que la CI tourne en Go 1.24 : job Go rouge sur la PR #165. Le step QUALIF utilise desormais PROTOC_GEN_GO_GRPC_PIN (v1.5.1) de .ci-versions.env ; le sync des pins capture donc v1.5.1 et ne propage plus v1.6.2. project-config.json : infrastructure.version_sources[0] server-go/config.json -> config.json (seul hunk commite ; le bump template_version v3.9.0 reste non commite). Ecarts restants traites dans l'issue #166. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Sync des agents/context/commands du template, bump template_version v3.9.0 dans project-config.json et mise a jour de .gitignore. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
CCoupel
added a commit
that referenced
this pull request
Oct 2, 2026
…2) (#167) * chore(contracts): add v2.2 Workflow Objets contracts (#136 #139 #140 #141 #142 #143 #144) - wails-bindings.md: GetFileState(backendID, localPath) → L/P/U/S/C/E/X - wails-events.md: sync:conflict + sync:file-state-changed specs - CHANGELOG.md: BREAKING changes (Rename/Copy interface), new types, new events Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * feat(sync): P1 fileStates cache + P2 Rename/Copy/ActionRename + intent queue (#136 #139 #140 #141 #142 #143 #144) P1 (#136) — per-file FileState cache in Engine: - types: FileState L/P/U/S/C/E/X constants - Engine.fileStates sync.Map + Get/SetFileState + makeStateUpdater - Dispatcher.stateUpdater callback (P→U→S/E transitions in execute()) - Reconciler.stateUpdater (L-state for local-only files) + conflictEmitter - App.GetFileState() Wails binding P2a (#139) — ActionRename with server-side rename: - plugins.ErrNotSupported sentinel - StorageBackend.Rename() interface method - Watcher: rename-pair detection with 150ms timer (Rename+Create → FileEventRenamed) - Dispatcher: ActionRename + fallback Upload+Delete on ErrNotSupported - Engine.handleLocalEvent: FileEventRenamed case - Plugins: Rename() in webdav (MOVE), moosefs (native), local (os.Rename), mock - Proto: rpc Rename(MoveRequest) returns (MoveResponse) + regenerated pb.go - gRPC: server.Rename + client.Rename; mapBackendError/mapGRPCError handle ErrNotSupported P2b (#142) — Atomic unpin guard: - App.PinFile: upload first if state != S before dehydrating CF placeholder - Dispatcher: CFSyncStateCloudOnly = 0 constant P2c (#143) — Pin intent queue: - CFManager.intentions sync.Map + QueuePinIntent + ConsumeIntent - Engine.SetIntentCallbacks: consumeIntent + pinFile injected from App - makeStateUpdater: consumes intent when U→S transition completes - App.PinFile: queues intent if state == U or P (active transfer) - App.StartSync: wires intent callbacks into engine P3b (#141) — Guard L-state delete: - Engine.handleLocalEvent FileEventDeleted: skips ActionDelete for L-state files P4 (#144) — Conflict event: - Reconciler emits sync:conflict via conflictEmitter (backendID, path, mods, resolution) - StorageBackend.Copy() interface method + ActionCopy in Dispatcher P2a also adds: - ActionCopy type + Dispatcher execute case (with ErrNotSupported fallback) - plugins.Copy() in webdav (HTTP COPY RFC 4918), moosefs (ErrNotSupported), local (io.Copy) - Proto: rpc Copy(CopyRequest) returns (CopyResponse) + pb.go regenerated - SDK echo plugin: Rename() + Copy() (returns ErrNotSupported as example) Test fixes: Rename/Copy stubs added to all test mocks (17 packages pass) Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * feat(ui): Toast conflict #144 — composant + hook + overlay (v2.2) - types/ghostdrive.ts : ConflictEvent interface + WailsEventMap['sync:conflict'] - components/ui/Toast.tsx : composant accessible (role=alert, aria-live=polite), auto-dismiss 5s, AlertTriangle amber, dismiss card/bouton × - hooks/useSyncStatus.ts : listener sync:conflict avec debounce 500ms/path, max 3 toasts individuels, regroupement ">3 conflits" en summary toast - App.tsx : overlay fixed bas-droit, rendu conditionnel des ConflictToast - 13 tests vitest (rendu, dismiss, auto-dismiss, cleanup timer) 42/42 tests pass — closes #144 (frontend) Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * test(v2.2): add tests and procedures for Workflow Objets (#136 #139 #140 #141 #142 #143 #144) - internal/sync/filestate_test.go: GetFileState/SetFileState concurrent, U→S→E transitions, L-guard (FileStateLocal skips remote delete), makeStateUpdater events + intent callback - internal/sync/dispatcher_test.go: ActionRename (native + ErrNotSupported fallback), ActionCopy (server-side + fallback), StateUpdater P→U→S/E lifecycle, nil-guard - internal/sync/watcher_rename_test.go: rename-pair detection → FileEventRenamed with OldPath, no-Create timeout → FileEventDeleted, cross-dir rename - internal/cfapi/manager_intent_test.go: QueuePinIntent (basic, overwrite, multi-path), ConsumeIntent (no-op, clears entry), concurrent thread-safety - plugins/webdav/rename_copy_test.go: Rename via MOVE (OK, notFound, notConnected), Copy via HTTP COPY (OK, notFound, overwrite, preservesSource, notConnected) - plugins/local/rename_copy_test.go: Rename via os.Rename (OK, notFound, subdir), Copy via io.Copy (OK, notFound, createsParentDirs, overwrite) - plugins/moosefs/copy_errnot_test.go: Copy returns ErrNotSupported (connected/disconnected/ cancelledCtx), Rename delegates to Move - tests/integration/workflow_v22_test.go: full dispatch workflows — ActionRename (native + fallback), ActionDelete L-guard, ActionCopy (server-side + fallback), ErrNotSupported detectability cross-package - frontend/src/hooks/useSyncStatus.test.ts: sync:conflict → toast, debounce 500ms, collapse >3, reset counter on dismiss-all - tests/procedures/v2.2-workflow-objets.md: 8 QA manual scenarios Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(sync): cycle-2 corrections review v2.2 — race watcher, CF badge copy, UploadFile 🔴 CRITIQUE — watcher.go: suppression race condition pendingRenames (AfterFunc goroutine vs boucle select) via canal expiredRenames (cap 16) — cleanup délégué au goroutine principal. 🟠 MAJEUR — dispatcher.go ActionCopy: ajout SetSyncState(CFSyncStateSynced) après copy réussie — badge Explorateur ✓✓ au lieu de ☁️ (identique à ActionDownload). 🟠 MAJEUR — engine.go: UploadFile(ctx, localPath) dispatch ActionUpload ciblé (1 fichier) — app.go PinFile remplace ForceSync() (réconciliation complète) par UploadFile() (#142). Optionnel — engine.go makeStateUpdater: guard prev==state évite flood events identiques. Optionnel — dispatcher.go ActionRename fallback: état E si delete échoue (doublon remote). Optionnel — frontend/src/App.tsx: DEFAULT_CONFIG version '' (était '0.6.0' hardcodé). Tests: 17/17 packages ok -race 0 races. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * docs(v2.2): CHANGELOG + versioning + docs plugin + états actions - Ajout entrée CHANGELOG v2.2.0 avec breaking changes (StorageBackend Rename/Copy), Added (7 issues #136-#144), Fixed (3 bugs), Tests (+67) - Mise à jour versions : config.json + frontend/package.json → 2.2.0 - Nouvelle matrice états/actions : états L/C/X, actions Rename/Copy natives, Guard L-state - docs/plugin-development.md : documentation Rename()/Copy() et ErrNotSupported sentinel - README.md : v2.2 marquée livrée avec highlights Co-Authored-By: Claude Haiku 4.5 <[email protected]> * fix(frontend): remove invalid toHaveLength second arg (Vitest compat) Vitest's toHaveLength matcher only accepts 1 argument; the message string accepted by Jest causes a tsc error in Vitest's type definitions. Removed the second argument on 3 assertions (lines 165, 183, 228). 51/51 tests pass. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(loader): #145 — filtre OS/arch avant chargement des plugins .ghdp Ajoute `isCompatibleBinary()` qui détecte le token OS dans le nom du fichier (ex: `-linux-`, `-windows-`) en se basant sur le pattern `-v<version>-<os>-<arch>.ghdp`. Tout binaire taggé avec un OS différent de `runtime.GOOS` est ignoré silencieusement avec un log WARN, et n'apparaît ni en « loaded » ni en « failed » dans `GetLoadedPlugins()`. Fichiers modifiés : - plugins/loader/grpc_loader.go : +runtime import, pluginOSRe, isCompatibleBinary, filtre dans Scan() - plugins/loader/export_test.go : expose IsCompatibleBinaryForTest - plugins/loader/grpc_loader_test.go : 5 nouveaux tests (unit + integration) Fixes #145 Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(app): #146 — log ERROR sur les échecs d'activation backend Les 3 chemins d'activation (Startup, SetBackendEnabled, UpdateBackend) émettent désormais `logger.Error(...)` avant de retourner l'erreur, ce qui garantit que les échecs apparaissent dans le log fichier et dans l'onglet Logs de l'UI — pas seulement dans l'événement Wails. Fichiers modifiés : - internal/app/app.go : logger.Error aux 3 sites d'activation - internal/app/app_test.go : 2 nouveaux tests + import bytes/logger Fixes #146 Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Revert "fix(loader): #145 — filtre OS/arch avant chargement des plugins .ghdp" This reverts commit 87ca2ae6110498938a54728749ce6129f0b77b61. * fix(app): #147 — première activation backend après démarrage sans backend Quand l'application démarre avec zéro backend activé, MountUnified n'est jamais appelé (Startup skip la section si len(unifiedBackends)==0). SetBackendEnabled appelait ensuite UpdateBackends qui échouait immédiatement avec "unified drive not mounted", bloquant définitivement l'activation manuelle. Fix : dans le chemin d'activation de SetBackendEnabled, on vérifie si le drive unifié est monté via GetUnifiedStatus() : - drive monté → UpdateBackends (chemin existant) - drive absent → MountUnified (premier backend activé manuellement) Test ajouté : TestSetBackendEnabled_FirstActivation_NoBrackendAtStartup — vérifie que l'erreur n'est plus "unified drive not mounted" et que le rollback est propre en cas d'échec de montage (NullDrive Linux CI). Fixes #147 Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(sync): #148 — ActionRename cassé dans les deux sens Deux root causes : 1. watcher.go émettait des chemins ABSOLUS (filepath.ToSlash(fsEvent.Name)). handleLocalEvent les traitait comme relatifs en préfixant e.remotePath / e.localDir → chemins distants doublés (/remote/tmp/xxx/new.txt) → backend.Rename échouait → la réconciliation ré-téléchargeait l'ancien nom → 2 fichiers coexistaient localement. Fix : filepath.Rel(w.dir, fsEvent.Name) avant ToSlash → chemins relatifs. 2. handleRemoteEvent n'avait aucun case pour FileEventRenamed → retour nil silencieux. Les backends émettaient FileEventCreated pour le nouveau nom → ActionDownload créait un nouveau fichier sans renommer l'existant. Fix : case FileEventRenamed → os.Rename local (fallback : download si l'ancien fichier local est absent). Tests ajoutés : - TestWatcherEmitsRelativePaths : vérifie que les chemins watcher sont relatifs - TestEngine_HandleLocalEvent_Rename_PropagatesRemote : rename local → backend - TestEngine_HandleRemoteEvent_Rename_RenamesLocally : rename distant → local - TestEngine_HandleRemoteEvent_Rename_FallbackDownload : fallback si absent 73/73 PASS internal/sync — 0 régression sur ./... Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(sync): #149 — 3 regressions causing write failures and delete-reverts Root causes introduced in bc86602 (#136 fileStates cache): 1. State inheritance bug (engine.go handleLocalEvent FileEventRenamed): SetFileState(newPath, oldState) propagated FileStateLocal to the renamed file's new path. The #141 guard then skipped ActionDelete for that file, causing deleted files to reappear after the next reconciliation. Fix: block inheritance when oldState == FileStateLocal. 2. Reconciler S→L race (engine.go makeStateUpdater): The reconciler could race with a concurrent upload and call updateState(FileStateLocal) on an already-Synced file (because its remote snapshot predated the upload completion). This again triggered the #141 delete-skip guard for a legitimately synced file. Fix: guard in makeStateUpdater prevents S→L downgrades. 3. Stale Create debounce timer (watcher.go): When a file was created and renamed within the 500ms debounce window, the original Create timer for the old path was not cancelled. It fired 500ms later emitting a stale FileEventCreated for a non-existent file, causing a spurious ActionUpload failure. Fix: stop and delete pending[path] when a Rename event is detected. Tests added (3): - TestEngine_HandleLocalEvent_Rename_DoesNotInheritLocalState - TestEngineMakeStateUpdater_NoDowngradeSyncedToLocal - TestWatcherCreateRename_NoStaleCreateEvent Results: 76/76 PASS internal/sync, 18/18 packages PASS, 0 regressions. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(sync): #150 — remote→local propagation failures (delete, rename feedback, Explorer refresh) Three sync remote→local bugs fixed: 1. **Delete propagation**: `handleRemoteEvent FileEventDeleted` previously emitted only a "V1 policy — manual review" warning with no `os.Remove`. Local file now deleted when the remote file is removed. 2. **Rename feedback loop**: `handleRemoteEvent FileEventRenamed` called `os.Rename` on the local filesystem, triggering fsnotify events that `handleLocalEvent` re-processed as local renames → spurious `ActionRename` to the backend. Both the old and new paths are now suppressed before `os.Rename` so the feedback loop is broken. Also: when Watch() emits Delete+Create pairs instead of FileEventRenamed, the Delete is now properly applied (fix 1 above), preventing both names from coexisting locally. 3. **Explorer refresh**: Added `LocalRefresher` interface (`NotifyLocalChange(path string)`) satisfied by `*cfapi.CFManager` via `ghd_notify_dir_change` (new C shim calling `SHChangeNotify(SHCNE_UPDATEDIR, SHCNF_PATH|SHCNF_FLUSH, dir, NULL)`). Called after ActionDownload and after each remote-triggered local filesystem change so Explorer refreshes without F5. Implementation: - `suppressedPaths gosync.Map` on Engine + `suppressLocalEvent` / `isLocalEventSuppressed` (TTL = debounceDuration + 1s slack). Checked at top of `handleLocalEvent` for main path and OldPath. - `cfapi/notify_windows.go` (Windows impl) + `cfapi/notify_stub.go` (non-Windows no-op) - `cgo_cfapi_windows.c/.h`: new `ghd_notify_dir_change(LPCWSTR dir)` function - 5 new tests: Delete_RemovesLocally, Delete_SuppressesLocalFeedback, Rename_SuppressesBothPaths, Delete_NonExistentLocal, SuppressExpiry 18/18 packages PASS — 0 regressions. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * test(e2e): add WSL CRUD test suite for GhostDrive sync validation Script bash exécutable depuis WSL qui valide les 5 opérations CRUD (New/Copy/Rename/Open/Delete) dans les deux sens : LOCAL→GhD: et GhD:→LOCAL. Remplace la validation manuelle Windows avec timeout 10s et analyse logs CF API. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(cfapi): convert regular files to CF placeholders before SetSyncState (#151) In v2.2 (bc86602 Phase 4), dispatcher.go calls CfSetInSyncState after ActionDownload to badge newly-downloaded files ✓✓. Download() writes files via os.Rename(tmp, dest) — the result is a regular NTFS file, never a CF placeholder. CfSetInSyncState on a non-placeholder writes invalid CF reparse-point data, corrupting the parent directory's CF state and causing 0x80070781 (ERROR_INVALID_REPARSE_DATA) + 0x8007017c (ERROR_CLOUD_FILE_ INVALID_REQUEST) on subsequent user operations (New File, Rename from local). Fix: add PlaceholderMaker interface to dispatcher.go; call ConvertToPlaceholder (→ ghd_convert_to_placeholder / CF_CONVERT_FLAG_MARK_IN_SYNC) before SetSyncState in ActionDownload and ActionCopy. Idempotent: if the file is already a CF placeholder the call succeeds as a no-op. Files: - internal/cfapi/provider.go — SyncProvider.ConvertToPlaceholder (Windows CGO) - internal/cfapi/provider_stub.go — no-op stub for non-Windows - internal/cfapi/manager.go — CFManager.ConvertToPlaceholder - internal/sync/dispatcher.go — PlaceholderMaker interface + wiring in ActionDownload and ActionCopy - internal/sync/dispatcher_test.go — 3 new tests: TestDispatcher_ActionDownload_ConvertBeforeSetSync TestDispatcher_ActionCopy_ConvertBeforeSetSync TestDispatcher_ActionDownload_NoPlaceholderMaker_NoPanic 18/18 PASS — 0 regressions Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(cfapi): guard SetSyncState on ConvertToPlaceholder failure (#152) If CfConvertToPlaceholder fails (handle invalid, file locked, etc.) the previous code still called CfSetInSyncState on the non-placeholder file. That writes invalid CF reparse-point data into the NTFS file, corrupting the parent directory's CF state → 0x80070781 on subsequent user ops. Introduce a placeholderOK bool in ActionDownload and ActionCopy: if ConvertToPlaceholder returns an error, emit sync:warn and skip SetSyncState entirely. Skipping the badge update is safe; calling CfSetInSyncState on a non-placeholder is the root cause of the regression. Closes #152 Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(sync): path normalisation for Windows backslash and leading-slash (#153 #154 #155) Three path-handling bugs in engine.go fixed in one commit as they all touch the same region of handleLocalEvent / handleRemoteEvent: #153 — Windows backslash in fsnotify evt.Path (handleLocalEvent) fsnotify on Windows returns paths like "subdir\file.txt" for files in subdirectories. path.Join (Unix semantics) treats "\" as a filename character, producing "/remote/subdir\file.txt" → backend.Delete/Move targets the wrong remote path → 404 → remote preserved → file revives. Fix: filepath.ToSlash(evt.Path) before path.Join. #154 — Double filepath.Dir in NotifyLocalChange calls (handleRemoteEvent) handleRemoteEvent was passing filepath.Dir(localPath) to NotifyLocalChange, which then applied filepath.Dir again → grandparent directory notified instead of the file's parent. Fix: pass localPath (file path) so NotifyLocalChange computes the parent. #155 — Leading "/" in remote Watch() evt.Path (handleRemoteEvent) Remote backends (WebDAV) may emit paths like "/file.txt". On Windows filepath.Join("C:\SyncRoot", "/file.txt") = "C:\file.txt" (absolute second component wins, drops sync root prefix) → os.Remove/Rename targets a wrong path outside the sync root → no-op → file not deleted. Fix: strings.TrimLeft(filepath.FromSlash(evt.Path), "/\") before Join. Added logger.Error/Info for os.Remove and os.Rename outcomes. Closes #153 Closes #154 Closes #155 Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(cfapi): trailing backslash for SHChangeNotify SHCNF_PATH (#154) SHChangeNotify(SHCNE_UPDATEDIR, SHCNF_PATH, dir, NULL) on Windows requires the directory path to end with a backslash. Without it the shell silently ignores the notification and Explorer never refreshes. filepath.Dir() does not append a trailing separator. Add it explicitly in NotifyLocalChange before converting to wide string for CGO. Closes #154 Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * test(sync): regression tests for path normalisation fixes (#152 #153 #155) Add 4 tests covering the post-151 regressions: TestDispatcher_ActionDownload_ConvertBeforeSetSync (#152) TestDispatcher_ActionCopy_ConvertBeforeSetSync (#152) Verify placeholderOK guard: ConvertToPlaceholder error must prevent SetSyncState from being called (already added in bfada8e, now with correct error-checking semantics). TestEngine_HandleLocalEvent_BackslashPath_Delete (#153) TestEngine_HandleLocalEvent_BackslashPath_Rename (#153) Windows-only (t.Skip on Linux): verify that backslash-separated fsnotify paths are converted to forward slashes before building the remote path. On Linux filepath.ToSlash is a no-op for backslashes (valid filename char), so fsnotify never produces such paths. TestEngine_HandleRemoteEvent_LeadingSlash_Delete (#155) TestEngine_HandleRemoteEvent_LeadingSlash_Rename (#155) Cross-platform: verify that leading "/" in remote Watch() evt.Path is stripped before filepath.Join so os.Remove/Rename targets the correct local path inside the sync root. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(cfapi): treat 0x8007017c as no-op for already-placeholder directories (#156) CF_CONVERT_FLAG_ENABLE_ON_DEMAND_POPULATION is not idempotent for directories: calling ghd_convert_dir_to_placeholder on a directory that is already a CF placeholder returns 0x8007017c (ERROR_CLOUD_FILE_INVALID_REQUEST). This is expected and benign on every FETCH_PLACEHOLDERS pass after the first conversion. The previous code logged this as an error on every FETCH_PLACEHOLDERS round-trip, flooding logs with misleading "CfConvertToPlaceholder isDir=true: HRESULT 0x8007017c" messages that suggest CF corruption when no corruption is occurring. Add hrAlreadyPlaceholder = 0x8007017c constant and detect it in the directory conversion branch of createPlaceholdersWithFlags. Treat it as a silent no-op: the directory is already properly set up as a CF placeholder with on-demand population — nothing to do. Unlike MARK_IN_SYNC (for files, truly idempotent), ENABLE_ON_DEMAND_POPULATION deliberately fails on already-converted directories to prevent accidentally clearing the on-demand population state. Note: this does NOT fix the 0x80070781 (ERROR_INVALID_REPARSE_DATA) on New File — that regression has a separate root cause not visible from these logs. The dir 0x8007017c errors are benign; no SetSyncState is called on these directories and the #152 placeholderOK guard does not apply to this code path. Closes #156 Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(cfapi): use DISABLE_ON_DEMAND_POPULATION in FETCH_PLACEHOLDERS ack (#157) ghd_cf_ack_placeholders was using CF_OPERATION_TRANSFER_PLACEHOLDERS_FLAG_NONE (0x0) in the CfExecute ack. FLAG_NONE leaves ENABLE_ON_DEMAND_POPULATION active on the directory → CF considers population incomplete → re-fires FETCH_PLACEHOLDERS on every directory access → 1550+ calls in minutes → concurrent CfCreatePlaceholders calls → CF state corruption → 0x80070781 on New File. Fix: use CF_OPERATION_TRANSFER_PLACEHOLDERS_FLAG_DISABLE_ON_DEMAND_POPULATION (0x2). Marks the directory as fully populated after each callback run. Subdirectories with ENABLE_ON_DEMAND_POPULATION still receive their own FETCH_PLACEHOLDERS on first access; the ack marks each one as done in turn. Ongoing remote changes continue to be delivered by the sync engine (Watch/reconciliation). 3 spec tests added (TestRegression157_*) — 18/18 packages PASS race-detector ✓. Closes #157. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(cfapi): revert DISABLE_ON_DEMAND_POPULATION, add cooldown dedup (#157 #158) Regression #158: after #157 (ada6568), bidirectional sync completely dead. Root cause: CfExecute with DISABLE_ON_DEMAND_POPULATION modifies the directory CF reparse point, interfering with ReadDirectoryChangesW (fsnotify) and/or blocking CF file operations in the sync root. Fix: revert to FLAG_NONE in ghd_cf_ack_placeholders (no reparse point change) and add a 30s cooldown deduplication guard in ghdOnFetchPlaceholders (provider.go): - First FETCH_PLACEHOLDERS per path: run OnFetchPlaceholders normally + store ts - Subsequent calls within 30s: ack immediately (S_OK, FLAG_NONE), skip backend.List() This stops the callback flood (#157 root cause) without touching CF directory state. Added SyncProvider.populatedDirs (sync.Map) for per-path cooldown tracking. Updated spec tests: replaced #157 DISABLE_ON_DEMAND assertions with #158 FLAG_NONE assertions + 5 new cooldown boundary tests. 18/18 packages PASS, race detector clean. Closes #158. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(cfapi): skip CfConvertToPlaceholder for files in hrAlreadyExists branch (#159) 0x80070781 persists on New File even with cooldown (#158): corruption happens during the first FETCH_PLACEHOLDERS pass before the cooldown is active. Root cause: in createPlaceholdersWithFlags, the hrAlreadyExists branch called CfConvertToPlaceholder for all entries including files. When the file is already a CF placeholder (from a prior FETCH_PLACEHOLDERS pass or ActionDownload), calling CfConvertToPlaceholder rewrites its Extended Attributes and can corrupt the parent directory's CF index → 0x80070781 on all subsequent CF operations in that dir. Fix (Option B): skip CfConvertToPlaceholder for files in the hrAlreadyExists branch. Files present when FETCH_PLACEHOLDERS fires are already CF placeholders or NTFS files handled by Watch() + ActionUpload. Directories still need conversion for ENABLE_ON_DEMAND_POPULATION (#156 guard still applies). This reverts the 69628d7 file conversion intentionally: the badge ✓✓ trade-off is worth avoiding CF index corruption that blocks Explorer file creation. Updated spec helpers (convertOnAlreadyExistsSpec, convertPathSpec, dirConvertResultSpec), 4 new TestRegression159_* tests, 6 updated existing tests. 18/18 packages PASS, race detector clean. Closes #159. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(cfapi): eliminate FETCH_PLACEHOLDERS race with per-directory mutex (#v2.2-bugA) Multiple simultaneous CGO callbacks for the same directory bypassed the 30s cooldown (all saw cache-miss before any stored the timestamp), causing concurrent backend.List() + CfCreatePlaceholders calls logged as "0s ago". Add fetchMu sync.Map of *sync.Mutex keyed by localPath. LoadOrStore ensures only one goroutine executes OnFetchPlaceholders at a time; concurrent ones block then hit the cooldown guard. Also add SetCompletionCallbacks to inject OnDeleteCompletion/OnRenameCompletion after Connect (used by bugD fix). Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(cfapi): wire OnDeleteCompletion/OnRenameCompletion to propagate CF ops to backend (#v2.2-bugD) CFCallbacks only wired OnFetchData, OnCancelFetch, and OnFetchPlaceholders. Delete and rename on GhD: (CF sync root) fired ghdOnDeleteCompletion / ghdOnRenameCompletion in C but the Go callbacks were nil — the operations were silently dropped and never propagated to the remote backend. Add CompletionCallbacks struct and CFManager.SetCompletionCallbacks() to inject these handlers post-Start. Internally delegates to SyncProvider.SetCompletionCallbacks which updates p.callbacks under the existing mutex. Called from app.go after the sync engine is started. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(sync): add HandleCFDelete/HandleCFRename for CF API completion events (#v2.2-bugD) When the user deletes or renames a file in GhD: (the CF sync root, not WinFsp), Windows fires NOTIFY_DELETE_COMPLETION and NOTIFY_RENAME_COMPLETION callbacks. These were unhandled — the remote backend never received the operation and the old path survived on the server, reappearing on the next reconciliation. Add Engine.HandleCFDelete and Engine.HandleCFRename which dispatch ActionDelete and ActionRename respectively to the backend. Both respect the existing file-state guard (#141) and transfer state across rename (#139/#149). Wired from app.go via cfManager.SetCompletionCallbacks after the engine starts. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(app): wire CF completion callbacks to sync engine after engine start (#v2.2-bugD) After starting the sync engine, inject OnDeleteCompletion and OnRenameCompletion into the CF provider via cfManager.SetCompletionCallbacks. These closures call engine.HandleCFDelete and engine.HandleCFRename respectively, propagating CF API delete/rename notifications from GhD: to the remote backend. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(cfapi): fix localToRemote case-insensitive path matching on Windows (#v2.2-bugC) strings.HasPrefix is case-sensitive. CF API NormalizedPath may return a drive letter with different casing than the sync root registered path (e.g. "c:\" vs "C:\"). A mismatch caused localToRemote to return the raw Windows absolute path as the remote path, making backend.ReadAt fail and FETCH_DATA transfer 0 bytes (copy/open of a remote placeholder showed 0-byte files). Fix: normalise both paths to lowercase + forward-slash before prefix comparison, then strip using the original byte count (preserves original casing for the relative remote path sent to the backend). Also strip trailing slash from root before comparison to avoid double-slash edge cases. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(cfapi): add -lole32 to CGO LDFLAGS for CLSIDFromString linkage cross-compile with mingw failed: undefined reference to __imp_CLSIDFromString. CLSIDFromString (used in cgo_cfapi_windows.c) lives in ole32.dll — adding -lole32 to the LDFLAGS resolves the missing symbol. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(cfapi): re-apply DISABLE_ON_DEMAND_POPULATION — test hypothesis that #158 sync breakage was Bug C/D, not the flag #158 reverted DISABLE_ON_DEMAND_POPULATION assuming the flag's reparse-point modification killed bidirectional sync. However, Bugs C and D were also present at the time of #158: - Bug C: localToRemote() case mismatch → uploads silently sent to wrong path - Bug D: OnDeleteCompletion/OnRenameCompletion never wired → deletes/renames dropped Now that C and D are fixed (c5875ec, 857f42d, 1005670, bdf0a43) and the per-directory mutex dedup is in place (1377bf5), re-applying the flag tests the hypothesis: if sync remains alive, #158 was caused by Bug C/D, not DISABLE_ON_DEMAND_POPULATION. If sync breaks again, the flag is genuinely at fault and must be reverted. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(moosefs): skip ANTOAN_NOP keepalive in ReadChunk + harden CS transport (#160) Root cause of #160: ReadChunk's frame loop treated the ANTOAN_NOP (cmd=0) keepalive as a fatal unknown opcode, aborting every read a chunk server interleaved a keepalive into — exactly what the reference Python POC skips but the Go port lost. isStaleConnErr then classified that fatal error as a "stale connection" and retried, masking the bug behind a retry that could never converge (the fresh connection gets a NOP too, deterministically). Phase 1 (root cause, csclient.go): - ReadChunk: skip ANTOAN_NOP frames, bounded by maxConsecutiveNOPs so a CS emitting nothing else cannot hang the caller (CA1, CA2, CA3) - errUnexpectedCmd removed from isStaleConnErr; detection is now typed (io.EOF, syscall.ECONNRESET/ECONNABORTED, net.Error) instead of the previous substring matching, which silently absorbed non-retryable errors too (CA4, CA7) - rewrote the file-header and errUnexpectedCmd comments that documented the disproven "half-closed socket" diagnosis Phase 2 (transport hardening, csclient.go/client.go/ecclient.go): - doCSRead: retry policy shared by Client.Read and readEC4At — attempt 0 may reuse a pooled connection, every retry forces a fresh dial (CA5), non-retryable errors fail fast without consuming budget (CA7), retryable ones back off exponentially under a hard cumulative budget kept well below the CF API's own Open() retry interval (CA6) - DialCS: dial timeout + TCP keepalive via net.Dialer - ReadChunk: single deadline covering the whole call; WriteChunk: idle deadline reset before every frame instead, so a large upload isn't falsely timed out by a global deadline (CA9, CA12) - csPool: idle connections are timestamped and evicted past maxIdleConnAge instead of being served stale (CA8); newCSPoolWithMaxAge added for tests go test ./plugins/... -race -cover: all green, mfsclient 74.3% (CA13), webdav/local untouched and green (CA11), WriteChunk suite green (CA12). Co-Authored-By: Claude Sonnet 5 <[email protected]> * fix(placeholder): stop serving truncated downloads as complete (#160/#162) D7 — a download interrupted partway left a partial file that was then served as if complete for up to cacheTTL: isCacheFresh only checked size > 0, and the failed Download() never cleaned up its output. - ensureDownloaded/downloadToCache now write to a private temp file and only rename it onto the final cache path on success, so a partial or failed download can never occupy the cache path (task 13, task 14). - isCacheFresh now compares the cached file's size against the remote size (via backend.Stat, reusing the Getattr metadata cache) instead of just checking size > 0; a mismatch invalidates and re-downloads, self-healing any already-truncated entry on next access (task 15). - downloadCoordinator deduplicates concurrent Open() calls on the same cache path so only one download is ever in flight per file (task 16). - Downloads are retried up to 3 times with capped exponential backoff instead of relying on the FUSE client to re-issue Open() in a cascade (task 17). - Each attempt is now bounded by a context timeout instead of context.Background(); sized well above WebDAV's existing 30s http.Client.Timeout so it never cuts off transfers earlier than WebDAV's own limit would (task 18). Fix lives entirely in ensureDownloaded, which is backend-agnostic (StorageBackend interface) — covers WebDAV, Local and MooseFS without touching any plugin. WebDAV/Local test suites green (no regression). Ref: _work/reports/plan-20260812-102022.md, tasks 13-18. * fix(moosefs): rev.2 — inactivity timeout, location invalidation, official backoff (#160) Revises the Phase 1+2 fix per the planner's revision 2, which corrects two things the first pass got wrong relative to the official MooseFS client (moosefs/mfsclient/readdata.c) used as the reference implementation: - ReadChunk previously used one static deadline covering the whole call. The official client instead uses an *inactivity* timeout — rearmed on every frame received (CHUNKSERVER_ACTIVITY_TIMEOUT, readdata.c:78/:1395) — so a read that keeps progressing is never killed, only a connection that goes silent. ReadChunk and WriteChunk now share one csInactivityTimeout (5s), both rearmed per-frame; the flood guard (maxConsecutiveNOPs) remains the only bound on a CS that never goes silent but never stops sending keepalives either. - A retry now invalidates the cached chunk location and re-queries the master (Client.locateChunk) before contacting a chunk server again, mirroring chunksdatacache_invalidate (readdata.c:1989) — the previous version kept re-attacking the same cached, possibly-faulty CS. doCSRead takes a locateFunc so both Client.Read and readEC4At (now taking nodeID) re-resolve on attempt > 0 instead of reusing the first ChunkInfo. Also per revision 2: - ANTOAN_NOP frames with a non-zero payload are now a hard error (readdata.c:1683) instead of being silently skipped like a real keepalive. - The unexpected-opcode error now includes the peer address (RemoteAddr), matching readdata.c:1694, to keep any residual desync diagnosable. - Retry backoff now follows the official schedule (readdata.c:1994) via a new internal/backoff package: 3 immediate attempts, then a 1ms+(n-3)*300ms ramp capped at 10s — exposed as backoff.MooseFS.Delay(n), a top-level package (not under plugins/moosefs/internal) so dev-backend can reuse it for issue #162 in internal/placeholder. - ReadAt now checks ctx.Err() before starting a read (it previously never checked at all); Download's existing between-block check is now backed by the inactivity timeout for the "stuck mid-block" case. Scope: plugins/moosefs/ only (mfsclient package + moosefs.go) + the new shared internal/backoff package. internal/cfapi and internal/placeholder are out of scope for this commit (Phase 3, dev-backend). go build ./plugins/... and go build ./... (module-wide, cgofuse's unrelated missing-header failure aside) are both green. go vet currently fails only inside csclient_test.go/ecclient_test.go — expected: those files are mid-rewrite by test-writer for revision 2 (readEC4At gained a nodeID parameter; csReadChunkDeadline was renamed to csInactivityTimeout). Co-Authored-By: Claude Sonnet 5 <[email protected]> * test(moosefs): tests for bugfix #160/#162 (NOP keepalive, retry/pool, cache integrity) Phase 4 (plan revision 2, _work/reports/plan-20260812-102022.md) — tests derived from docs/diagrams/moosefs-ec4-read-statemachine.md invariants, covering both dev-plugin's mfsclient protocol fix (#160) and dev-backend's placeholder cache-integrity fix (#162). plugins/moosefs/internal/mfsclient (csclient_test.go, ecclient_test.go): - TestReadChunk_NOPskip / NOPBadLength / NOPFlood / UnknownCmd (CA1-CA4) - TestReadEC4Cmd0_NOPIgnored + TestReadEC4Cmd0_TrueStaleness_TriggersRetry, replacing the old TestReadEC4StaleConnection_Cmd0 which encoded the erroneous "NOP = stale connection" diagnosis (CA1, CA5) - TestRetry_ForcesFreshDial / InvalidatesChunkLocation / BackoffBounded / NonRetryableFailsFast (CA6-CA8) - TestReadChunk_InactivityTimeout (progressing read never interrupted vs. silent connection bounded) / TestCSPool_MaxIdleAge (CA9, CA10) - makeECClient upgraded to back tests with a stable in-memory master so the new locateChunk-based retry path (#160 D3bis) works in EC4-level tests internal/placeholder (download_integrity_test.go, new): - TestEnsureDownloaded_PartialRemovedOnError (CA11) - TestEnsureDownloaded_RedownloadsTruncatedCache — end-to-end complement to dev-backend's TestIsCacheFresh_SizeMismatch_ReturnsFalse, proving isCacheFresh/remoteSize/downloadToCache are wired together correctly (CA12) - TestEnsureDownloaded_ConcurrentOpensSingleDownload (CA13) - metacache_vfs_test.go: added ReadAt/ChunkSize/Rename/Copy stubs to the shared mockStorageBackend (interface had grown without every mock being updated — pre-existing gap, fixed incidentally to unblock compilation) tests/procedures/ec4-chunk-read.md: - Scenario 6 (#160 perf) and Scenario 7 (#162 integrity after induced failure) with read-only MooseFS access reminder Non-regression: full mfsclient suite (74 tests) green, -race, 21.6s. internal/placeholder verified via GOOS=windows go vet (Windows-only build tag, cannot execute on this host). Co-Authored-By: Claude Sonnet 5 <[email protected]> * fix(moosefs): truncate final zero-padded block in Download() (#160/#162) QA (qa-20260812-112858.md) found a preexisting bug that the new strict isCacheFresh size check (CA12) turned blocking: Download() writes every 64 KiB chunk read verbatim, but a real MooseFS chunk server zero-pads a read within a chunk boundary — the last block of a file whose size is not a multiple of chunkSize therefore always arrives padded to a full chunkSize. The local cache file ended up larger than the remote size on essentially every file, so the strict size-equality freshness check could never pass: every Open() redownloaded the whole file, silently defeating #160. Truncate the block to the file's real remaining length before writing it, right after the read. offset/done then advance by the real byte count, so the loop's own EOF guard still terminates cleanly on the next iteration. Also fixes the structural test blind spot QA flagged: the in-memory fake chunk server (moosefs_test.go) always returned exactly the stored data length, never padding like the real protocol does, so no existing test could have caught this. Added an opt-in padReadsToRequestedSize flag on integFakeCSServer (default false — every other test keeps its current exact-length semantics) and TestDownload_TruncatesFinalPaddedBlock, which fails without the fix (verified by reverting moosefs.go locally) and passes with it. go test ./plugins/... -race -cover: all green, mfsclient 74.3%, moosefs 75.3%, webdav/local unaffected. Co-Authored-By: Claude Sonnet 5 <[email protected]> * chore(release): bump version to 2.2.1 (bugfix #160/#162) Fix protocole MooseFS EC4 (keepalive ANTOAN_NOP, timeout inactivite, invalidation localisation, backoff officiel) et perte de donnees silencieuse du cache placeholder (ecriture atomique, validation stricte taille distante, deduplication, relances bornees). Z-segment bugfix, milestone v2.2 actif. QUALIF v2.2.1 : build frais valide (72.5% cov, -race clean), scenarios manuels 6/7 (tests/procedures/ec4-chunk-read.md) valides par QA. Co-Authored-By: Claude Sonnet 5 <[email protected]> * test(moosefs): #163 Phase 0 — measured baseline before B1/B2 (chunk location + read granularity) Blocking prerequisite (plan task 2, _work/reports/plan-20260812-155238.md): without a measured "before" baseline, CA7/CA8 (throughput/mass-copy gains) are not demonstrable. Adds readChunkCount (master, CLTOMA_FUSE_READ_CHUNK) and csReadCount (chunk server, CLTOCS_READ) atomic counters to the existing in-memory fake server, and TestDownload_NetworkRoundtripBaseline, which downloads a reference file (2,109,497 B — deliberately not a chunkSize multiple, single MooseFS chunk) and asserts the current 1:1 ratio: one locateChunk call and one ReadChunk call per 64 KiB block (33 blocks -> 33 + 33 calls). This is the exact mechanism the plan's arithmetic proof (from the issue's production logs) attributes the latency plateau to. Measured now, pre-fix: - file=2,109,497 B (33 blocks of 65,536 B) - locateChunk_calls=33, ReadChunk_calls=33 (1:1 with block count) - elapsed=~43ms on this loopback fake server — NOT representative of real network latency; the authoritative duration/throughput baseline remains the issue's own production logs (~4.03-4.25 ms/roundtrip, ~8 MB/s, P2 S14 T2.MP4 and P2 S15 T1.MP4), cited in the plan and repeated in the dev-plugin handoff for #163. The assertions are intentionally pre-fix expectations: they will be updated (not deleted) once the Phase 1 location cache and Phase 2 read-granularity change land, so the improvement is proven by a diff in this same test rather than assumed. Co-Authored-By: Claude Sonnet 5 <[email protected]> * test(moosefs): tests for bugfix #163 (chunk-location cache, read granularity) Phase 4 (_work/reports/plan-20260812-155238.md) — tests for B1 (chunk location cache) and B2 (larger multi-shard reads), plus non-regression of #160/#162 at the new granularity and a performance measurement procedure. plugins/moosefs/internal/mfsclient (new: chunklocation_cache_test.go): - TestChunkLocationCache_HitAvoidsMasterRoundtrip (CA1) - TestChunkLocationCache_InvalidatedOnReadError (CA2) - TestChunkLocationCache_VersionMismatchRefetches (CA3) - TestChunkLocationCache_Eviction + _LRUOrderPreservedOnHit (CA6) — white-box against chunkLocationCache directly (fast, precise, uses the real chunkLocationCacheMaxEntries constant) rather than thousands of network round-trips through Client.Read() ecclient_test.go: - TestReadEC4At_MultiMiBRead — large single-shard read == concatenation of 64 KiB reads, byte for byte (CA4) - TestReadEC4At_ShardBoundarySplit — a read straddling two shards is split and reassembled correctly (CA4) csclient_test.go: - TestReadChunk_MultiFrameCRC — CRC checked on every frame of a multi-frame response, corrupted frame detected even after valid ones (CA5) - TestReadChunk_NOPSkip_LargeMultiFrame — #160 NOP-skip replayed at B2's larger multi-frame granularity (task 19 non-regression) moosefs_test.go: - TestDownload_LastBlockTruncation — table-driven complement to TestDownload_TruncatesFinalPaddedBlock across chunkSize-relative boundary sizes, granularity-agnostic (references the chunkSize constant, never a literal) (CA11) tests/procedures/: - mass-copy-performance-163.md (new) — reproducible before/after throughput protocol (single file + full folder copy), mandatory measurement table, read-only MooseFS access reminder (CA7, CA8) - ec4-chunk-read.md — cross-reference to the new procedure Non-regression: full mfsclient suite 83/83 PASS, -race, 25.5s (includes existing #160/#162 tests + all new #163 tests). moosefs package: 47/48 PASS; the one failure (TestWatch_adaptiveReset) is a pre-existing order-dependent flake in unrelated Watch/polling code (passes 3/3 in isolation, unrelated to #163's read path) — flagged in the handoff, not fixed here (out of scope). Written in parallel with dev-plugin actively landing B1 (chunklocationcache.go) and B2 (ecclient.go shard-splitting) — most tests found the target behaviour already implemented and passed on first run; TestChunkLocationCache_Eviction's LRU math was corrected once against the real cache implementation. Co-Authored-By: Claude Sonnet 5 <[email protected]> * fix(moosefs): chunk-location cache (B1) + shard-aware multi-MiB reads (B2) (#163) Issue #163 blamed the lack of inter-file pipelining for slow mass copies. The plan's arithmetic proof of the issue's own logs shows otherwise: two files of very different sizes both plateau at ~4.1 ms per network roundtrip (~8 MB/s), which is a latency ceiling driven by roundtrip COUNT, not bandwidth — pipelining files that are each individually this slow would not fix it. Root cause is two compounding inefficiencies in plugins/moosefs, both intra-file: B1 — no chunk-location cache. Client.Read called locateChunk (a master CLTOMA_FUSE_READ_CHUNK roundtrip) before every chunk-server read, regardless of size. A 64 MiB MooseFS chunk read in 64 KiB blocks meant requesting the SAME location up to 1024 times. Worse, locateChunk held Client.mu — the single master connection's mutex — serializing every other master operation (GetAttr, Lookup, ReadDir) across the whole plugin, not just within one file. #160 already ported the *invalidation* half of the official client's chunksdatacache model (a retry re-queries the master instead of reusing a stale location) without the cache that invalidation was meant to protect; this is the other half. Adds chunklocationcache.go: a bounded (8192 entries), TTL-aware (30s) LRU cache indexed by (nodeID, chunkIndex), guarded by its own mutex (never Client.mu, so a cache hit never contends with other master traffic). Client.locateChunk(nodeID, index, forceRefresh) checks the cache first unless forceRefresh (used exclusively by the #160 D3bis retry path, which now also invalidates the entry before re-querying — the crucial wiring this task required: without it, a cache would have silently defeated #160's "never retry against the same bad location" guarantee). No separate chunk-version validation before use: the wire protocol already carries the cached Version in every CLTOCS_READ, so a stale entry surfaces as a chunk-server status error, which the existing #160 error-driven invalidation path already handles. B2 — 64 KiB read granularity despite the protocol already supporting far larger single requests. ReadChunk already collects multiple CSTOCL_READ_DATA frames (each CRC-checked) per CLTOCS_READ; only readEC4At's documented precondition (size <= one 64 KiB block) forced the caller to chop every read that small. readEC4At now accepts up to a full shard per call and splits a request across shard boundaries as needed (one CS request per shard crossed — a CLTOCS_READ cannot itself read past the shard/physical chunk it names). moosefs.go's shared Upload/Download block size goes from 64 KiB to 4 MiB (divides a full chunk's 16 MiB shard evenly). The #160 zero-pad truncation in Download (4e0cbd3) is untouched and remains size-agnostic, so it keeps working unmodified at the new granularity. Incidentally also resolves the cfapi ReadAt/ChunkSize() 64 MiB-vs-64 KiB precondition mismatch the plan flagged as latent (not exercised by #163's own FUSE path) — full cfapi verification stays with #161. Measured (plugins/moosefs, in-memory fake server, call counts are exact and reproducible; wall-clock is not representative of real network latency — see the issue's own production logs for that, cited in the plan and the dev-plugin handoff for #163): before: 2,109,497 B file, 33 x 64 KiB blocks -> 33 locateChunk calls, 33 ReadChunk calls (1:1 with block count) after: 13,360,689 B file (3 x 4 MiB + partial), still 1 MooseFS chunk -> 1 locateChunk call (cache hit for blocks 2-4), 4 ReadChunk calls (one per Download iteration, as expected — B1 collapses the master roundtrips specifically, B2 raises the bytes moved per chunk-server roundtrip) go test ./plugins/... -race -cover -count=1: all green (mfsclient 74.9%, moosefs 75.5%, webdav/local unaffected). One unrelated, pre-existing flake observed under full-suite -race load (TestWatch_adaptiveReset, Watch/ polling code untouched by this change) — passes 3/3 in isolation, not a regression from this commit. Co-Authored-By: Claude Sonnet 5 <[email protected]> * fix(moosefs): bound chunk-location cache staleness on server status rejection (#163) code-reviewer (code-reviewer-20260813-093106.md) found a MAJOR: the chunklocationcache.go design comment claimed a stale cached location "naturally surfaces as a non-OK server status ... already routed through doCSRead's re-locate-on-retry path" — but doCSRead only retries (and thus only forces a relocate) for isStaleConnErr-classified errors (typed connection failures), and a non-OK CSTOCL_READ_STATUS was, since #160, a deliberately non-retryable fmt.Errorf (CRC/status errors must fail fast). #163 added a second, unrelated meaning to that same error class (a stale cached placement) without changing its classification: a chunk relocated or rewritten during the cache's 30s TTL window would make a read fail outright instead of self-correcting via a fresh master query, exactly as the comment implied it would. Fail-safe (no wrong data, CA3 holds in the strict sense) but not the self-healing robustness claimed. - csclient.go: new errServerStatus sentinel, wrapped by ReadChunk's non-OK CSTOCL_READ_STATUS branch. doCSRead grants it exactly ONE forced chunk-location refresh (locationRefreshUsed flag) before treating it as terminal — never the full isStaleConnErr retry/backoff budget, so a persistent non-location status rejection still fails fast on its second occurrence rather than burning through maxCSAttempts pointlessly. - chunklocationcache.go: design comment corrected to describe the actual (bounded, error-triggered) mechanism instead of an inaccurate "already handled by the existing retry path" claim. - chunklocation_cache_test.go: two new tests exercising the REAL scenario end to end through Client.Read() — TestChunkLocationCache_Version- MismatchRefetches only ever exercised a connection-error trigger, as its own comment admitted. TestChunkLocationCache_ServerStatusRejection- ForcesRelocate (a rejecting CS -> one relocate -> a working CS succeeds, exactly 2 master roundtrips) and TestChunkLocationCache_ServerStatus- RejectionIsBoundedNotLooped (both locations reject -> fails with errServerStatus after exactly 2 master roundtrips, never more). Also code-reviewer MINOR-2: ecclient.go readEC4At's multi-segment loop (#163 B2) resolved every segment's shard geometry from the ORIGINAL info parameter, ignoring a fresher ChunkInfo obtained by an earlier segment's retry. Unreachable before #163 (a read never crossed a shard boundary); reachable now that one call can span multiple shards. Fixed by tracking currentInfo across loop iterations, updated by the locate closure whenever a segment's retry re-queries the master, so later segments start from whatever the most recent segment actually learned. go test ./plugins/... -race -cover -count=1: all green (mfsclient 75.3%, moosefs 75.5%). gofmt -l flags csclient.go, confirmed pre-existing at de4dbc6 (before this commit) — not introduced here, left as-is per code-reviewer's MINOR-1 disposition (separate cleanup commit). Co-Authored-By: Claude Sonnet 5 <[email protected]> * fix(moosefs): missing EOF check in locateEC4Shard breaks every non-shard-aligned file (#163) QA (qa-20260813-101152.md) found a QUALIF-blocking regression on v2.2.2: 8/8 real large files failed to download, all with the identical signature "shardIdx 4 out of range" at chunkOffset == exactly 4*shardSize. Worse than the performance problem #163 set out to fix — these same files downloaded fine (slowly) before this branch. Root cause: locateEC4Shard only treated chunkDataSize == 0 as EOF. shardSize is rounded UP (divCeil(chunkDataSize, 4) then aligned to a 64 KiB block), so for a chunk whose real data is not an exact multiple of 4*shardSize — the last MooseFS chunk of essentially every real file — 4*shardSize overshoots chunkDataSize. readEC4At's segmented read loop (#163 B2) is bounded by the caller's requested size, not by chunkDataSize, and Download() always requests a fixed block size without knowing in advance how much of the current chunk is real (exactly like the #160 whole-chunk zero-pad case) — so the loop kept walking curOffset forward until it landed precisely on 4*shardSize, one past the 4 valid shards (0-3), and locateEC4Shard computed shardIdx=4 and failed hard instead of recognizing EOF. Before #163 (64 KiB requests, one CLTOCS_READ at a time, no internal segment loop) this path could not be reached the same way. B2's segmented loop is what turned a previously-unreachable arithmetic edge into a guaranteed crash on any non-shard-aligned file. Fix (ecclient.go, locateEC4Shard): check chunkOffset >= chunkDataSize right after computing chunkDataSize, before any shardSize-based division, so EOF is detected at the chunk's real length regardless of the shardSize rounding margin. Test gap this exposed: TestReadEC4At_MultiMiBRead and TestReadEC4At_ShardBoundarySplit both use an info.Length that IS an exact multiple of 4*shardSize, so neither exercised a partial last chunk. TestReadEC4At_PartialLastChunkEOF reproduces the exact QA failure (verified failing pre-fix with the identical "shardIdx 4 out of range" message, verified passing post-fix) with a chunkDataSize deliberately not a multiple of 4*shardSize. go test ./plugins/... -race -cover -count=1: all green (mfsclient 75.5%, moosefs 75.5%). Co-Authored-By: Claude Sonnet 5 <[email protected]> * test(moosefs): pin locateEC4Shard boundary cases (code-review follow-up, #163) code-reviewer (code-reviewer-20260813-103203.md, APPROVED) independently wrote and ran a 7-case white-box boundary sweep against locateEC4Shard while verifying 91844b5 (1-byte chunk, empty chunk, exact-4*mfsBlockSize boundary with no rounding margin, etc.) — all passed, then discarded the test. Non-blocking recommendation: pin at least the tiny-final-chunk case permanently, since this same function has now produced two successive edge-case regressions in a row (the server-status-staleness MAJOR in 77ffe04's predecessor, and the QUALIF-blocking crash fixed in 91844b5). TestLocateEC4Shard_BoundaryCases reproduces all 7 of the reviewer's cases as a permanent, table-driven test — cheap insurance against a future change to locateEC4Shard silently breaking one of these magnitudes again without any test catching it. go test ./plugins/... -race -cover -count=1: all green (mfsclient 75.5%, moosefs 75.5%). Co-Authored-By: Claude Sonnet 5 <[email protected]> * chore(release): bump version to 2.2.2 (bugfix #163) Fix performance copie de masse MooseFS : cache de localisation de chunk (LRU+TTL 30s) + granularite de lecture 64 KiB -> 4 MiB (alignee sur shards 16 MiB). Regression critique (locateEC4Shard EOF check manquant, fichiers non-alignes illisibles) trouvee et corrigee avant merge. Z-segment bugfix, milestone v2.2 actif. QUALIF v2.2.2 : build frais valide (72.5% cov, -race clean), mesure perf reelle validee par QA (fichier unique x2.63, copie masse x1.5, integrite complete). Co-Authored-By: Claude Sonnet 5 <[email protected]> * chore(claude): sync template v3.3.1 (2fc09471) — teamleader/agents table + versioning docs Migre les contextes partages legacy (agents/context/*.md sans compagnon) vers le pattern *.template.md, met a jour la table "Agents Disponibles" et la section Versioning dans CLAUDE.md, et resynchronise init-project.md lui-meme. Le schema infrastructure (binaires GitHub Releases) et stack.plugins (StorageBackend Go) restent inchanges — pas d'equivalent dans le nouveau modele environments[] generique du template. Co-Authored-By: Claude Sonnet 5 <[email protected]> * chore(claude): sync template v3.8.0 — environments[] schema, protocol block, mockup/tests indexes Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * fix(ci): pin protoc-gen-go-grpc to go1.24-compatible version QUALIF installait protoc-gen-go-grpc@latest (v1.6.2, requiert Go >= 1.25) alors que la CI tourne en Go 1.24 : job Go rouge sur la PR #165. Le step QUALIF utilise desormais PROTOC_GEN_GO_GRPC_PIN (v1.5.1) de .ci-versions.env ; le sync des pins capture donc v1.5.1 et ne propage plus v1.6.2. project-config.json : infrastructure.version_sources[0] server-go/config.json -> config.json (seul hunk commite ; le bump template_version v3.9.0 reste non commite). Ecarts restants traites dans l'issue #166. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * chore(claude): sync template v3.9.0 Sync des agents/context/commands du template, bump template_version v3.9.0 dans project-config.json et mise a jour de .gitignore. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * chore: commit local workflow state, QUALIF artifacts and wails bindings Etat de workflow, artefacts QUALIF 2.2.x (hors binaires >5 Mo : *.exe et *.ghdp), couverture QA, diagrammes, bindings Wails et package-lock. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> --------- Co-authored-by: Claude Sonnet 4.6 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Résumé
StorageBackend), cache d'états fichiers, toast de conflit (feat(cfapi): cache d'état fichier L/P/U distinct pour badges UI GhostDrive #136–feat(ui): avertissement UI à l'ouverture d'un fichier en état Conflict #144)environments[], bloc protocole)Plan de test
🤖 Generated with Claude Code