Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 1 addition & 13 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,6 @@
# Changelog

## Unreleased

* `state.RootLock.acquire` no longer truncates the file behind a hard link
planted at `.lock`. It opens the state root `O_DIRECTORY|O_NOFOLLOW` and
both lock files relative to it with `O_NOFOLLOW`, creating them mode
0600; a lock file name that is not a regular file with one link is
refused before it is opened, so a FIFO or symbolic link there is never
followed; and once both locks are held, the gate must still be a regular
file whose one link is `.lock` before it is truncated.
* A refused gate is the new `RootLock.AcquireError.GateNotPrivate`
(`NotPrivate` stays the sidecar's), a symbolic link as the state root is
`NotDir`, and `LockOps.open_sidecar` takes the state root's descriptor
and a name.
<!-- towncrier release notes start -->

## 0.8.1

Expand Down
6 changes: 4 additions & 2 deletions COMPAT.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ breaks a tool shows before the tool moves to it.

`sys.describe` prints glibc's `strerror` text, as Python's `os.strerror` does,
for every errno the Zig standard library's `linux.E` names. An errno it does
not name prints `[Errno N] Unknown error N`; on x86_64 that is errno 134,
which glibc calls "Memory page has hardware error".
not name prints `[Errno N] Unknown error N`. On x86_64, errno 133 is
`HWPOISON`, `[Errno 133] Memory page has hardware error`, and errno 134 is
not a named errno: both glibc and this library print `[Errno 134] Unknown
error 134`.

4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,11 +44,11 @@ the entry and its hash.
| module | what it does |
|---|---|
| `sys` | `std.os.linux` calls with EINTR retried and errno mapped to a Zig error set |
| `fsx` | open without following a final symbolic link, checks on the descriptor (`fstat`), cp-style copy, line append, atomic replace keeping owner, mode and access ACLs, directory iteration, `rm -rf` that never follows links, `realpath` through `/proc/self/fd` |
| `fsx` | open without following a final symbolic link, checks on the descriptor (`fstat`), a copy that renames into place so a short write does not destroy the destination, line append, atomic replace keeping owner, mode and access ACLs, directory iteration, `rm -rf` that never follows links, `realpath` through `/proc/self/fd` |
| `proc` | run a program by absolute path with an empty signal mask, default handlers and no descriptors above 2, by default in its own session; stdin fed and stdout and stderr read at once, each with a cap; an optional deadline, SIGTERM, a grace period, SIGKILL to the process group; a cancel flag a signal handler can set; group members left after the child exits are killed before it is reaped; `which` looks a name up in an explicit PATH |
| `logs` | `gzip -9` in place with gzip's header (name, mtime, OS byte), owner, mode and times kept; rewrite of `.licensed-logs` entries |
| `env` | the environment as raw bytes with getenv's first-entry rule, duplicate names found, a child environment built with each name once, shell name check |
| `state` | private files in a shared directory: trusted only when the open file is a regular file of this user with one link and no group or other bits, a lock trusted the same way, atomic 0600 writes; `RootLock`, a flock gate plus a POSIX record lock on a sidecar, with a deadline |
| `state` | private files in a shared directory: trusted only when the open file is a regular file of this user with one link and no group or other bits, a lock trusted the same way, atomic 0600 writes; `RootLock`, a flock gate plus a POSIX record lock on a sidecar, with a deadline read from the clock. The gate and the sidecar must each be a regular file of this user with one link |
| `json` | a JSON reader with Python's two string encodings (surrogateescape text, one byte per code point), Python values, and `json.dumps(indent=N, sort_keys=True)` byte for byte, float repr included |
| `pytext` | byte strings as Python text: the surrogateescape character, the 'replace' decoder, str.isspace, strip and split, shlex.quote |
| `atomic` | `atomic_write`: replace a file only after its bytes, owner, mode and every ACL form (GPFS NFSv4, NFSv4, POSIX access) are durable and read back; fault points before write, fsync and rename |
Expand Down
1 change: 1 addition & 0 deletions changelog.d/+append-short-write.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`fsx.appendLine` rolls a short write back, so a full disk does not leave a partial line.
1 change: 1 addition & 0 deletions changelog.d/+atomic-eintr.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`atomic` retries an ACL read interrupted by a signal, instead of refusing the replacement.
1 change: 1 addition & 0 deletions changelog.d/+atomic-leak.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`atomicWrite` frees the path, the step names and the spare ACL read it allocates.
1 change: 1 addition & 0 deletions changelog.d/+copy-short-write.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`fsx.copyTo` renames the new bytes into place, so a short write leaves the existing file unchanged.
1 change: 1 addition & 0 deletions changelog.d/+errno-133.doc.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`COMPAT.md` names errno 133 as the hardware-error text. Errno 134 is unnamed, which is what glibc prints too.
1 change: 1 addition & 0 deletions changelog.d/+group-alive-eintr.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`proc.groupAlive` retries a `/proc` read interrupted by a signal, instead of reporting that the process group is gone.
1 change: 1 addition & 0 deletions changelog.d/+gzip-partial.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`logs.gzipInPlace` writes the archive under a temporary name and renames it into place, so a crash does not leave a partial `.gz` at the final name.
1 change: 1 addition & 0 deletions changelog.d/+json-escape.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`json` returns an encoding error for a truncated or unknown escape, instead of reading past the string.
1 change: 1 addition & 0 deletions changelog.d/+lock-gate.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`state.RootLock` also refuses a gate or sidecar that another user owns, as `GateNotPrivate` and `NotPrivate`: each must be a regular file of this user with one link.
1 change: 1 addition & 0 deletions changelog.d/+lock-timeout.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`state.lockExclusive` measures its deadline on the monotonic clock, so a signal that cuts a sleep short does not expire the wait.
1 change: 1 addition & 0 deletions changelog.d/+proc-std-fds.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`proc.run` gives the child /dev/null or its pipes on fds 0 to 2 when the caller started with one of them closed: the child's stderr no longer lands in the captured stdout, and its stdin or stderr is no longer closed at exec.
1 change: 1 addition & 0 deletions changelog.d/+proc-wait-status.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`proc.run` returns `error.NoChild` when the child's status is lost because the caller ignores SIGCHLD, instead of reporting a kill by signal 9.
1 change: 1 addition & 0 deletions changelog.d/+pysplit-oom.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`pytext.pySplit` frees its list when an allocation fails, and the slice it returns is the whole allocation.
1 change: 1 addition & 0 deletions changelog.d/+rootlock-api.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
A refused gate is the new `RootLock.AcquireError.GateNotPrivate` (`NotPrivate` stays the sidecar's), a symbolic link as the state root is `NotDir`, and `LockOps.open_sidecar` takes the state root's descriptor and a name.
1 change: 1 addition & 0 deletions changelog.d/+rootlock-truncate.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`state.RootLock.acquire` no longer truncates the file behind a hard link planted at `.lock`. It opens the state root `O_DIRECTORY|O_NOFOLLOW` and both lock files relative to it with `O_NOFOLLOW`, creating them mode 0600; a lock file name that is not a regular file with one link is refused before it is opened, so a FIFO or symbolic link there is never followed; and once both locks are held, the gate must still be a regular file whose one link is `.lock` before it is truncated.
108 changes: 101 additions & 7 deletions src/atomic.zig
Original file line number Diff line number Diff line change
Expand Up @@ -94,11 +94,19 @@ pub const Why = struct {

pub const Error = error{ Refused, OutOfMemory } || sys.Error;

const static_refused: []const u8 = "refused";

fn refuse(a: Allocator, why: *Why, comptime f: []const u8, args: anytype) Error {
why.msg = a.print(f, args) catch "refused";
why.msg = a.print(f, args) catch static_refused;
return error.Refused;
}

/// Free a message `refuse` stored, including its out-of-memory fallback.
fn freeMsg(a: Allocator, msg: []const u8) void {
if (msg.len == 0 or msg.ptr == static_refused.ptr) return;
a.free(msg);
}

/// The value of each ACL attribute present, keyed by its name.
pub const Acls = struct {
present: [acl_attributes.len]?[]const u8 = .{ null, null, null },
Expand Down Expand Up @@ -159,12 +167,27 @@ fn errnoText(e: linux.E) []const u8 {
};
}

fn freeAcls(a: Allocator, acls: Acls) void {
for (acls.present) |p| if (p) |s| a.free(s);
}

/// One `fgetxattr`, retrying when a signal interrupts it. `op` cannot be
/// used here: the errno itself decides whether a missing attribute is an
/// error, and `op` collapses errnos into `sys.Error`.
fn xattrGet(fd: sys.fd_t, name: [:0]const u8, buf: []u8) usize {
while (true) {
const rc = ops.fgetxattr(fd, name, buf);
if (linux.errno(rc) != .INTR) return rc;
}
}

fn readAllAcls(a: Allocator, fd: sys.fd_t, why: *Why) Error!Acls {
const listed = try listedAcls(a, fd, why);
var acls: Acls = .{};
errdefer freeAcls(a, acls);
var buf: [64 * 1024]u8 = undefined;
for (acl_attributes, 0..) |name, i| {
const rc = ops.fgetxattr(fd, name, &buf);
const rc = xattrGet(fd, name, &buf);
const e = linux.errno(rc);
if (e == .SUCCESS) {
acls.present[i] = try a.dupe(u8, buf[0..rc]);
Expand All @@ -189,7 +212,9 @@ fn readAllAcls(a: Allocator, fd: sys.fd_t, why: *Why) Error!Acls {
pub fn readAcls(a: Allocator, fd: sys.fd_t, why: *Why) Error!Acls {
const before = try fsx.fstat(fd);
const acls = try readAllAcls(a, fd, why);
errdefer freeAcls(a, acls);
const again = try readAllAcls(a, fd, why);
defer freeAcls(a, again);
if (!acls.eql(again) or !sameIdentity(try fsx.fstat(fd), before))
return refuse(a, why, "access metadata changed during inspection", .{});
return acls;
Expand Down Expand Up @@ -225,13 +250,16 @@ fn restoreMetadata(a: Allocator, fd: sys.fd_t, metadata: ?Metadata, mode: u32, w
break;
};
if (primary) |i| {
if (!(try readAcls(a, fd, why)).eql(m.acls)) {
const now = try readAcls(a, fd, why);
defer freeAcls(a, now);
if (!now.eql(m.acls)) {
const v = m.acls.present[i].?;
_ = try op(ops.fsetxattr, .{ fd, acl_attributes[i], v });
}
}
for (acl_attributes) |_| {
const present = try readAcls(a, fd, why);
defer freeAcls(a, present);
var extra: ?usize = null;
for (present.present, m.acls.present, 0..) |p, want, i| if (p != null and want == null) {
extra = i;
Expand All @@ -243,7 +271,9 @@ fn restoreMetadata(a: Allocator, fd: sys.fd_t, metadata: ?Metadata, mode: u32, w
const restored = try fsx.fstat(fd);
if (restored.uid != m.info.uid or restored.gid != m.info.gid or restored.perm() != mode)
return refuse(a, why, "ownership or mode did not round-trip: expected {d}:{d} {o}, found {d}:{d} {o}", .{ m.info.uid, m.info.gid, mode, restored.uid, restored.gid, restored.perm() });
if (!(try readAcls(a, fd, why)).eql(m.acls)) return refuse(a, why, "access-control attributes did not round-trip", .{});
const round = try readAcls(a, fd, why);
defer freeAcls(a, round);
if (!round.eql(m.acls)) return refuse(a, why, "access-control attributes did not round-trip", .{});
}

/// Called before each named step; a test hook can stop the program there.
Expand Down Expand Up @@ -359,12 +389,17 @@ pub fn atomicWrite(a: Allocator, path: []const u8, data: []const u8, opts: Write
} else |_| {}
const metadata_path: ?[]const u8 = opts.metadata_from orelse (if (exists(path)) path else null);
var metadata: ?Metadata = null;
defer if (metadata) |m| freeAcls(a, m.acls);
if (metadata_path) |mp| {
var inner: Why = .{};
metadata = readMetadata(a, mp, &inner) catch |err| return refuse(a, why, "cannot read access metadata for {s}: {s}", .{ mp, if (err == error.Refused) inner.msg else describe(err) });
metadata = readMetadata(a, mp, &inner) catch |err| {
defer freeMsg(a, inner.msg);
return refuse(a, why, "cannot read access metadata for {s}: {s}", .{ mp, if (err == error.Refused) inner.msg else describe(err) });
};
}
const mode: u32 = opts.mode orelse (if (metadata) |m| m.info.perm() else fileMode(path));
const tmp = try a.print("{s}/.{s}.new", .{ parent, std.fs.path.basename(path) });
defer a.free(tmp);
if (fsx.statAt(fsx.cwd, tmp, false)) |stale| {
// A privileged writer hands its temporary file to the metadata owner
// before a crash can interrupt it.
Expand All @@ -377,6 +412,7 @@ pub fn atomicWrite(a: Allocator, path: []const u8, data: []const u8, opts: Write

{
const step = try a.print("{s}.write", .{opts.step});
defer a.free(step);
if (opts.faults.fails(step)) return injected(a, why, step);
}
const fd = try sys.openat(fsx.cwd, tmp, .{ .ACCMODE = .WRONLY, .CREAT = true, .EXCL = true, .NOFOLLOW = true }, 0o600);
Expand All @@ -392,20 +428,29 @@ pub fn atomicWrite(a: Allocator, path: []const u8, data: []const u8, opts: Write
defer sys.close(fd);
try sys.writeAll(fd, data);
var inner: Why = .{};
restoreMetadata(a, fd, metadata, mode, &inner) catch |err| return refuse(a, why, "cannot preserve access metadata for {s}: {s}", .{ path, if (err == error.Refused) inner.msg else describe(err) });
restoreMetadata(a, fd, metadata, mode, &inner) catch |err| {
defer freeMsg(a, inner.msg);
return refuse(a, why, "cannot preserve access metadata for {s}: {s}", .{ path, if (err == error.Refused) inner.msg else describe(err) });
};
{
const step = try a.print("{s}.fsync", .{opts.step});
defer a.free(step);
if (opts.faults.fails(step)) return injected(a, why, step);
}
_ = try op(ops.fsync, .{fd});
}
if (metadata) |m| {
var inner: Why = .{};
const current = readMetadata(a, metadata_path.?, &inner) catch |err| return refuse(a, why, "cannot recheck access metadata for {s}: {s}", .{ metadata_path.?, if (err == error.Refused) inner.msg else describe(err) });
const current = readMetadata(a, metadata_path.?, &inner) catch |err| {
defer freeMsg(a, inner.msg);
return refuse(a, why, "cannot recheck access metadata for {s}: {s}", .{ metadata_path.?, if (err == error.Refused) inner.msg else describe(err) });
};
defer freeAcls(a, current.acls);
if (!sameMetadata(current, m)) return refuse(a, why, "access metadata of {s} changed before {s} was replaced", .{ metadata_path.?, path });
}
{
const step = try a.print("{s}.rename", .{opts.step});
defer a.free(step);
if (opts.faults.fails(step)) return injected(a, why, step);
}
try sys.renameat(fsx.cwd, tmp, fsx.cwd, path);
Expand Down Expand Up @@ -474,6 +519,55 @@ test "a step a test makes fail leaves the destination and no temporary file" {
}
}

test "atomicWrite frees what it allocates" {
const gpa = std.testing.allocator;
var cwd_buf: [sys.path_max]u8 = undefined;
const cwd = try fsx.realpath(fsx.cwd, ".", &cwd_buf);
const dir = try std.fmt.allocPrint(gpa, "{s}/.zig-cache/atomic-noleak-{d}", .{ cwd, linux.getpid() });
defer gpa.free(dir);
try makedirs(dir);
defer fsx.deleteTree(gpa, fsx.cwd, dir) catch {};
const path = try std.fmt.allocPrint(gpa, "{s}/j.json", .{dir});
defer gpa.free(path);
var why: Why = .{};
try atomicWrite(gpa, path, "one\n", .{}, &why);
try atomicWrite(gpa, path, "two\n", .{}, &why);
try std.testing.expectEqual(@as(usize, 0), why.msg.len);
}

test "an interrupted ACL read is retried" {
const gpa = std.testing.allocator;
var cwd_buf: [sys.path_max]u8 = undefined;
const cwd = try fsx.realpath(fsx.cwd, ".", &cwd_buf);
const dir = try std.fmt.allocPrint(gpa, "{s}/.zig-cache/atomic-eintr-{d}", .{ cwd, linux.getpid() });
defer gpa.free(dir);
try makedirs(dir);
defer fsx.deleteTree(gpa, fsx.cwd, dir) catch {};
const path = try std.fmt.allocPrint(gpa, "{s}/f", .{dir});
defer gpa.free(path);
const created = try sys.openat(fsx.cwd, path, .{ .ACCMODE = .WRONLY, .CREAT = true }, 0o600);
sys.close(created);
const fd = try sys.openat(fsx.cwd, path, .{}, 0);
defer sys.close(fd);
const old = ops.*;
defer ops.* = old;
const S = struct {
var left: u8 = 0;
fn get(fd_: sys.fd_t, name: [:0]const u8, buf: []u8) usize {
if (left > 0) {
left -= 1;
return @bitCast(-@as(isize, @backingInt(linux.E.INTR)));
}
return linux.fgetxattr(fd_, name.ptr, buf.ptr, buf.len);
}
};
S.left = 2;
ops.fgetxattr = S.get;
var why: Why = .{};
const acls = try readAcls(gpa, fd, &why);
defer freeAcls(gpa, acls);
}

test "the umask is read without changing it" {
const old: u32 = @intCast(linux.syscall1(.umask, 0o027));
defer _ = linux.syscall1(.umask, old);
Expand Down
Loading
Loading