Repository navigation
Determine and document the distinction between I/O safety and regular safety on unsafe stdlib APIs #434
Description
Previously: rust-lang/rust#72175, rust-lang/rust#93562
The stdlib has a bunch of unsafe APIs (e.g. FromRawFd) that are primarily unsafe because they care about "I/O safety".
Rust libraries are free to expand the scope of what they consider unsafe, but this is typically a crate-local decision. The stdlib due to its special status risks imposing this on other crates.
Basically, if a crate or project wishes to not consider I/O safety a problem (which is often necessary in more complicated I/O code! Exactly the kind of code that would wish to use these APIs) these APIs are not useful to them: it is currently unclear as to what usages of these APIs are undefined behavior vs a violation of I/O safety.
There is a valid optimization that could be performed here in the future which would be to use niches for -1 fds on Unix (etc), so there is a potential for this API being Actually UB, but that's actually something that can be checked (unlike "is a real owned FD").
It would be nice if we could settle on what is Actually UB in these APIs, and what is "a violation of I/O safety", and document it.
cc @thomcc
Activity
Manishearth commentedon Jul 26, 2023
The API that triggered this question is the following:
https://github.com/google/fleetspeak-rs/blob/d031cfc8ee07aa0670e0040d24ab99dd689e4124/crates/fleetspeak/src/lib.rs#L264-L282
(constructing a file from an fd from an env var. Reasonable to do for IPC cases, but a violation of the API requirements, currently.
bjorn3 commentedon Jul 26, 2023
Violating I/O safety can trigger UB. For example if you have some code that has a userfaultfd open and then some other code violates I/O safety to close the userfaultfd fd (eg by creating a
Filefrom it which is then dropped) will unmap the memory backed by the userfaultfd, resulting in UB on the next access.ChrisDenton commentedon Jul 26, 2023
I think "I/O safety" is a red herring here. It has nothing to do with the reasons that
from_raw_fdis unsafe (see above). It has been unsafe a long long time before the term "I/O safety" was coined.Manishearth commentedon Jul 26, 2023
@bjorn3 Okay, but isn't that analogous to the
/proc/mem/selfissue? There are lots of weird things you can do with specific kinds of fds that are UB in Rust already.the8472 commentedon Jul 26, 2023
Not quite, opening
/proc/mem/selfis reaching outside Rust.Using
from_raw_fdis passing something around inside Rust, into a type that has preconditions or from a type that says "you can use this Fd only for specific operations".You can make up any RawFd you want, they're inert, like invalid pointers. To actually use them you need to access them, so converting a RawFd to some type that could cause any kinds of effects depending on what that Fd is. It might even be one that's already open and in use for memory-mapped stuff.
Manishearth commentedon Jul 26, 2023
Right but ... I can do that by opening the memory-mapped file directly with safe File APIs, too. That's kinda what I'm getting at: we do not seem to have a principled approach to this, it seems strange to single this out as potentially creating undefined behavior.
The annoying thing here is not just that this method has preconditions, it has preconditions that cannot be checked.
the8472 commentedon Jul 26, 2023
You can hide your memory-mapped files, unlink them (on unix) or make the directory non-traversible if you're paranoid. But again, it's basically reaching out to a system that has insufficient sandboxing. The OS offers you footguns, we can't do anything about that without shutting down all IO. Inside Rust we do care.
the8472 commentedon Jul 26, 2023
You kind of can if you're
main. No other threads are running yet, only FDs 0,1,2 should be open. So anything else was explicitly passed from the parent and you can take ownership of it. Once you have a raw fd you can probe whether it is valid then use it.That still leaves the uncheckable condition that nothing else on the system is using it, but that does fall under the "insufficiently paranoid OS" problem.
thomcc commentedon Jul 26, 2023
Yeah, I was kind of opposed to codifying I/O safety as a type of safety for these reasons. It is subtle, as has been discussed here, because there is real UB that can occur if I/O safety is violated.
I agree that it's not great that we have APIs with preconditions that cannot be checked (in practice anyway -- being early main is not an option most of the time).
Manishearth commentedon Jul 26, 2023
And then you need to worry about other crates doing stuff; which you can control by not using other crates but it's still an action-at-a-distance thing.
(And yeah, even if you whittle this down to "you only call trusted APIs before doing this", your dependencies are free to link in life-before-main C code)
I think now that this can of worms has been opened we need to either fully specify it or decide it is an additional library invariant that callers are not required to follow; the status quo does not make much sense.
the8472 commentedon Jul 26, 2023
Well sure, but C is never safe. We can't do anything about that. C programs would also break if other threads start mucking around with random file descriptors they don't semantically own.
thomcc commentedon Jul 26, 2023
Yeah, I think I lean towards the "decide it is an additional library invariant that callers are not required to follow", but with a long list of caveats, which definitely do need to be worked though.
I think it's worth CCing @sunfishcode, who was the author of the RFC, to see if they have a more nuanced take here, I suspect that they will.
ChrisDenton commentedon Jul 26, 2023
Note that to the extent that "I/O safety" is documented, it's documented here: https://doc.rust-lang.org/std/os/unix/io/index.html
One problem with
file_from_env_varis that it allows creating twostd::fs::Files from the same fd, with both expecting to have exclusive access (e.g. they close on drop).Manishearth commentedon Jul 26, 2023
That's still a library invariant though, not UB
the8472 commentedon Jul 26, 2023
AsRawFdis safe, aMmaptype can implement it and it would be doing nothing wrong. Now you have aRawFd. This is analogous tovec.as_mut_ptr(). Both of them are inert. The UB only happens when you pass it to something else that does things with them they shouldn't. That's why that isunsafe.We could have put the
unsafeon the extraction instead. But neither did we do that for raw pointers.The only difference is that there are some Fds which are harmless to use with some syscalls. But I guess you could do the same with pointers, like calling madvise on them or something.
37 remaining items
thomcc commentedon Jul 29, 2023
Yes, we never close the file descriptor. More generally, I like this formulation, but yeah:
This is true, and it's hard to imagine
set_var, even once we make itunsafe(or more likely, deprecate it in favor of an unsafe version), having a precondition that "also you can't set vars in a way that would cause IO unsafety elsewhere in the program" -- it's far to vague.This doesn't concern me because the person spawning the program has an arbitrary number of ways they can cause UB when spawning the program.
CAD97 commentedon Jul 30, 2023
I don't think it needs to be that specific, though. Instead, being more general gets the point across perfectly well; something along the lines of "the environment acts as an open set of global variables in a flat namespace; setting variables used elsewhere in the code can lead to arbitrary misbehavior of that code, potentially even including unsound behavior if that code makes unsafe assumptions which no longer hold." It's not just about IO safety, but also e.g.
var("…")?.parse().unwrap_unchecked(), as unwise as that would be to trust.In a way, w.r.t. safety, setting environment variables is no different from choosing a random bytes in the global variable section of memory and writing to them. The one way that it's actually different is that there's a greater expectation of trustable isolation between standard globals than there is of the
Map<String, String>environment variables.Any scheme which passes fds via envvar is putting a global proof burden that no code abuses the relevant envvar names, as the compiler doesn't offer any assistance in isolation there. It's equivalent to name mangling and name collisions of
#[no_mangle], and that's at least partially outside the Rust model currently.RalfJung commentedon Jul 31, 2023
@Manishearth
Yes, it is intended to constrain the ecosystem to not do
loop { close(rand()); }.Note that the RFC didn't introduce any new language UB. The operational semantics is unaffected. Whether a concrete program has UB or not is unaffected.
It introduced a new reasoning principle, and it changed/clarified which APIs are sound. It introduced new terminology that libraries can use to define whether something is or is not library UB. (That's probably what you meant but I feel precise terminology can help us tease out the core questions here.)
That probably means this decision should have involved T-lang, not just T-libs-api. And yes if T-lang were to do this arbitrarily, that would be a backwards compatibility hazard. Was this particular RFC a breaking change? I don't know. Was
loop { close(rand()); }sound before the RFC? The ownership semantics were not clarified, butfrom_raw_fdwasunsafe, so arguably the intent was there to make this loop unsound -- it just wasn't spelled out properly, and the RFC fixed that.Do they feel that? I'd like to learn more about that.
nix predates the RFC, and the nix issue nix-rust/nix#1750 so far looks to me like it is totally possible to make nix IO-safe. I don't see nix as having actively decided against IO safety, they just didn't have
OwnedFdwhen the crate was created.But yes if
loop { close(rand()); }must truly be considered sound, then I/O safety needs to be fundamentally revised.It's not great, yeah. We could say that, as @CAD97 suggests, changing any variable requires knowing all invariants associated with that variable and proving they are being maintained -- but then code still has to argue why it can know that a given variable doesn't have an invariant associated with it in another part of the program. It's not great.
We could also give up on what I called "strong ownership" of an FD, and say that read/write/etc are free game on arbitrary FDs, and only
close/dup2are restricted. Basically it would be sound to convert any integer to aBorrowedFd<'static>.Manishearth commentedon Aug 12, 2023
yeah I think that's part of the source of my confusion
While digging for this i did seem to find discussions amongst the nix maintainers that seemed to indicate otherwise but I can't find them now, I could just have misread.
I do think it is worth talking to the tokio and other async library maintainers about this though. I don't think I saw much from them on that RFC.
RalfJung commentedon Aug 13, 2023
If major I/O stakeholders feel like I/O safety (even the weaker form that is just about not closing other file descriptors) gets in their way, that would absolutely be a good reason to re-evaluate this RFC, yes. I would expect that to take the form of a new RFC that describes the usecases blocked by the I/O safety RFC and suggests to abolish the concept and describes how that impacts standard library APIs.
Meanwhile I don't see a ton for us on the UCG side to do here. I/O safety is clearly an intended reasoning principle right now, and some hints of it go back all the way to Rust 1.0 (with
from_raw_fdbeing unsafe). The questions here are not on the operational side (so this is outside of t-opsem's realm).Is this a documentation issue? I don't see any mention of I/O safety in the standard library docs, so that seems like something that should definitely be improved.
RalfJung commentedon Aug 13, 2023
rust-lang/rust#114780 should fix the documentation issue.
Manishearth commentedon Aug 13, 2023
Yes, it's mostly a documentation issue.
sunfishcode commentedon Aug 13, 2023
I also made a list of things I plan to improve about the documentation, based on the discussion in this thread.
Auto merge of rust-lang#114780 - RalfJung:io-safety, r=Amanieu
RalfJung commentedon Sep 22, 2023
The most pressing docs issue should hopefully be fixed by rust-lang/rust#114780, though if @sunfishcode gets around to improving them further based on their list of course that would be even better. :) Still, the issue here can be closed now I think.
RalfJung commentedon Sep 22, 2023
For the question of how
cc::Build::newcan be safe, I have opened rust-lang/rust#116059.Auto merge of #114780 - RalfJung:io-safety, r=Amanieu
Auto merge of #114780 - RalfJung:io-safety, r=Amanieu
Auto merge of #114780 - RalfJung:io-safety, r=Amanieu