Error: Windows stdio in console mode does not support writing non-UTF-8 byte sequences #8523

Activity

1. cakebaker

   added

   U - tail

   P - Windows

   U - head

   on Aug 26, 2025
2. ChrisDenton commented on Aug 26, 2025

   

   ChrisDenton

   on Aug 26, 2025

   Last edited by ChrisDenton

   Contributor

   More actions

   This is a consequence of Rust's standard library needing to convert to UTF-16 for console output and without knowing the encoding, UTF-8 is always assumed. The restriction on the encoding could be relaxed (e.g. by being lossy) but that requires changes in the standard library unless uutils wants to workaround it by manually doing console output.
3. Artoria2e5 commented on Sep 17, 2025

   

   Artoria2e5

   on Sep 17, 2025

   Last edited by Artoria2e5

   More actions

   I think lossy console output is fine as long as false positives for console detection are rare OR we can always override what it does.

   That said I’m not sure relying on the "graceful" thing like Cygwin is the best way, since the two squares look an awful lot like a byte order mark to me. It’s likely undesirable on other systems, but on Windows the BOM is/was king and we should probably do very lightweight encoding detection with it. I do hope it dies but Powershell 2.x as a preinstall isn’t going away.h
4. Skrity commented on Sep 20, 2025

   

   Skrity

   on Sep 20, 2025

   Last edited by Skrity

   More actions

   We can force windows to use UTF-8 stdin/stdout for our application. Here is an example code for that:

   fn force_utf8_on_windows() {
       use windows_sys::Win32::Foundation::TRUE;
       use windows_sys::Win32::Globalization::{CP_UTF8, IsValidCodePage};
       use windows_sys::Win32::System::Console::{SetConsoleCP, SetConsoleOutputCP};
       // SAFETY: argument passed by value and result is checked
       if unsafe { IsValidCodePage(CP_UTF8) } != TRUE {
           return;
       }
       // set stdin codepage
       // SAFETY: argument passed by value and result is checked
       if unsafe { SetConsoleCP(CP_UTF8) } != TRUE {
           return;
       };
       // set stdout codepage
       // SAFETY: argument passed by value and result is checked
       if unsafe { SetConsoleOutputCP(CP_UTF8) } != TRUE {
           return;
       };
   }

   This way Windows handles the BOM mark and (I assume) encoding by itself:

   PS C:\__git\coreutils> .\target\debug\coreutils.exe head test
   foobar

   This sets chcp for the whole console host session, so we may need to get previous codepage via GetConsoleCP and set it back on exit.

   Scratch all of that, when running in powershell it caches the codepage at the session start.
   Changes to codepoint from SetConsole*CP allows console to handle BOM, but UTF-16 content doesn't work anyway.

   Here's an example how to put raw bytes to stdout via win32 api for cygwin-like behaviour. (this fails to print non-ascii stuff properly)
   raw_stdout.txt

   TLDR: Conhost expects UTF-8 output, so the only way to correctly output to console (without squares and with proper encoding of non-ascii stuff) is to convert whatever we are given to UTF-8.

   User side workaround for profile.ps1 (forces all ps applets to write in utf-8):
   $PSDefaultParameterValues['*:Encoding'] = 'utf8'

   This one might also be needed on older versions (forces powershell/stdin/stdout to use utf-8, default in pscore6+):
   $OutputEncoding = [console]::InputEncoding = [console]::OutputEncoding = New-Object System.Text.UTF8Encoding

   Added: This also affects cat
5. KrushnXGIT commented on Nov 6, 2026

   

   KrushnXGIT

   on Nov 6, 2026

   More actions

   Still reproduces on current main (81572a4) for head, tail and cat when stdout is a Windows console. Redirecting to a file or piping works fine. rust-lang/rust#134534 would fix it in std but is still open.

   I'd like to work around it in uucore until then: a small writer that is only used when stdout is a real console (checked with GetConsoleMode, as suggested in #12236, so MSYS2/Cygwin terminals still get raw bytes) and that replaces invalid UTF-8 with U+FFFD instead of failing. Files and pipes are untouched.

   I have a working prototype for head. A file written by PowerShell 5's echo foo > file now prints as ��foo instead of erroring, and output to files and pipes stays byte-identical. If this approach is OK, I'd send head first and then tail and cat as separate PRs.

   One question: the writer holds back an unfinished UTF-8 sequence at the end of a write (at most 3 bytes) and joins it with the next one, so characters split across buffers survive. I saw @ChrisDenton prefers avoiding hidden buffers like that in #134534. Would you rather I leave the incomplete-sequence handling to std?
6. ChrisDenton commented on Nov 6, 2026

   

   ChrisDenton

   on Nov 6, 2026

   Last edited by ChrisDenton

   Contributor

   More actions

   This should now have been fixed in nightly (rust version 1.101).