This package provides dependency-free TypeScript implementations of the native text encoding classes, designed as ponyfills for the following standard APIs.
| API | Description | Notes |
|---|---|---|
TextEncoder |
Encodes strings into UTF-8 byte sequences. | -- |
TextDecoder |
Decodes UTF-8 byte sequences into strings. | Currently only supports UTF-8. |
TextEncoderStream |
Full-duplex text-to-bytes encoding stream. | Requires TransformStream API. |
TextDecoderStream |
Full-duplex bytes-to-text decoding stream. | Requires TransformStream API. |
deno add jsr:@nick/utf8
npx jsr add @nick/utf8
bunx jsr add @nick/utf8
pnpm dlx jsr add @nick/utf8
yarn add @nick/utf8Mirrored on NPM as @nberlette/utf8:
npm install @nberlette/utf8import { TextDecoder, TextEncoder } from "@nick/utf8";
const encoder = new TextEncoder();
const decoder = new TextDecoder();
const encoded = encoder.encode("Hello, World!");
const decoded = decoder.decode(encoded);
console.log(decoded); // Hello, World!The TextEncoder class encodes strings into UTF-8 byte sequences.
Creates a new TextEncoder instance.
new TextEncoder();encode 📚 MDN
Encodes the given input into a new Uint8Array.
encode(input: string): Uint8Array;input: The string to encode.
A new Uint8Array containing the encoded bytes.
import { TextEncoder } from "@nick/utf8";
const encoder = new TextEncoder();
const encoded = encoder.encode("Hello, World!");
console.log(encoded); // Uint8Array([...])encodeInto 📚 MDN
Encodes an input string into an existing Uint8Array output buffer.
encodeInto(input: string, output: Uint8Array): TextEncoderEncodeIntoResult;input: The string to encode.output: The output buffer to write the encoded bytes into.
A TextEncoderEncodeIntoResult object, containing the number of characters
read and number of bytes written.
Note
Refer to the MDN documentation for more information.
import { TextEncoder } from "@nick/utf8";
const encoder = new TextEncoder();
const output = new Uint8Array(16);
const input = "Hello, my name is Nick!"; // 23 characters
const { read, written } = encoder.encodeInto(input, output);The TextDecoder class decodes encoded byte sequences into strings.
Creates a new TextDecoder instance with the given encoding and options.
new TextDecoder(encoding?: string, options?: TextDecoderOptions)encoding: The encoding to use. Currently, only"utf-8"is supported.options: An optionalTextDecoderOptionsobject.
decode 📚 MDN
Decodes UTF-8 bytes from the given BufferSource into a string.
decode(input?: BufferSource, options?: TextDecodeOptions): string;input: TheBufferSourcecontaining the UTF-8 bytes to decode. If omitted, defaults to an emptyUint8Array.options: An optionalTextDecodeOptionsobject.
The decoded bytes as a string.
import { TextDecoder } from "@nick/utf8";
const decoder = new TextDecoder();
const encoded = new Uint8Array([72, 101, 108, 108, 111, 33]);
const decoded = decoder.decode(encoded);
console.log(decoded); // Hello!Provides a full-duplex decoding stream, allowing UTF-8 bytes to be written to its writable side, and the decoded strings to be read from its readable side.
Creates a new TextDecoderStream instance with an optional encoding standard
and options to configure the underlying TextDecoder instance.
new TextDecoderStream(encoding?: string, options?: TextDecoderOptions)This class supports the same arguments as the TextDecoder API, which it uses
under the hood to perform the decoding. The fatal and ignoreBOM options,
just like in the TextDecoder class, go on to become read-only properties of
the same name on the new TextDecoderStream instance.
encoding: The encoding to use. Currently, only"utf-8"is supported.options: An optionalTextDecoderOptionsobject.
encoding: string 📚 MDN
The encoding used by the underlying decoder. Represents the value passed to the
constructor as the encoding parameter.
fatal: boolean 📚 MDN
Whether to throw an error if the input contains invalid bytes. Represents the
value passed to the constructor as the fatal option.
ignoreBOM: boolean 📚 MDN
Whether to ignore a leading BOM character in the input. Represents the value
passed to the constructor as the ignoreBOM option.
readable: ReadableStream<string> 📚 MDN
The output side of the duplex stream, from which decoded strings are read.
writable: WritableStream<BufferSource> 📚 MDN
The input side of the duplex, into which BufferSource objects are written.
Just like the TextDecoder API, the writable stream supports
any type of BufferSource object (an ArrayBuffer or a view of one) as input.
Important
TextDecoderStream requires runtime support for TransformStream.
Provides a full-duplex encoding stream, allowing strings to be written to its writable side, and the encoded bytes to be read from its readable side.
Creates a new TextEncoderStream instance with an optional encoding standard
and options to configure the underlying TextEncoder instance.
new TextEncoderStream();encoding: string 📚 MDN
The encoding used by the underlying encoder. Represents the value passed to the
constructor as the encoding parameter.
readable: ReadableStream<Uint8Array> 📚 MDN
The output side of the duplex stream, from which encoded chunks are read.
writable: WritableStream<string> 📚 MDN
The input side of the duplex, into which strings are written.
Important
TextEncoderStream requires runtime support for TransformStream.
Options that can be passed to TextDecoder.decode.
interface TextDecodeOptions {
stream?: boolean;
}Boolean flag that indicates the call to decode is part of a stream, which
affects the behavior of the decoder.
When set to true, incomplete byte sequences will be buffered internally and
their errors will be suppressed, allowing the stream to continue processing. The
next call to decode will resume decoding from the buffered bytes.
Tip
It is important to flush any buffered bytes from the TextDecoder internal
state once the stream is complete. This can be done by calling decode with
no arguments, as shown in the example below.
import { TextDecoder } from "@nick/utf8";
const decoder = new TextDecoder();
const stream = new Uint8Array([0xF0, 0x9F, 0x98, 0x8A]);
let result = "";
for (const chunk of stream) {
result += decoder.decode(chunk, { stream: true });
}
// Flush any remaining bytes from the internal state.
result += decoder.decode();Options that can be passed to the TextDecoder and
TextDecoderStream class constructors to configure the
behavior of the decoder instance.
interface TextDecoderOptions {
fatal?: boolean;
ignoreBOM?: boolean;
}Boolean flag that indicates whether to throw an error if the input contains
invalid bytes. The value passed to this option will be exposed as the fatal
property on the decoder instance (read-only).
Default: false
Instructs the TextDecoder to ignore a leading BOM character in the input. The
value passed to this option will be exposed as the ignoreBOM property on the
decoder instance (read-only).
Default: false
TextEncoderEncodeIntoResult 📚 MDN
The object returned by TextEncoder.encodeInto,
containing the number of characters read from the input string and the number of
bytes written to the output buffer.
interface TextEncoderEncodeIntoResult {
read: number;
written: number;
}The number of characters read from the input string.
The number of bytes written to the output buffer.
This package is not a polyfill, but rather a [ponyfill] that doesn't overwrite the native implementation. It provides a drop-in replacement for the native APIs, allowing you to use them in environments that don't support them.
That being said, some users and use cases may indeed require a side-effecting
polyfill that patches the native APIs. For those cases, you can import the
./shim module, which will gracefully patch the native APIs as needed.
If the APIs already exist on the global scope, no changes will be made. If the
TransformStream API is not available, the streaming APIs will not be patched.
import "@nick/utf8/shim";
// The native APIs are now patched if needed.
console.log(new TextEncoder().encode("Hello, World!"));JSR, the primary distribution channel for this package, does not support type
augmentation on the global scope. As a result, this package cannot provide an
"all-in-one" polyfill experience from a single import of the ./shim module.
If you need type definitions for the patched APIs, or if for some reason you're
only looking for type definitions alone, the @nick/utf8/shim.d.ts module has
ambient declarations for all of the APIs provided by this package.
import type {} from "@nick/utf8/shim.d.ts";/// <reference types="@nick/utf8/shim.d.ts" />Deno users will need to include the
.d.tsextension as seen above. Users of TypeScript in Node.js / Bun environments might be able to omit that in their triple-slash references, but I'm not 100% certain in that regard.
This package is compatible with all modern browsers, Deno, Node.js, Bun, and
Cloudflare Workers. The streaming APIs require support for the
TransformStream interface, which is available in all of the previously
mentioned environments.
If you're running in an environment that doesn't support the
TransformStreaminterface, you can find a full-featured polyfill for it in core-js.
The implementations in this package are highly optimized for performance. They are written in a way that minimizes the number of allocations and copies, and they take advantage of the fastest available APIs in the environment.
Take a look at the benchmarks below for a performance sample comparing this package side-by-side with the native APIs in Deno v2.1.2.
While benchmarks are far from a definitive measure of performance, they're a good indicator of general performance characteristics. The results may vary depending on the environment, machine, workload, and other factors.
View Benchmarks: @nick/utf8 vs. Deno v2.1.2
> deno bench -A --no-check
CPU | Apple M1 Pro
Runtime | Deno 2.1.2+7c03677 (x86_64-apple-darwin)
benchmark time/iter (avg) iter/s (min … max) p75 p99 p995
---------------------------- ----------------------------- --------------------- --------------------------
Native TextDecoder 1.8 µs 543,200 ( 1.4 µs … 13.7 ms) 1.6 µs 2.9 µs 3.6 µs
Ponyfill TextDecoder 769.0 ns 1,300,000 (583.0 ns … 1.5 ms) 708.0 ns 1.4 µs 1.8 µs
summary
Ponyfill TextDecoder
2.39x faster than Native TextDecoder
Native TextDecoderStream 24.6 µs 40,730 ( 13.3 µs … 5.6 ms) 18.9 µs 219.8 µs 574.2 µs
Ponyfill TextDecoderStream 5.4 µs 185,700 ( 4.5 µs … 1.5 ms) 5.1 µs 7.4 µs 9.2 µs
summary
Ponyfill TextDecoderStream
4.56x faster than Native TextDecoderStream
Native TextEncoder 1.1 µs 926,900 (630.1 ns … 1.6 µs) 1.3 µs 1.6 µs 1.6 µs
Ponyfill TextEncoder 1.1 µs 870,300 (708.0 ns … 6.9 ms) 1.0 µs 3.6 µs 6.1 µs
summary
Ponyfill TextEncoder
1.06x slower than Native TextEncoder
Native TextEncoderStream 8.0 µs 124,500 ( 4.9 µs … 3.6 ms) 6.0 µs 19.9 µs 44.6 µs
Ponyfill TextEncoderStream 5.8 µs 171,100 ( 4.7 µs … 1.1 ms) 5.4 µs 9.2 µs 14.9 µs
summary
Ponyfill TextEncoderStream
1.37x faster than Native TextEncoderStream