Grade S3 metadata as header bytes, and let clients encode or refuse it

[tiptenbrink]

A general-purpose bucket reads metadata header bytes as ISO-8859-1. é sent as UTF-8 is stored as é, and S3 returns it as the RFC 2047 encoded word =?UTF-8?Q?caf=C3=83=C2=A9?=. The grader read header values as UTF-8, so it passed that corrupting request and errored on the byte e9, which stores é. The S3 metadata cases now pin down what a client must send and read, measured live on a general-purpose bucket and a directory bucket.

Two client policies

• Encode. Send a value that HTTP cannot carry as it is as an RFC 2047 encoded word, and decode encoded words on a read.
• Keep to what reads back. Refuse a value that the service would not give back as sent. Another spelling of the same text is not a loss, such as é as e9 or as an encoded word.

Where no request reads back as sent, the case expects a refusal.

Cases

• Outside ASCII. é passes as the byte e9 (general-purpose only), as an encoded word, or refused. 雪 passes as an encoded word or refused. Raw UTF-8 fails everywhere.
• Reads. A general-purpose bucket re-encodes values in its own spelling, so a read must decode, including a long value that S3 splits into several words. A directory bucket stores what was sent, so a read may decode or return it as stored. A malformed encoded word stays as it is.
• Whitespace and control characters. A tab inside a value passes raw. Whitespace at either end, NUL and DEL must be encoded or refused. A general-purpose bucket stores CR and LF as spaces even from an encoded word, so only a refusal passes there.
• Text that reads as an encoded word. A general-purpose bucket reads each space-separated token that starts with =? and ends with ?= as an encoded word: it decodes it, or answers 400 or 500. Such a value must be wrapped in an encoded word of its own, or refused. A value like a=?b?=c x=?UTF-8?Q?caf=C3=A9?=y holds no such token, and must be sent: a refusal fails.
• Names. A name may hold any HTTP token character. Two names that differ only in case must be refused: a general-purpose bucket merges them into one,two, and a directory bucket answers 400.
• Azure refuses non-ASCII metadata in either byte form, so its case lists both.

A case past every refusal

Each case that permits a refusal passes a client that refuses too much. So each refusal now names a neighbor just past its boundary that a client must send, and the case generator rejects a refusal without one. The audit added four cases, verified live:

• stage-part-10000 on S3 and S3 Express, past stage-part-10001.
• stage-id-64-bytes on Azure, past the 65-byte stage-id-too-long.
• list-page-size-one on Azure, past list-max-keys-zero.
• put-metadata-identifier-name (_a1) on Azure, past put-metadata-invalid-name.

Grader

• The transport hands request header values to the grader as bytes, one ISO-8859-1 character per byte.
• A wrong or failed verdict carries the case's purpose, which states the rule it grades.
• A connection that closes or times out before its first byte is no transport failure. The AWS SDK's idle pool connections made reports differ between runs.

Every request path was verified live. The AWS C++ SDK fails 8 of the metadata cases on a general-purpose bucket and 9 on a directory bucket, and passes the new neighbor cases, as the Azure C++ SDK does. The pinned borink adapter passes the Azure neighbors and reports every S3 case unsupported, so its grading shows no wrong case.

Co-Authored-By: Claude Opus 5.5 (1M context) [email protected]