Reject HelloRetryRequest lacking supported_versions before downgrade

[embhorn]

Description

A downgrade capable TLS 1.3 client (for example wolfSSLv23_client_method) detects the HelloRetryRequest sentinel Random in DoTls13ServerHello, but a missing supported_versions extension forced a downgrade to TLS 1.2 that delegated to ServerHello. That path reparses the extensions as a plain server_hello and loses the HelloRetryRequest context, so a recognized but not permitted extension such as server_name produced an unsupported_extension alert (110) instead of the illegal_parameter alert (47) required by RFC 8446 4.2.

A HelloRetryRequest is a TLS 1.3 message whose sentinel Random is reserved (RFC 8446 4.1.3) and which MUST carry supported_versions (4.1.4/4.2.1). Reject a message bearing the sentinel without supported_versions during version negotiation, before the downgrade discards the HelloRetryRequest context. The case with other extensions present returns EXT_NOT_ALLOWED and the case with no extensions field returns INVALID_PARAMETER; both map to the illegal_parameter alert.

Fixes #10792

Testing

Add test_tls13_hrr_recognized_ext_downgrade covering both malformed HelloRetryRequest shapes through the memio harness.

How did you test?

Checklist

• added tests
• updated/added doxygen
• updated appropriate READMEs
• Updated manual and documentation

[Copilot]

Pull request overview

This PR fixes TLS 1.3 downgrade-capable client handling of malformed HelloRetryRequest (HRR) messages that carry the HRR sentinel Random but omit the mandatory supported_versions extension, ensuring the client rejects the message before the TLS 1.2 downgrade path loses the HRR context and emits the wrong alert.

Changes:

• Add early rejection in DoTls13ServerHello() when the HRR sentinel is present but supported_versions is missing, preventing TLS 1.2 downgrade from misclassifying the message.
• Ensure the resulting error maps to the RFC 8446-required illegal_parameter alert rather than unsupported_extension in the downgrade-capable client scenario.
• Add a memio regression test covering malformed HRR shapes (with extensions and with no extensions field).

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
tests/api/test_tls13.h	Registers the new TLS 1.3 regression test in the API test list.
tests/api/test_tls13.c	Adds a memio-based regression test exercising malformed HRR downgrade behavior and expected alerts/errors.
src/tls13.c	Adds early HRR-without-supported_versions rejection in the TLS 1.3 ServerHello/HRR parsing path before downgrade occurs.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

MemBrowse Memory Report

gcc-arm-cortex-m4-openssl-compat

• FLASH: .text +64 B (+0.0%, 771,212 B / 1,048,576 B, total: 74% used)

gcc-arm-cortex-m4-pq

• FLASH: .text +64 B (+0.0%, 279,460 B / 1,048,576 B, total: 27% used)

gcc-arm-cortex-m4-tls13

• FLASH: .text +64 B (+0.0%, 236,146 B / 262,144 B, total: 90% used)

gcc-arm-cortex-m7-pq

• FLASH: .text +64 B (+0.0%, 280,036 B / 1,048,576 B, total: 27% used)
  No memory changes detected for:
• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m3
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-dtls13
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-rsa-only
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls12
• gcc-arm-cortex-m7
• gcc-arm-cortex-m7-tls13
• linuxkm-pie
• linuxkm-standard
• stm32-sim-stm32h753

[Frauschi]

Jenkins retest this please

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10892

Scan targets checked: wolfcrypt-rs-bugs, wolfssl-bugs, wolfssl-src

Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Findings are non-blocking.