From 95d3a061f22b2e509294591977444e418929701e Mon Sep 17 00:00:00 2001 From: Kaushik Agrawal Date: Mon, 19 Oct 2026 20:59:15 -0400 Subject: [PATCH 1/3] docs(pmtu): document adaptive path MTU for long-lived sessions (I95-60371) - Add comprehensive Adaptive Path MTU coverage for SSR 7.2.0 (I95-60371) - Unify narrative around Client -> Spoke SSR -> Underlay -> Hub SSR -> Server topology - Detail in-flight flow MTU updates, SVR overhead subtraction, and client notification - Contrast TCP MSS resizing vs UDP/DF=0 fabric fragmentation fallback - Scrub internal terminology in favor of RFC 1191 standard phrasing - Verify Mermaid sequence diagrams compile cleanly to SVG Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- docs/config_pmtu.md | 252 ++++++++++++++++++++++++++++++++++++-------- 1 file changed, 208 insertions(+), 44 deletions(-) diff --git a/docs/config_pmtu.md b/docs/config_pmtu.md index 8b259b105d9..608d4f43efc 100644 --- a/docs/config_pmtu.md +++ b/docs/config_pmtu.md @@ -3,90 +3,229 @@ title: Path MTU Discovery sidebar_label: Path MTU Discovery --- -The SSR performs Path MTU Discovery (PMTUD) along the overlay to determine the correct maximum transmission unit (MTU) for each peer path. By default, this test runs every ten minutes. If a change in the underlay reduces the available path MTU between two SSRs, the new value is not discovered until the next PMTUD cycle. Additionally, existing sessions continue to use the previous MTU value until the next time those sessions are rebuilt. +## Periodic Path MTU Discovery + +The SSR performs Path MTU Discovery (PMTUD) along the overlay to determine the maximum transmission unit (MTU) for each peer path. Peering SSR routers send packets of various sizes to discover the path MTU. By default, this test runs every ten minutes to account for path changes between peers. | Direction | Port/Proto | Client Payload (bytes) | Server Payload (bytes) | Default Interval | Notes | | --- | --- | --- | --- | --- | --- | | bidirectional | 1280/UDP | 2945 | 90 | 600s | Interval is configurable within `path-mtu-discovery/interval`, or disabled `path-mtu-discovery/enabled`. | -Peering SSR routers will perform path MTU discovery on each peer path between each other. This test is run every ten (10) minutes by default, to adjust in the event of path changes between peering devices. During the test, SSR routers send packets of various sizes to discover the MTU of the path. However, in some government deployments the use of MTU discovery is not possible. +In deployments where ICMP Destination Unreachable: Fragmentation Needed and DF Set response messages are not generated or are suppressed, the SSR uses probe non-responses to infer the MTU. Three consecutive non-responses are considered equivalent to ICMP responses for the purposes of driving the discovery algorithm. See [RFC 1191](https://www.rfc-editor.org/rfc/rfc1191) for ICMP-based Path MTU Discovery. + +Use `show peers` to view the discovered MTU. + +Prior to SSR 7.2.0, a reduced underlay MTU was not discovered until the next PMTUD cycle. Existing sessions continued to use the previous MTU until they were rebuilt, even after periodic PMTUD discovered the change. + +Devices in the underlay report an ICMP Destination Unreachable: Fragmentation Needed and DF Set message (Type 3, Code 4) when a packet with DF set exceeds the next-hop MTU. Prior to SSR 7.2.0, the SSR forwarded these messages to the endpoint without acting on the reported MTU. The affected session retained its previous path MTU. -In order to accommodate these deployments where “ICMP Destination Unreachable - Fragmentation Needed” response messages are not generated (RFC1911 is not followed), three successive non-responses are considered equivalent to ICMP responses for the purposes of driving the algorithm with an inferred MTU. +SSR 7.2.0 introduces two complementary enhancements under Adaptive Path MTU (I95-60371): -The discovered MTU is viewable in the output of `show peers`. +1. **Underlay ICMP Reaction** — The ingress SSR intercepts the ICMP Fragmentation Needed message, updates the affected flow, and generates a corrected ICMP Fragmentation Needed message toward the original sender. The corrected MTU accounts for SVR overhead. +2. **Session Refresh** — The flow that carried the dropped packet is updated in place to use the MTU reported by the underlay. The session does not need to be rebuilt. -Devices in the underlay may report an ICMP Destination Unreachable / Fragmentation Needed (type 3, code 4) error, referred to here as a _TooBig_ packet, to indicate they could not forward a packet due to an undersized MTU. Prior to SSR 7.2.0, these messages were forwarded to the correct endpoint, but the SSR itself did not act on the MTU value contained in the message, leaving existing sessions with an incorrect PMTU. +See [Adaptive Path MTU for Long-Lived Sessions](#adaptive-path-mtu-for-long-lived-sessions) for a hub-and-spoke example. -SSR 7.2.0 introduces two complementary enhancements to address the gaps mentioned above: +For TCP flows, configure `enforced-mss automatic` on the egress `network-interface`. This setting, known as MSS clamping, adjusts the advertised TCP maximum segment size (MSS) to reduce fragmentation. See [Configuring `enforced-mss`](#configuring-enforced-mss-recommended-for-tcp). -1. **Underlay ICMP reaction** — When the SSR receives a TooBig packet from the underlay, it updates the affected overlay flow and generates a corrected TooBig packet toward the original packet sender, allowing the sender to adjust its segment size. +## Adaptive Path MTU for Long-Lived Sessions -2. **Session Refresh** - The flow which was traversed to trigger the TooBig response from the underlay is now updated to use the MTU reported in the TooBig packet. +Database replication, backups, and tunnels can maintain sessions across an underlay MTU change. If a route change or additional encapsulation reduces the path MTU, packets sized for the previous MTU no longer fit. Adaptive Path MTU allows the SSR to correct an established flow when the underlay reports the change, without waiting for periodic discovery or a session rebuild. -For TCP flows, setting `enforced-mss automatic` on the egress `network-interface` is the recommended complement to these features. It adjusts the TCP MSS advertised at the interface boundary to avoid fragmentation in the first place. See [Configuration](#configuration) for details. +### Example Topology and Initial State -## How The SSR Reacts to Underlay ICMP TooBig Messages +The following example uses one client-to-server session through a hub-and-spoke overlay: -The following sequence illustrates what happens when the underlay path MTU changes after a session is already established. +- **Client** — Sends traffic from the spoke LAN. +- **Spoke SSR R1** — Encapsulates client traffic into SVR at ingress. +- **Underlay Device R2** — Forwards traffic between the spoke and hub. +- **Hub SSR** — Reassembles and decapsulates SVR traffic at egress. +- **Server** — Receives traffic on the hub LAN. -### Initial State +Initially, every hop supports a 1500-byte MTU. Both SSRs use a path MTU of 1500 for the session's fabric flows. ```mermaid sequenceDiagram participant Client + participant Spoke as Spoke SSR (R1) + participant R2 as Underlay Device (R2) participant Hub as Hub SSR - participant R1 as Spoke SSR participant Server - Client->>Hub: Data (MTU 1500) - Hub->>R1: SVR overlay packet (MTU 1500) - R1->>Server: Data (MTU 1500) - Note over Hub,R1: Underlay MTU = 1500. Session PMTU on both SSRs = 1500. + Note over Client,Server: All links support MTU 1500 + Note over Spoke,Hub: Fabric flow MTU = 1500 in both directions + Client->>Spoke: Application data (1428-byte IP packet) + Spoke->>R2: SVR-encapsulated fabric packet (1468 bytes) + R2->>Hub: Forward 1468-byte fabric packet + Hub->>Server: Decapsulated original packet (1428 bytes) ``` -The client and server are communicating through two peering SSRs over the overlay. The PMTU is consistent at 1500 across all hops, and both SSRs have applied an MTU of 1500 to the forward flow actions for this session. +The client sends 1428-byte IP packets. The spoke SSR encapsulates them into SVR. With SVR encapsulation overhead added, the corresponding fabric packet is 1468 bytes, which fits within the initial underlay MTU. R2 forwards it to the hub SSR, which decapsulates it and delivers the original packet to the server. + +### The Solution in Short + +When the spoke SSR receives an ICMP Fragmentation Needed message from R2, it: + +- Updates the affected flow MTU in place, without rebuilding the session. +- Subtracts SVR overhead from the reported MTU and sends a corrected ICMP Fragmentation Needed message to the client. +- Fabric-fragments subsequent oversized packets if the client does not reduce its packet size. The hub SSR reassembles these packets before delivery. + +The flow update and client notification provide two ways to handle the reduced MTU: the sender can reduce its packet size, or the SSRs can fragment and reassemble traffic within the fabric. -### Underlay MTU Drops — First TooBig Received by Hub +:::info Introduced In SSR 7.2.0 +Adaptive Path MTU applies to overlay (SVR/fabric) sessions between peering SSR routers and requires no configuration. It depends on receiving an ICMP Fragmentation Needed message from the underlay. +::: + +| Session State | Behavior | +| --- | --- | +| Existing session | The SSR updates the affected flow in place when it processes an ICMP Fragmentation Needed message. | +| New session | The SSR uses the peer path MTU available at session setup. After periodic PMTUD discovers the reduced MTU, new sessions use that value. A session established before discovery can still require adaptive correction. | + +### Detailed Walkthrough + +R2's outgoing link MTU toward the hub drops from 1500 to 1000 bytes while the session remains established. The following sequence shows how the spoke SSR processes the next oversized packet and the resulting ICMP response. ```mermaid sequenceDiagram participant Client + participant Spoke as Spoke SSR (R1) + participant R2 as Underlay Device (R2) participant Hub as Hub SSR - participant R2 as Underlay Device - participant R1 as Spoke SSR participant Server - Note over R2,R1: Underlay MTU between Hub and R2 drops to 1300 - Hub->>R2: SVR packet hub-WAN to spoke-WAN (over 1300 bytes) - R2-->>Hub: ICMP TooBig type 3 code 4, reported MTU = 1300 - Note over Hub: DivertedPacketHandler finds reverse flow. Updates Hub-to-Spoke PMTU to 1300. - Hub-->>Server: New TooBig toward Server - Note over Server: Server adjusts MSS if TCP-capable + Note over R2: Underlay MTU at R2 drops from 1500 to 1000 + Client->>Spoke: Application data (1428-byte IP packet) + Spoke->>R2: SVR-encapsulated fabric packet (1468 bytes) + Note over R2: 1468 bytes exceeds 1000-byte MTU. Packet dropped. + R2-->>Spoke: ICMP Fragmentation Needed (Type 3, Code 4), next-hop MTU = 1000 + Note over Spoke: Identify flow from quoted packet header + Note over Spoke: Update forward-flow MTU in place to 1000 + Note over Spoke: Subtract 48-byte SVR overhead. 1000 minus 48 = 952. + Spoke-->>Client: Corrected ICMP Fragmentation Needed (576 bytes), next-hop MTU = 952 ``` -When R2 (an underlay device) cannot forward an oversized packet, it sends a TooBig packet to the Hub's WAN interface. The SSR processes this message and does the following: +#### Step 1: R2 Reports the Reduced MTU + +The client continues sending 1428-byte IP packets. The spoke SSR encapsulates the next packet into a 1468-byte fabric packet with DF set. R2 cannot forward it over the 1000-byte link, so it drops the packet. + +R2 returns an ICMP Destination Unreachable: Fragmentation Needed and DF Set message (Type 3, Code 4) to the spoke SSR's WAN interface. The message reports a next-hop MTU of 1000 and quotes the dropped packet's IP header. + +#### Step 2: The Spoke SSR Updates the Flow + +The spoke SSR intercepts the ICMP message and reads the inner, quoted packet header to identify the affected SVR session. It locates the corresponding flow and updates the spoke-to-hub forward-flow MTU to 1000 in place. -1. It extracts the encapsulated IP header from the TooBig body to identify the affected overlay session. -2. It finds the reverse flow using that header and updates the Hub → Spoke forward flow's PMTU to the value reported by the underlay. -3. It constructs a new TooBig packet directed toward the original packet sender (the Server), so the server's TCP stack can reduce its MSS. +The `flow-mtu-updates` counter in `show stats icmp` increments when the flow MTU changes. See [Checking Adaptive Path MTU Activity](#checking-adaptive-path-mtu-activity). -:::note -The MTU value propagated in the new TooBig packet reflects the underlay-reported value. On paths with encryption, HMAC, FEC, or BFD tunneling overhead, the effective usable MTU will be lower than the raw underlay value. The SSR accounts for these overheads when setting the MTU on forward flow actions. +#### Step 3: The Spoke SSR Notifies the Client + +The reported MTU applies to encapsulated traffic. The spoke SSR must account for SVR overhead before notifying the client; otherwise, a client packet sized to 1000 bytes would exceed the underlay MTU after encapsulation. + +In this example, the SSR subtracts a 48-byte SVR overhead allowance and sends a corrected ICMP Fragmentation Needed message to the client with a next-hop MTU of **952 bytes** (1000 − 48). Note that the wire length of the ICMP packet itself (576 bytes in this example) is separate from the MTU value it reports inside its payload. + +:::info +The SVR overhead allowance used for MTU calculation is not necessarily a fixed difference between an individual client packet and its resulting fabric packet. Encryption, HMAC, forward error correction (FEC), and BFD tunneling can affect the usable MTU. Rely on the corrected MTU calculated and reported by the SSR rather than estimating from a single observed packet. ::: +### Scenarios: Post-Correction Traffic Handling + +After the flow update, packet handling depends on whether the client reduces its packet size. + +#### Scenario 1: TCP Traffic — Sender Adjusts MSS + +In this scenario, the TCP client processes the corrected ICMP message and lowers its path MTU to 952 bytes. Its effective send MSS becomes 912 bytes: 952 minus 40 bytes for IPv4 and TCP headers, assuming no options. This changes the segment size used for the established connection; it does not renegotiate the MSS option exchanged during connection setup. + +The resulting packets fit within the 1000-byte underlay MTU after SVR encapsulation. + +```mermaid +sequenceDiagram + participant Client as Client (TCP) + participant Spoke as Spoke SSR (R1) + participant R2 as Underlay Device (R2) + participant Hub as Hub SSR + participant Server + + Note over Client: Corrected ICMP received. Path MTU = 952. + Client->>Client: Reduce effective send MSS to 912 bytes + Client->>Spoke: TCP segment (IP packet of 952 bytes or less) + Spoke->>R2: SVR-encapsulated fabric packet (1000 bytes or less) + Note over R2: Fits within 1000-byte MTU. No fragmentation. + R2->>Hub: Forwarded whole + Hub->>Server: Decapsulated original segment, unfragmented +``` + +No fabric fragmentation or reassembly is required. Configure [`enforced-mss automatic`](#configuring-enforced-mss-recommended-for-tcp) to clamp the advertised MSS during TCP connection setup. MSS clamping does not replace ICMP-based correction when the path MTU drops during an established connection. + +#### Scenario 2: UDP or DF=0 Traffic — Fabric Fragmentation Fallback + +In this scenario, the UDP client sends with DF=0 and does not act on the corrected ICMP message. It continues sending 1428-byte IP packets. UDP applications can use ICMP feedback, but UDP does not automatically resize datagrams as TCP resizes segments. DF=0 permits IP fragmentation; it does not itself determine whether the sender responds to ICMP. + +The spoke SSR fragments the oversized packets within the fabric, and the hub SSR reassembles them: + +```mermaid +sequenceDiagram + participant Client as Client (UDP, DF=0) + participant Spoke as Spoke SSR (R1) + participant R2 as Underlay Device (R2) + participant Hub as Hub SSR + participant Server + + Note over Client: ICMP ignored. Packet size unchanged. + Client->>Spoke: Application data (1428-byte IP packet) + Note over Spoke: Encapsulated size exceeds flow MTU of 1000 + Spoke->>Spoke: Fabric-fragment packet + Spoke->>R2: Fabric fragment 1 (998 bytes, 970-byte payload) + Spoke->>R2: Fabric fragment 2 (550 bytes, 522-byte payload) + Note over R2: Both fragments fit within 1000-byte MTU + R2->>Hub: Forward fabric fragment 1 (998 bytes) + R2->>Hub: Forward fabric fragment 2 (550 bytes) + Hub->>Hub: Reassemble fragments + Hub->>Server: Original 1428-byte IP packet delivered intact +``` + +The 998- and 550-byte fabric fragments both fit within R2's 1000-byte MTU. Their payloads are 970 and 522 bytes, respectively. The hub SSR reassembles them and delivers the original 1428-byte IP packet to the server. The fragmentation remains within the fabric, but adds processing at both SSRs for each oversized packet. + +The following table traces the same sequence at the packet level. Steps 1 through 3 establish the baseline flow MTU update and the ICMP notification sent to the client; steps 4 and 5 show the resulting fabric fragmentation once the client keeps sending oversized packets: + +| Step | Packet | Size | Notes | +| --- | --- | --- | --- | +| 1 | Client-to-server IP packet (pre-encapsulation) | 1428 bytes total (1388-byte payload) | Sent from the client to the spoke SSR. | +| 2 | SVR-encapsulated fabric packet | 1468 bytes total (1440-byte fabric payload) | Sent by the spoke SSR toward R2. Exceeds R2's new 1000-byte MTU and is dropped. | +| 3 | Corrected ICMP Fragmentation Needed (Type 3, Code 4) | 576 bytes | Sent by the spoke SSR to the client. Reports next-hop MTU 952 after subtracting the 48-byte SVR overhead allowance. | +| 4 | Fabric fragment 1 | 998 bytes total (970-byte payload) | Sent by the spoke SSR for a later 1428-byte client packet. Fits within R2's 1000-byte MTU. | +| 5 | Fabric fragment 2 | 550 bytes total (522-byte payload) | Second fragment of the same packet. R2 forwards both fragments to the hub SSR for reassembly and delivery to the server. | + +#### Scenario 3: Asymmetric Paths and Directional Behavior + +Adaptive Path MTU updates each direction independently. An ICMP Fragmentation Needed message updates the flow that carried the dropped packet; it does not also change the reverse flow. + +After the spoke SSR processes R2's message: + +| Direction | Flow MTU After Correction | Why | +| --- | --- | --- | +| Client to server (spoke SSR to hub SSR) | 1000 | The spoke SSR received the ICMP message from R2 and updated this flow. | +| Server to client (hub SSR to spoke SSR) | 1500 (unchanged) | The hub SSR has not received an ICMP message for this flow. | + +If server-to-client traffic also crosses a reduced-MTU link, an oversized packet in that direction causes an ICMP response to the hub SSR. The hub SSR updates its flow and sends the corrected ICMP message to the server. + +If the reverse path still supports 1500 bytes, its flow MTU remains unchanged. Check both SSRs when troubleshooting asymmetric paths; `flow-mtu-updates` increments on the SSR that processes the ICMP response. + ## Fabric Fragmentation and Oversize Packet Behavior -When the PMTU on an overlay (SVR/fabric) path is lower than the MTU of the segment immediately preceding the Hub, packets larger than the PMTU will require fragmentation along the overlay. The SSR always fragments fabric packets when necessary, even when the incoming packet carries the Don't Fragment (DF) bit. This preserves packet delivery but prevents the sender from learning about the smaller path MTU and adjusting its segment size. +When an encapsulated packet exceeds the overlay path MTU, the SSR fragments it within the fabric, even if the original packet has the Don't Fragment (DF) bit set. The egress peer reassembles the fabric fragments before delivering the original packet. + +Fabric fragmentation is separate from fragmentation of the client's IP packet by an underlay router. It allows traffic to use a smaller underlay MTU without exposing fabric fragments to the endpoint. It also adds fragmentation and reassembly work at the SSRs. -:::note -For TCP traffic, setting `enforced-mss automatic` on the egress `network-interface` is the most reliable way to avoid this scenario. When set, the SSR rewrites the TCP MSS at the interface boundary to match the session MTU (including the path MTU for SVR sessions). This is commonly known as `MSS Clamping` and is not the default; it must be explicitly configured. +:::info +For TCP traffic, use `enforced-mss automatic` on the egress `network-interface` to reduce fragmentation when connections are established. This setting is not enabled by default. ::: ## Configuration +Adaptive Path MTU requires no configuration. You can configure MSS clamping and the periodic PMTUD interval separately. + ### Configuring `enforced-mss` (Recommended for TCP) -Set `enforced-mss` to `automatic` on egress interfaces to avoid fabric fragmentation for TCP traffic. The SSR calculates the correct MSS from the interface or path MTU for SVR sessions. +Set `enforced-mss` to `automatic` on the egress `network-interface`. The SSR calculates the MSS from the interface's session MTU, which may be the interface MTU or the path MTU for an SVR session, and rewrites the advertised MSS in TCP packets leaving that interface. ``` config @@ -106,7 +245,7 @@ exit ### Configuring PMTUD Interval -The PMTUD interval (how frequently the SSR probes each overlay path) is configurable at the router level and can be overridden per neighborhood or per adjacency. +Configure `path-mtu-discovery interval` at the router level to control how often the SSR probes each overlay path. You can override this setting per neighborhood or per adjacency. ``` config @@ -151,23 +290,48 @@ exit ## Verification -Use `show peers` to confirm the currently discovered path MTU for each peer path: +Use `show peers` to confirm the discovered path MTU for each peer path. The following example shows the reduced MTU after periodic PMTUD completes: ```text -admin@node1.router1# show peers +admin@node1.spoke# show peers Peer Node Network Interface Destination Status Hostname Path MTU ------------------------ --------- ------------------- --------------- -------- ------------ ---------- -router2 node1 wan0 192.0.2.10 Up router2.lab 1300 +hub node1 wan0 192.0.2.10 Up hub.lab 1000 ``` -A `Path MTU` value of `0` indicates PMTUD is disabled or has not yet completed a test cycle. +A `Path MTU` value of `0` indicates that PMTUD is disabled or has not yet completed a test cycle. It does not indicate a usable MTU of zero. + +The peer path MTU and an adaptive flow update are separate observations. A flow can be corrected by an ICMP response before the next periodic test updates the value in `show peers`. + +### Checking Adaptive Path MTU Activity + +Use `show stats icmp` and check `flow-mtu-updates`. You can also request the counter directly: + +```text +admin@node1.spoke# show stats icmp flow-mtu-updates +Field Value +-------------------- ----- +flow-mtu-updates 3 +``` + +The [`flow-mtu-updates`](cli_stats_reference.md#show-stats-icmp-flow-mtu-updates) counter increments when the SSR adjusts a flow's egress MTU in response to an ICMP Fragmentation Needed message. Compare the counter before and after the underlay change to confirm an update. The counter resets when the system resets and is not persisted across restarts. + +### Capturing Packets To Confirm Behavior + +Capture on both the LAN and WAN interfaces of the spoke SSR: + +- **LAN side:** Check that the SSR sends the corrected ICMP Fragmentation Needed message with next-hop MTU 952 toward the client. Check subsequent client packets to determine whether the sender reduces its packet size. An SSR-side capture alone does not prove the message reached the client. +- **WAN side:** Check for R2's ICMP Fragmentation Needed message reporting MTU 1000. If the client continues sending oversized packets, check for fabric fragments. In this scenario, oversized packets are split into fragments (such as the 998- and 550-byte fragments in this example). -The [`show stats icmp flow-mtu-updates`](cli_stats_reference.md#show-stats-icmp-flow-mtu-updates) provides a count of flows that have had their MTU updated at runtime via a TooBig packet. This counter is reset when the system resets (not persisted). +Capture on the hub LAN if you need to confirm reassembly and delivery of the original 1428-byte packet. ## Troubleshooting - If the path MTU shown by `show peers` does not reflect the expected value, verify that `path-mtu-discovery > enabled` is `true` on both sides of the adjacency. -- If TCP sessions continue to fragment after configuring `enforced-mss automatic`, confirm the setting is applied to the correct egress interface and that both peers have completed a PMTUD cycle. +- If TCP sessions continue to fragment after you configure `enforced-mss automatic`, confirm the setting is on the correct egress interface and that periodic PMTUD has discovered the expected MTU. Check the advertised MSS when a new connection is established; the setting does not renegotiate an existing connection. +- If `flow-mtu-updates` does not increment after an underlay MTU change, verify that an oversized fabric packet triggers an ICMP Destination Unreachable: Fragmentation Needed and DF Set response. Confirm that the response reaches the ingress SSR. Adaptive Path MTU cannot react to a message that is suppressed or filtered; periodic PMTUD can still infer a lower MTU from probe non-responses. +- If only one direction of a session appears corrected, check the counter on the peer SSR as well. Each direction is corrected independently; see [Scenario 3](#scenario-3-asymmetric-paths-and-directional-behavior). +- If fabric fragmentation continues, check whether the sender is receiving and acting on the corrected ICMP message. A sender that keeps sending oversized packets continues to require fabric fragmentation. For TCP, check for ICMP filtering between the SSR and the sender. ## Related Topics From 641e6482a65513b272f225f00d08ab10516e9fcf Mon Sep 17 00:00:00 2001 From: Kaushik Agrawal Date: Mon, 19 Oct 2026 23:07:20 -0400 Subject: [PATCH 2/3] docs(pmtu): polish voice and phrasing in adaptive path MTU guide - Tighten passive constructions and bureaucratic phrasing - Maintain direct senior-engineer whiteboard register - Preserve all technical invariants, parameters, and Mermaid diagrams Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- docs/config_pmtu.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/config_pmtu.md b/docs/config_pmtu.md index 608d4f43efc..dea35b6d30d 100644 --- a/docs/config_pmtu.md +++ b/docs/config_pmtu.md @@ -11,7 +11,7 @@ The SSR performs Path MTU Discovery (PMTUD) along the overlay to determine the m | --- | --- | --- | --- | --- | --- | | bidirectional | 1280/UDP | 2945 | 90 | 600s | Interval is configurable within `path-mtu-discovery/interval`, or disabled `path-mtu-discovery/enabled`. | -In deployments where ICMP Destination Unreachable: Fragmentation Needed and DF Set response messages are not generated or are suppressed, the SSR uses probe non-responses to infer the MTU. Three consecutive non-responses are considered equivalent to ICMP responses for the purposes of driving the discovery algorithm. See [RFC 1191](https://www.rfc-editor.org/rfc/rfc1191) for ICMP-based Path MTU Discovery. +In deployments where ICMP Destination Unreachable: Fragmentation Needed and DF Set response messages are not generated or are suppressed, the SSR uses probe non-responses to infer the MTU. The SSR treats three consecutive non-responses as equivalent to an ICMP response, which keeps the discovery algorithm moving forward. See [RFC 1191](https://www.rfc-editor.org/rfc/rfc1191) for ICMP-based Path MTU Discovery. Use `show peers` to view the discovered MTU. @@ -19,10 +19,10 @@ Prior to SSR 7.2.0, a reduced underlay MTU was not discovered until the next PMT Devices in the underlay report an ICMP Destination Unreachable: Fragmentation Needed and DF Set message (Type 3, Code 4) when a packet with DF set exceeds the next-hop MTU. Prior to SSR 7.2.0, the SSR forwarded these messages to the endpoint without acting on the reported MTU. The affected session retained its previous path MTU. -SSR 7.2.0 introduces two complementary enhancements under Adaptive Path MTU (I95-60371): +SSR 7.2.0 adds two complementary enhancements under Adaptive Path MTU (I95-60371): 1. **Underlay ICMP Reaction** — The ingress SSR intercepts the ICMP Fragmentation Needed message, updates the affected flow, and generates a corrected ICMP Fragmentation Needed message toward the original sender. The corrected MTU accounts for SVR overhead. -2. **Session Refresh** — The flow that carried the dropped packet is updated in place to use the MTU reported by the underlay. The session does not need to be rebuilt. +2. **Session Refresh** — The SSR updates the flow that carried the dropped packet in place, using the MTU the underlay reported. The session does not need to be rebuilt. See [Adaptive Path MTU for Long-Lived Sessions](#adaptive-path-mtu-for-long-lived-sessions) for a hub-and-spoke example. @@ -120,7 +120,7 @@ The `flow-mtu-updates` counter in `show stats icmp` increments when the flow MTU The reported MTU applies to encapsulated traffic. The spoke SSR must account for SVR overhead before notifying the client; otherwise, a client packet sized to 1000 bytes would exceed the underlay MTU after encapsulation. -In this example, the SSR subtracts a 48-byte SVR overhead allowance and sends a corrected ICMP Fragmentation Needed message to the client with a next-hop MTU of **952 bytes** (1000 − 48). Note that the wire length of the ICMP packet itself (576 bytes in this example) is separate from the MTU value it reports inside its payload. +In this example, the SSR subtracts a 48-byte SVR overhead allowance and sends a corrected ICMP Fragmentation Needed message to the client with a next-hop MTU of **952 bytes** (1000 − 48). The wire length of the ICMP packet itself (576 bytes in this example) is separate from the MTU value it reports inside its payload — do not confuse the two. :::info The SVR overhead allowance used for MTU calculation is not necessarily a fixed difference between an individual client packet and its resulting fabric packet. Encryption, HMAC, forward error correction (FEC), and BFD tunneling can affect the usable MTU. Rely on the corrected MTU calculated and reported by the SSR rather than estimating from a single observed packet. From e5b656959e8dd2817e6454582a46e05cae41139c Mon Sep 17 00:00:00 2001 From: Chris Date: Tue, 20 Oct 2026 09:36:51 -0400 Subject: [PATCH 3/3] very minor text edits, mostly identifying the whole ICMP Destination Unreachable: message as a message and not just text (this was not initally clear to me). --- docs/config_pmtu.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/config_pmtu.md b/docs/config_pmtu.md index dea35b6d30d..aaae3d9399b 100644 --- a/docs/config_pmtu.md +++ b/docs/config_pmtu.md @@ -11,17 +11,17 @@ The SSR performs Path MTU Discovery (PMTUD) along the overlay to determine the m | --- | --- | --- | --- | --- | --- | | bidirectional | 1280/UDP | 2945 | 90 | 600s | Interval is configurable within `path-mtu-discovery/interval`, or disabled `path-mtu-discovery/enabled`. | -In deployments where ICMP Destination Unreachable: Fragmentation Needed and DF Set response messages are not generated or are suppressed, the SSR uses probe non-responses to infer the MTU. The SSR treats three consecutive non-responses as equivalent to an ICMP response, which keeps the discovery algorithm moving forward. See [RFC 1191](https://www.rfc-editor.org/rfc/rfc1191) for ICMP-based Path MTU Discovery. +In deployments where `ICMP Destination Unreachable: Fragmentation Needed and DF Set` response messages are not generated or are suppressed, the SSR uses probe non-responses to infer the MTU. The SSR treats three consecutive non-responses as equivalent to an ICMP response, which keeps the discovery algorithm moving forward. See [RFC 1191](https://www.rfc-editor.org/rfc/rfc1191) for ICMP-based Path MTU Discovery. Use `show peers` to view the discovered MTU. Prior to SSR 7.2.0, a reduced underlay MTU was not discovered until the next PMTUD cycle. Existing sessions continued to use the previous MTU until they were rebuilt, even after periodic PMTUD discovered the change. -Devices in the underlay report an ICMP Destination Unreachable: Fragmentation Needed and DF Set message (Type 3, Code 4) when a packet with DF set exceeds the next-hop MTU. Prior to SSR 7.2.0, the SSR forwarded these messages to the endpoint without acting on the reported MTU. The affected session retained its previous path MTU. +Devices in the underlay report an `ICMP Destination Unreachable: Fragmentation Needed and DF Set` message (Type 3, Code 4) when a packet with DF set exceeds the next-hop MTU. Prior to SSR 7.2.0, the SSR forwarded these messages to the endpoint without acting on the reported MTU. The affected session retained its previous path MTU. SSR 7.2.0 adds two complementary enhancements under Adaptive Path MTU (I95-60371): -1. **Underlay ICMP Reaction** — The ingress SSR intercepts the ICMP Fragmentation Needed message, updates the affected flow, and generates a corrected ICMP Fragmentation Needed message toward the original sender. The corrected MTU accounts for SVR overhead. +1. **Underlay ICMP Reaction** — The ingress SSR intercepts the ICMP Fragmentation Needed message, updates the affected flow, and generates a corrected `ICMP Fragmentation Needed` message toward the original sender. The corrected MTU accounts for SVR overhead. 2. **Session Refresh** — The SSR updates the flow that carried the dropped packet in place, using the MTU the underlay reported. The session does not need to be rebuilt. See [Adaptive Path MTU for Long-Lived Sessions](#adaptive-path-mtu-for-long-lived-sessions) for a hub-and-spoke example. @@ -73,12 +73,12 @@ When the spoke SSR receives an ICMP Fragmentation Needed message from R2, it: The flow update and client notification provide two ways to handle the reduced MTU: the sender can reduce its packet size, or the SSRs can fragment and reassemble traffic within the fabric. :::info Introduced In SSR 7.2.0 -Adaptive Path MTU applies to overlay (SVR/fabric) sessions between peering SSR routers and requires no configuration. It depends on receiving an ICMP Fragmentation Needed message from the underlay. +Adaptive Path MTU applies to overlay (SVR/fabric) sessions between peering SSR routers and requires no configuration. It depends on receiving an `ICMP Fragmentation Needed` message from the underlay. ::: | Session State | Behavior | | --- | --- | -| Existing session | The SSR updates the affected flow in place when it processes an ICMP Fragmentation Needed message. | +| Existing session | The SSR updates the affected flow in place when it processes an `ICMP Fragmentation Needed` message. | | New session | The SSR uses the peer path MTU available at session setup. After periodic PMTUD discovers the reduced MTU, new sessions use that value. A session established before discovery can still require adaptive correction. | ### Detailed Walkthrough @@ -108,7 +108,7 @@ sequenceDiagram The client continues sending 1428-byte IP packets. The spoke SSR encapsulates the next packet into a 1468-byte fabric packet with DF set. R2 cannot forward it over the 1000-byte link, so it drops the packet. -R2 returns an ICMP Destination Unreachable: Fragmentation Needed and DF Set message (Type 3, Code 4) to the spoke SSR's WAN interface. The message reports a next-hop MTU of 1000 and quotes the dropped packet's IP header. +R2 returns an `ICMP Destination Unreachable: Fragmentation Needed and DF Set` message (Type 3, Code 4) to the spoke SSR's WAN interface. The message reports a next-hop MTU of 1000 and quotes the dropped packet's IP header. #### Step 2: The Spoke SSR Updates the Flow @@ -120,7 +120,7 @@ The `flow-mtu-updates` counter in `show stats icmp` increments when the flow MTU The reported MTU applies to encapsulated traffic. The spoke SSR must account for SVR overhead before notifying the client; otherwise, a client packet sized to 1000 bytes would exceed the underlay MTU after encapsulation. -In this example, the SSR subtracts a 48-byte SVR overhead allowance and sends a corrected ICMP Fragmentation Needed message to the client with a next-hop MTU of **952 bytes** (1000 − 48). The wire length of the ICMP packet itself (576 bytes in this example) is separate from the MTU value it reports inside its payload — do not confuse the two. +In this example, the SSR subtracts a 48-byte SVR overhead allowance and sends a corrected `ICMP Fragmentation Needed` message to the client with a next-hop MTU of **952 bytes** (1000 − 48). The wire length of the ICMP packet itself (576 bytes in this example) is separate from the MTU value it reports inside its payload — do not confuse the two. :::info The SVR overhead allowance used for MTU calculation is not necessarily a fixed difference between an individual client packet and its resulting fabric packet. Encryption, HMAC, forward error correction (FEC), and BFD tunneling can affect the usable MTU. Rely on the corrected MTU calculated and reported by the SSR rather than estimating from a single observed packet. @@ -184,7 +184,7 @@ sequenceDiagram The 998- and 550-byte fabric fragments both fit within R2's 1000-byte MTU. Their payloads are 970 and 522 bytes, respectively. The hub SSR reassembles them and delivers the original 1428-byte IP packet to the server. The fragmentation remains within the fabric, but adds processing at both SSRs for each oversized packet. -The following table traces the same sequence at the packet level. Steps 1 through 3 establish the baseline flow MTU update and the ICMP notification sent to the client; steps 4 and 5 show the resulting fabric fragmentation once the client keeps sending oversized packets: +The following table traces the same sequence at the packet level. Steps 1 through 3 establish the baseline flow MTU update and the ICMP notification sent to the client; steps 4 and 5 show the resulting fabric fragmentation when the client continues to send oversized packets: | Step | Packet | Size | Notes | | --- | --- | --- | --- | @@ -196,7 +196,7 @@ The following table traces the same sequence at the packet level. Steps 1 throug #### Scenario 3: Asymmetric Paths and Directional Behavior -Adaptive Path MTU updates each direction independently. An ICMP Fragmentation Needed message updates the flow that carried the dropped packet; it does not also change the reverse flow. +Adaptive Path MTU updates each direction independently. An `ICMP Fragmentation Needed` message updates the flow that carried the dropped packet; it does not also change the reverse flow. After the spoke SSR processes R2's message: @@ -211,7 +211,7 @@ If the reverse path still supports 1500 bytes, its flow MTU remains unchanged. C ## Fabric Fragmentation and Oversize Packet Behavior -When an encapsulated packet exceeds the overlay path MTU, the SSR fragments it within the fabric, even if the original packet has the Don't Fragment (DF) bit set. The egress peer reassembles the fabric fragments before delivering the original packet. +When an encapsulated packet exceeds the overlay path MTU, the SSR fragments it within the fabric, even if the original packet has the `Don't Fragment` (DF) bit set. The egress peer reassembles the fabric fragments before delivering the original packet. Fabric fragmentation is separate from fragmentation of the client's IP packet by an underlay router. It allows traffic to use a smaller underlay MTU without exposing fabric fragments to the endpoint. It also adds fragmentation and reassembly work at the SSRs. @@ -329,7 +329,7 @@ Capture on the hub LAN if you need to confirm reassembly and delivery of the ori - If the path MTU shown by `show peers` does not reflect the expected value, verify that `path-mtu-discovery > enabled` is `true` on both sides of the adjacency. - If TCP sessions continue to fragment after you configure `enforced-mss automatic`, confirm the setting is on the correct egress interface and that periodic PMTUD has discovered the expected MTU. Check the advertised MSS when a new connection is established; the setting does not renegotiate an existing connection. -- If `flow-mtu-updates` does not increment after an underlay MTU change, verify that an oversized fabric packet triggers an ICMP Destination Unreachable: Fragmentation Needed and DF Set response. Confirm that the response reaches the ingress SSR. Adaptive Path MTU cannot react to a message that is suppressed or filtered; periodic PMTUD can still infer a lower MTU from probe non-responses. +- If `flow-mtu-updates` does not increment after an underlay MTU change, verify that an oversized fabric packet triggers an `ICMP Destination Unreachable: Fragmentation Needed and DF Set` response. Confirm that the response reaches the ingress SSR. Adaptive Path MTU cannot react to a message that is suppressed or filtered; periodic PMTUD can still infer a lower MTU from probe non-responses. - If only one direction of a session appears corrected, check the counter on the peer SSR as well. Each direction is corrected independently; see [Scenario 3](#scenario-3-asymmetric-paths-and-directional-behavior). - If fabric fragmentation continues, check whether the sender is receiving and acting on the corrected ICMP message. A sender that keeps sending oversized packets continues to require fabric fragmentation. For TCP, check for ICMP filtering between the SSR and the sender.