Skip to content

Add hw_ppe.qos: shape in the switch where the switch offloads the flow - #193

Open
JuliusBairaktaris wants to merge 2 commits into
tohojo:mainfrom
JuliusBairaktaris:hw-ppe-offload
Open

Add hw_ppe.qos: shape in the switch where the switch offloads the flow#193
JuliusBairaktaris wants to merge 2 commits into
tohojo:mainfrom
JuliusBairaktaris:hw-ppe-offload

Conversation

@JuliusBairaktaris

@JuliusBairaktaris JuliusBairaktaris commented Aug 20, 2026

Copy link
Copy Markdown

A connection the kernel has offloaded to a switch's flow engine is never handed to a qdisc again, so a shaper built on cake and an ifb does not shape it — the CPU never sees those packets. Where the switch can be told a rate instead, it can still be governed. hw_ppe.qos does that: a root tbf on the port, which a driver implementing TC_SETUP_QDISC_TBF programs into the port's hardware token bucket, and a matchall policer on the port's ingress in place of the redirect to an ifb — a switch cannot queue an arriving packet in order to shape it, but it can meter one. No ifb is created and nothing is redirected. The ingress filter is skip_sw, so a kernel or switch that cannot take it fails loudly instead of quietly policing on the CPU.

Two defaults do the heavy lifting, both measured:

  • The tbf carries an explicit limit of 1 ms of the shaped rate (never less than eight full frames) instead of the customary latency 300ms. The limit is how deep the queue behind the shaper may get, and a driver that drains a real queue sizes that queue from it: at 165 Mbit/s the wide limit held 5.4 ms of standing queue under upload saturation — the bufferbloat the shaper was installed against — where the 1 ms limit holds the added latency under measurement noise at the same throughput.
  • The policer's burst defaults to 10 ms of the rate instead of the generic 1 ms. A meter has no queue to absorb a flow's bursts — it can only drop, and TCP pays every drop with a retransmit: on a 336 Mbit/s line policed at 335 Mbit/s, 1 ms of burst costs 43% of the throughput, 10 ms delivers 96%, and neither adds measurable latency.

The ingress side is still a trade (thanks @moeller0 for keeping this honest): a policer drops where a shaper would have delayed, so it adds no queueing delay but not no delay, and ECN cannot be honoured. What it buys is that offloaded connections are governed at all, at zero CPU. Raise ishaper_burst_dur_us to tolerate more, or set download to 0 to keep only hardware egress shaping.

Configure it on the physical port (wan), not on a pppoe or vlan device above it — the hardware only knows the port. That placement is also what covers traffic the switch cannot offload, Wi-Fi included: it shares the port.

What a user can configure

All ordinary /etc/config/sqm options; run-openwrt.sh now exports the burst durations, so those work from uci (and LuCI's dropdown picks the script up from /usr/lib/sqm/).

option reaches effect
interface both the physical switch port — not pppoe-wan or a VLAN above it
upload / download hardware token bucket / policer rate in kbit/s; 0 disables that direction
eshaper_burst_dur_us hardware egress bucket depth as a duration (default 1 ms of the rate)
ishaper_burst_dur_us hardware policer burst as a duration (default 10 ms of the rate, see above)
qdisc, elimit, etarget, eecn, eqdisc_opts CPU path the qdisc under the tbf, carrying what is not offloaded (Wi-Fi, local traffic); cake falls back to fq_codel with a warning
linklayer, overhead, tcMTU, tcMPU, tcTSIZE neither warned about and ignored — the hardware meters the wire frame including preamble, gap and CRC
ilimit, itarget, iecn, iqdisc_opts neither ingress is a meter, not a queue; nothing for them to configure

Testing

Qualcomm IPQ8074 (Xiaomi AX3600), kernel 6.18, a driver offloading both directions, on a 336 Mbit/s PPPoE line shaped to 335000/165000. Latency probed at 20 Hz from a wired LAN host, load from public speed endpoints:

latency avg / worst throughput
idle 10.0 / 10.5 ms
upload saturated (3 streams) 9.7 / 10.6 ms 159 Mbit/s
download saturated (6 streams) 10.0 / 11.4 ms 325 Mbit/s
both directions at once 10.0 / 11.9 ms 318 + 153 Mbit/s

Zero added latency at 96–97% of the configured rates, both directions loaded. The rates and the queue limit read back exactly in the hardware registers; stop clears the token bucket and the meter (verified by register readback); the uci burst knobs were verified end to end (option → filter burst → register).

The functions.sh change

get_ifb_associated_with_if() treats any filter on parent ffff: that does not name an ifb as a parse failure and prints the cut-here block asking the user to open an issue. It now looks for the redirect before deciding the name should have been there; a real parse failure still reports.

Copilot AI lite review requested due to automatic review settings August 20, 2026 16:29

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@moeller0

Copy link
Copy Markdown
Collaborator

So for what it is worth, I am all for more approaches here and more diverse qos scripts and if this works well for you, you can ignore the rest.
That said, policers are indeed less heavy weight than shapers, but that comes at a cost, a policer will drop packets more readily than a traffic shaper, especially one using ECN, and a dropped packet often is retransmitted, so instead of adding zero delay, the policer adds at least one RTT in such cases, e.g. for TCP traffic. A policer might still be what you want in your script, but I would not try to sell this as a unproblematic latency optimisation.

@JuliusBairaktaris
JuliusBairaktaris force-pushed the hw-ppe-offload branch 5 times, most recently from 27ce43c to 6b32d92 Compare August 26, 2026 07:31
@JuliusBairaktaris

JuliusBairaktaris commented Aug 26, 2026

Copy link
Copy Markdown
Author

@moeller0 saw the comment on the typo, do you want me to fix that?

Also please wait until: openwrt/openwrt#24806 is merged. Thanks!

@sppmasterspp

Copy link
Copy Markdown

saw the comment on the typo, do you want me to fix that?

What typo?

Comment thread src/functions.sh
if [ ! -z "${TMP}" ] && echo "${TMP}" | grep -q mirred; then
# oops, a redirect is there but we failed to parse it? Ask for a user report
sqm_error "#---- CUT HERE ----#"
sqm_error "get_ifb_associated_with_if failed to extrect the ifb name from:"

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one

get_ifb_associated_with_if() treats any filter on parent ffff: that does not
name an ifb as a parse failure and prints a cut-here block asking the user to
file an issue. A script that puts something else on the ingress qdisc - a
policer, say - therefore prints that block on every stop.

Look for the redirect before deciding the name should have been there. A real
parse failure still reports.

Signed-off-by: Julius Bairaktaris <[email protected]>
…ower

Connections a switch has offloaded to its flow engine never pass a
software qdisc again, so a shaper that lives in a qdisc governs an
ever-smaller share of the traffic the moment offloading is on. This
script shapes in the switch instead, written for and measured on the
Qualcomm PPE the qca_ppe driver programs, and driven entirely through
existing offload uAPIs: tbf for the egress rate, mqprio for the
download band's rate, flower with skbedit for the classification that
feeds it.

Egress is a root tbf on the WAN port, which the driver turns into the
port's hardware token bucket; the qdisc beneath carries what still
travels the CPU path. The tbf limit is the queue depth the driver
programs, sized by eshaper_queue_dur_us of the rate - deep enough for a
single flow to reach the shaped rate, while frames the switch
classifies as latency-sensitive ride higher-priority queues past it.

The download cannot be shaped where it arrives - the switch cannot
queue on ingress - so it is shaped where it leaves, and only it. Every
user port's scheduler keeps a band of queues no frame selects on its
own; a flower rule on the WAN port marks each arriving frame addressed
to the router - every routed download frame, nothing bridged - and the
mark selects that band on whichever port the frame leaves by. An
mqprio class spanning the band carries the download rate, which the
driver programs into the band's scheduler node. A transfer between two
local ports never wears the mark and passes at line rate.

What the switch does not forward in hardware reaches the host, and
only that: those flows are shaped by the classic redirect of the WAN
port's ingress to an ifb, which therefore governs exactly the download
remainder - the flows the flow table does not hold, and the path to
Wi-Fi clients. The band's node shaper and the ifb each carry the full
download rate, so a workload split across both can exceed the rate
while the split lasts; sustaining that needs a flow table too full to
take new flows.

Measured on an IPQ8074 AX3600 (kernel 6.18, PPPoE line shaped
305/165 Mbit/s), against the same script shaping every user port's
egress instead, arms interleaved run for run. A wired transfer between
two switch ports reads 941/940/939 Mbit/s where a per-port egress
shaper holds it to 287/287/287, against an unshaped 932-939 baseline.
The download direction is unchanged: 8-stream download
286.1/286.0/286.1 against 286.1/286.1/286.0 Mbit/s, ICMP under that
load 19.54/19.62/19.59 against 19.69/19.55/19.63 ms, idle 19.90 ms.
Run concurrently, the download holds 285.0 Mbit/s at 19.7 ms while the
local transfer runs 915-934 Mbit/s beside it. The switch's per-queue
counters attribute the download to the band's four queues (1506-byte
average) with the acks promoted past it (80-byte average), and during
a download to a Wi-Fi client the ifb's counters grow by every packet
the WAN port's hook sees, 51669 of 51669.

The download band and the mqprio rate need a driver that serves
internal priorities eight and up from a schedulable node of their own;
the qca_ppe driver does. On a driver without that the mqprio install
fails and the script fails loudly with it, the same stance skip_sw
takes for the filter.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Julius Bairaktaris <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants