Add hw_ppe.qos: shape in the switch where the switch offloads the flow - #193
Add hw_ppe.qos: shape in the switch where the switch offloads the flow#193JuliusBairaktaris wants to merge 2 commits into
Conversation
fc46927 to
e39091e
Compare
|
So for what it is worth, I am all for more approaches here and more diverse qos scripts and if this works well for you, you can ignore the rest. |
27ce43c to
6b32d92
Compare
|
@moeller0 saw the comment on the typo, do you want me to fix that? Also please wait until: openwrt/openwrt#24806 is merged. Thanks! |
What typo? |
| if [ ! -z "${TMP}" ] && echo "${TMP}" | grep -q mirred; then | ||
| # oops, a redirect is there but we failed to parse it? Ask for a user report | ||
| sqm_error "#---- CUT HERE ----#" | ||
| sqm_error "get_ifb_associated_with_if failed to extrect the ifb name from:" |
get_ifb_associated_with_if() treats any filter on parent ffff: that does not name an ifb as a parse failure and prints a cut-here block asking the user to file an issue. A script that puts something else on the ingress qdisc - a policer, say - therefore prints that block on every stop. Look for the redirect before deciding the name should have been there. A real parse failure still reports. Signed-off-by: Julius Bairaktaris <[email protected]>
…ower Connections a switch has offloaded to its flow engine never pass a software qdisc again, so a shaper that lives in a qdisc governs an ever-smaller share of the traffic the moment offloading is on. This script shapes in the switch instead, written for and measured on the Qualcomm PPE the qca_ppe driver programs, and driven entirely through existing offload uAPIs: tbf for the egress rate, mqprio for the download band's rate, flower with skbedit for the classification that feeds it. Egress is a root tbf on the WAN port, which the driver turns into the port's hardware token bucket; the qdisc beneath carries what still travels the CPU path. The tbf limit is the queue depth the driver programs, sized by eshaper_queue_dur_us of the rate - deep enough for a single flow to reach the shaped rate, while frames the switch classifies as latency-sensitive ride higher-priority queues past it. The download cannot be shaped where it arrives - the switch cannot queue on ingress - so it is shaped where it leaves, and only it. Every user port's scheduler keeps a band of queues no frame selects on its own; a flower rule on the WAN port marks each arriving frame addressed to the router - every routed download frame, nothing bridged - and the mark selects that band on whichever port the frame leaves by. An mqprio class spanning the band carries the download rate, which the driver programs into the band's scheduler node. A transfer between two local ports never wears the mark and passes at line rate. What the switch does not forward in hardware reaches the host, and only that: those flows are shaped by the classic redirect of the WAN port's ingress to an ifb, which therefore governs exactly the download remainder - the flows the flow table does not hold, and the path to Wi-Fi clients. The band's node shaper and the ifb each carry the full download rate, so a workload split across both can exceed the rate while the split lasts; sustaining that needs a flow table too full to take new flows. Measured on an IPQ8074 AX3600 (kernel 6.18, PPPoE line shaped 305/165 Mbit/s), against the same script shaping every user port's egress instead, arms interleaved run for run. A wired transfer between two switch ports reads 941/940/939 Mbit/s where a per-port egress shaper holds it to 287/287/287, against an unshaped 932-939 baseline. The download direction is unchanged: 8-stream download 286.1/286.0/286.1 against 286.1/286.1/286.0 Mbit/s, ICMP under that load 19.54/19.62/19.59 against 19.69/19.55/19.63 ms, idle 19.90 ms. Run concurrently, the download holds 285.0 Mbit/s at 19.7 ms while the local transfer runs 915-934 Mbit/s beside it. The switch's per-queue counters attribute the download to the band's four queues (1506-byte average) with the acks promoted past it (80-byte average), and during a download to a Wi-Fi client the ifb's counters grow by every packet the WAN port's hook sees, 51669 of 51669. The download band and the mqprio rate need a driver that serves internal priorities eight and up from a schedulable node of their own; the qca_ppe driver does. On a driver without that the mqprio install fails and the script fails loudly with it, the same stance skip_sw takes for the filter. Assisted-by: Claude:claude-fable-5 Signed-off-by: Julius Bairaktaris <[email protected]>
6b32d92 to
51a4009
Compare
A connection the kernel has offloaded to a switch's flow engine is never handed to a qdisc again, so a shaper built on cake and an ifb does not shape it — the CPU never sees those packets. Where the switch can be told a rate instead, it can still be governed.
hw_ppe.qosdoes that: a roottbfon the port, which a driver implementingTC_SETUP_QDISC_TBFprograms into the port's hardware token bucket, and amatchallpolicer on the port's ingress in place of the redirect to an ifb — a switch cannot queue an arriving packet in order to shape it, but it can meter one. No ifb is created and nothing is redirected. The ingress filter isskip_sw, so a kernel or switch that cannot take it fails loudly instead of quietly policing on the CPU.Two defaults do the heavy lifting, both measured:
limitof 1 ms of the shaped rate (never less than eight full frames) instead of the customarylatency 300ms. The limit is how deep the queue behind the shaper may get, and a driver that drains a real queue sizes that queue from it: at 165 Mbit/s the wide limit held 5.4 ms of standing queue under upload saturation — the bufferbloat the shaper was installed against — where the 1 ms limit holds the added latency under measurement noise at the same throughput.The ingress side is still a trade (thanks @moeller0 for keeping this honest): a policer drops where a shaper would have delayed, so it adds no queueing delay but not no delay, and ECN cannot be honoured. What it buys is that offloaded connections are governed at all, at zero CPU. Raise
ishaper_burst_dur_usto tolerate more, or setdownloadto 0 to keep only hardware egress shaping.Configure it on the physical port (
wan), not on a pppoe or vlan device above it — the hardware only knows the port. That placement is also what covers traffic the switch cannot offload, Wi-Fi included: it shares the port.What a user can configure
All ordinary
/etc/config/sqmoptions;run-openwrt.shnow exports the burst durations, so those work from uci (and LuCI's dropdown picks the script up from/usr/lib/sqm/).interfacepppoe-wanor a VLAN above itupload/downloadeshaper_burst_dur_usishaper_burst_dur_usqdisc,elimit,etarget,eecn,eqdisc_optscakefalls back tofq_codelwith a warninglinklayer,overhead,tcMTU,tcMPU,tcTSIZEilimit,itarget,iecn,iqdisc_optsTesting
Qualcomm IPQ8074 (Xiaomi AX3600), kernel 6.18, a driver offloading both directions, on a 336 Mbit/s PPPoE line shaped to 335000/165000. Latency probed at 20 Hz from a wired LAN host, load from public speed endpoints:
Zero added latency at 96–97% of the configured rates, both directions loaded. The rates and the queue limit read back exactly in the hardware registers;
stopclears the token bucket and the meter (verified by register readback); the uci burst knobs were verified end to end (option → filter burst → register).The functions.sh change
get_ifb_associated_with_if()treats any filter onparent ffff:that does not name an ifb as a parse failure and prints the cut-here block asking the user to open an issue. It now looks for the redirect before deciding the name should have been there; a real parse failure still reports.