diff --git a/content/newsletters/2026-week-43.md b/content/newsletters/2026-week-43.md index 03ee29f01..aa87e4215 100644 --- a/content/newsletters/2026-week-43.md +++ b/content/newsletters/2026-week-43.md @@ -31,7 +31,7 @@ We ran the four DORA metrics over a full year of git history and found that the ![Page, Ticket or Ignore: Is the Cheap Model Good Enough for Alert Triage?](/images/posts/cheap-model-alert-triage-digitalocean-inference.png) -Six models and two model routers on DigitalOcean Serverless Inference got the same written on-call policy and had to sort alerts into page, ticket or ignore, and the results show whether the cheap option holds up. +Six models on DigitalOcean Serverless Inference got the same written on-call policy and had to sort alerts into page, ticket or ignore, and the results show whether the cheap option holds up. ### [DevOps Interviews Now Test How You Decide. How to Practice Situational Questions](https://devops-daily.com/posts/devops-interviews-situational-questions) diff --git a/content/posts/cheap-model-alert-triage-digitalocean-inference.md b/content/posts/cheap-model-alert-triage-digitalocean-inference.md index a59ff9c16..1d38a42ed 100644 --- a/content/posts/cheap-model-alert-triage-digitalocean-inference.md +++ b/content/posts/cheap-model-alert-triage-digitalocean-inference.md @@ -1,12 +1,12 @@ --- title: 'Page, Ticket or Ignore: Is the Cheap Model Good Enough for Alert Triage?' -excerpt: 'We gave six models and two model routers on DigitalOcean Serverless Inference one written on-call policy and 200 alerts, three times each. A model at $0.16 per 1,000 alerts made every call the same way as one at $1.64, and the general router picked the expensive one. The real differences were latency, missed pages and a bug in our own test data that the models caught.' +excerpt: 'We gave six models on DigitalOcean Serverless Inference one written on-call policy and 200 alerts, three times each. A model at $0.16 per 1,000 alerts made every call the same way as one at $1.64. The real differences were latency, missed pages and a bug in our own test data that the models caught.' category: name: 'DevOps' slug: 'devops' date: '2026-10-14' publishedAt: '2026-10-14T09:00:00Z' -updatedAt: '2026-10-14T09:00:00Z' +updatedAt: '2026-10-19T09:00:00Z' readingTime: '14 min read' author: name: 'DevOps Daily Team' @@ -22,14 +22,13 @@ tags: sponsorCampaign: 'digitalocean' --- -We gave six models and two model routers on DigitalOcean Serverless Inference the same written on-call policy and the same 200 alerts, three times each, and asked one question per alert: page, ticket or ignore. DeepSeek V4 Flash made the policy's call on all 600 for an estimated **$0.16 per 1,000 alerts**. Qwen3.8-Max, the most expensive direct model we tested, also got all 600, for **$1.64 per 1,000**. Routing on the alert's own `severity` label, with no model at all, got 102 of 200, but we built that label to mislead. +We gave six models on DigitalOcean Serverless Inference the same written on-call policy and the same 200 alerts, three times each, and asked one question per alert: page, ticket or ignore. DeepSeek V4 Flash made the policy's call on all 600 for an estimated **$0.16 per 1,000 alerts**. Qwen3.8-Max, the most expensive model we tested, also got all 600, for **$1.64 per 1,000**. Routing on the alert's own `severity` label, with no model at all, got 102 of 200, but we built that label to mislead. So on this test the cheap model was good enough. That is not the whole answer. DeepSeek V4 Flash took 4.4 seconds at the median and once took a full minute. Two other cheap models each let through alerts the policy says must page. And 200 clean results still leave room for a real error rate of 1.5 percent. The most useful thing we learned came from a bug in our own test data: most of the models found it, and our second reviewer did not. ## TL;DR -- With a clear written policy, DeepSeek V4 Flash, Qwen3.8-Max and `router:general` made the policy's call on all 600 alert runs. `gpt-oss-120b` got 599 (one empty reply). Qwen3.8-Max costs about ten times as much per alert as DeepSeek V4 Flash, and it answered about three times faster. -- `router:general` sent 571 of 600 calls to Qwen3.8-Max, so it cost almost what Qwen3.8-Max costs. `router:software-engineering` sent most calls to DeepSeek V4 Flash and got 4 wrong, all 4 from a model it reported as `deepseek-4-flash`. +- With a clear written policy, DeepSeek V4 Flash and Qwen3.8-Max made the policy's call on all 600 alert runs. `gpt-oss-120b` got 599 (one empty reply). Qwen3.8-Max costs about ten times as much per alert as DeepSeek V4 Flash, and it answered about three times faster. - Llama 4 Maverick failed to page on 2 different page-level alerts (3 times in all) and `gpt-oss-20b` on 3 (once each). Ministral 3 14B missed 4.7 page-level alerts per run out of 50, and 17 of its 600 replies contained two different decisions. - DeepSeek V4 Flash spends most of its output on reasoning: p95 latency of 10.7 seconds, 33 calls over 10 seconds and one at 60.6 seconds. - At 10,000 alerts a month, the token cost is about $1.57 for the cheap model and $16.40 for the expensive one. Choose on missed pages and latency, not on price. @@ -38,7 +37,7 @@ So on this test the cheap model was good enough. That is not the whole answer. D - A DigitalOcean account with a Serverless Inference model access key - Node.js 20 or later (the scripts have no dependencies) -- About $3 of inference if you rerun all eight configurations +- About $1.50 of inference if you rerun all six models The code, the alerts, every recorded decision and reply, and the scoring are in the repository: @@ -109,21 +108,19 @@ To check that the text carries what the labels assume, we gave one AI second rea ## Results -Eight configurations, three runs each, 4,800 calls. The six direct models are open-weight models. Our account's key returned `403 this model is not available for your subscription tier` for the GPT-5 and Claude models we tried, so they are not in this test (the check is saved in the repo). The two routers are DigitalOcean's Inference Router, in public preview, which picks a model per request and reports which one it used. +Six models, three runs each, 3,600 calls. All six are open-weight models. Our account's key returned `403 this model is not available for your subscription tier` for the GPT-5 and Claude models we tried, so they are not in this test (the check is saved in the repo). DigitalOcean's Inference Router is in public preview, and its preview terms do not allow publishing test results without DigitalOcean's written consent, so this post covers the direct models only. -| Configuration | Correct (of 600) | Page-level alerts missed in any run (of 50) | False pages per run | $ per 1,000 alerts | p50 | p95 | -| ----------------------------- | ---------------- | ------------------------------------------- | ------------------- | ------------------ | ----- | ------ | -| DeepSeek V4 Flash | 600 | 0 | 0 | 0.157 | 4.4 s | 10.7 s | -| Qwen3.8-Max | 600 | 0 | 0 | 1.640 | 1.3 s | 2.6 s | -| `router:general` | 600 | 0 | 0 | 1.565 | 1.8 s | 3.5 s | -| `gpt-oss-120b` | 599 | 0 | 0 | 0.194 | 3.1 s | 5.1 s | -| Llama 4 Maverick | 597 | 2 | 0 | 0.189 | 1.9 s | 3.3 s | -| `router:software-engineering` | 596 | 0 | 0.7 | 0.305 | 4.2 s | 8.9 s | -| `gpt-oss-20b` | 595 | 3 | 0 | 0.144 | 2.4 s | 5.3 s | -| Ministral 3 14B | 532 | 8 | 6.0 | 0.149 | 0.8 s | 1.8 s | -| Severity label, no model | 102 of 200 | 14 | 39 | 0 | | | +| Configuration | Correct (of 600) | Page-level alerts missed in any run (of 50) | False pages per run | $ per 1,000 alerts | p50 | p95 | +| ------------------------ | ---------------- | ------------------------------------------- | ------------------- | ------------------ | ----- | ------ | +| DeepSeek V4 Flash | 600 | 0 | 0 | 0.157 | 4.4 s | 10.7 s | +| Qwen3.8-Max | 600 | 0 | 0 | 1.640 | 1.3 s | 2.6 s | +| `gpt-oss-120b` | 599 | 0 | 0 | 0.194 | 3.1 s | 5.1 s | +| Llama 4 Maverick | 597 | 2 | 0 | 0.189 | 1.9 s | 3.3 s | +| `gpt-oss-20b` | 595 | 3 | 0 | 0.144 | 2.4 s | 5.3 s | +| Ministral 3 14B | 532 | 8 | 6.0 | 0.149 | 0.8 s | 1.8 s | +| Severity label, no model | 102 of 200 | 14 | 39 | 0 | | | -Costs are estimates: recorded token usage times DigitalOcean's list prices on 2026-10-14. One model the router reported, `deepseek-4-flash`, is not on the price page under that name, and we priced it as DeepSeek V4 Flash. +Costs are estimates: recorded token usage times DigitalOcean's list prices on 2026-10-14. ```chart { @@ -133,16 +130,11 @@ Costs are estimates: recorded token usage times DigitalOcean's list prices on 20 "rows": [ { "label": "DeepSeek V4 Flash", "value": 0, "series": "direct" }, { "label": "Qwen3.8-Max", "value": 0, "series": "direct" }, - { "label": "router:general", "value": 0, "series": "router" }, { "label": "gpt-oss-120b", "value": 1, "series": "direct" }, { "label": "Llama 4 Maverick", "value": 3, "series": "direct" }, - { "label": "router:software-engineering", "value": 4, "series": "router" }, { "label": "gpt-oss-20b", "value": 5, "series": "direct" } ], - "series": [ - { "name": "direct", "color": "#0080ff" }, - { "name": "router", "color": "#f59e0b" } - ] + "series": [{ "name": "direct", "color": "#0080ff" }] } ``` @@ -151,39 +143,25 @@ Costs are estimates: recorded token usage times DigitalOcean's list prices on 20 "type": "bar", "title": "Estimated cost per 1,000 alerts", "unit": "$", - "caption": "Recorded token usage times DigitalOcean list prices checked 2026-10-14. A routed call is billed at the price of the model that served it; the router itself is free during the public preview.", + "caption": "Recorded token usage times DigitalOcean list prices checked 2026-10-14.", "rows": [ { "label": "gpt-oss-20b", "value": 0.144, "series": "direct" }, { "label": "Ministral 3 14B", "value": 0.149, "series": "direct" }, { "label": "DeepSeek V4 Flash", "value": 0.157, "series": "direct" }, { "label": "Llama 4 Maverick", "value": 0.189, "series": "direct" }, { "label": "gpt-oss-120b", "value": 0.194, "series": "direct" }, - { "label": "router:software-engineering", "value": 0.305, "series": "router" }, - { "label": "router:general", "value": 1.565, "series": "router" }, { "label": "Qwen3.8-Max", "value": 1.64, "series": "direct" } ], - "series": [ - { "name": "direct", "color": "#0080ff" }, - { "name": "router", "color": "#f59e0b" } - ] + "series": [{ "name": "direct", "color": "#0080ff" }] } ``` -Three configurations scored 600 of 600, four more got between one and five wrong, and one is clearly not good enough. Ministral 3 14B is the fastest and among the cheapest, but it got 42 of the 200 alerts wrong in at least one run, missed 4.7 page-level alerts per run and gave the same answer in all three runs for only 80 percent of the alerts. It also ignored the format: 593 of its 600 replies had text around the JSON, and 17 contained two different decisions, typically an answer followed by a "correction". +Two models scored 600 of 600, three more got between one and five wrong, and one is clearly not good enough. Ministral 3 14B is the fastest and among the cheapest, but it got 42 of the 200 alerts wrong in at least one run, missed 4.7 page-level alerts per run and gave the same answer in all three runs for only 80 percent of the alerts. It also ignored the format: 593 of its 600 replies had text around the JSON, and 17 contained two different decisions, typically an answer followed by a "correction". That raises a scoring question, so here is the rule: the scorer takes the first decision in the reply, because an automated router would act on the first answer it can parse, and an empty reply counts as wrong. The repo records the first 400 characters of each reply from this run. The runner now keeps the full text for future runs. The misses from the near-perfect models are worth reading, because they are the errors that hurt. Llama 4 Maverick called a Postgres database that would run out of connections in about 34 minutes a ticket, twice. Once it answered ticket for a public certificate expiring in 16 hours, then wrote "Wait, the correct response should be based on the given policy" and started a second answer, which we score as the ticket it gave first. `gpt-oss-20b` let three different Redis memory alerts through, all in its second run. Each of those replies would have suppressed a page that the policy requires. -## What the routers picked - -The Inference Router reports the model that served each call in the response's `model` field, so you can see exactly what it chose: - -- **`router:general`** sent 571 calls to Qwen3.8-Max and 29 to DeepSeek V4 Flash. It made every call correctly and cost $1.57 per 1,000 alerts, about 5 percent less than calling Qwen3.8-Max directly. For this task, the general router is mostly the expensive model. -- **`router:software-engineering`** sent 465 calls to `deepseek-v4-flash-0731`, 72 to a model it reported as `deepseek-4-flash` and 63 to Qwen3.8-Max. It cost about $0.31 per 1,000. Its 4 wrong answers all came from the 72 `deepseek-4-flash` calls: two pages and two ignores, all on ticket-level alerts. The 465 `deepseek-v4-flash-0731` calls were all correct, as were all 600 of our direct calls to that model. We do not know what `deepseek-4-flash` is, and the price page does not list it. - -The router is free during the preview, and a routed call costs whatever the chosen model costs. In this test, calling DeepSeek V4 Flash directly was both cheaper and more accurate than the software-engineering router, and the general router cost about the same as calling Qwen3.8-Max directly. - ## Latency is where the cheap model pays DeepSeek V4 Flash matched Qwen3.8-Max's accuracy at about a tenth of the token cost. Latency goes the other way. It is a reasoning model, and 81 percent of its output tokens were reasoning, a median of 143 output tokens per call against Qwen3.8-Max's 33 tokens of plain JSON. @@ -197,11 +175,9 @@ DeepSeek V4 Flash matched Qwen3.8-Max's accuracy at about a tenth of the token c "rows": [ { "label": "Ministral 3 14B", "value": 0.8, "tick": 1.8 }, { "label": "Qwen3.8-Max", "value": 1.3, "tick": 2.6 }, - { "label": "router:general", "value": 1.8, "tick": 3.5 }, { "label": "Llama 4 Maverick", "value": 1.9, "tick": 3.3 }, { "label": "gpt-oss-20b", "value": 2.4, "tick": 5.3 }, { "label": "gpt-oss-120b", "value": 3.1, "tick": 5.1 }, - { "label": "router:software-engineering", "value": 4.2, "tick": 8.9 }, { "label": "DeepSeek V4 Flash", "value": 4.4, "tick": 10.7 } ] } @@ -211,7 +187,7 @@ DeepSeek V4 Flash had 33 calls over 10 seconds and one at 60.6 seconds. For a ti ## The bug the models found -In the first run, nobody scored 600. Qwen3.8-Max, `gpt-oss-120b` and both routers "got wrong" the same two alerts every time. This was one of them: +In the first run, nobody scored 600. Qwen3.8-Max and `gpt-oss-120b` "got wrong" the same two alerts every time. This was one of them: ```text status: firing @@ -235,12 +211,12 @@ llama-4-maverick: {"decision": "ignore", "reason": "service is in an active maintenance window"} ``` -Across the two broken alerts and three runs, Qwen3.8-Max, `gpt-oss-120b` and both routers paged all 6 times. Llama 4 Maverick and Ministral 3 14B trusted the word "active" all 6 times. DeepSeek V4 Flash paged three times, ignored twice and once used up its 2,000-token allowance on reasoning without giving a decision. `gpt-oss-20b` paged four times and ignored twice. The second reader that "agreed on all 200" had trusted the word too. After the fix, it labelled the two corrected alerts ignore, which matches their labels. +Across the two broken alerts and three runs, Qwen3.8-Max and `gpt-oss-120b` paged all 6 times. Llama 4 Maverick and Ministral 3 14B trusted the word "active" all 6 times. DeepSeek V4 Flash paged three times, ignored twice and once used up its 2,000-token allowance on reasoning without giving a decision. `gpt-oss-20b` paged four times and ignored twice. The second reader that "agreed on all 200" had trusted the word too. After the fix, it labelled the two corrected alerts ignore, which matches their labels. Three lessons from that: 1. **Agreement between two readers is not proof the labels are right.** Both can take the same shortcut. Here, the disagreements between models found what our check missed, so read them before you trust a score. -2. **A score is only as good as its labels.** With the bug in place, Llama 4 Maverick ranked first (99.5 percent) and Qwen3.8-Max tied for third (99.0 percent), marked down for reading the timestamps. +2. **A score is only as good as its labels.** With the bug in place, Llama 4 Maverick ranked first (99.5 percent) and Qwen3.8-Max third (99.0 percent), marked down for reading the timestamps. 3. **Real alerts can contain this kind of contradiction.** A silence that expired, a maintenance flag that nobody cleared. The models that read the times treated stale context the way an engineer should. We fixed the inputs, not the labels: the generator now keeps maintenance alerts inside their window, only those two alerts changed (their start times moved to 03:00 and 04:35), and we reran them on every configuration. This correction came after we had seen the results, so treat the final numbers as exploratory rather than a clean holdout. The original two alerts and every configuration's answers to them are in `results/v1-maintenance-bug/` in the repo. The window text still has an end time but no date, which a stronger dataset would fix. @@ -249,15 +225,15 @@ We fixed the inputs, not the labels: the generator now keeps maintenance alerts Perfect scores on 200 alerts do not mean a perfect model. Alerts are the unit here (three runs of one alert are not three independent samples), so we count the alerts a model got wrong in at least one run and compute a one-sided 95 percent upper confidence limit on that share. These limits assume the alerts are a random sample, and they are not: they come from 31 templates. Read them as a sense of scale, not as a bound on production error rates: -| Configuration | Alerts wrong in any run (of 200) | Upper 95% limit | Page-level alerts missed in any run (of 50) | Upper 95% limit | -| ------------------------------------------------ | -------------------------------- | --------------- | ------------------------------------------- | --------------- | -| DeepSeek V4 Flash, Qwen3.8-Max, `router:general` | 0 | 1.5% | 0 | 5.8% | -| `gpt-oss-120b` | 1 | 2.3% | 0 | 5.8% | -| Llama 4 Maverick | 2 | 3.1% | 2 | 12.1% | -| `gpt-oss-20b` | 5 | 5.2% | 3 | 14.8% | -| Ministral 3 14B | 42 | 26.3% | 8 | 27.0% | +| Configuration | Alerts wrong in any run (of 200) | Upper 95% limit | Page-level alerts missed in any run (of 50) | Upper 95% limit | +| ------------------------------ | -------------------------------- | --------------- | ------------------------------------------- | --------------- | +| DeepSeek V4 Flash, Qwen3.8-Max | 0 | 1.5% | 0 | 5.8% | +| `gpt-oss-120b` | 1 | 2.3% | 0 | 5.8% | +| Llama 4 Maverick | 2 | 3.1% | 2 | 12.1% | +| `gpt-oss-20b` | 5 | 5.2% | 3 | 14.8% | +| Ministral 3 14B | 42 | 26.3% | 8 | 27.0% | -So "missed no pages" means "missed none of these 50", which is still consistent with missing about one in seventeen. Compared with Qwen3.8-Max, the other configurations except Ministral 3 14B were between 0 and 0.83 percentage points less accurate. A paired bootstrap over alerts puts `router:software-engineering` at -0.67 (95% interval -1.33 to -0.17) and `gpt-oss-20b` at -0.83 (-1.67 to -0.17), while the intervals for `gpt-oss-120b` and Llama 4 Maverick reach zero. When two models made identical calls, the bootstrap has nothing to resample, and that is not evidence that they are equivalent. We cannot rank DeepSeek V4 Flash, Qwen3.8-Max and `router:general` against each other from this data. +So "missed no pages" means "missed none of these 50", which is still consistent with missing about one in seventeen. Compared with Qwen3.8-Max, the other models except Ministral 3 14B were between 0 and 0.83 percentage points less accurate. A paired bootstrap over alerts puts `gpt-oss-20b` at -0.83 (95% interval -1.67 to -0.17), while the intervals for `gpt-oss-120b` and Llama 4 Maverick reach zero. When two models made identical calls, the bootstrap has nothing to resample, and that is not evidence that they are equivalent. We cannot rank DeepSeek V4 Flash and Qwen3.8-Max against each other from this data. There is also what this test does not cover: @@ -267,9 +243,9 @@ There is also what this test does not cover: ## Cost in perspective -Each call used about 720 input tokens and a median of 20 to 200 output tokens, depending on the model. At 10,000 alerts a month, which is a busy on-call rotation, the estimated token cost is $1.57 for DeepSeek V4 Flash and $16.40 for Qwen3.8-Max. The whole test, 4,800 scored calls plus 48 reruns, came to an estimated $2.63. +Each call used about 720 input tokens and a median of 20 to 200 output tokens, depending on the model. At 10,000 alerts a month, which is a busy on-call rotation, the estimated token cost is $1.57 for DeepSeek V4 Flash and $16.40 for Qwen3.8-Max. The whole test, 3,600 scored calls plus 36 reruns, came to an estimated $1.50. -That changes the question. Neither bill matters next to a single missed page, so price should be the last thing you choose on. We tested each configuration on all three decisions, not a cascade where cheap rules or a cheap model sort alerts first and only some go to a second model. On this test, DeepSeek V4 Flash and Qwen3.8-Max made the same calls. Qwen3.8-Max answered in 1.3 seconds at the median and never took more than 6.6. DeepSeek V4 Flash cost a tenth as much and sometimes took ten seconds or more. If a few extra seconds before a page are acceptable to you, the cheap model is the better buy. Either way, test it on your own alerts first. +That changes the question. Neither bill matters next to a single missed page, so price should be the last thing you choose on. We tested each model on all three decisions, not a cascade where cheap rules or a cheap model sort alerts first and only some go to a second model. On this test, DeepSeek V4 Flash and Qwen3.8-Max made the same calls. Qwen3.8-Max answered in 1.3 seconds at the median and never took more than 6.6. DeepSeek V4 Flash cost a tenth as much and sometimes took ten seconds or more. If a few extra seconds before a page are acceptable to you, the cheap model is the better buy. Either way, test it on your own alerts first. ## Run it on your own alerts