Repository navigation
Experiment: Prompt optimization - #801
mrfootoyou wants to merge 9 commits into
Conversation
|
Are there static suggested prompts or is it dynamic based on # of windows etc? |
@upmcplanetracker Unlike the current system prompt which varies depending on addressing mode, segment detection settings, and The user prompt, on the other hand, is still quite variable, just like the current user prompt. Depending on which features are enabled, the user prompt will have different instructions inserted, including window instructions. |
|
@mrfootoyou give me some time to review this and test with some additional models. After experimenting with Jev, I ended up with a simpler prompt for ad detection, so there is some evidence this could be simplified. Also, this prompt structure is from about a year ago, and LLMs have gotten significantly better. On a somewhat related note, have you experimented with which "keep content only" mode vs. the standard of detect ads?
|
|
@ttlequals0 Sure, take your time. I tried "content only" mode briefly but quickly changed back because it did not give good results. I don't remember the details, but I do remember thinking that I must not understand what that mode does. If you have a paypal/venmo/etc account, I will contribute to the benchmark fund. |
|
not @ttlequals0/MinusPod ***@***.***> but he put this on
his main github page https://buymeacoffee.com/ttlequals0
…On Mon, Oct 5, 2026 at 8:47 AM John Belcher ***@***.***> wrote:
*mrfootoyou* left a comment (ttlequals0/MinusPod#801)
<#801 (comment)>
@ttlequals0 <https://github.com/ttlequals0> Sure, take your time.
I tried "content only" mode briefly but quickly changed back because it
did not give good results. I don't remember the details, but I do remember
thinking that I must not understand what that mode does.
If you have a paypal/venmo/etc account, I will contribute to the benchmark
fund.
—
Reply to this email directly, view it on GitHub
<#801?email_source=notifications&email_token=BUKFPFAXFKDA6JQHITC5PYT5SOJ67A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKOJZGQ3TINJTGIY2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-5994745321>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BUKFPFCVMXJFTGHZAXWLCUL5SOJ67AVCNFSNUABGKJSXA33TNF2G64TZHMYTCMBUGMYDCNBSHA5US43TOVSTWNJWGQ3TKMJUGE2DTILWAI>
.
Triage notifications, keep track of coding agent tasks and review pull
requests on the go with GitHub Mobile for iOS
<https://github.com/notifications/mobile/ios/BUKFPFBQGN6VIVD6JBWVU5T5SOJ67A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKOJZGQ3TINJTGIY2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y>
and Android
<https://github.com/notifications/mobile/android/BUKFPFBSOWSAC6IONMXVHHL5SOJ67A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKOJZGQ3TINJTGIY2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>.
Download it today!
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Starting some tests I plan to test these models, the ones I use on my setup, current best and and current worst models.
|
|
I think some of us could nerd out about models and other minutia, but I
always feel bad opening an issue for that. Or adding into an issue or PR
when I screw something up. Any thoughts about opening the Discussion area
for the minuspod+/- minuspodjev repo for these type of conversations? This
way also if there's an ongoing discussion you don't have to keep a PR or
issue open for weeks or months.
…On Wed, Oct 7, 2026, 9:01 AM Dominick Krachtus ***@***.***> wrote:
*ttlequals0* left a comment (ttlequals0/MinusPod#801)
<#801 (comment)>
Starting some tests I plan to test these models, the ones I use on my
setup, current best and and current worst models.
- claude-opus-5-5 (my reviewr model)
- claude-fable-5-1
- claude-sonnet-5
- claude-haiku-4-5-20251001 (best, #1
<#1> current prompt, F0.5
0.908)
- claude-opus-5 (older data to compare)
- claude-opus-4-8 (older data to compare)
- qwen/qwen3.8-flash (my alternate reviewer model)
- qwen/qwen3.5-plus-02-15 (best, #2
<#2>, F0.5 0.865)
- google/gemini-3.5-flash (best, #5
<#5>, F0.5 0.851)
- x-ai/grok-4.3 (best, #6
<#6>, F0.5 0.847)
- bytedance-seed/seed-2-1-turbo (worst, #83
<#83>, F0.5 0.052)
- mistralai/mistral-small-2603 (worst, #84
<#84>, F0.5 0.049)
- qwen/qwen3-8b (worst, #85
<#85>, F0.5 0.000)
—
Reply to this email directly, view it on GitHub
<#801?email_source=notifications&email_token=BUKFPFGIQSR7JRLDSU3B3DL5SY5CZA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTMMBTHA2DKOBTGI4KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-6038458328>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BUKFPFGFBBC7ZQXZRISDVRL5SY5CZAVCNFSNUABGKJSXA33TNF2G64TZHMYTCMBUGMYDCNBSHA5US43TOVSTWNJWGQ3TKMJUGE2DTILWAI>
.
Triage notifications, keep track of coding agent tasks and review pull
requests on the go with GitHub Mobile for iOS
<https://github.com/notifications/mobile/ios/BUKFPFGIBSKN6MYRKGWX4G35SY5CZA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTMMBTHA2DKOBTGI4KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y>
and Android
<https://github.com/notifications/mobile/android/BUKFPFBGVLSLOYBMHTOQ6F35SY5CZA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTMMBTHA2DKOBTGI4KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>.
Download it today!
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
I ran this prompt (417a039, verbatim) through a separate harness that scores seconds of programme cut rather than matched ad events, in case it's useful next to the F0.5 numbers. Setup: 26 episodes (This American Life, The Moth, Joe Rogan; 30.7 h) with second-level human labels. Gemini 3.8 Flash, Parakeet transcripts, index addressing, whole episode in one call (no windows, no description, no audio context, none of the sentence re-splitting from
What I took from it:
Caveats: my labels and taxonomy aren't MinusPod's, and it's 26 episodes from three shows. Happy to share the longer prompt I compared against, and to run other variants of this one through the same harness if that's helpful. |
|
Just saw #824. Our numbers line up once the categories match: my "sponsor only" row hid it. With |
Append the 1880 call records and response-shard lines PR #801 ran under the segmentation prompt variant. Co-authored-by: John Belcher <[email protected]>
|
@mrfootoyou thanks for the work on this. Results are in #824, which adds a Short version:
@johnfewell's seconds-cut harness points at the same cause. With Worth keeping from this PR, separate from the classify-everything approach: index addressing, the smaller system prompt structure (strict schema, single Not merging this as-is. If you want to try a sponsor-only or tightened- |
|
Thanks for the thorough comparison in #824, and for the mention. Following up on index addressing: I reran my longer prompt with one change, numbered transcript lines (
So index addressing carries most of this PR's saving on its own (input tokens dropped from 3.76M to 1.74M over 3 runs), with recall within about a point and none of the precision loss. One side effect: when credits are skipped, the closing-credits block now sometimes runs over the closing song or music next to it (6–23 s in three or four episodes). That doesn't matter for Happy to run more variants through the same harness. |
|
@johnfewell, that matches the live A/B. MinusPod already supports both modes, and the
The same volume reaches the pipeline either way. Segment-id mode proposes a few more, and the first-line filters remove them. Some are invented IDs, which a timestamp cannot reveal. The rest are spans too long for the window, which fits a wrong end ID that lands in bounds. Your credits block running over the closing music has the same effect at the line level. That log cannot show what happens next. Boundary snapping to audio cues, the reviewer pass and manual corrections all run after this point, and none of them records the mode. I am adding that stamp in a logging change so hold rate, reviewer adjustment size, and correction rate can be split by mode from the same random draw. The most useful run from your harness is the numbered-lines variant with |
|
No new model calls needed for that, so I rescored the same 3 runs. My taxonomy has no separate
Your reading holds. Every numbered-lines cut is the credits block overrunning into the music beside it: a closing song between the sign-off and the credits on The Moth, and closing music after the credits on This American Life (1–23 s per episode). The same spans recur in every run; run 3 adds one more episode. The PR's One caveat: my labels count closing songs and music as programme. If yours treat a closing song as part of the outro, the numbered-lines figure mostly disappears; the PR's doesn't. |
|
Thanks for rescoring. MinusPod defines outro as closing credits, sign-off, or theme music, so under our labels, most of the numbered lines are intended for removal. The PR's cuts do not collapse the same way. A closing conversation and a storyteller bio are program under any definition, and that is the precision loss #824 measured. Show-segment detection is also opt-in here and off by default, so only feeds that asked for outro removal see any of this. The credits-to-music overrun lands on a silence edge, which is what boundary snapping cleans up after detection. Index addressing stays in. |
…eparately The detection runs added for the prompt-variant comparison had regenerated results/report.md and its charts. That file is the last full campaign and the next one is weeks out, so it returns to the main version. The regenerated detection-cell report is kept as results/report-prompt-testing.md with its own chart directory, labeled as the PR #801 / #824 prompt test.
|
Hey @ttlequals0 @johnfewell. I'm still here, just processing your fantastic feedback (I should have paid more attention in stats class) and trying to figure out what can be salvaged.
I'll convert this PR to a draft and rebase with |
417a039 to
1c18234
Compare
It can be disabled via configuration, so no need to remove from prompt
- Omit `main_content` segments to reduce output tokens. - Simplify category descriptions
- Normalizes all segments immediately after transcription: - Converts start & end to floats - Ensures end >= start - Sorts all segments and words - Merges word-pairs - Ensures words are space-prefixed - Ensures segment text matches words. - Uses a more nuanced algorithm to split transcribed segments into sentences: - Supports additional end-of-sentence (EOS) punctuation - Supports EOS quotes - Improved (but still not perfect) false positive detection - [parakeet] Use `split_segments_into_sentences` to generate the final transcript segments.
Inserts lines into the transcript prompt describing gaps above a certain duration (1.5s by default), enabling the LLM to reason about long pauses between cues. The first "gap" line includes a description.
|
@ttlequals0 @johnfewell I was so confused as to why my real-world results seem to be so much better than the benchmarks suggest. I think I figured out why. My system prompt is based on the actual transcript data included in the user prompt. In the actual transcript, each cue is a single "utterance", i.e., a single sentence split into 25-word parts. While this transcript format consumes a lot of tokens, especially in timestamp mode, the benefit is that the LLM will almost never encounter a cue which spans multiple topics. Because of this reality, my prompt contains two paragraphs describing how to handle these cues:
Notice that last instruction: "always favor main content". Now let's look at some of the "utterances" used in the benchmarks: From ep-andy-and-ari-e774e8022fab. Main content is displayed with
Just about every sponsor cue includes some main_content in it. So, it is of little surprise that my system prompt's recall score suffers in the benchmarks, yet in real world experience, works great! I will push my latest iteration. It applies your suggested prompt improvements (intro/outro definitions), discourages output of Results Settings Performance Costs |
1c18234 to
516fa99
Compare
|
Thanks for digging into this. I tested it. Our eval (corpus 1.0 dev) was already on sentence-level cues. The transcripts are Parakeet, split into sentences, median cue 2.4 s. Only 5 of 180 ad boundaries fall inside a cue (26 of 2,518 ad seconds), so mixed cues don't explain our numbers. Your examples come from the #824 benchmark, where the Whisper cues are about 24 s, so I tested it there. I ran the
So you're right that sentence cues raise recall. Every sentence-cue trial found more ad time than every original-cue trial, and Andy & Ari improved. Precision drops more, though, so F0.5 goes down. Most of the extra cuts are self_promo, such as Patreon and merch segments, which the #824 truth files don't count as ads. Whether that's a real loss depends on whether you want those cut. Sentence cues also cost about 20% more in tokens. One thing in your favor: on the original cues, one drink-champs window came back empty in all 3 trials, and it was fine once split. Worth noting too: in the #824 table, your prompt's recall was already higher than the current prompt's (median 0.904 vs 0.842 across 17 models). The gap there is mostly precision. In production, I've moved to 90-minute windows after trying your settings on This American Life. Cost per episode went from 31¢ to 10¢, the ads it found were unchanged, and processing time dropped from 26 minutes to under 5. On costs: 2.100.18 still records them per call for me, in |
|
@mrfootoyou thanks for tracing it to the cue shape, and @johnfewell for testing it the same day. Where the pieces stand: #884 (gap cues) and #886 (minimal timestamps) are merged and in 2.100.19 and later. #887 is waiting on the rebase down to On the mixed-cue explanation, John's run answers it for the benchmark corpus. Sentence cues lift recall (0.898 to 0.931) and cost precision (0.805 to 0.757), so F0.5 drops and tokens go up about 20%. The Parakeet side of the eval was already sentence-level, so this is a Whisper-transcript effect. It does not close the precision gap on The per-mode stamp I mentioned on Oct 13 shipped in 2.100.13, so here is the live
Same volume of ads reaches the pipeline either way. Segment ids lose about 5% at the contract check, the reviewer adjusts them less often on a small sample, and the shift size is similar. No quality gain on this instance in either direction. The saving John measured is input tokens, and the spend ledger does not record the addressing mode, so there is no production number for that side. On prompt size: a detection window here runs about 11k input tokens, and the system prompt is about 2.6k of that. The transcript is the bulk, which is why #886 went after timestamps first. On native Anthropic the system prompt is also cached for 5 minutes, so with 4 parallel windows it is written once per pass and read three times at the cache rate. Trimming it buys less than the token count suggests. Nothing shows cache hits yet, so I opened #897 to surface them in the LLM and System One stats. On costs: 2.100.22 still records cost per call. A row that shows as incomplete rather than a dollar figure means the price lookup for that model id failed, and the Incomplete chip on the episode lists the unpriced calls. |
|
Closing PR. I'll the keep the branch around for a while. |
Summary
This is a POC to validate the outcome of using a different LLM prompt philosophy: Short and simple is better (in this case, at least). This PR should not be merged -- the unit tests are destroyed, and the other prompts should be investigated.
I ran a subset of the benchmarks using 7 low-cost models over 8 podcasts in the corpus (the shorter ones). All models improved significantly, but I'll let @ttlequals0 judge the attached results.
I only modified the ad detection and verification prompts. Both prompts are very similar. Verification has a couple extra sentences describing 2nd pass and ad fragments, but otherwise identical.
I also focused on using index-based addressing since 1) an index requires approximately 1/12 the number of tokens, and 2) they have a smaller cognitive load than timestamps (at least for me) and are almost certainly easier for LLMs to identify and reference.
Prompt Philosophy
Simplify LLM's job by asking it to segment and categorize the entire transcript, including main content. MP simply discards the segments it doesn't need.
Trust that the model understands podcasting concepts and terminology.
Avoid multiple choice instructions when one precise instruction will do.
Use
json_schemarecommended by all providers: strict, all properties required, no additional properties.Describe the full schema in the prompt, including exact field names and enum values. This works for all models and all providers.
Omit examples. Turns out they are not needed when the job and schema are simple enough.
Use enum for
confidencewith a small number of descriptive options. I am attempting to solicit less than absolute certainty in the model's confidence. It is not working so far.Tell the LLM it is a "JSON API" to encourage valid JSON.
Tell the LMM to return a JSON object, not an array. I got very bad results when asking for an array (even when the json_schema specified an array), but asking for an object works much more reliably.
Use a single prompt for both transcript addressing modes. The difference isn't that significant and is easy to describe.
Use
startandendfor both addressing modes. Having two sets of fields needlessly complicates things, and the LLM has no problem putting the cue index in the 'start' field.Use
sponsor_namefield instead of the 7 or so variants. MP still supports the variants, but the json_schema only supportssponsor_name.Use positive examples and avoid negative ones (e.g. "...that is not an ad", "NOT AD EXAMPLE").
Avoid terminology which only make sense to a listener ("voiced by", "tonally distinct", etc.)
Avoid unimportant details. For example, the statement "any product promotion for compensation is an ad." The LLM does not know if the host was compensated for the ad, nor does it matter, so just omit that detail.
Don't include instructions which don't apply:
Omit the common sponsors/brands.
System prompt size: (without dynamic sponsors)
Based on https://token-calculator.net/.