Version 1.3.20 - #651
Merged
Merged
Version 1.3.20#651
Conversation
A setting under Settings → Captions, off by default, that gives the .vtt and .srt downloads a colour per speaker. Nothing else changes: the live <track>, the saved captions, burn-in and the interactive-transcript sidecar stay plain. Export only, and deliberately. The caption editor is the source of truth and a caption row is four <input> values, so a STYLE block would be erased by the first caption edit, a <v Name> tag would show up literally in a text box and count against the reading-rate limit, burn-in would paint the tag as words, and our own VTT import would chop it into fabricated word spans. A downloaded file is a leaf, so the colour is added at the moment of writing. The speaker is recorded when the captions are generated, not looked up when the file is downloaded. Captions diverge from the transcript on the first edit, and a lookup at download time would let a later transcript edit silently recolour captions the user had already finished. caption.js starts a new segment on every speaker change, so each cue sits inside one speaker's run; it drops the name and it is vendored, so the run is re-derived from the same struck-word-free transcript copy the cues were built from, and lands on the row as data-speaker. Insert inherits from the row above, merge keeps the survivor's, the row cache carries it, and the project saves the list beside the captions file rather than inside it. A saved list that does not match the cue count is dropped rather than applied to the wrong lines. WebVTT gets a STYLE block and voice tags, SRT gets font tags. Measured across engines: Chromium and WebKit colour both voice and class selectors and hide the voice tag itself; Firefox ignores every mechanism and renders plain white. The setting's hint says so, SRT's player dependence included. Also fixes a flake in the 1.3.19 tab test: it measured the modal with a bounding rect, which reports the opening transform rather than the layout.
The .vtt and .srt links took their address from the two caption writers alone, and those run only on transcription, caption-editor entry, regenerate and sanitise. None runs at boot — the boot pass builds the track straight from the generator — and the project-restore and VTT-import routes wrote at most the WebVTT link. So a freshly opened page handed you an empty .vtt and .srt, and a reloaded project an empty .srt, with nothing on screen to say so. The captions were never missing: the live track has had them all along. One helper now writes both links from whatever VTT its caller holds, with the SRT coming from media-export's existing cue serialiser rather than a second one. Restore and import call it with the file they were handed; a capture-phase click backstop fills anything still empty from the live track, early enough that the speaker colouring reads the real file later in the same click. That backstop made the colour setting reachable on a page whose captions have never been built, where nothing has recorded a speaker. With no caption rows there have been no caption edits either, so the transcript is still the whole truth and the speakers are derived for that one export. Only for that case: a recorded list always wins, which is what keeps a later transcript edit from recolouring captions someone had already finished.
Transcribing a URL produced a project with the previously open project's name AND its media: the new transcript was filed against the old recording, and the player was left pointing at it. hyperaudio-save captures what is being transcribed at setTranscriptBusy(true) — the player's src and the File behind it — so a completion arriving after the user has switched away is still filed against the right project, and so a recovered birth cannot steal the identity of whatever is on screen (#529, #612). The contract is that the engine puts the transcription's media on the player first. All three cloud engines raised the busy signal before assigning player.src, so the capture took the OUTGOING project's media and File, named the new project after them, and — because the captured src is restored when the transcript lands — actively overwrote the real URL on the player. Deepgram, AssemblyAI and Parakeet (HuggingFace) now signal busy after the media is on the player. Local files were never affected: the file picker's own change handler puts them on the player when the file is chosen, long before TRANSCRIBE is pressed, which is why this only showed with a URL. The local Whisper and Parakeet engines already had the order right. Two regression tests: a Deepgram URL run end to end against a stubbed API, which now names the project after its own media and leaves that media on the player; and the contract itself, checked for all three cloud engines by observing the player at the moment each signals busy. Both fail without the reorder, the first with exactly the reported symptom. Fixes #643
A cloud recording is still downloading while it is transcribed, so the player never decodes a frame from it. The picture frozen at loadstart to cover the loading gap — the PREVIOUS project's — then stayed for good: URL media stores no file, so the capture pipeline has nothing to read and applyStoredPoster can never answer for it, and the handler that would swap in a real picture gave up without a fallback. The project wore the last recording's frame while Recents drew its glyph beside it, which is the same disagreement #603 fixed for audio. The glyph is the fallback here too, so the two agree. Only a picture belonging to ANOTHER medium gives way: this medium's own frame, when it managed to draw one, is the better answer and stays, which is what #619 and #582 put there. Telling them apart needs the load token each was made under, since the stale freeze and the honest own-frame are both data: JPEGs. The handler also stopped gating on videoWidth. A medium still arriving reports none, which is exactly the case this exists for; waiting for a capture stays gated on it, because there is nothing to grab without a frame (#603). Two tests: a cloud recording whose bytes arrive only after the transcription is over, which must end on its own glyph rather than the previous project's frame; and one that does decode, which must keep its own frame at its own size. The first fails without the fix.
Every colour now clears 9.7:1 against black where the weakest was 6.9:1, and the set averages 12.6:1 rather than 11.2:1, measured as relative luminance. Captions are painted on a translucent black ground, so brightness is what carries them. The hues are spread as before and it is still not red-then-green, which the commonest colour blindness would make the hardest first pair to tell apart. The pale yellow that sat next to the amber is gone — two yellows out of eight was a wasted slot — and a periwinkle takes its place at the far end, so a conversation has to reach eight speakers before the two blues meet.
Downloading the word-level ("karaoke") WebVTT while the caption editor was
open produced a file with the header and nothing under it. The exporter always
read #hypertranscript straight out of the document, and in caption mode the
transcript is not there — the caption rows have the screen and the transcript
is held in the cache — so it found no words and wrote a bare WEBVTT line. No
error, no empty-file warning: just a seven-byte download.
editor-core now answers where the transcript is, by the same rule
captionSourceWithoutStruckWords already follows, and the exporter asks rather
than assumes. A caller that names its own source still gets exactly that one:
the burn-in renderer passes a re-timed copy and is untouched.
Test: the same file from both views, which fails without the fix with the
caption-editor download empty.
The earlier fix caught only one of the ways a previous project's picture stays over a new recording. It asked whether the poster was a STAND-IN, which is true of a frozen frame and false of a stored capture — so a project opened from Recents, wearing its own saved capture, lent that capture straight to the next cloud transcription. Reported from real use, and the case my reproduction had missed. Worse, the picture was not merely left in place, it was re-applied. The library announces a change before library.currentId() moves on, so applyStoredPoster ran with the id of the project the user had been on, fetched ITS capture, and put it on the new recording. Every poster this module writes is now recorded with the load it was written for, the project it was fetched for, and whether a real capture could still improve on it. The question the handler asks became "is this picture this medium's and this project's?" — which a capture, a frozen frame, a first frame and a glyph cannot be told apart by any other means. A capture fetched for the outgoing project no longer counts as this one's, so the next change asks again and lands on the glyph, the picture Recents draws. Marking captures final and everything else provisional keeps #619's upgrade: a stand-in still gives way to a real capture when one arrives, and a picture belonging to the medium on the player is never traded down for a glyph. Test: a local project reopened from Recents, then a cloud recording still downloading, which must end on its own glyph rather than the reopened project's capture. It fails against the previous fix.
hyperaudioPosterDebug() in the console prints what the poster is, what this module last wrote and for which load and project, whether the loader owns the screen, and for every project whether the player is showing its glyph or its stored capture. The state is closed over, so a stale picture cannot be diagnosed from the element alone — and the long console snippets that would otherwise do this keep wrapping into syntax errors when pasted.
The other half of #643. That fix put the URL on the player before the busy signal, so the new project got the right NAME and the right src — but the previous project's File still sat in the session through the birth and was written as the new project's media: a copy of the last recording inside a project that plays from a link. The poster pipeline then captured that file, so the new project wore the previous recording's picture in the player and in Recents alike. No poster-side rule could catch it, because by every record it WAS the new project's own capture. Diagnosed from a real session with hyperaudioPosterDebug(): current project, showing its own capture, and the capture a frame of the recording before. Three places, one rule — a URL on the player is URL-mode and the URL wins, which currentMediaDescriptor already said: the identity captured at the busy signal carries no File for a URL transcription; the birth drops a File that is not this medium's; and writeMediaOnce refuses one as a backstop. The one exception stays: a file fetched from this very URL is this project's media. The two earlier poster commits on this branch stand. They close the other paths — a frozen frame or a stand-in left behind by a recording that never decodes — and this closes the one that had made them look ineffective. Test: a local-file project, then a URL transcription from a video that loads fully. The new project must store no media and no poster, and wear its own glyph rather than the local clip's capture. Fails without the fix with the previous project's file in the new project's media directory. Fixes #644
A link project stores no file, so it had no picture of its own. It wore a soundwave whether it was audio or video, and no frame could be read from the player because the server had not allowed a cross-origin read. The library entry now records whether the medium has a picture. Metadata for a remote recording often lands after the birth committed the entry, so the save module also records it the moment the player knows, for the project that owns the medium and never while a transcription has the screen — reading before writing, because the index updater writes and notifies unconditionally and an unconditional write on the notification looped thousands of times a second. A link video draws as a strip of film on the same pastel ground; audio, and entries from before the field existed, keep the wave. A glyph drawn before the entry knew is redrawn when the entry changes. The poster pipeline asks the server for a CORS-readable copy, on the detached capture element only and never on the live player: a server that refuses makes that element fail, which resolves null and costs nothing, where on the player it would have stopped the video. A refusal is remembered per URL for the session and audio links are not asked at all — without both, the intro's mp3 was asked, and refused, on every library change. The five-second stand-in during the loader phase asks the same way. Where the server allows it, a link project now earns a real capture and wears it in the player and in Recents; where it does not, the film glyph. Tests run against a real second-origin server, because a routed response is fulfilled with permissive headers and would have let the "refusing" server allow the capture. The media-leak test (#644) now accepts a poster the project captured from its own URL and keeps asserting what matters: no leaked file, and never the previous project's picture.
…ntred A screen recording of switching between projects showed the markup logo for about seven tenths of a second, and a glyph for a frame, before the right picture arrived. Every poster write during real switches was traced to its line; three writers were at fault. The reveal fallback treated any data: poster as another project's picture and replaced it with the markup poster. By now a data: poster can be this project's own glyph, so it painted the logo over the glyph for the length of a slow load — and the logo is branding, not the project's picture (#603). A foreign picture now gives way to this project's glyph, and its own is left alone. The audio re-seed handler acted on library changes before metadata, when a video reports no width and looks like audio, and re-seeded a glyph over a video project's capture for one frame at every switch. It waits for metadata now; settleAudio does the job properly once the medium is known. A slow link showed the outgoing project's capture for the length of its load, because the loadstart freeze could not read a cross-origin frame and left what was up. A project change is now covered at loadstart by the incoming project's own picture: its capture if one has been read this session, synchronously from the poster cache, else its glyph, provisional, so a capture or a frame replaces it the moment either exists. Within one project the outgoing frame still covers the gap. The film glyph is redrawn as a short length of film centred on the wave's footprint, so the play badge sits over it the same way. Test: a local project and a slow, uncacheable, CORS-refusing link, switched between five times with every poster write recorded and judged — never the logo, never another project's picture, never a soundwave on a video.
…cents A glyph in the player was a flash of colour on every project switch before the capture landed, and for a link the server will not let a canvas read it was the picture for good. Black is what an empty player looks like and is what a frame arrives over, so it is less jarring both as a cover and as an end state. The film glyph stays where a picture is wanted for its own sake: the Recents rollover. The poster attribute is never removed to get there. #575 measured what happens in WebKit when a posterless element paints a frame: its display mode advances and never comes back, and every project opened afterwards in that tab paints nothing. Black is a poster like any other. At loadstart the cover for an incoming project is its capture if one has been read this session, else black for video and the wave for audio, decided from the library as last announced so nothing waits on a read. Audio keeps the wave throughout: settleAudio and the audio re-seed are untouched. The switch test now judges every poster write with black as an allowed cover and no glyph of any kind allowed in the player.
The parser walked two lists in lockstep: the word array, which carries the timings and the speakers, and the formatted transcript split on spaces, which supplied the visible text — matched by position alone, with nothing checking they were the same length. smart_format, which we ask for, is exactly what breaks that. It rewrites spoken numbers and the like into forms with a different token count, so from the first mismatch every word after it is drawn from the wrong slot: the text slides against the timings, and a speaker label, placed where the word array says the speaker changed, lands against whatever text has drifted into that position. Reported as speaker names appearing mid-sentence. smart_format already puts the formatted text on each word as punctuated_word. Taking it from there puts text, timing and speaker together on one object, the way the AssemblyAI parser has always done it — which is why AssemblyAI does not show this, and why its being clean did not settle where the fault was. Whether this was the whole of the reported problem is now answerable by looking: if labels still land mid-sentence, the speaker changed there and it is Deepgram's diarization. Test: a response whose formatted transcript is two tokens shorter than its word array, as a spoken year makes it. Against the old parser the last two words render as "undefined" and the label moves; now every word carries its own text and the label sits on the word the speaker changed on.
Taking each word's text from the word itself did not settle the misplaced labels, so the question is now whether Deepgram's flags really do flip where a label appears. Only their raw flags answer that, and the response is not kept anywhere a report can reach. The response is kept for the session, and hyperaudioSpeakerDebug() prints, for each change: the words either side, the gap before it, whether the previous word ended a sentence, and how many words the new speaker holds before changing back. Above that, the counts that decide it — how many changes fall mid-sentence, and how many are one-word turns, which is the signature of diarization noise rather than a turn. Enough to say whose fault it is without passing a whole transcript around.
`diarize` tags every word on its own, with no regard for what was being said, so a boundary lands a few words short of where the turn actually ended and the next speaker is credited with the tail of the previous one's sentence. That is the reported fault: the label reads as mid-sentence. Measured on a six-way debate, eight or nine of about thirty changes fell inside a sentence, against essentially none from AssemblyAI on the same audio — which is not a matter of quality but of what each returns. AssemblyAI gives speaker-tagged utterances. We were asking Deepgram only for per-word flags. Deepgram has the same thing behind `utterances`: segments cut on pauses and turns, each with one speaker. The request asks for them, and a word takes the speaker of the utterance its own start time falls inside, so nothing depends on the two lists being the same length or in step. A word in no utterance keeps the flag it had rather than being swept into the nearest segment. Best-effort throughout: a response without utterances leaves every per-word flag exactly as it was, so an older account or a model that does not support them transcribes as before. The single-word heuristic stays for that case — it patches the same fault a word at a time. This does not recover a change Deepgram missed outright, and there were some: two speakers merged into one block where AssemblyAI separated them. Nothing on our side can. Four tests: the boundary moving to the end of the turn; a response with no utterances behaving as before; a word in a gap keeping its flag; and the report naming its source. hyperaudioSpeakerDebug() now says whether the speakers came from utterances or from per-word flags.
Taking each word's speaker from the utterance it fell in was wrong, and the debate audio showed it plainly: 374 words re-assigned, and whole turns merged — the host swallowed into the answer before it, two speakers the per-word flags had kept apart reduced to one. Deepgram's utterance segmentation runs straight through speaker changes when the pauses are short, which a six-way debate with crosstalk supplies constantly. It is coarser than their own per-word flags, so trusting it loses distinctions those flags had. What the segments are good for is WHERE a change belongs, not WHO is speaking. Every word-level change is kept, exactly as many as before, and each is nudged onto the nearest utterance edge within three seconds — an edge being a pause or a turn, so the label lands at the start of what someone said rather than four words in. A change with no edge near it stays put, and none may cross its neighbours: this can move a boundary, never remove one, and never merge two speakers. Tests cover the case that broke it: a single segment spanning both turns, from which both speakers must still emerge. The others fix the position of a change onto a pause, leave a change alone when no edge is near, and keep a response without utterances behaving exactly as before.
Both attempts to correct Deepgram's mid-sentence speaker placement failed against the audio that showed the fault, so the parser draws their boundaries exactly as they report them, as it did before this line of work. Taking each word's speaker from the `utterances` segment it falls in merged distinct people: 374 words moved and the host was swallowed into the answer before it. Their segmentation runs through a speaker change when the pause is short, so it is coarser than their own per-word flags. Moving each change onto the nearest utterance edge instead, keeping every change and correcting only its position, moved 16 of about 30 and changed nothing anyone could see. At roughly one utterance every nine seconds the edges are too sparse to land on, and where one is near it is generally mid-sentence too, being cut from the same diarization. `utterances` is no longer requested, since nothing reads it. The narrow single-word repair stays: it predates this and addresses a signature that is demonstrably theirs. What survives from the work is the real bug found on the way — each word's text now comes off the word itself — and hyperaudioSpeakerDebug(), which reports what their flags say. A test pins the decision, so anything that starts editing their speakers again has to come past it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Speaker-coloured caption exports, and a run of poster and transcription fixes found in real use.
Fixes #643
Fixes #644
Fixes #645
Fixes #646
Fixes #647
Fixes #648
Fixes #649
Fixes #650
Gate: 139 unit; 423 e2e passed, 1 skipped (the known WebKit OPFS skip).