[Feature Request] Add automatic voice cloning and the ability to return to the dubbing phase.

[invio-a11y]

The program is very comprehensive, but in my opinion, two key features are missing:

1 - the ability to automatically clone the vocal timbre of individual sentences: other free programs like Pyvideotrans and Voicestudio don't need to load sample voices to generate the clone, while in Subtitle Edit, in addition to loading the sample voice, you then have to manually edit who says what. I think this is feasible, since others are already doing it without any problems.

2 - Once the dubbed audio is generated, in Subtitle Edit you have to edit and possibly re-dub the individual sentence in the same session. From what I understand, you can't exit that phase, otherwise you have to start from scratch. And this is a problem with long videos, because the work can't necessarily be finished in one sitting.

The rest of the program is fantastic, very fast, and above all professional.

I hope these two improvements can be incorporated, as they would significantly improve production.

[niksedk]

Thanks for the detailed feedback!

2 - returning to the dubbing phase: this already works today, it's just not named well. After generating, the review window has an Export... button - it writes SubtitleEditTts.json plus a wav folder with all generated audio. Next session, open "Text to speech" again and use Import... to pick that json: you get every line back with its audio, its engine/voice/speed, the include check boxes and the actor->voice cast mapping, and you can re-generate individual lines from there. So a long video can be done over several sittings. If the "Export/Import" wording is what made this hard to find, that's fair feedback - better names welcome.

1 - automatic voice cloning: a few questions so we build the right thing:

• Do you mean automatic speaker detection (the tool finds each speaker and clones them separately), or just a faster way to clone from the video you already have open - e.g. select a region in the waveform, right click -> "Clone to <TTS engine>"?
• In pyvideotrans/VoiceStudio, do you get one cloned voice per detected speaker, or is each sentence cloned from its own audio?
• Which language(s), and roughly how many speakers, do your videos have? And which TTS engine are you using in Subtitle Edit today?

Using subtitleformat "Advanced Sub Station Alpha" or "WebVTT" is probably required for this to work as these formats support actor/voice.

[invio-a11y]

I mean this: if I upload a video, for example, in English with two or three people (or more), other programs automatically clone the individual voices for each individual sentence thanks to zero-shot. Basically, before dubbing the individual sentence, the model also analyzes the timbre of the individual sentence's voice. It's very convenient, and with the current level of certain cloning (such as Omnivoice and others), the results are astonishing. Instead, imagine uploading a sample audio: apart from finding the audio, cutting it, importing it, giving it a name that ends up in a long list of names, then once the cloning starts, the same voice will be present throughout the video, and, manually, you will have to edit and redub the individual voices of the other characters in the video (that is, repeat the whole process again: finding the voice, extracting it, cutting it, importing it, giving it a name that adds to the already long list, etc. etc.). In short, I use a lot of programs, and it's truly a unique convenience that helps you produce more (and also makes you want to work more, because doing everything by hand honestly takes away a bit of the desire). I'm passionate about subtitling and dubbing, but I'm not a technician, but I can contribute my experience to improve my work; that's what I can do. I've been trying out every model for years, and I've even tried many paid ones. Lately, I've become passionate about dubbing. In my opinion, Subtitle Edit is the best when it comes to subtitling, interface, convenience, professionalism... and I've seen it's also expanding to dubbing: the models work very well, they're very fast, easy to download, etc., but in my opinion, this automatic cloning feature (the template recognizes phrase by phrase which stamp to clone) would be a turning point that would take the program to another level. I repeat, other models already do it, but the interface and completeness of Subtitle Edit is unique, so trust me, it could easily become the top software. The others have major shortcomings in terms of interface, subtitle functionality, etc., etc.

I downloaded all the models with cloning on Subtitle Edit to test, and none of them have the automatic cloning feature (it's actually a command (request) that needs to be added to Subtitle Edit on the software side, since the templates are already ready for this feature). In fact, the exact same templates, on the other software mentioned, work very well with automatic recognition of the timbre of the individual sentences to be dubbed, precisely because they were designed for this purpose. It's the software side that needs to be configured to do this. But again, I'm not a techie, but I've seen that it can be done.

I'm mainly using these templates to translate English videos into Italian.
Thank you so much for your valuable efforts!