I think a lot of people have heard of OpenAI’s local-friendly Whisper model, but I don’t see enough self-hosters talking about WhisperX, so I’ll hop on the soapbox:
Whisper is extremely good when you have lots of audio with one person talking, but fails hard in a conversational setting with people talking over each other. It’s also hard to sync up transcripts with the original audio.
Enter WhisperX: WhisperX is an improved whisper implementation that automatically tags who is talking, and tags each line of speech with a timestamp.
I’ve found it great for DMing TTRPGs — simply record your session with a conference mic, run a transcript with WhisperX, and pass the output to a long-context LLM for easy session summaries. It’s a great way to avoid slowing down the game by taking notes on minor events and NPCs.
I’ve also used it in a hacky script pipeline to bulk download podcast episodes with yt-dlp, create searchable transcripts, and scrub ads by having an LLM sniff out timestamps to cut with ffmpeg.
Privacy-friendly, modest hardware requirements, and good at what it does. WhisperX, apply directly to the forehead.
What would be some use cases for WhisperX? I’m struggling to envision how I would use that in a selfhosting/homelabbing environment.
half sarcastic but the overall premise of rigging something in to a local voice assistant, when an arguement starts “Ok nabu record this conversation”. then 2 weeks later on another arguement… “OK nabu search our last arguement for the cabinet”. Would be like having a court transcriber on call.
I have a lady friend that does quite a good enough job of that. LOL
‘You remember back in 1979…it was a Friday at 2:11 PM, and you said…’ ‘Babe, I don’t remember what I had for breakfast yesterday.’
I’m personally looking at setting up whisper or whisperx with bazarr, to get subtitles for movies and series that I can’t find any to download.
Long videos or voice notes where you’re usually just looking for a small snippet.
Now that’s an interesting angle. I am a mediocre musician on my best day, but sometimes I incorporate phrases and lyric snippits in a piece. I wonder if I could use WhisperX to find those words or phrases from a stack of songs. For instance, I did a piece that used a line from Jimi Hendrix’s ‘If 6 were 9’ where he says ‘I’m the one who’s gotta die when it’s time for me to die. So let me live my life the way I want to.’ I wonder if WhisperX could pick that out of a stack of Jimi Hendrix songs.
It might take a while, but when your PC is working on it you are not and searching for words might be easier ^^
I’m excited to hear how well it works ^^
I’m always excited to try new stuff. You never know. A use case might develop that you didn’t think of.
You should be able to get decent results if you pipe your tracks through demucs first to isolate the vocals.
https://github.com/adefossez/demucs
Vanilla whisper will probably be better than whisperX for that use case though.
Depending on how esoteric your music library is, you can also build a lyrics DB with beets: https://beets.readthedocs.io/en/stable/plugins/lyrics.html
I use UVR for vocal isolation. It just works, but that shouldn’t be a problem. I’ll check it out. At the worst, I’ll learn something.