How to Use WhisperX: The Beginner's Guide (First Transcription, the Diarization Lesson, and the HF-Token Lesson)

CodexFuse Team · 2026-11-06T07:21:01.467Z · 21 min read min read

How to Use WhisperX: The Beginner's Guide (First Transcription, the Diarization Lesson, and the HF-Token Lesson)

If you want to learn how to use whisperx, the short answer is that you install the Python package, run one command on an audio file, and get back a transcript with word-level timestamps, then add speaker labels with one more flag when you are ready. WhisperX is Max Bain's free, open-source Python library for fast automatic speech recognition with word-level timestamps and speaker diarization, built on OpenAI's Whisper models through the faster-whisper backend. This whisperx tutorial walks the whole arc in order: install, first transcription, word timestamps, the diarization lesson with the Hugging Face token, and the Python API.

Two names to get straight before anything else. First, WhisperX is not OpenAI's Whisper: it is a separate open-source project by Max Bain, released under the BSD-2-Clause licence, that builds on OpenAI's Whisper model weights through the faster-whisper backend. Second, WhisperX Assistant is a different product entirely: it is a Visual Studio Code extension by mwhesse that turns your speech into text inside the editor, and it has nothing to do with the m-bain/WhisperX transcription library this guide covers.

This guide is about m-bain's WhisperX only, and every command, version, and number below comes from WhisperX's own docs and the sources listed at the end, checked on 6 November 2026. Before you install anything from the internet, build one safety habit: if you download a tool, a script, or an installer from a third-party mirror, check the file first with a SHA-256 hash generator before you run it. And if you plan to do your transcription on a machine you control rather than your laptop, a VPS hosting plan gives you a server of your own, which is the kind of setup that GPU work needs.

How to Use WhisperX in 5 Steps

If you only need the shape of the whole job, here it is in five steps. Each step has its own section below with the exact commands and the mistakes to avoid.

1. Install WhisperX with pip install whisperx on Python 3.10 to 3.13, after installing the CUDA toolkit 12.8 if you have a GPU.
2. Run your first transcription with whisperx interview.wav and read the transcript it writes.
3. Ask for a bigger model when accuracy matters, such as whisperx interview.wav --model large-v2 --batch_size 4.
4. Add word-level timestamps with the alignment pass and --highlight_words True for subtitle-ready output.
5. Label the speakers with whisperx interview.wav --model large-v2 --diarize --hf_token "$HF_TOKEN" after accepting the pyannote model terms on Hugging Face.

What Do You Need Before You Start?

You need Python 3.10, 3.11, 3.12, or 3.13, because the current release supports Python 3.10 to 3.13 only and Python 3.14 is not supported yet. You also need pip, a terminal, and an audio file to transcribe, plus disk space for model weights that run to several gigabytes for the large models. If you have an NVIDIA GPU, install the CUDA toolkit 12.8 before WhisperX, because the GPU path needs it first.

You also need ffmpeg on your system, since WhisperX's README points to the OpenAI Whisper setup instructions that require it, even though the faster-whisper backend itself decodes audio without a separate ffmpeg install. If ffmpeg is new to you, our FFmpeg beginner guide walks through the install on every platform. And if you plan to use speaker diarization, you need a free Hugging Face account, because the diarization model is gated behind accepted terms and a read token.

One honest note about hardware: large-v2 needs less than 8 GB of GPU memory with beam_size=5, according to the README, but bigger batches and bigger alignment models need more. If you have no GPU at all, you can still run everything in this guide on the CPU, only much more slowly, as the CPU section below shows.

How Do You Install WhisperX?

A whisperx install takes one command for most beginners: pip install whisperx. Install it inside a fresh virtual environment so its pinned dependencies, including torch 2.8, do not clash with your other projects.

If you have an NVIDIA GPU, install the CUDA toolkit 12.8 first, on Linux from NVIDIA's install guide or on Windows from NVIDIA's CUDA 12.8 download archive, and skip that step entirely if you only use the CPU. After installing, check the version with whisperx --version, which should report 3.8.6, the current stable release from 25 May 2026.

Do not install a yanked or unofficial version: PyPI versions 3.1.1 through 3.1.6 are yanked unofficial third-party releases, 3.3.2 was yanked for dependency issues, and 3.8.3 is yanked, so a plain pip install whisperx plus the version check is the safe path. A pre-release, 3.8.7rc1, exists on PyPI from 26 June 2026, but it is not a stable release and this guide does not use it.

If you want the fastest alignment step for long files, add Numba with pip install "whisperx[fast]", which the README says accelerates forced alignment by 30 to 100 times on GPU. If you would rather skip installing anything, the uvx form runs it without a permanent install: uvx whisperx.

Which Model Should You Start With?

Start small, then go big only when the transcript needs it. The CLI default model is small, not large-v2, so a first run with no --model flag uses small and will look less accurate than the large-v2 benchmarks quoted in this guide and elsewhere.

Whisper ships in six sizes, from tiny at 39 million parameters to large at 1,550 million, and large-v2 is the large-size model the WhisperX README's speed claim uses. That makes large-v2 about 6.35 times the size of small and about 39.74 times the size of tiny, which is why it also needs about 10 GB of video memory against small's 2 GB. The table below lists every size with its memory need and relative speed, measured by OpenAI on an A100.

For a first transcription, small is fine, because it downloads fast and runs on modest hardware. When accuracy matters, switch to large-v2 with --model large-v2, and when you need speed on a smaller GPU, turbo is an optimised large-v3 that runs about 8 times faster than large at roughly half the parameters.

How Do You Run Your First Transcription?

A first transcription is one command: whisperx interview.wav. WhisperX detects the language, transcribes the file with the small model by default, and writes its outputs into the current directory.

By default you get every output format at once: the .srt and .vtt subtitle files, a plain .txt transcript, a .tsv table of segments, and a .json file that carries the per-word timings and speaker fields. The table below says what each file is for, so you can ignore the ones you do not need.

The --output_format flag accepts all, srt, vtt, txt, tsv, json, and aud, and the default all writes the five formats in the table. Add --highlight_words True to underline each word as it is spoken in the .srt and .vtt files, which is what makes the subtitles feel synced. To transcribe in another language, pass its code, such as --language de for German, and to translate speech into English instead of transcribing it, pass --task translate, because transcription is the default and translation is a separate task.

How Do You Get Word-Level Timestamps?

WhisperX gets whisperx word timestamps from a second pass called forced alignment: after Whisper writes the text, a phoneme-based wav2vec2 model snaps each word to its exact moment in the audio. That is the difference between subtitles that feel synced and ones that drift, and VexaScribe's guide puts the result at roughly 30 milliseconds of accuracy per word. OpenAI's Whisper alone only gives utterance-level timestamps that the WhisperX README says can be inaccurate by several seconds.

You do not have to ask for alignment: it runs by default, and the default English alignment model is WAV2VEC2_ASR_LARGE_LV60K_960H when you pass a bigger model. To name the alignment model yourself, pass --align_model WAV2VEC2_ASR_LARGE_LV60K_960H, and to control how words that cannot be aligned are handled in subtitles, use --interpolate_method nearest, linear, or ignore. Pass --no_align only when you want segment timestamps and no word timings at all.

The alignment model is language-specific, and WhisperX ships with built-in models for 41 languages: 5 through torchaudio pipelines, including English, French, German, Spanish, and Italian, and 36 more through Hugging Face. If your language is not in that list, the README says to find a phoneme-based ASR model on Hugging Face and test it on your data. And one limit to know: words with characters outside the alignment model's dictionary, such as 2014. or £13.60, cannot be aligned and get no timing at all.

How Does Speaker Diarization Work?

Speaker diarization answers who spoke when: it splits the audio into turns and labels each one SPEAKER_00, SPEAKER_01, and so on, which are anonymous IDs, not real names. WhisperX bundles the pyannote diarization pipeline behind a single flag, so one command adds the labels to the same transcript.

The whisperx speaker diarization lesson starts before the command, with the Hugging Face token. Create a free Hugging Face account, open the pyannote/speaker-diarization-community-1 model page, accept its user conditions, and create a read token at huggingface.co/settings/tokens. Without the accepted terms, the gated model will not download and --diarize fails even though plain transcription works fine, and you should never publish or share your token.

Then run whisperx interview.wav --model large-v2 --diarize --hf_token "$HF_TOKEN", passing the token through the whisperx hf_token flag instead of typing it into the command where it could end up in your shell history. If you know how many speakers are in the recording, say so with --min_speakers 2 --max_speakers 2, because a known speaker count sharpens the result.

The default diarization model is pyannote/speaker-diarization-community-1, not the older 3.1 pipeline, and the community model beats 3.1 on nearly every benchmark the Hugging Face page reports. The table below shows the diarization error rate, where lower is better, for both models, and the worked numbers after it show how big the gains are.

AliMeeting improves the most, from 24.5 to 20.3, a drop of 4.2 points or 17.14 percent in relative terms, while AMI SDM drops 2.8 points and AMI IHM drops 1.8 points. Two rows break the pattern, and they are worth knowing: VoxConverse is unchanged at 11.2 for both models, and REPERE phase 2 is worse on the community model, rising from 7.9 to 8.9. Diarization is far from perfect, in the README's own words, so treat the labels as a strong first draft and check them on any recording that matters.

WhisperX vs Faster-Whisper vs OpenAI Whisper

WhisperX, faster-whisper, and OpenAI Whisper share the same model weights but do different jobs, and picking the wrong one is the most common beginner mistake. OpenAI Whisper is the original model and library: audio in, text out, with segment timestamps that can drift by seconds and no native batching. Faster-whisper is a reimplementation of the same model on the CTranslate2 engine, up to 4 times faster than openai/whisper for the same accuracy while using less memory.

WhisperX sits on top of faster-whisper and adds the three things the other two do not give you in one place: VAD-based chunking with batched inference, wav2vec2 forced alignment for word-level timestamps, and built-in pyannote speaker diarization. The speed comes mostly from the batching: the README claims 70x realtime with large-v2, so a 60-minute recording finishes in about 51.43 seconds of processing and a 90-minute podcast in about 77.14 seconds, while a 10-hour archive takes about 514.29 seconds, or 8.57 minutes.

The faster-whisper benchmark table makes the engine difference concrete: on 13 minutes of audio with large-v2, openai/whisper takes 2 minutes 23 seconds and 4,708 MB of video memory, faster-whisper takes 1 minute 3 seconds and 4,525 MB, and faster-whisper with batch_size=8 takes 17 seconds and 6,090 MB. That is 8.41 times faster than openai/whisper for the batched run, at the cost of 1,382 MB more video memory, a 29.35 percent increase. If you only need a plain transcript, our Whisper beginner guide covers OpenAI Whisper on its own, and the whisperx vs faster-whisper choice is simple: take WhisperX when you need word timings, speaker labels, or both.

One honest limit from a production guide: WhisperX does not improve the transcript itself, because it uses the same Whisper weights, so if you do not need timings, labels, or batch speed, plain Whisper is simpler. The worked numbers in the code block below recompute every speed and diarization figure in this article from the source tables, so you can check the arithmetic yourself.

# Numbers checked 6 November 2026 from the WhisperX README,
# the faster-whisper benchmark table, and the pyannote
# speaker-diarization-community-1 benchmark table.
from datetime import date
print("60-min audio at 70x realtime: %.2f s" % (60 * 60 / 70))
print("90-min podcast at 70x realtime: %.2f s" % (90 * 60 / 70))
print("10-hour audio at 70x realtime: %.2f s" % (10 * 3600 / 70))
print("13-min audio, openai/whisper 143 s to faster-whisper batch 17 s: %.2fx faster" % (143 / 17))
print("13-min audio, openai/whisper 143 s to faster-whisper 63 s: %.2fx faster" % (143 / 63))
print("VRAM for the batched run: 4708 to 6090 MB, %+d MB (%+.2f%%)" % (6090 - 4708, (6090 - 4708) / 4708 * 100))
print("large-v2 1550M vs small 244M parameters: %.2fx" % (1550 / 244))
print("large-v2 1550M vs tiny 39M parameters: %.2fx" % (1550 / 39))
print("DER AliMeeting: 24.5 to 20.3, %.1f points (%.2f%%)" % (20.3 - 24.5, (20.3 - 24.5) / 24.5 * 100))
print("DER AMI SDM: 22.7 to 19.9, %.1f points (%.2f%%)" % (19.9 - 22.7, (19.9 - 22.7) / 22.7 * 100))
print("DER REPERE phase 2: 7.9 to 8.9, %.1f points (%.2f%%) - worse" % (8.9 - 7.9, (8.9 - 7.9) / 7.9 * 100))
print("Days since WhisperX 3.8.6 (25 May 2026): %d" % ((date(2026, 11, 6) - date(2026, 5, 25)).days))
# Expected output:
# 60-min audio at 70x realtime: 51.43 s
# 90-min podcast at 70x realtime: 77.14 s
# 10-hour audio at 70x realtime: 514.29 s
# 13-min audio, openai/whisper 143 s to faster-whisper batch 17 s: 8.41x faster
# 13-min audio, openai/whisper 143 s to faster-whisper 63 s: 2.27x faster
# VRAM for the batched run: 4708 to 6090 MB, +1382 MB (+29.35%)
# large-v2 1550M vs small 244M parameters: 6.35x
# large-v2 1550M vs tiny 39M parameters: 39.74x
# DER AliMeeting: 24.5 to 20.3, -4.2 points (-17.14%)
# DER AMI SDM: 22.7 to 19.9, -2.8 points (-12.33%)
# DER REPERE phase 2: 7.9 to 8.9, 1.0 points (12.66%) - worse
# Days since WhisperX 3.8.6 (25 May 2026): 165

Running it prints 51.43 s for the 60-minute file, 77.14 s for the 90-minute podcast, and 514.29 s for the 10-hour archive, plus 8.41x faster and 2.27x faster for the two faster-whisper runs and +1382 MB (+29.35%) of extra video memory. It also prints 6.35x and 39.74x for the model sizes, -4.2 points (-17.14%) and -2.8 points (-12.33%) for the two diarization gains, 1.0 points (12.66%) worse on REPERE, and 165 days since the 3.8.6 release.

How Do You Speed Up WhisperX?

WhisperX is already fast, but three levers make it faster still, and all three trade something for speed. Raise the batch size with --batch_size 16, which packs more audio into each GPU pass, and lower it to 4 when video memory runs short. Switch the compute type to int8 with --compute_type int8, which cuts video memory by about a third on the faster-whisper benchmark, 2,926 MB against 4,525 MB for fp16, at some cost to accuracy.

Add Numba with pip install "whisperx[fast]" for the alignment step, which the README says runs 30 to 100 times faster with JIT-compiled loops. Turn on interleaved context with --interleaved_context when your audio has technical language, because it re-runs the first batch with wrap-around context for better punctuation and proper nouns.

How Do You Use WhisperX in Python?

The whisperx python api follows the same three steps as the CLI, only in code: transcribe, align, then diarize. Load the model with whisperx.load_model("large-v2", device, compute_type="float16"), read the audio with whisperx.load_audio(audio_file), and transcribe it with model.transcribe(audio, batch_size=16), which returns segments before alignment.

Then load the alignment model with whisperx.load_align_model(language_code=result["language"], device=device) and align the segments with whisperx.align(result["segments"], model_a, metadata, audio, device). The README's example also passes interleaved_context=True to the transcribe call, which keeps context flowing between segments for better punctuation and proper nouns.

Finally, build the diarization pipeline with DiarizationPipeline, passing your Hugging Face read token when you build it, run it on the audio to get the speaker segments, and merge the labels into the transcript with whisperx.assign_word_speakers(diarize_segments, result). Each segment then carries a speaker field alongside its text and word timings, which is the same output the CLI writes to JSON. Delete each model from GPU memory with del model plus torch.cuda.empty_cache() when you are short on video memory, as the README's example shows.

How Do You Run WhisperX on CPU or a Mac?

Run WhisperX on the CPU with whisperx interview.wav --compute_type int8 --device cpu, which is also the path for running on Mac OS X. The int8 compute type keeps memory low, at the cost of some accuracy, and it is the setting the README recommends when video memory is tight.

CPU mode works, but it is much slower, so do not promise yourself GPU speed on a CPU: the faster-whisper benchmark for the small model on CPU takes 6 minutes 58 seconds with openai/whisper, 2 minutes 37 seconds with faster-whisper, and 1 minute 6 seconds with faster-whisper batched, which is 6.33 times faster than the reference run but still far from the 70x realtime the GPU claims. A community fork, whisperx-mlx, ports WhisperX to Apple Silicon through MLX for M1, M2, and M3 Macs, if you want a native Mac path.

If you have no GPU and no patience for CPU speeds, try WhisperX without installing anything through the Replicate demos the README links, which cover large-v3, large-v2, and medium. And if running models locally is becoming a habit, our Ollama beginner guide shows the same local-first pattern for large language models.

What Are WhisperX's Limits?

WhisperX is not a live-transcription tool: it transcribes finished audio files, so live audio needs a streaming speech recogniser instead. Overlapping speech is not handled particularly well by Whisper or WhisperX, and words with characters outside the alignment dictionary get no timing, so a crosstalk-heavy panel recording will always need a human pass.

The README is blunt that diarization is imperfect, and the labels are anonymous IDs, so a two-speaker interview can still come back with a third phantom speaker or a merged one. Subtitle .ass output was removed in version 3, and each language needs its own alignment model, so a language outside the 41 built-in models needs a Hugging Face model you find and test yourself.

CodexFuse does not offer a WhisperX tool, and this guide does not promise one: it teaches you to run the real thing on your own machine. If the limits above rule WhisperX out, the honest alternatives are plain Whisper for a simple transcript, faster-whisper alone for speed without diarization, or a hosted transcription service when you would rather pay than install.

Sources

• WhisperX README, m-bain/WhisperX on GitHub — install, CLI usage, Python API, limitations, and the 70x realtime claim, fetched 6 November 2026
• PyPI JSON API for whisperx — version 3.8.6, release dates, Python requirement, and dependencies, fetched 6 November 2026
• GitHub API for m-bain/whisperX — 25,108 stars, 2,533 forks, creation date, and last push date, fetched 6 November 2026
• faster-whisper README, SYSTRAN/faster-whisper on GitHub — benchmark table for 13 minutes of audio and the 4-times-faster claim, fetched 6 November 2026
• Whisper README, openai/whisper on GitHub — model sizes, video memory, relative speeds, and ffmpeg setup, fetched 6 November 2026
• pyannote/speaker-diarization-community-1 model page on Hugging Face — setup steps and the diarization error rate table, fetched 6 November 2026
• VexaScribe, pyannote.audio Guide, Open-Source Speaker Diarization Pipeline 2026 — word-timestamp accuracy and the WhisperX versus Whisper comparison, fetched 6 November 2026
• Bain, Huh, Han, and Zisserman, WhisperX: Time-Accurate Speech Transcription of Long-Form Audio, INTERSPEECH 2023, arXiv:2303.00747 — cited in the WhisperX README

Frequently Asked Questions

Is WhisperX free?

Yes. WhisperX is free and open source under the BSD-2-Clause licence, so you can install it, use it, and change it without paying the project anything. You pay only for your own hardware and electricity, and the Replicate demos the README links are third-party services with their own terms.

Do I need a GPU to use WhisperX?

No. WhisperX runs on the CPU with --device cpu and --compute_type int8, which is also how it runs on a Mac, but it is much slower than the GPU path and the 70x realtime figure does not apply there. A GPU is strongly recommended for long recordings or batch work, and the large-v2 model needs less than 8 GB of video memory at beam_size=5.

What is the difference between WhisperX and Whisper?

OpenAI Whisper is the original model and library: audio in, text out, with segment-level timestamps. WhisperX is a separate project that runs the same Whisper weights through the faster-whisper backend and adds VAD-based batching, wav2vec2 word-level alignment, and built-in pyannote speaker diarization. The transcript itself is no more accurate than Whisper's, because the weights are the same.

Why does --diarize fail with a token or 401 error?

The diarization model is gated, so three things must line up: a Hugging Face account, the user conditions accepted on the pyannote/speaker-diarization-community-1 model page, and a read token from that same account passed with --hf_token. Accepting the terms with one account and using another account's token fails, and the token must stay private. If you switched the diarization model to 3.1, accept both its page and the segmentation-3.0 page, because 3.1 needs two acceptances.

Is WhisperX the same as the WhisperX Assistant extension?

No. The extension is a voice-to-text coding tool for Visual Studio Code and Cursor by a different author, mwhesse, which records your microphone and pastes the text into the editor. The library in this guide is m-bain's WhisperX, a Python transcription pipeline with word timestamps and speaker diarization, and the two projects share only part of a name.