Build guide
Build a local, private, multilingual Wispr Flow-style dictation tool.
How to build dictation that feels like Wispr Flow, but runs on a normal laptop with no cloud. Round 1 scores Hindi+English: land a fast, faithful final within about 2 seconds of stopping, and keep both languages as spoken.
← See the challenge spec, benchmark & how it's scoredA repo that implements draft(audio_buffer, is_final) in solution/draft.py, plus the models you used and their licenses.
No cloud, no API keys, no paid services — it's scored with the network off. Your code stays yours.
One combined score: final meaning and facts 70 points, final paste latency 30 points — and your spot on the board.
Your first 30 minutes (zero to a scored engine)
The whole loop runs offline on the sample clips that ship with the template:
git clone https://github.com/builderr-ai/builderr-speech-to-text cd builderr-speech-to-text pip install -r requirements.txt python preview.py # transcribes the sample clips, prints your result
- Read the getting-started guide and the build skill — the high-level architecture to follow (it's the recipe, not boilerplate to copy).
- Fork the template. It already wraps a strong open model — that's your starting point, and it already gets plain English about right.
- Run
python preview.pyon the included sample clips (English + Hindi+English). Everything is offline. - Now spend your time where the prize is: keep both Hindi and English exactly as spoken instead of translating everything to English (see below).
- Email your repo + your model licenses to submit@builderr.ai. We clone it, run it offline on the hidden set, and you land on the board. Revise any time before the cutoff.
You can't win on English alone. Off-the-shelf models can handle plain English, but they usually fail when Hindi and English are mixed. The leaders already beat the open-source baselines we tested; the prize bar is RambleFix on the hidden run. The reference engine is in the reference-bot write-up.
What you're optimizing for — and what this is NOT
- A cloud-API wrapper — it's scored with the network off, on one pinned laptop (Apple M1 Pro · macOS).
- An English-only transcriber — that ties the free tools and ranks below the benchmark.
- A translator — turning “mujhe ek meeting schedule karni hai” into English loses the point.
- A research model you can't ship — non-commercial licenses can't become a free product.
- Local + offline — runs on a laptop, no internet, nothing leaves the machine.
- Faithful to Hindi+English — writes what was actually said.
- Fast — a clean final lands within ~2s of you stopping (we measure this — see the streaming contract).
- Shippable — commercial-friendly models, so the winner can be released for free.
Two interfaces — one easy, one for the win
Batch option: implement solution.transcribe(wav_path) → return the text. That alone puts you on the board for accuracy. Test it on your own recordings, in your own languages.
Real-time (where the speed score is won): also expose a simple function that takes audio as it comes in — no servers, no networking. We do the heavy lifting: our harness plays real speech into your engine at real time, on the single pinned scoring machine (Apple M1 Pro · 32 GB · macOS · accelerator on · network blocked), warmed up, and measures how fast a clean final lands after you stop (target ~2s). We score that final — not the live preview while you're still talking. The exact shape, latency targets and weights are in the streaming contract.
Fork the template → — it ships with a working reference wrapper + sample clips, so you start from something that already runs.
Where the prize is: Hindi+English
Plain English is mostly solved by the base model. Your score is won or lost on code-switching — when someone slides between Hindi and English mid-sentence. A few directions builders take:
Decode as audio arrives, finalize only the risky tail
the winning shapeThe architecture that wins both speed and quality: decode the easy parts as the audio comes in, run your heavier Hindi+English finalizer only on uncertain bits, and normalize on stop — so a clean final lands fast.
Start from a strong multilingual base
start hereOpen ASR models like Whisper large-v3 / faster-whisper already handle accented English and a lot of Hindi. That's your floor. The work is stopping them from translating or dropping the Hindi — not training from scratch.
Stop it translating the Hindi
biggest winThe open engines quietly translate Hindi into English and 'lose it'. Force transcription mode, pin the task to verbatim, and keep Hindi words as they were said (Roman or Devanagari as the reference expects). This single fix is the biggest score jump.
Finish the final quickly
Keep the model warm, reuse encoded audio where possible, and run heavier cleanup only where the final needs it. Only time after the user stops is scored; first-word speed is not.
A light language-aware post-step
optionalA small pass that fixes common Hindi+English boundaries, numbers, and work jargon can improve output without retraining the acoustic model. Keep it light so it doesn't overfit the samples.
Common traps (good to know going in)
- Translating instead of transcribing. Aces a translation demo, fails the gate here. Keep the actual words.
- Any network call. Scored offline — a single cloud request and the run fails. Pin all weights locally.
- Non-commercial model license. The whole point is a shippable free tool — declare licenses, and they must be commercial-friendly.
- Hallucination / repetition loops. Long or empty audio makes weak setups repeat gibberish. Cap and guard it.
- Overfitting the sample clips. The hidden set is different — tune for faithful Hindi+English, not the exact samples.
The bar to beat: keep both Hindi and English, land the final fast, and beat RambleFix on the hidden run. Full table + scoring on the challenge page.