Build guide

Build the best local Hindi + English speech-to-text.

How to build a dictation engine that runs offline on a normal laptop, gets the Hindi+English mix right, and lands a fast, faithful final — within about 2 seconds of you stopping, like Wispr Flow feels in real use, but local. No ML PhD needed — most of the work is wiring a strong open model correctly and making it fast.

← See the challenge spec, benchmark & how it's scored
2 minutes: why builders are entering — and what you walk away with.
What you give

A repo that implements draft(audio_buffer, is_final) in solution/draft.py, plus the models you used and their licenses.

What we never need

No cloud, no API keys, no paid services — it's scored with the network off. Your code stays yours.

What you get back

One combined score — ~60% accuracy + mix, ~35% speed of the final, ~5% reliability — and your spot on the board.

Your first 30 minutes (zero to a scored engine)

The whole loop runs offline on the sample clips that ship with the template:

git clone https://github.com/builderr-ai/builderr-speech-to-text
cd builderr-speech-to-text
pip install -r requirements.txt
python preview.py           # transcribes the sample clips, prints your scorecard
  1. Read the getting-started guide and the build skill — the high-level architecture to follow (it's the recipe, not boilerplate to copy).
  2. Fork the template. It already wraps a strong open model — that's your starting point, and it already gets plain English about right.
  3. Run python preview.py on the included sample clips (English + Hindi+English) and read your scorecard. Everything is offline.
  4. Now spend your time where the prize is: the Hindi+English mix — keeping what was actually said instead of translating it to English (see below).
  5. Email your repo + your model licenses to submit@builderr.ai. We clone it, run it offline on the hidden set, and you land on the board. Revise any time before the cutoff.

You can't win on English alone.Wrapping an off-the-shelf model aces English and flunks the mix — and the mix is the gate. The English bar is “tie the best free tools”; the points are in the mix. The exact thresholds and how the reference engine is built are in the reference-bot write-up.

What you're optimizing for — and what this is NOT

This is NOT
  • A cloud-API wrapper — it's scored with the network off, on one pinned laptop (Apple M1 Pro · macOS).
  • An English-only transcriber — that ties the free tools and ranks below the benchmark.
  • A translator — turning “mujhe ek meeting schedule karni hai” into English loses the point.
  • A research model you can't ship — non-commercial licenses can't become a free product.
It IS
  • Local + offline — runs on a laptop, no internet, nothing leaves the machine.
  • Faithful to the mix — writes what was actually said, keeping the meaning.
  • Fast — a clean final lands within ~2s of you stopping (we measure this — see the streaming contract).
  • Shippable — commercial-friendly models, so the winner can be released for free.

Two interfaces — one easy, one for the win

Batch option: implement solution.transcribe(wav_path) → return the text. That alone puts you on the board for accuracy. Test it on your own recordings, in your own languages.

Real-time (where the speed score is won): also expose a simple function that takes audio as it comes in — no servers, no networking. We do the heavy lifting: our harness plays real speech into your engine at real time, on the single pinned scoring machine (Apple M1 Pro · 32 GB · macOS · accelerator on · network blocked), warmed up, and measures how fast a clean final lands after you stop (target ~2s). We score that final — not the live preview while you're still talking. The exact shape, latency targets and weights are in the streaming contract.

Fork the template → — it ships with a working reference wrapper + sample clips, so you start from something that already runs.

Where the points actually are: the mix

Plain English is mostly solved by the base model. Your score is won or lost on code-switching — when someone slides between Hindi and English mid-sentence. A few directions builders take:

Decode as audio arrives, finalize only the risky tail

the winning shape

The architecture that wins both speed and the mix: decode the easy parts incrementally as the audio comes in, run your heavier verbatim/Hinglish finalizer only on the uncertain bits, and normalize on stop — so a clean final lands fast, instead of running one big slow model over the whole clip after you stop.

Start from a strong multilingual base

start here

Open ASR models like Whisper large-v3 / faster-whisper already handle accented English and a lot of Hindi. That's your floor. The work is stopping them from doing the wrong thing on the mix — not training from scratch.

Stop it translating the mix

biggest win

The open engines quietly translate Hindi into English and 'lose it'. Force transcription mode, pin the task to verbatim, and keep Hindi words as they were said (Roman or Devanagari as the reference expects). This single fix is the biggest score jump.

Chunk + voice-activity detection for speed

Split on silence (VAD) and stream chunks so short English feels instant. Latency is scored — a model that's accurate but slow to first word loses points it didn't need to.

A light language-aware post-step

optional

A small pass that fixes common Hindi+English boundaries, numbers, and work jargon can lift 'meaning' without retraining the acoustic model. Keep it rule-light so it doesn't overfit the samples.

Common traps (good to know going in)

  • Translating instead of transcribing. Aces a translation demo, fails the gate here. Keep the actual words.
  • Any network call. Scored offline — a single cloud request and the run fails. Pin all weights locally.
  • Non-commercial model license. The whole point is a shippable free tool — declare licenses, and they must be commercial-friendly.
  • Hallucination / repetition loops. Long or empty audio makes weak setups repeat gibberish. Cap and guard it.
  • Overfitting the sample clips. The hidden set is different — tune for the behavior (faithful mix), not the exact samples.

The bar to beat: RambleFix must be scored on the same hidden corpus as entrants before any prize decision. Its older OpenSLR-only check was useful evidence, but not the payout line. Match English, beat the mix, and land the final fast. Full table + scoring on the challenge page.

Enter the speech-to-text challenge →

Stuck? Join the Discord → or email inquiries@builderr.ai. We want you to succeed.