Method

How we turn a messy problem into something you can score.

Most real problems arrive as a single sentence. “Watch my kitchen.” “We need better company data.” Below are the seven questions we ask before any of it becomes a challenge, and four problems that went through it, with the scoring we actually published.

Every example links to the live challenge page it came from.

The seven questions

If a problem fails one of these, it is not ready to be scored yet.

What is the one number?

One primary number and at most one guardrail. If you need a weighted index of five things, you have five problems, not one.

Fails as: “Improve kitchen operations.” Nothing here can be counted.

Where do the right answers come from?

Name the thing you compare against, and name its gap out loud. An answer key someone wrote. A set of records. A panel of judges with a stated agreement rate.

Fails as: “The AI decides if it did a good job.” Then the AI is marking its own work.

What already happens without any of this?

The starting score to beat: what a person does today, or the simplest script that could work. A result only means something next to the thing it beat.

Fails as: “85% accuracy.” With no starting score this means little: if guessing the most common answer gets 80%, 85% is a small gain.

How would someone cheat this, and what stops them?

Write the cheat down first. Then write the guard. Every measure gets gamed the moment it becomes a target, so the guard is part of the design, not a patch.

Fails as: No cheat listed. That means nobody has looked yet.

What does “I don’t know” look like?

There has to be a way to answer honestly when the evidence is not there, and it has to score better than a confident guess. Otherwise you are paying for made-up answers.

Fails as: Every question must get an answer. You will get made-up answers and score them as correct.

Is there enough of it to tell two entries apart?

Count the units, then work out the smallest gap you could actually see. Publish the result as a range, not a position: at 60 items a score of 70% is really somewhere around 58% to 82%, so places two through six are not separable and should be shown as a band. Score everyone on the same items.

Fails as: “Did we close the deal?” Four deals a quarter cannot separate two solutions.

What does a do-nothing submission score?

Run the lazy entries yourself before anyone else does. Answer “don’t know” to everything. Answer “yes” to everything. Dump the whole input back. Then push a known-perfect answer through and check it scores full marks. Then deliberately break something and check the score drops. Five cheap runs, and they anchor every number on the board.

Fails as: Nobody has run them. Then you do not know what a high score means: a do-nothing entry may already score well.

Four that went through it

The sentence we got, and the scoring we ran.

“I want AI to watch my kitchen and tell me when something goes wrong.”

Became: Answer hidden questions about kitchen CCTV, with the timestamp you got it from.

The one number: 80 of 100 points for correct answers against a hidden key, across yes/no, counts, event order, timestamp, duration and not-visible questions.
Right answers from: An answer key written by a person watching the footage first. The questions are not published.
Guardrail: A hard cap of $0.30 estimated model and API cost per 60 minutes of source video. Cost only counts once the answer is already valid, so nobody wins by being cheap and wrong.
The cheat, and the guard: Guess “yes” on every yes/no question. Guarded by mixing in counts, ordering and duration questions that a guess cannot hit, and by requiring a supporting timestamp for each answer.
“I don’t know”: not_visible is a scored answer. Returning it when the footage genuinely does not show the moment beats inventing a timestamp.

Read the full challenge →

“There is no good company database for Norway.”

Became: Given a company number, find the public facts, show where each came from, and keep the profile current.

The one number: A 100-point score. 50 for how much checked information you found, 30 for whether it is correct, 12 for whether the summary is useful, and 8 for whether it is easy to verify. Refresh is a separate harness check.
Right answers from: Every submission pooled together, plus our own crawlers, de-duplicated and source-checked. The known gap is stated on the page: the set is only as good as what came in, and it is not a claim that we found everything online.
The bar: 65 out of 100 on an official run. Builderr supplies the current company batch after the cutoff and gives the same batch to every eligible entry.
The cheat, and the guard: Profile easy companies and stop. Guarded by a Builderr-supplied batch that can grow from 1,000 to 1,100, so covering your own list buys you nothing.
“I don’t know”: Every one of the 100 must return a result, including not_available, blocked and ambiguous. Missing values must never be filled with zero, and a material wrong-company match blocks qualification no matter how high the score.

Read the full challenge →

“Can an AI actually trade?”

Became: Percentage return on a $100,000 simulated account, measured forward from the entry’s first scored session.

The one number: Percentage return. $100,000 becoming $103,000 is 3%.
Right answers from: Real market prices, recorded afterwards.
The cheat, and the guard: Two of them. Peek at future prices, guarded by running forward only from the day you submit and by auditing the top three against the exact code that produced the result. Or take one enormous bet and hope, guarded by a 30% cap per name and 1.5x gross.
The bar: Beat the previous round’s winning entry. A leaderboard place is provisional until the code is verified.
What makes it honest: Bots start on different dates, so every result is published with its own window. Comparing two returns over different windows is the mistake the format is built to prevent.

Read the full challenge →

“Dictation is useless when I switch between Hindi and English.”

Became: A 100-point score on the text itself and on how fast that text is ready to paste.

The one number: 70 points for the final transcript. 30 points for how quickly that final is ready after you stop talking. Speed is scored, not assumed.
Right answers from: Transcripts of real clips, including the Hindi-heavy and fast-switching ones, which is where entries differed most.
The bar: Open-source baselines run on the same harness, and RambleFix, our own tool. The leading submissions already beat the open-source baselines on sample clips.
The cheat, and the guard: Hard-code the answers for the sample clips. Guarded by a pinned scoring machine that runs the final review offline, with the network blocked.
Constraint that shapes everything: It has to run on a normal computer, on Mac and Linux, using only models that are free to use in a real product. A cloud API that is 2% better does not qualify, because the winning tool has to be releasable.

Read the full challenge →

Why it is built this way

People game any score once they aim at it.

Bing once shipped a bug that made search results noticeably worse. Queries per user went up more than 10% and revenue per user went up more than 30%, because people who cannot find what they want search again. Both of those were top-level company goals.

That is why question 04 exists, and why every challenge here names the cheat before it names the guard. It is also why the guardrail is a separate number rather than a weight inside the main one. A weighted index lets a good score on one part pay for a bad score on another. A guardrail does not.

And it is why “I don’t know” is always a scored answer. If the only way to score is to produce an answer, you will get answers whether or not the evidence is there, and the scoring will call that a success.

Question 07 is the cheapest one on the list. Run the lazy entry yourself before anyone else does. Until you have, you do not know what your own top score means.

The deliberate-break run is the same idea pointed the other way. Move the thing being measured in a direction whose sign you already know, and check the score moves the way it should. When Bing did this across a corpus of past experiments, scored against human judgments of whether each change was actually good for users, it found that clicks on ads tracked user value in the negative direction. More engagement with ads meant a worse result. A measure nobody has ever pointed at a known-bad change is a measure nobody has tested.

Send us a problem that looks hard to measure.

Send it as one sentence. We will tell you which of the seven questions it fails and what we would need to make it scorable. If it cannot be scored, we will say that too.

Send us the problem →