# Kitchen Video QA Challenge - Draft

Status: upcoming  
Prize: $300  
Public page: https://builderr.ai/kitchen-video

## Task

Build an agent that answers structured questions from short fixed-camera kitchen or restaurant videos.

Input:

- 7-8 video clips, ideally 5-10 minutes each.
- A JSON file of questions for each clip.

Output:

- One JSON file with answers.
- One log file showing runtime, frame sampling, model/API calls, and estimated cost.

Example run:

```bash
python answer.py --videos ./clips --questions questions.json --out answers.json
```

## What The Questions Test

The task is not freeform video captioning. It is operational QA:

- Did a required step happen?
- Who did what, and when?
- How many visible objects or people were present?
- Was an item sealed, moved, spilled, left unattended, or handed off?
- Was a cap, hairnet, glove, or container visible?
- Was the question impossible to answer from the footage?

Answer types:

- yes/no
- multiple choice
- count
- timestamp
- duration
- short structured object
- not visible

## Dataset Split

- Public sample clips: 1-2 clips with questions and example answers.
- Public validation clips: optional clips with questions only.
- Hidden final clips: 4-5 clips that are rights-cleared, anonymized, and not discoverable online.

Public examples may use public fixed-camera footage. Hidden scoring should use sponsor-owned or rights-cleared footage, with faces, payment screens, receipts, phone numbers, and order IDs blurred or excluded.

## Budget

Each submission must run within a fixed budget. Draft limits:

- 20 minutes wall-clock for the full eval.
- $3 model/API equivalent maximum, if external APIs are allowed.
- 1 frame per second average, or an equivalent fixed frame budget.
- One reproducible command.
- No manual inspection during evaluation.

Submissions that exceed the hard budget are ineligible.

## Scoring

Total: 100 points.

- 65 points - answer correctness.
- 15 points - budget efficiency.
- 15 points - hidden generalization across layouts, lighting, camera quality, and cuisines.
- 5 points - reproducibility and auditability.

Timestamp answers:

- Full credit within 2 seconds.
- Partial credit within 5 seconds.
- Zero beyond that, unless the event span is long and the rubric says otherwise.

## Baseline

Before launch, builderr should run a simple baseline:

1. sample frames at 1 fps,
2. run a common vision-language model or video model,
3. run OCR where relevant,
4. answer each question independently,
5. write structured JSON.

The target baseline score should be roughly 35-50%. If it is above 70%, the task is too easy. If it is below 25%, the questions are probably too ambiguous.

## Current Video Leads

These are research/sample leads, not final hidden data:

- Rail Drishti - Railway Kitchen Live CCTV video: https://www.youtube.com/watch?v=LYRWZ_Gf6_E
- Jane at the Marketplace restaurant surveillance video - full raw: https://www.youtube.com/watch?v=FsKal7K2zJY
- Commercial Kitchen Breakfast / Opening / Prep: https://www.youtube.com/watch?v=pIsRSX0C4jo
- KitchenGuard - AI for Compliance Monitoring in Restaurants: https://www.youtube.com/watch?v=AcwKckfiVMs
- Bon Appetit - 11 cameras in NYC's busiest brunch restaurant: https://www.youtube.com/watch?v=eKQWVkrJyzE
- Bon Appetit - 12 cameras in a tiny restaurant kitchen: https://www.youtube.com/watch?v=j5pcpuTWP6U
- Bon Appetit - 13 cameras in New York's busiest ramen restaurant: https://www.youtube.com/watch?v=iIi73dlsay8
- Bon Appetit - 19 cameras in a Michelin-starred restaurant: https://www.youtube.com/watch?v=klfxQuXT66s
- Security Camera B&W Restaurant: https://www.youtube.com/watch?v=Phl0xV-MQrM
- Elevated kitchen fire: CCTV footage: https://www.youtube.com/watch?v=1ORc1JPm3n0
- LIVEFEED Fauna - 3 hours of real kitchen service: https://www.youtube.com/watch?v=2pUQ8wZcFec
- Kaggle Kitchen Video in Restaurants: https://www.kaggle.com/datasets/naoamscoltd/kitchen-video-in-restaurants
- COM Kitchens reference dataset: https://www.nii.ac.jp/dsc/idr/en/rdata/COM_Kitchens/
- EPFL Smart Kitchen reference dataset: https://github.com/amathislab/EPFL-Smart-Kitchen

Public URLs cannot be hidden final eval clips unless rights are cleared and the eval clips are held back from participants.