Kitchen video QA — $300 bounty · upcoming: answer operational questions from fixed-camera restaurant and kitchen clips. Dataset being prepared · How scoring works. Two useful views: the draft page (objective, prize, candidate clips) or the build guide (first implementation shape). See live challenges →
Upcoming challenge - fixed-camera video QA
Build an agent that answers questions from messy kitchen video.
The target is practical: a small restaurant or cloud kitchen already has camera footage and wants plain answers, not a demo. Who did what? Did a required step happen? Was something visible, missing, late, or not possible to tell?
What you build
You submit one reproducible program. It receives short fixed-camera clips and a JSON file of questions, then writes a JSON file of answers. No manual review during evaluation.
python answer.py --videos ./clips --questions questions.json --out answers.json
The clips should feel like ordinary operations footage: prep counter, packing station, fryer line, handoff shelf, dish area, storage corner. Public examples may use fixed production cameras; final hidden scoring should use rights-cleared or sponsor-owned footage.
Question types
Objective answers
Yes/no, multiple choice, counts, timestamps, durations, and not-visible answers. Freeform scene descriptions do not decide the winner.
Operational facts
Cap or hairnet visible? Container sealed before handoff? Tray unattended too long? Which station was active? When did a bottleneck start?
No guessing credit
If the order number, face, label, or action is not visible, the right answer is not visible. Guessing should lose points.
Temporal reasoning
Many questions need the order of events, not just a single frame: first sealed bag, last item added, duration unattended, or step before serving.
Scoring draft
Total score is 100 points. The benchmark should be a simple frame-sampling VLM/OCR baseline; the prize should require beating that baseline on the hidden clips.
- 65 points - answer correctness: structured answers across yes/no, counts, visible states, event order, timestamps, and not-visible cases.
- 15 points - budget efficiency: valid runs stay under the hard runtime, frame, and model/API budget; cheaper valid runs get credit only if accuracy holds.
- 15 points - hidden generalization: stable performance across layouts, lighting, cuisines, camera angles, and video quality.
- 5 points - reproducibility: one-command run, logged frame sampling/model calls, deterministic or near-deterministic outputs.
Candidate video plan
Public raw kitchen CCTV that is both long and usable is scarce. The honest plan is: use public fixed-camera footage for examples, then score final entries on hidden rights-cleared clips. These are the current leads.
| Research lead | Rail Drishti - Railway Kitchen Live CCTV video | 10:25 operational kitchen CCTV/dashboard-style footage. Strongest public kitchen-CCTV lead, but it is a screen recording. |
| Research lead | Jane at the Marketplace restaurant surveillance video - full raw | 9:54 raw restaurant surveillance. Good duration and camera feel; may be front-of-house or incident-focused, so it needs visual QA. |
| Fallback lead | Commercial Kitchen Breakfast / Opening / Prep | 9:51 real commercial kitchen prep. More routine operations, but likely creator-shot rather than CCTV. |
| Fallback lead | KitchenGuard - AI for Compliance Monitoring in Restaurants | 6:42 compliance-monitoring demo with real kitchen views. Relevant to hygiene questions, but not raw eval footage. |
| Fallback lead | Bon Appetit - 11 cameras in NYC's busiest brunch restaurant | 15:12 busy restaurant-service footage. Useful for public workflow questions, but produced and too long unless clipped. |
| Fallback lead | Bon Appetit - 12 cameras in a tiny restaurant kitchen | 11:36 fixed multi-camera kitchen service. Good for workflow and station questions, but produced footage. |
| Fallback lead | Bon Appetit - 13 cameras in New York's busiest ramen restaurant | 9:06 active kitchen line and service. Useful for event ordering and who-did-what public examples. |
| Anomaly sample | Elevated kitchen fire: CCTV footage | 18:38 fixed CCTV-style incident footage. Useful for anomaly questions after clipping; not routine operations. |
| Long sample lead | LIVEFEED Fauna - 3 hours of real kitchen service | Long, real working-kitchen footage from Gusto TV. Not CCTV; useful for sample clips if rights are cleared. |
| Fallback lead | Bon Appetit - 19 cameras in a Michelin-starred restaurant | 12-minute fixed-angle restaurant operations. Useful for public examples, not final hidden scoring. |
| Fallback lead | Security Camera B&W Restaurant | 5:51 fixed-camera restaurant footage. Good camera feel, but it is more front-of-house than kitchen. |
| Dataset lead | Kaggle - Kitchen Video in Restaurants | Claims restaurant-kitchen video for HACCP analysis. Needs access and license check before use. |
| Reference only | COM Kitchens | 177 unedited fixed-view cooking videos, but restricted to academic research. Good reference, not a public challenge asset. |
| Reference only | EPFL Smart Kitchen | Long multi-view kitchen action dataset. Lab setting, not a restaurant, but useful for benchmark design. |
Why this maps to a real business
A restaurant owner does not need a general video captioner. They need cheap, repeatable checks from footage they already collect: food sitting at handoff, missed sealing steps, cap/hairnet compliance, bottlenecks, spills, or staff activity at a station.
The cost cap matters. A system that spends heavily on every frame is less useful to a small kitchen than one that samples intelligently, asks the right visual questions, and admits when the video does not show enough.
Current status
- Challenge mechanics are drafted.
- Public sample candidates are identified, but rights and clip cuts still need a final pass.
- Final hidden clips should be rights-cleared, anonymized, and not discoverable online.
- The full machine-readable spec will be published once the clip set and baseline are locked.