Head to head: Kimi-K2.7-Code vs mistral-medium-3-5
Kimi-K2.7-Code vs mistral-medium-3-5
This wasn’t a contest so much as a sweep. Kimi-K2.7-Code dominated the matchup on both score and consistency, beating mistral-medium-3-5 across nearly every task type with a decisive 100% confidence verdict.
Kimi-K2.7-Code wins this head-to-head outright: **116.0 to 81.0**, with **11 task wins to 0** and **1 tie**. That’s not a narrow edge or a vibes-based preference; it’s a comprehensive result backed by a **100% confidence** statistical verdict. If you’re looking for the model that more reliably lands the assignment, this matchup answered the question emphatically. What stands out is how often Kimi won on the boring but essential stuff that separates useful models from annoying ones: following output constraints, keeping formats clean, and not smuggling in extra explanation where it wasn’t asked for. It was better on strict JSON extraction, messy contact normalization, scheduling, clinic assignment, release-note formatting, and the Python bugfix largely because it actually respected the prompt shape. Mistral-medium-3-5 repeatedly lost points by wrapping answers in markdown, adding prose, or drifting from exact structural requirements. Kimi also looked stronger on implementation-heavy work. It took the TypeScript LRU cache, the concurrency bug fix, and the Go LRU task, where correctness under edge cases and adherence to requested structure mattered more than sounding polished. Even when both models reached the right underlying answer—as in the scheduling and assignment tasks—Kimi was the one more likely to deliver it in the form the user actually requested. The lone tie, on the unit-aware math problem, says something useful: when the task is straightforward and the format is forgiving, both models can perform competently. But this benchmark wasn’t decided on easy arithmetic. It was decided on whether a model can be trusted across messy real-world asks—code repair, constrained generation, exact extraction, and format compliance—and on that measure, mistral-medium-3-5 was simply outclassed. **Final call: Kimi-K2.7-Code is the clear, decisive winner. mistral-medium-3-5 never found a lane in this matchup, while Kimi combined stronger technical execution with much better instruction discipline.**
python_fee_bugfix
Debugging & code review — Language: Python 3. The function should total invoice lines, applying each line's discount, then adding tax once to the discounted subtotal. Find the bug, fix it, and return only the corrected code. ```python def invoice_total(lines, tax_rate): total = 0 for qty, unit_price, discount in lines: line_total = qty * unit_price total += line_total * (1 - tax_rate) * (1 - discount) return round(total * (1 + tax_rate), 2) items = [ (2, 19.95, 0.10), (1, 120.00, 0.25), (3, 4.50, 0.00), ] print(invoice_total(items, 0.0825)) # expected 156.16 ```
Both outputs fix the tax/discount bug correctly, but Model A follows the instruction to return only corrected code, while Model B adds explanatory prose and omits the provided usage example. Model A is therefore the better fit for the task. (Order-swapped judge pass: Both outputs fix the tax/discount bug correctly, but Model B better follows the instruction to return only the corrected code. Model A adds explanatory prose and markdown, which violates the requested output format.)
proofread_policy_note
Editing & proofreading — Correct the passage for grammar, punctuation, and clarity without changing meaning. Then provide 2 brief bullet points noting the main changes. Passage: "Before anyone submits receipts for the North Quay visit, make sure they was uploaded in Concur by Friday. The finance team dont reimburse late meals unless a director approve it in writing, however taxi fares after 9pm are allowed if theres no train running. Please attached the hotel folio, not just the card slip, because thats what audit ask for last quarter."
Model A is more accurate and polished overall: it fixes the grammar cleanly, improves punctuation, and its notes accurately describe the edits. Model B leaves weaker choices such as "in Concur," adds emphasis formatting and quotation marks unnecessarily, and its change note incorrectly labels "that's" as possessive. (Order-swapped judge pass: Model B provides a cleaner, more polished correction and keeps the meaning intact while improving clarity. Model A introduces a likely error by changing "audit" to "audits" and is slightly less natural in a few phrasing choices.)
Unit-aware math
A pump moves 3.5 liters every 8 seconds. A tank holds 0.9 cubic meters. Starting empty, how long to fill it, in minutes and seconds (mm:ss), rounded to the nearest second? Show the key steps, then give the final answer on its own line.
Both outputs are mathematically correct, show the key steps clearly, and provide the final answer on its own line in the requested mm:ss format. The differences are purely stylistic and do not materially affect quality. (Order-swapped judge pass: Both outputs are correct, show the key steps clearly, and provide the final answer on its own line in the requested mm:ss format. The differences are purely stylistic and do not materially affect quality.)
LRU cache
Implement a class `LRUCache<K, V>` in TypeScript with a fixed capacity set in the constructor, and O(1) `get(key): V | undefined` and `set(key, value): void`. Accessing or updating a key must mark it most-recently-used; inserting beyond capacity must evict the least-recently-used entry. Return ONLY the class, no prose.
A is concise, valid TypeScript, and correctly implements LRU behavior in O(1) using Map insertion order. B violates the requirement to return only the class by adding a separate top-level helper class, and its capacity-0 case can crash due to dereferencing tail! when null. (Order-swapped judge pass: Model B is simpler and correctly implements LRU behavior in O(1) average time using Map insertion order, while Model A has a serious edge-case bug when capacity is 0 and also violates the 'return ONLY the class' instruction by including an extra class. Both are readable, but B adheres better to the requested output format.)
messy_contacts_to_json
Data wrangling / structured output — Convert the messy contact notes below into valid JSON only. Schema: {"contacts":[{"name":string,"company":string,"email":string,"phones":[string],"tags":[string]}]} Rules: - Output contacts sorted by name ascending - Normalize emails to lowercase - Keep phone numbers as strings exactly as written except trim surrounding spaces - Tags must be lowercase, unique, sorted alphabetically - If a contact has no phone, use an empty array - Ignore lines marked "duplicate" Data: 1) Nia Vale | Larkspur Systems | NIA.VALE@LARKSPUR.IO | +44 20 7946 1182 | Tags: Procurement; Q3; procurement 2) duplicate -> Nia Vale | Larkspur Systems | nia.vale@larkspur.io | +44 20 7946 1182 | Tags: q3 3) Omar Sen | Brindle Health | omar.sen@brindlehealth.com | | Tags: Renewal ; VIP 4) Petra Ko | Aster Dock | PETRA.KO@ASTERDOCK.CO | (312) 555-0199, 312-555-0111 | Tags: west, Logistics, west 5) Eli Dunn | Northline Civic | eli.dunn@northline.example | 0780 123 900 | Tags: board; urgent
Both outputs extract and normalize the contacts correctly, but Model A better follows the requirement to return valid JSON only. Model B wraps the JSON in Markdown code fences, which violates the output-format instruction. (Order-swapped judge pass: Both outputs correctly parse, normalize, deduplicate, and sort the contacts, but Model A wraps the JSON in a Markdown code fence, violating the requirement to output valid JSON only. Model B provides clean valid JSON directly and is therefore better.)
Concurrency bug fix
This TypeScript function is meant to memoize an async loader but has a race: concurrent callers can each trigger the underlying fetch. Fix it so the fetch runs at most once per key, and a rejected fetch does NOT poison the cache (a later call must retry). Return ONLY the corrected function. ```ts const cache = new Map<string, Promise<string>>(); async function load(key: string, fetcher: (k: string) => Promise<string>) { if (cache.has(key)) return cache.get(key)!; const value = await fetcher(key); cache.set(key, Promise.resolve(value)); return value; } ```
Model A correctly stores the in-flight promise before awaiting, ensuring only one fetch per key, and removes failed promises so retries work. Model B also coalesces concurrent calls and clears failures, but it unnecessarily changes the cache shape, deletes successful results so it no longer memoizes, and is more complex than needed. (Order-swapped judge pass: Model B correctly stores the in-flight promise immediately so concurrent callers share one fetch, and it removes failed promises so later calls retry. Model A avoids the race but incorrectly deletes successful results from the cache, breaking memoization semantics and adding unnecessary complexity.)
strict_release_note
Instruction following — Write a release note for an internal app update. Requirements: - Exactly 4 bullet points - Each bullet must start with "- " - Each bullet must be 7 to 10 words long - Include these exact version and date strings somewhere: "v2.8.1" and "2026-03-14" - Mention all three features exactly once each: offline drafts, SSO, audit export - Do NOT use the words: improved, seamless, bug, issue, fix - After the bullets, add a final line exactly: Owner: Mara Iqbal
Model A satisfies all structural constraints, includes both required strings, and mentions each required feature exactly once. Model B omits the exact version/date pairing in one bullet set and violates the 7-10 word requirement on multiple bullets. (Order-swapped judge pass: Both outputs satisfy the structural constraints and include the required version, date, features, and owner line. Model B is stronger because all bullets meet the 7-10 word requirement, while Model A has one bullet with only six words.)
Strict JSON extraction
Extract every meeting from this text into a JSON array of {"title","day","start24","durationMinutes"}. Use 24-hour "HH:MM" for start24, integer minutes for duration, and title-case titles. Return ONLY valid JSON, no commentary. Text: "Standup is daily at 9am for a quarter hour. The design review runs thursday from 2:30-4pm. Payroll sync — first monday, 11:00 to noon."
A is fully correct and returns only valid JSON. B incorrectly drops the ordinal in "First Monday" and wraps the JSON in a markdown code fence, violating the output constraint. (Order-swapped judge pass: Both extract the meetings and times correctly, but B better preserves the source text by keeping "First Monday" rather than collapsing it to "Monday." A also wraps its JSON in a markdown code fence, which violates the instruction to return only valid JSON.)
go_lru_cache
Algorithm & data structures — Return code only. Implement an LRU cache in Go with this API: - `type LRUCache struct { ... }` - `func NewLRU(capacity int) *LRUCache` - `func (c *LRUCache) Get(key string) (int, bool)` - `func (c *LRUCache) Put(key string, value int)` Requirements: - Both Get and Put must be O(1) average time - Evict the least recently used item when capacity is exceeded - Updating an existing key must refresh its recency - Use only the Go standard library - Include a minimal `main` demonstrating a few operations
Model A provides a complete, compilable Go program that correctly implements O(1) average-time LRU behavior and includes a minimal main demonstration. Model B is incomplete and incorrect as written: it omits the package/imports, fails to initialize the sentinel links in NewLRU, and can panic during add/remove operations. (Order-swapped judge pass: Model B provides a complete, compilable Go program that correctly implements O(1) average-time LRU behavior and includes a minimal demonstration. Model A is incomplete and incorrect because it omits the package declaration/imports and fails to initialize the sentinel links, causing runtime panics in normal use.)
Localization with tone
Translate this app onboarding line into natural, friendly European Spanish suitable for a mobile toast (keep it under 60 characters, no exclamation marks): "You're all set — your first backup starts tonight." Return only the translation, then the character count in parentheses.
A is concise, natural, under the limit, and preserves the key meaning including that it starts tonight. B is also natural and within length, but it changes "tonight" to "today," which makes it less correct for the source text. (Order-swapped judge pass: B is more accurate and natural for European Spanish, especially by translating “tonight” as “esta noche” and keeping a friendly mobile-toast tone under the limit. A is understandable but mistranslates “tonight” as “hoy” and sounds slightly less natural with the longer phrasing.)
clinic_shift_assignment
Constraint satisfaction — Determine the only valid assignment of four staff to four clinic roles for Tuesday morning: Triage, Phones, Intake, Discharge. Staff: Amina, Benoit, Carla, Dev. Constraints: 1. Amina cannot do Phones. 2. Benoit must be either Triage or Discharge. 3. Carla cannot do Intake. 4. Dev must be assigned to Phones or Intake. 5. The person on Triage is not Carla. 6. If Dev is on Phones, then Amina is on Intake. 7. Benoit is not on Discharge. 8. Carla is on Phones. Return just the final mapping as four lines in the format `Role: Name`.
Both models give the same correct assignment, but Model A follows the instruction to return just the final mapping, while Model B adds unnecessary explanation and formatting beyond the requested output. Model B’s reasoning is clear, but it is less compliant with the prompt. (Order-swapped judge pass: Both models give the correct unique assignment, but Model B follows the instruction exactly by returning just the final mapping. Model A adds unnecessary reasoning and formatting beyond what was requested.)
Constraint scheduling
Four talks (A, B, C, D) fill four 1-hour slots 9,10,11,12. Constraints: A is before D; C is not first; B is immediately after A; D is not at 12. Give the ONE valid schedule as 'slot: talk' lines, then a one-line justification. If impossible, say so and explain.
Both outputs give the correct unique schedule, but Model A follows the requested format exactly with slot lines plus a one-line justification. Model B is correct and clear, but it ignores the instruction to provide only the schedule and a one-line justification by adding extensive step-by-step analysis. (Order-swapped judge pass: Both models give the correct unique schedule, but Model B follows the requested format exactly by providing only the slot lines and a one-line justification. Model A is correct but adds unnecessary step-by-step reasoning instead of the concise requested output.)
Matchup powered by OpenRouter.