Head to head: Kimi-K2.7-Code vs gpt-5.4-nano
Kimi-K2.7-Code vs gpt-5.4-nano
This wasn’t a squeaker. Kimi-K2.7-Code controlled the matchup on both score and task wins, beating gpt-5.4-nano 104.0 to 92.9 with a 95% confidence verdict and a 7–2 edge in task wins.
Kimi-K2.7-Code takes this head-to-head cleanly. The aggregate margin is substantial, the task record is lopsided at 7 wins to 2, and the statistical readout matters here: **95% confidence for a clear win**. That’s not a vibes-based verdict; it’s backed by repeated better performance across the set. Where Kimi separated itself was instruction discipline plus technical correctness. It was better on **precise proofreading**, where gpt-5.4-nano made more unnecessary rewrites; on **vendor-record normalization**, where B needlessly altered capitalization and even wrapped JSON in code fences in one judge pass; on the **SQL window query**, where the losing output used the wrong ranking logic for duplicate top values; on the **concurrency bug fix**, where Kimi delivered the minimal correct function without adding extra baggage; and on **constraint scheduling**, where B’s contradictory reasoning undercut an otherwise solvable task. Kimi also looked stronger on style-sensitive writing work. In the **internal delay update**, it better matched the requested calm, accountable, no-blame tone. In **meeting-note summarization**, it more reliably hit the exact formatting constraints and preserved the action-oriented structure the prompt demanded. These are not glamorous wins, but they’re exactly the kind that separate a model that merely sounds competent from one that actually follows directions under pressure. To gpt-5.4-nano’s credit, it did take **strict JSON extraction** and **unit-aware math** was effectively even, while **shipment extraction** and **helpdesk classification** ended in ties. But its only other nominal win, **conference catering cost**, is flatly contradicted by the accompanying rationale: Kimi found the true minimum and B did not. Even allowing for a few noisy order-swapped judge reversals, the larger pattern doesn’t budge. **Final call: Kimi-K2.7-Code is the better model here, and not by a hair. It was more reliable on exacting instructions, more correct on code-heavy tasks, and more composed on constrained writing—an unambiguous win over gpt-5.4-nano.**
Precise proofreading
Correct ONLY the grammar, spelling, and punctuation errors in the sentence below without changing wording, tone, or meaning, then on a second line list each change as 'was -> now'. Sentence: "Its been a long time since we seen there new office, and me and him was suprised how quite it was."
Model A makes the needed grammar, spelling, and punctuation fixes and provides a mostly accurate change list, though it changes wording more than strictly necessary. Model B introduces more unnecessary wording changes, includes incorrect/non-changes in its list, and adds quotation marks not requested. (Order-swapped judge pass: Model A is better because it preserves more of the original wording while correcting the clear grammar, spelling, and punctuation errors. Model B makes larger wording changes ('seen' to 'saw', 'me and him' to 'he and I') that go beyond the instruction to correct only errors without changing wording.)
Summarize meeting notes in bullets
Summarize the meeting notes below into exactly 4 bullet points. Each bullet must be 9–14 words, factual, and include no speculation. Notes: “Monday launch review for the Larkspur mobile app. Nina said Android crash-free sessions improved from 96.8% to 98.9% after last week’s hotfix. iOS still has a checkout spinner affecting about 3% of sessions on version 5.4. Support volume dropped from 184 tickets to 121 week-over-week, but refund requests rose from 9 to 17, mostly tied to duplicate orders. The team agreed to delay the social-share feature to the June 3 release and prioritize checkout instrumentation first. Omar will own a dashboard by Thursday; Mei will draft the app store update text today.”
Model A satisfies the exactly-4-bullets requirement and keeps each bullet within 9–14 words while accurately covering key decisions and action items. Model B has only 8 words in its first bullet, violating the length constraint, and omits the assigned owners/action-item detail. (Order-swapped judge pass: Model B better satisfies the content coverage while remaining factual and concise, including the assigned action items omitted by A. Both violate the 9–14 words per bullet requirement on multiple bullets, but B is stronger overall.)
Strict JSON extraction
Extract every meeting from this text into a JSON array of {"title","day","start24","durationMinutes"}. Use 24-hour "HH:MM" for start24, integer minutes for duration, and title-case titles. Return ONLY valid JSON, no commentary. Text: "Standup is daily at 9am for a quarter hour. The design review runs thursday from 2:30-4pm. Payroll sync — first monday, 11:00 to noon."
Both outputs are valid JSON and correctly parse times and durations, but B better preserves the source text by capturing "First Monday" rather than reducing it to "Monday." (Order-swapped judge pass: Model A preserves the recurrence detail in "first monday" as the day value "First Monday," while Model B loses that information by reducing it to "Monday." Both follow the JSON-only instruction and correctly parse the other meetings.)
Rewrite internal delay update
Rewrite this into a polished Slack update for an internal product team. Audience: engineering + support. Tone: calm, accountable, no blame. Length: 90–120 words. Include: what happened, customer impact, current workaround, next update time. Draft: “Heads up, the Atlas import thing is still broken. We changed the queue settings around 9:40 and now some CSV imports are hanging forever, mostly the ones over 20k rows. Support has 6 tickets so far and two customers are pretty mad. We rolled back one worker group but it didn’t fix the stuck jobs already in there. For now people can split files smaller and retry. I think Priya is patching it but not sure when. I’ll send another note after lunch, probably 2 pm.”
Model A is more polished and better matches the requested calm, accountable, no-blame tone while covering all required points clearly. Model B is accurate but less polished, uses "still intermittently broken" and "investigating/patching" language that feels rougher and less aligned with the tone brief. (Order-swapped judge pass: Model B is more polished and better aligned to the requested calm, accountable internal Slack tone while clearly covering what happened, customer impact, workaround, and next update time. Model A is solid but slightly less polished and includes more uncertainty around ownership/status.)
Unit-aware math
A pump moves 3.5 liters every 8 seconds. A tank holds 0.9 cubic meters. Starting empty, how long to fill it, in minutes and seconds (mm:ss), rounded to the nearest second? Show the key steps, then give the final answer on its own line.
Both outputs are correct and follow the instructions, but B is slightly better because it presents the arithmetic more cleanly and explicitly, including the exact fraction step before rounding. A is also strong, just a bit less polished. (Order-swapped judge pass: Both outputs are fully correct, show the key steps clearly, and provide the final answer on its own line as requested. Model A is slightly more detailed, while Model B is slightly more concise, but neither is meaningfully better overall.)
Normalize vendor records to JSON
Convert the messy vendor notes below into valid JSON only. Output an array of objects using this exact schema and key order: [{"vendor_id":"string","company":"string","contact":{"name":"string","email":"string|null"},"phone":"string|null","active":true,"tags":["string"]}] Rules: trim spaces; lowercase emails; phone must keep digits and leading + only; active is true unless note says inactive/paused/closed; tags must be lowercase, unique, sorted alphabetically. Notes: - ID vx-208 ; Company: Northloop Imaging ; Contact= Rina Sol ; Email= RSol@NorthLoop.co ; Phone: +1 (415) 555-0188 ; Tags: MRI, urgent, west - Company: Elm & Cipher LLC | id=ec-77 | contact: Mateo Dunn | phone: none | email: | tags: legal; compliance; Legal | status: paused - id: q9-004 / company “Harbor Grain” / contact “Asha Pike” / email asha@harborgrain.io / tel 020 7946 1182 / tags: food, supplier - company: BlueMesa Studio, id bm-19, contact J. Vale, email JVale@BlueMesa.Studio, phone +44-20-7000-9911, tags design, beta, design
Model A preserves the source company and contact capitalization while correctly normalizing emails, phones, active status, and tags. Model B unnecessarily lowercases company and contact names, which was not requested and reduces fidelity to the input. (Order-swapped judge pass: Model A better follows the requirement to return valid JSON only, while Model B incorrectly wraps the JSON in markdown code fences. Both are largely correct structurally, but A is stronger on instruction adherence despite unnecessarily lowercasing company and contact names.)
SQL window query
Given Postgres tables `orders(id, customer_id, created_at, total_cents)` and `customers(id, name)`, write ONE SQL query returning each customer's name and their SECOND-largest order total (in dollars, 2 decimals), excluding customers with fewer than two orders. Order by that value descending. Return ONLY the SQL.
Model A correctly returns the second-largest order per customer using ROW_NUMBER and excludes customers with fewer than two orders. Model B uses DENSE_RANK, which returns the second distinct total rather than the second-largest order, so it can fail when a customer has duplicate top order amounts. (Order-swapped judge pass: Model B correctly returns the second-highest order per customer using ROW_NUMBER and excludes customers with fewer than two orders. Model A uses DENSE_RANK, which returns the second distinct total rather than the second-largest order, so it can be wrong when the top total is duplicated.)
Compute conference catering cost
A team coordinator is finalizing catering for a 2-day workshop. Day 1 attendance: 18 people. Day 2 attendance: 23 people. Order rules: - Sandwich boxes cost $12 each and serve 1 person. - Salad trays cost $28 each and serve 4 people; you can only order whole trays. - On each day, at least 30% of attendees must be covered by salad trays. - Everyone must be covered by exactly one meal’s worth of food total per day. - Delivery fee is $14 per day if that day’s food subtotal is under $180; otherwise no delivery fee. Question: What is the minimum total cost for both days combined? Show your working briefly and give the final dollar amount.
Model A finds the true minimum by considering additional salad trays, which lower food cost enough to outweigh delivery effects; Model B stops too early and gives an incorrect total. Both show brief working, but A is substantially more correct and complete. (Order-swapped judge pass: Model A correctly enforces exact coverage and finds the valid minimum total of $426 with clear brief working. Model B violates the exact-coverage rule by allowing oversupply from extra salad trays, leading to an incorrect total of $340.)
Concurrency bug fix
This TypeScript function is meant to memoize an async loader but has a race: concurrent callers can each trigger the underlying fetch. Fix it so the fetch runs at most once per key, and a rejected fetch does NOT poison the cache (a later call must retry). Return ONLY the corrected function. ```ts const cache = new Map<string, Promise<string>>(); async function load(key: string, fetcher: (k: string) => Promise<string>) { if (cache.has(key)) return cache.get(key)!; const value = await fetcher(key); cache.set(key, Promise.resolve(value)); return value; } ```
Model A gives a minimal correct fix that stores the in-flight promise before awaiting and deletes it on rejection, which directly solves both the race and cache-poisoning issues. Model B is also logically correct, but it violates the instruction to return only the corrected function by re-declaring the cache and adding extra content outside the function. (Order-swapped judge pass: Both outputs correctly store the in-flight promise before awaiting, ensuring at most one fetch per key, and both remove rejected promises so later calls retry. Both follow the instruction to return only the corrected function; A redundantly redeclares cache while B is slightly more concise, but neither has a meaningful quality advantage.)
Extract shipment facts
From the shipping thread below, extract the requested facts and return exactly one JSON object with these keys: {"order_id":"string","customer":"string","items":[{"sku":"string","qty":number}],"ship_to":{"name":"string","street":"string","city":"string","postal_code":"string","country":"string"},"dispatch_date":"string","carrier":"string","tracking":"string|null","delivery_window":"string|null","special_instructions":"string|null"} Use null if truly unavailable. Thread: “Re: order HZ-4418 for customer Maren Iqbal. Please send 2x SKU AX9-B and 1 x TT-44. Ship to: Maren Iqbal, 18 Kirkwall Mews, Bristol BS7 9PL, UK. Warehouse note: leave with reception if office is closed. Dispatch on 2026-04-18 via DPD. Tracking not generated yet. Customer asked if it can arrive Tue–Thu next week, morning preferred if possible.”
Both outputs correctly extract all shipment facts and adhere to the required JSON schema. The minor phrasing differences in delivery_window and special_instructions do not materially affect correctness. (Order-swapped judge pass: Both outputs correctly extract the shipment facts and return a single valid JSON object. Model B is slightly better because its delivery_window preserves the source wording more faithfully, whereas Model A adds parenthetical reformatting.)
Constraint scheduling
Four talks (A, B, C, D) fill four 1-hour slots 9,10,11,12. Constraints: A is before D; C is not first; B is immediately after A; D is not at 12. Give the ONE valid schedule as 'slot: talk' lines, then a one-line justification. If impossible, say so and explain.
Model A gives the unique valid schedule and a concise correct justification that matches the requested format. Model B eventually reaches the same schedule, but includes contradictory reasoning, an incorrect interim claim that no valid schedule exists, and does not adhere cleanly to the instruction to give the one valid schedule followed by a one-line justification. (Order-swapped judge pass: Model B gives the unique valid schedule directly and concisely. Model A eventually reaches the same correct answer, but includes an invalid schedule, contradictory reasoning, and unnecessary self-correction that weakens correctness and adherence.)
Classify helpdesk tickets
Classify each ticket into exactly one category: Billing, Access, Bug, Feature Request, or Account Closure. For each, give a 1-sentence justification. Return exactly 5 lines in this format: <ID> | <Category> | <Justification> T1: “I was charged twice for the March Pro plan after updating my card.” T2: “The reset-password link says expired immediately, and I still can’t sign in.” T3: “Please add an option to export weekly summaries as .docx, not just PDF.” T4: “On Android 14, tapping ‘Save draft’ closes the app without saving my notes.” T5: “We shut down our studio last month; please permanently delete the workspace and stop future invoices.”
Both outputs classify all five tickets correctly, provide exactly one category per ticket with concise one-sentence justifications, and follow the required 5-line format. Writing quality is equally clear and polished in both. (Order-swapped judge pass: Both outputs classify all five tickets correctly, follow the required 5-line format with one-sentence justifications, and are equally clear and concise. Any wording differences are stylistic and do not materially affect quality.)
Matchup powered by OpenRouter.