Head to head: grok-4.6 vs Codestral-2501
grok-4.6 vs Codestral-2501
By Ryan Merket · Published
This wasn’t a close call. grok-4.6 dominated the practical workload, winning 29 tasks to 4 with a 95% confidence verdict and a massive aggregate-score gap.
grok-4.6 wins this matchup cleanly. The topline is brutal for Codestral-2501: **109.4 to 79.7** on aggregate score, with a **29–4 task record** and **95% confidence** behind the result. That is not a vibes-based edge or a narrow points decision; it’s a broad, repeatable advantage across the set. Where grok-4.6 separated itself was on the unglamorous stuff that actually matters: constraint handling, correctness, and format discipline. It repeatedly beat Codestral-2501 on both Python and JavaScript **LRU cache** tasks by meeting the required **O(1)** behavior while Codestral often reached for list operations or scans that broke the core requirement. It also kept winning on extraction and formatting-heavy work — the **shipping incident JSON** task, the **classification** task, the **project summary**, the **Mexican Spanish localization**, and the **constraint scheduling** puzzle — because it followed the brief tightly instead of drifting into code fences, extra prose, wrong labels, or invalid schedules. The pattern is even clearer in coding. On the **concurrency bug fix**, the models were effectively even: both fixed the race correctly, and all three versions landed as ties. But once the task demanded not just plausible code, but code that satisfied performance or output constraints exactly, grok-4.6 pulled away. Codestral-2501 repeatedly lost on avoidable violations: non-O(1) eviction logic, extra scaffolding when only a function was requested, markdown wrappers around strict JSON, and edge-case bugs like mishandling zero capacity. Codestral-2501 did have a real bright spot: it won the **faithful rewrite** task consistently, where its Slack-style prose came off warmer and more natural. It also stole one **Go slugify** variant, though that result was mixed across repeats and undercut by other runs where it violated the ASCII-only requirement or added unnecessary harness code. That’s not enough to change the overall read. Codestral showed occasional stylistic polish; grok-4.6 was the model you’d trust to actually satisfy the assignment. **Final call: grok-4.6 is the clear winner — substantially more reliable on correctness, stricter about constraints, and decisively better across the benchmark.**
Concurrency bug fix
This TypeScript function is meant to memoize an async loader but has a race: concurrent callers can each trigger the underlying fetch. Fix it so the fetch runs at most once per key, and a rejected fetch does NOT poison the cache (a later call must retry). Return ONLY the corrected function. ```ts const cache = new Map<string, Promise<string>>(); async function load(key: string, fetcher: (k: string) => Promise<string>) { if (cache.has(key)) return cache.get(key)!; const value = await fetcher(key); cache.set(key, Promise.resolve(value)); return value; } ```
Model A and Model B both correctly store the in-flight promise before awaiting completion, preventing duplicate concurrent fetches, and both delete the cache entry on rejection so failures do not poison the cache. Model A is slightly simpler, while Model B is a bit more explicit, but both fully satisfy the prompt. (Second judge pass, order swapped — scores are the average of both: Model B and Model A both fix the race by storing the in-flight promise before awaiting, ensuring only one fetch per key, and both remove failed promises so rejections do not poison the cache. Model B is slightly more explicit about success handling, while Model A is simpler, but both fully satisfy the task.)
Python LRU cache
Write Python code only. Implement an `LRUCache` class with methods `get(key)` and `put(key, value)` and constructor `LRUCache(capacity: int)`. Requirements: both operations must be O(1) average time; evict the least recently used item when capacity is exceeded; updating an existing key must refresh its recency. Do not use `functools.lru_cache`. Use only the standard library.
Model A correctly uses OrderedDict from the standard library to achieve O(1) average-time get and put, refreshes recency on access/update, and evicts the least recently used item properly. Model B is readable but does not meet the O(1) requirement because list remove/pop(0) are O(n), so it fails a core constraint of the prompt. (Second judge pass, order swapped — scores are the average of both: Model A is better because it satisfies the required O(1) average-time operations using a standard-library OrderedDict and correctly refreshes recency on both get and put. Model B uses a list with remove and pop(0), which makes operations O(n) and therefore fails a core requirement despite otherwise implementing basic LRU behavior.)
Localize app notice to Mexican Spanish
Translate into Mexican Spanish for an in-app notice. Tone: friendly and clear, suitable for everyday users. Keep it to at most 26 words. Source: "We couldn't back up your photos last night because your phone was in battery saver mode. Plug it in and try again tonight."
Model A is concise, natural for Mexican Spanish, and stays within the 26-word limit while preserving the full meaning. Model B is also accurate and clear, but it exceeds the word limit and adds an unnecessary apology, making it less compliant with the instructions. (Second judge pass, order swapped — scores are the average of both: Model A is better localized for Mexican Spanish because "celular" sounds more natural than "teléfono" in this context, and it stays friendly and clear without adding extra wording. Model B is accurate, but "Lo sentimos" is unnecessary and makes the notice slightly less concise for an in-app message.)
Nuanced classification
Classify each review's sentiment as "positive", "negative", or "mixed", and give a 6-word-max reason. Return ONLY a JSON array of {"text","label","reason"} in input order. Reviews: ["Fast shipping but the fabric feels cheap.", "Absolutely love it, wearing it daily!", "It broke after a week. Refund was quick and painless though."]
Model A is better because all three sentiment labels are appropriate and each reason stays within the six-word limit while remaining clear. Model B misclassifies the third review as negative instead of mixed and also wraps the output in a code block rather than returning only the JSON array. (Second judge pass, order swapped — scores are the average of both: Model A is better because it correctly classifies the third review as mixed, reflecting both the product failure and the positive refund experience. Model B follows the format well but makes that key sentiment error and also uses a first reason that exceeds the six-word limit.)
Customer delay apology email
Draft an email to a small-business customer, Maya Chen at Larkspur Studio, whose custom signage order is delayed because one powder-coat color failed inspection. Audience: customer. Tone: accountable, calm, professional; do not sound defensive. Length: 120–150 words. Must include: apology, plain-English cause, revised ship date of 22 September, assurance that the rest of the order passed QA, and a goodwill offer of free expedited shipping. Subject line required.
Model A better satisfies the brief with a clear apology, a plain-English explanation of the failed powder-coat finish, the revised 22 September ship date, confirmation that the rest passed QA, and the goodwill offer, all in an accountable, calm tone. Model B is solid but slightly less direct and more generic, and phrases like "technical issue" and "This is a rare occurrence" make it less plain-English and a bit more defensive than requested. (Second judge pass, order swapped — scores are the average of both: Model A better matches the requested accountable, calm, professional tone and explains the delay in clearer plain English without sounding defensive. Model B includes all required elements, but phrases like "This is a rare occurrence" and the more generic wording make it slightly less aligned with the brief.)
LRU cache
Implement a class `LRUCache<K, V>` in TypeScript with a fixed capacity set in the constructor, and O(1) `get(key): V | undefined` and `set(key, value): void`. Accessing or updating a key must mark it most-recently-used; inserting beyond capacity must evict the least-recently-used entry. Return ONLY the class, no prose.
Model A correctly implements LRU behavior in O(1) average time using Map insertion order, and it follows the instruction to return only the class. Model B does implement eviction logic, but eviction is O(n) due to scanning timestamps, so it fails the required O(1) set behavior. (Second judge pass, order swapped — scores are the average of both: Model A correctly achieves O(1) get and set by using Map insertion order to track recency, and it cleanly evicts the least-recently-used entry. Model B violates the O(1) requirement because eviction scans all entries, making set O(n) when at capacity.)
Constrained project summary
Write a project status summary that follows ALL rules exactly: - Exactly 4 bullet points - Each bullet begins with one of these tags in this order: `[Scope]`, `[Risk]`, `[Next]`, `[Ask]` - Each bullet is 11–15 words long - Total response under 90 words - Include the numbers `37%`, `11`, and `Q4` somewhere in the response - Do not use the words `delay`, `problem`, `issue`, or `urgent` - Context to summarize: migration of 11 retail kiosks to a new payment gateway is 37% complete; certificate renewal is the main risk; next milestone is pilot in Q4; team needs one security reviewer for two days.
Model A better satisfies the constraints: it has exactly four bullets, uses the required tags in order, stays under 90 words, includes 37%, 11, and Q4, and each bullet appears to meet the 11–15 word requirement. Model B is concise and accurate, but several bullets are far under the required 11–15 words, so its instruction following is substantially weaker. (Second judge pass, order swapped — scores are the average of both: Model A better satisfies the strict length constraints: all four bullets appear to be within 11–15 words, while Model B has multiple bullets that are too short. Both are factually aligned and avoid banned words, but Model A follows the formatting and constraint requirements more completely and reads more smoothly.)
Extract shipping incident fields
Extract the requested facts from this messy note and return a single JSON object with keys exactly: incident_id, customer, callback_number, damaged_items, replacement_by, refund_amount_usd, address, tracking_numbers. Text: "Spoke w/ Nira Patel from Juniper Dental at 8:12a. Ref: INC-48271-B. She says 2 boxes arrived wet; contents damaged: 1x sterilization pouch roll 200m, 3x Sapphire bib clips. Please send replacements before Tue 14 May 2024 if possible. Refund approved for freight overcharge: $18.40. Call back on (312) 555-0189 ext 4 — but if unreachable use her cell 312.555.0190. Ship to: Juniper Dental, Attn Nira Patel, 1840 W Cermak Rd, Suite 2B, Chicago, IL 60608. Tracking stickers mentioned: 1Z9X44A20301984562 and 9405 5118 9956 1034 7712 88. Ignore prior draft with wrong ref INC-48217-B."
Model A better follows the likely normalization expected for extracted facts by converting the replacement deadline to an ISO-style date while preserving all requested fields accurately. Model B is also largely correct, but it wraps the JSON in a code fence and leaves the date less normalized, which is a weaker fit for a strict extraction task requesting a single JSON object. (Second judge pass, order swapped — scores are the average of both: Model A is slightly better because it normalizes the replacement date to an unambiguous ISO-style format while preserving all requested facts accurately. Model B is also highly accurate, but its date string is less standardized for downstream JSON use.)
Constraint scheduling
Four talks (A, B, C, D) fill four 1-hour slots 9,10,11,12. Constraints: A is before D; C is not first; B is immediately after A; D is not at 12. Give the ONE valid schedule as 'slot: talk' lines, then a one-line justification. If impossible, say so and explain.
Model A gives the unique valid schedule and a concise justification that matches all constraints. Model B contradicts itself, ends with an invalid schedule placing D at 12 and C first, and does not follow the requested output format. (Second judge pass, order swapped — scores are the average of both: Model A gives the unique valid schedule and a concise justification that matches all constraints. Model B contradicts itself and presents an invalid schedule with D at 12 and C first, so it fails both correctness and instruction adherence.)
Slugify product titles in Go
Write Go code only. Implement `func SlugifyTitle(s string) string` for an e-commerce ingest service. Rules: lowercase; trim outer space; convert any run of spaces or underscores to a single `-`; remove all characters except ASCII letters, digits, and `-`; collapse repeated `-`; strip leading/trailing `-`. Examples: `" ACME__Bolt 3/8\" - Zinc " -> "acme-bolt-38-zinc"`, `"Kids' Rain_Jacket (Blue)" -> "kids-rain-jacket-blue"`, `"---" -> ""`.
Model A is closer to the requested function-only Go solution and correctly handles the provided examples by treating spaces, underscores, and hyphens as separators while removing other punctuation. Model B violates the prompt by including a main function and comments/tests, and it is less correct because it allows non-ASCII letters despite the requirement to keep only ASCII letters, digits, and hyphens. (Second judge pass, order swapped — scores are the average of both: Model A better matches the required slug rules with a simple ASCII-only implementation that trims, lowercases, normalizes separators, removes disallowed characters, and avoids leading/trailing hyphens. Model B violates the ASCII-only requirement by using unicode letter/digit checks, includes extra code beyond the requested function, and produces the wrong result for inputs like "3/8\"" because it drops the slash without inserting a separator as required by the example.)
Unit-aware math
A pump moves 3.5 liters every 8 seconds. A tank holds 0.9 cubic meters. Starting empty, how long to fill it, in minutes and seconds (mm:ss), rounded to the nearest second? Show the key steps, then give the final answer on its own line.
Model A is slightly better because it is fully correct, concise, and follows the requested format closely by showing the key steps and then giving the final answer cleanly on its own line. Model B also gets the correct result, but it is more verbose than needed and introduces minor rounding imprecision in the intermediate interval calculation. (Second judge pass, order swapped — scores are the average of both: Model A is better because it is fully correct, concise, and follows the requested format by showing the key steps and then giving the final answer cleanly on its own line. Model B is also correct, but it is more verbose than needed and includes a slightly clunkier conversion sequence.)
Faithful rewrite
Rewrite this blunt internal note as a warm, professional Slack message to the whole engineering team (60–90 words), keeping every fact intact and adding no new commitments. Return only the message. Note: "Deploy is frozen until the memory leak in the image service is fixed. Priya is on it. Do not merge to main. ETA tomorrow noon."
Model B is warmer and more natural for Slack while preserving the key facts with minimal embellishment. Model A exceeds the 60–90 word limit and adds extra language that edges toward new implications, while Model B stays concise and professional despite slightly softening the exact ETA wording. (Second judge pass, order swapped — scores are the average of both: Model B is warmer and professional while preserving the core facts with only a slight softening of the ETA into an update expectation. Model A keeps the facts more literally but violates the 60–90 word limit and adds extra sentiment and implications beyond the note, so Model B is the better overall fit.)
Matchup powered by OpenRouter.