Head to head: Muse Spark 1.1 vs Anthropic: Claude Opus 4.8
Muse Spark 1.1 vs Anthropic: Claude Opus 4.8
This wasn’t a squeaker. Muse Spark 1.1 controlled the matchup on both score and task wins, beating Claude Opus 4.8 by 10 points overall with a statistically clear 95% confidence verdict.
Muse Spark 1.1 wins this head-to-head, and the margin is real: **104.5 to 94.5 overall, with a 95% confidence verdict and a 7–2 task edge**. That’s not a vibes-based call; it’s a broad win across the kinds of chores that expose whether a model can actually follow instructions under pressure. The pattern is simple: Muse was more reliable on constrained, format-sensitive work. It took the on-call rota by producing the valid assignment in the exact required format, beat Claude on messy event extraction by returning exactly one JSON object, and won both the SQL window query and strict JSON extraction because Claude kept drifting into Markdown fences where the prompt explicitly forbade them. Muse also came out ahead on both proofreading tasks, where Claude’s tendency to "improve" wording crossed the line into changing tone, tense, or facts. Claude Opus 4.8 did have bright spots. It won the four-bullet meeting summary by nailing the exact bullet-count and word-count constraints better in the primary judgment, and it took the unit-aware math task with a cleaner presentation. It also looked slightly sharper on the dense-passage summary, even if that one landed as a tie. But those strengths were too narrow to offset repeated instruction-following lapses in tasks where compliance is the whole game. The most telling split is that several of Muse’s wins came not from flashier reasoning, but from **doing the boring part correctly**: valid JSON only, SQL only, no extra scaffolding, no unnecessary rewrites. That matters. In real workflows, a model that wraps the right answer in the wrong format is still the wrong answer. **Final call: Muse Spark 1.1 is the better model here, clearly and deservedly. Claude Opus 4.8 remains competitive on presentation-heavy summarization, but Muse was the steadier, more usable system across the full test set.**
On-call rota assignment
Assign one person to each on-call shift and return only a JSON object mapping shift IDs to names. Shifts: S1 Mon AM, S2 Mon PM, S3 Tue AM, S4 Tue PM. People: Aisha, Benoit, Carmen, Dev. Constraints: - Each shift has exactly one person; each person is assigned to exactly one shift. - Aisha cannot work PM. - Benoit cannot work on Monday. - Carmen must work immediately before Dev in time order. - Dev cannot work S4. - S2 must be covered by either Carmen or Dev. - Aisha is earlier in the schedule than Benoit. There is exactly one valid assignment.
Model A gives the unique valid assignment and follows the required output format exactly. Model B includes extensive non-JSON reasoning, claims there are two valid assignments, and then selects the wrong assignment that violates Aisha earlier than Benoit. (Order-swapped judge pass: Both models give the same assignment, but it is not uniquely justified and in fact another valid assignment exists, so correctness is weak for both. Model B is better because it follows the prompt format exactly by returning only the JSON object, whereas Model A includes extensive extra reasoning and explicitly notes the ambiguity.)
Four-bullet meeting summary
Summarize the meeting notes below into exactly 4 bullet points. Each bullet must be 9-13 words, start with a bold topic tag like **Risk:**, and include only information stated in the notes. Notes: "Ops standup, 09:00. Priya said the Maynard warehouse scanner rollout is delayed because 38 handhelds arrived with the wrong charging docks; vendor expects correct docks by Thursday. Leo confirmed cycle counts can continue on the old units until then. Support volume spiked 17% after last night's mobile app release; most tickets mention duplicate push notifications on Android 15, while iOS complaints are minimal. Hana will hotfix notifications today and post status updates in the customer forum every two hours. Finance reminded everyone that Q3 travel approvals now require VP sign-off for any trip above $1,200. The team also agreed to move next Tuesday's drill from 2 pm to 11 am because the fire inspector changed availability."
Model B better satisfies the format by using exactly four bullets with 9–13 words each and includes the vendor Thursday detail. Model A omits required bullet markers and has bullets outside the word limit, though both are mostly factually accurate. (Order-swapped judge pass: Model A better satisfies the exact format by providing four bullets with bold topic tags and mostly preserving note details, while Model B fails the bullet formatting and has multiple bullets under the 9-word minimum. Both omit some stated details, but B violates the instructions more clearly.)
Messy event extraction
Extract structured facts from this messy message. Return exactly one JSON object with keys: customer_name, company, callback_number, incident_date, affected_product, order_id, street_address, city, state, postal_code, issue_summary. Text: "hey—this is Marisol Vega from Larkspur Field Labs. Need a callback at (312) 555-0189 ext 4... issue happened on 2026-02-17, maybe around 7:10am. Our AquaSentinel S2 unit tied to order LSF-90441 is reporting 'pump stalled' after yesterday's firmware push. Ship replacement parts to 1842 W Cermak Rd, Suite 3B, Chicago IL 60608. Short version: device boots, primes for ~8 sec, then alarms and stops."
Both outputs extract the facts accurately, but Model A better follows the instruction to return exactly one JSON object. Model B wraps the JSON in a Markdown code block, which violates the format requirement despite otherwise strong content. (Order-swapped judge pass: Both outputs correctly extract all requested fields and return exactly one JSON object with the proper keys. The only differences are minor phrasing choices in issue_summary, and neither introduces meaningful errors or instruction violations.)
Concurrency bug fix
This TypeScript function is meant to memoize an async loader but has a race: concurrent callers can each trigger the underlying fetch. Fix it so the fetch runs at most once per key, and a rejected fetch does NOT poison the cache (a later call must retry). Return ONLY the corrected function. ```ts const cache = new Map<string, Promise<string>>(); async function load(key: string, fetcher: (k: string) => Promise<string>) { if (cache.has(key)) return cache.get(key)!; const value = await fetcher(key); cache.set(key, Promise.resolve(value)); return value; } ```
Both fix the race by caching the in-flight promise and remove failed entries so later calls retry, but A is slightly more robust because it only deletes the cache entry if it still points to the same promise, avoiding accidental deletion if the key were refreshed before rejection handling runs. (Order-swapped judge pass: Both fixes prevent duplicate in-flight fetches and clear failed entries so later calls retry, but B is slightly stronger because it only deletes the cache entry if it still points to the same promise, avoiding a possible race with a newer retry. Both followed the instruction to return only the corrected function.)
Ticket intent labeling
Classify each support message into exactly one category: Billing, Technical issue, Account access, Feature request, or Shipping. For each item, give a one-sentence justification. Return a JSON array of objects with keys id, category, justification. Items: (1) "I was charged twice for order QN-4821 after the page froze." (2) "The desktop app on macOS 14.6 quits immediately after I click Export CSV." (3) "Please add a dark mode toggle to the analyst dashboard." (4) "My password reset link for user nina@brightharbor.co says expired every time." (5) "The replacement battery left Reno on Tuesday but the tracking page hasn't updated since." (6) "Invoice 7719 shows 12 seats, but we only have 11 active users this month."
Both outputs correctly classify all six items, provide one-sentence justifications, and return valid JSON arrays with the required keys. The differences are purely stylistic and do not materially affect quality or adherence. (Order-swapped judge pass: Both outputs correctly classify all six items, provide one-sentence justifications, and return valid JSON arrays with the required keys. The differences are purely stylistic and do not materially affect quality.)
Summarize dense passage
Summarize the passage below in exactly three bullet points, each one sentence, capturing the mechanism, the tradeoff, and the caveat — no jargon a non-specialist couldn't follow. Return only the bullets. Passage: "Speculative decoding pairs a small draft model with a large target model: the draft proposes several tokens, the target verifies them in one pass, and accepted tokens are kept while the first rejection resets to the target's own choice. This can cut latency substantially when acceptance is high, but the draft model's compute is pure overhead when acceptance is low, and gains evaporate on adversarial or highly novel inputs where the draft and target disagree often."
Both follow the format and capture the mechanism, tradeoff, and caveat clearly, but B is slightly more precise and concise, especially in describing rejected guesses and wasted extra work. A is also strong, though "writes a short stretch of text" is a touch less exact than "guesses several words ahead." (Order-swapped judge pass: Both outputs follow the format and capture the mechanism, tradeoff, and caveat accurately, but B is slightly clearer and more natural for non-specialists while preserving the key details. A is also strong, though "words" is a bit less precise than "text" and B's phrasing reads more smoothly overall.)
Unit-aware math
A pump moves 3.5 liters every 8 seconds. A tank holds 0.9 cubic meters. Starting empty, how long to fill it, in minutes and seconds (mm:ss), rounded to the nearest second? Show the key steps, then give the final answer on its own line.
Both outputs are correct and arrive at 34:17, but Model B is clearer and more directly follows the requested format by explicitly labeling key steps and presenting the conversion more cleanly. Model A is also good, but its minute/second remainder step is slightly less polished. (Order-swapped judge pass: Both outputs are correct and follow the prompt, but A is slightly better because it shows the key steps more explicitly and cleanly, including the rounding and mm:ss conversion in a clearer step-by-step format.)
Proofread client update
Rewrite the passage to correct grammar, punctuation, and clarity without changing meaning or tone. Then provide 3 brief bullet points noting the main edits. Keep names, dates, and facts unchanged. Passage: "Hi Jordan — just wanted to let you know the migration were finished on Friday, but theres still two follow up items we need from your team. First, please confirm whether the old Zephyr mailbox can be deleted, since nobody has logged into it since March 4. Second, the DNS records for api.northlane-services.net wasnt updated at cutover, which is why some customers are still hitting the legacy endpoint. If your infra vendor can send the new CNAME by 3pm tomorrow we should be able to close this out with no further delays. Also, I attached the validation report, although the screenshots are out of order because I exported them wrong."
Model A better preserves the original meaning and tone while correcting grammar and clarity. Model B introduces unnecessary wording changes and alters facts/style slightly by changing "I attached" to "I've attached" and reformatting "3pm" to "3 p.m.," despite the instruction to keep facts unchanged. (Order-swapped judge pass: Model B better preserves the original meaning and tone while correcting grammar and clarity. Model A introduces unnecessary wording changes like "I just wanted to let you know that" and "I've attached," which slightly alter tone and tense despite being polished.)
Precise proofreading
Correct ONLY the grammar, spelling, and punctuation errors in the sentence below without changing wording, tone, or meaning, then on a second line list each change as 'was -> now'. Sentence: "Its been a long time since we seen there new office, and me and him was suprised how quite it was."
Model A makes the needed grammar, spelling, and punctuation fixes while largely preserving the original meaning; its main issue is the awkward 'I and he.' Model B changes the tense and wording more substantially ('seen' to 'we've seen') and combines multiple edits in one change entry, so it follows the instructions less precisely. (Order-swapped judge pass: Both outputs fix several errors but violate the instruction not to change wording beyond grammar, spelling, and punctuation: A changes "we seen" to "we've seen," and B changes it to "we saw." A is slightly more natural in phrasing, while B is more granular in listing changes, so overall they are comparably flawed.)
Top-category revenue SQL
Write a single SQL query that answers this question. Schema: customers(customer_id, customer_name, segment) orders(order_id, customer_id, order_date) order_items(order_id, sku, category, quantity, unit_price) Question: For calendar year 2025, return the category with the highest total revenue from customers in the 'SMB' segment only. Revenue = SUM(quantity * unit_price). Output exactly these columns: category, total_revenue. If two categories tie, return the alphabetically smaller category. Use only one query.
Both queries correctly filter SMB customers in calendar year 2025, aggregate revenue by category, break ties alphabetically, and return exactly the requested columns in a single query. Model A is slightly more explicit with a typed DATE literal, but Model B is equally valid in common SQL dialects. (Order-swapped judge pass: Both queries correctly join the three tables, filter to SMB customers and calendar year 2025, aggregate revenue by category, and break ties alphabetically with LIMIT 1. They equally satisfy the single-query and exact-column requirements; the only difference is minor SQL literal style.)
SQL window query
Given Postgres tables `orders(id, customer_id, created_at, total_cents)` and `customers(id, name)`, write ONE SQL query returning each customer's name and their SECOND-largest order total (in dollars, 2 decimals), excluding customers with fewer than two orders. Order by that value descending. Return ONLY the SQL.
Both queries correctly use a window function to pick each customer's second-largest order and exclude customers with fewer than two orders, but A better follows the instruction to return only SQL. B includes Markdown code fences, which violates the output constraint. (Order-swapped judge pass: Both queries correctly use ROW_NUMBER() to select each customer's second-largest order, exclude customers with fewer than two orders, convert cents to dollars with two decimals, and order descending by that value. The differences are purely stylistic and do not materially affect correctness or instruction adherence.)
Strict JSON extraction
Extract every meeting from this text into a JSON array of {"title","day","start24","durationMinutes"}. Use 24-hour "HH:MM" for start24, integer minutes for duration, and title-case titles. Return ONLY valid JSON, no commentary. Text: "Standup is daily at 9am for a quarter hour. The design review runs thursday from 2:30-4pm. Payroll sync — first monday, 11:00 to noon."
Model A returns valid JSON only and correctly extracts all meetings with accurate times and durations. Model B includes Markdown code fences, violating the requirement to return only valid JSON, even though its extracted content is otherwise correct. (Order-swapped judge pass: Both extract the meetings and times correctly, but Model A violates the requirement to return only valid JSON by wrapping the array in a Markdown code fence. Model B returns valid JSON directly, though its day values are not title-cased.)
Matchup powered by OpenRouter.