Head to head: Llama-4-Scout-17B-16E-Instruct vs cohere-command-a

Llama-4-Scout-17B-16E-Instruct vs cohere-command-a

By Ryan Merket · Published Aug 22, 2026, 9:04am CT

This one is closer than the aggregate score suggests. Llama-4-Scout-17B-16E-Instruct posts the slightly higher raw total, but cohere-command-a wins more tasks and takes the verdict on confidence-adjusted performance.

The topline is simple: **cohere-command-a takes this matchup on an 83% confidence lean**, even though Llama-4-Scout-17B-16E-Instruct edges it on aggregate score, 101.0 to 99.6. That sounds contradictory until you look at the distribution. Cohere wins **6 tasks to 3**, with 3 ties, and its edge comes from being a little more reliable across a wider spread of everyday instruction-following work rather than running up the score on a few prompts. Where Cohere earns the nod is in the unglamorous stuff that matters in production: cleaner **strict JSON extraction**, better prompt-discipline on **messy-orders-to-json**, more faithful and polished **status writing**, sharper **nuanced classification**, and slightly stronger presentation on **unit-aware math** and **warehouse-pick-time**. None of those are blowouts, but together they paint a consistent picture: Cohere is usually the model that loses less information, adds less noise, and lands closer to the requested format. Llama’s wins are real, and they are not trivial. It was clearly better on **sql-top-region-q2**, where Cohere’s tie-handling logic was simply not reliable, and it also took **Faithful rewrite** by avoiding the extra promises and implications Cohere introduced. It also gets credit on **oncall-shift-assignment** in the published tally, though that result was less clean than ideal across judge passes. In other words, Llama shows higher upside on tasks where correctness and faithfulness matter more than polish. The ties reinforce the overall read. On **python-invoice-bugfix** and **Find the contradiction**, both models were effectively even. On **LRU cache**, both missed the O(1) requirement in different ways, which is a useful reminder that neither model was especially trustworthy on data-structure implementation details here. **Final call: cohere-command-a wins, but only narrowly.** If you care most about day-to-day instruction following, formatting discipline, and polished business-facing output, Cohere is the safer pick in this head-to-head. If your workload leans harder toward SQL correctness and tightly faithful rewriting, Llama remains very competitive—but not enough to overturn the match result.

Strict JSON extraction

Extract every meeting from this text into a JSON array of {"title","day","start24","durationMinutes"}. Use 24-hour "HH:MM" for start24, integer minutes for duration, and title-case titles. Return ONLY valid JSON, no commentary. Text: "Standup is daily at 9am for a quarter hour. The design review runs thursday from 2:30-4pm. Payroll sync — first monday, 11:00 to noon."

Llama-4-Scout-17B-16E-Instruct: ```json [ { "title": "Standup", "day": "daily", "start24": "09:00", "durationMinutes": 15 }, { "title": "Design Review", "day": "thursday", "start24": "14:30", "durationMinutes": 90 }, { "title": "Payroll Sync", "day": "monday", "start24": "11:00", "durationMinutes": 60 } ] ```

cohere-command-a: ```json [ {"title": "Standup", "day": "Daily", "start24": "09:00", "durationMinutes": 15}, {"title": "Design Review", "day": "Thursday", "start24": "14:30", "durationMinutes": 90}, {"title": "Payroll Sync", "day": "First Monday", "start24": "11:00", "durationMinutes": 60} ] ```

Model B better preserves the source text by keeping day values title-cased and retaining the recurrence detail "First Monday" for Payroll Sync, while Model A loses that qualifier and does not title-case day fields. Both include valid JSON and correctly compute times and durations. (Second judge pass, order swapped — scores are the average of both: Model B is better because it correctly preserves the recurrence/day information as "Daily," "Thursday," and especially "First Monday," while Model A loses specificity on the payroll meeting and uses lowercase day values. Both extract times and durations correctly, but Model B is more faithful to the source and cleaner overall.)

Unit-aware math

A pump moves 3.5 liters every 8 seconds. A tank holds 0.9 cubic meters. Starting empty, how long to fill it, in minutes and seconds (mm:ss), rounded to the nearest second? Show the key steps, then give the final answer on its own line.

Llama-4-Scout-17B-16E-Instruct: To find out how long it takes to fill the tank, we first need to convert all measurements to the same units. Let's convert everything to liters because the pump's rate is given in liters per 8 seconds. 1 cubic meter = 1000 liters So, the tank holds 0.9 cubic meters * 1000 liters/cubic meter = 900 liters. The pump moves 3.5 liters every 8 seconds. First, let's find out how many 8-second intervals it takes to fill the tank: 900 liters / 3.5 liters per 8 seconds = 900 / 3.5 = 257.14 intervals of 8 seconds. To find the total time in seconds: 257.14 intervals * 8 seconds/interval = 2057.12 seconds. To convert seconds into minutes and seconds: 2057.12 seconds / 60 = 34.2853 minutes. Converting the decimal part of the minutes into seconds: 0.2853 minutes * 60 = 17.12 seconds. So, it takes approximately 34 minutes and 17 seconds to fill the tank. Rounded to the nearest second, the time is 34:17. 34:17

cohere-command-a: To determine how long it takes to fill the tank, we need to follow these steps: 1. **Convert the tank's volume to liters:** \[ 0.9 \text{ cubic meters} = 0.9 \times 1000 \text{ liters} = 900 \text{ liters} \] 2. **Determine the pump's rate in liters per second:** \[ \text{Rate} = \frac{3.5 \text{ liters}}{8 \text{ seconds}} = 0.4375 \text{ liters/second} \] 3. **Calculate the total time required to fill the tank in seconds:** \[ \text{Time} = \frac{900 \text{ liters}}{0.4375 \text{ liters/second}} = 2057.142857 \text{ seconds} \] 4. **Convert the total time to minutes and seconds:** \[ \text{Minutes} = \left\lfloor \frac{2057.142857}{60} \right\rfloor = 34 \text{ minutes} \] \[ \text{Remaining seconds} = 2057.142857 - (34 \times 60) = 2057.142857 - 2040 = 17.142857 \text{ seconds} \] 5. **Round the remaining seconds to the nearest second:** \[ 17.142857 \approx 17 \text{ seconds} \] Thus, the time to fill the tank is: \[ \boxed{34:17} \]

Model B is slightly better because it presents the unit conversions and rate calculation more cleanly and precisely while arriving at the same correct final answer. Model A is also correct and follows the prompt well, but it is a bit less polished and slightly less concise in its intermediate conversion steps. (Second judge pass, order swapped — scores are the average of both: Model B is slightly better because it is fully correct, shows the key steps cleanly, and presents the final answer clearly on its own line. Model A reaches the same final result, but it rounds intermediate values a bit loosely and is less precise in its calculations and formatting.)

oncall-shift-assignment

Assign one person to each on-call shift: Mon AM, Mon PM, Tue AM, Tue PM. People: Imani, Jorge, Keiko, Lev. Constraints: - Each person gets exactly one shift. - Imani cannot do any PM shift. - Jorge cannot do Mon AM. - Keiko must be scheduled later in the week than Imani. - Lev cannot do Tue AM. - Tue PM must be Jorge or Lev. Provide just the final assignment.

Llama-4-Scout-17B-16E-Instruct: ## Step 1: Analyze the constraints for each person and shift Given the constraints: - Imani cannot do any PM shift. - Jorge cannot do Mon AM. - Keiko must be scheduled later in the week than Imani. - Lev cannot do Tue AM. - Tue PM must be Jorge or Lev. ## 2: Assign Imani Imani can only do Mon AM since she cannot do any PM shift. So, Imani is assigned to Mon AM. ## 3: Assign Keiko Keiko must be scheduled later in the week than Imani. This means Keiko can be assigned to Mon PM, Tue AM, or Tue PM. However, we need to consider other constraints as well. ## 4: Assign Jorge Jorge cannot do Mon AM, which is already taken by Imani. Jorge can potentially do Mon PM, Tue AM, or Tue PM. ## 5: Assign Lev Lev cannot do Tue AM. Lev can potentially do Mon PM, Tue PM. ## 6: Consider Tue PM constraint Tue PM must be Jorge or Lev. This will help in narrowing down the assignments. ## 7: Trial and Error with Constraints - Imani is on Mon AM. - Since Keiko must be later than Imani and Imani is on Mon AM, Keiko can be on Mon PM, Tue AM, or Tue PM. - Jorge can't be on Mon AM, so he could be on Mon PM, Tue AM, or Tue PM. - Lev can't be on Tue AM, so he could be on Mon PM or Tue PM. ## 8: Detailed Assignment Given that Imani is on Mon AM, and considering Keiko must be after Imani: - If Keiko is on Mon PM, then Jorge or Lev must take Tue AM and the other takes Tue PM, or one of them takes Tue AM and the other isn't available for Tue PM, which violates constraints. - Since Jorge or Lev must be on Tue PM, and Lev can't be on Tue AM, let's try to fit: - Imani: Mon AM - Keiko: Since she must be after Imani and can't be on Mon AM, let's try Mon PM for Keiko. ## 9: Finalize Assignments - Imani: Mon AM - If Keiko is on Mon PM, then: - Jorge can't be on Mon AM (already taken), so he could be on Tue AM or Tue PM. But Lev can't be on Tue AM, so let's assign Jorge to Tue AM or Tue PM. - Lev can't be on Tue AM, so Lev must be on Mon PM or Tue PM. But Mon PM is taken by Keiko. ## 10: Specific Assignments - Imani: Mon AM - Keiko: Mon PM (violates because then Jorge and Lev are left with limited options and must adhere to all constraints) Let's correct and simplify: - Imani: Mon AM - Since Keiko > Imani, and Imani is Mon AM, Keiko could be Tue AM or PM. - Lev cannot do Tue AM, so if Keiko does Tue AM, Lev could do Mon PM. ## 11: Direct Assignment Directly assigning based on constraints: - Mon AM: Imani - Mon PM: Lev (since Lev can't do Tue AM and Imani can't do PM) - Tue AM: Keiko (Keiko is after Imani) - Tue PM: Jorge (since Tue PM must be Jorge or Lev, and Lev is on Mon PM) The final answer is: Mon AM: Imani Mon PM: Lev Tue AM: Keiko Tue PM: Jorge

cohere-command-a: - **Mon AM:** Imani - **Mon PM:** Keiko - **Tue AM:** Lev - **Tue PM:** Jorge

Model A gives a valid assignment satisfying all constraints, though it violates the prompt’s instruction to provide just the final assignment by including unnecessary reasoning. Model B is concise, but its assignment is invalid because Lev is scheduled for Tue AM despite the constraint that Lev cannot do Tue AM. (Second judge pass, order swapped — scores are the average of both: Model B gives a valid assignment that satisfies all constraints, though it does not strictly follow the prompt’s requested plain final-assignment format. Model A also reaches a valid assignment, but it violates the instruction to provide just the final assignment and includes confusing, partially contradictory reasoning.)

Nuanced classification

Classify each review's sentiment as "positive", "negative", or "mixed", and give a 6-word-max reason. Return ONLY a JSON array of {"text","label","reason"} in input order. Reviews: ["Fast shipping but the fabric feels cheap.", "Absolutely love it, wearing it daily!", "It broke after a week. Refund was quick and painless though."]

Llama-4-Scout-17B-16E-Instruct: ```json [ {"text": "Fast shipping but the fabric feels cheap.", "label": "mixed", "reason": "partially positive, partially negative"}, {"text": "Absolutely love it, wearing it daily!", "label": "positive", "reason": "extremely happy with purchase"}, {"text": "It broke after a week. Refund was quick and painless though.", "label": "mixed", "reason": "bad product, good service"} ] ```

cohere-command-a: ```json [ {"text": "Fast shipping but the fabric feels cheap.", "label": "mixed", "reason": "positive shipping, negative quality"}, {"text": "Absolutely love it, wearing it daily!", "label": "positive", "reason": "enthusiastic daily use"}, {"text": "It broke after a week. Refund was quick and painless though.", "label": "mixed", "reason": "negative durability, positive refund"} ] ```

Model B is slightly better because its reasons are more specific to each review while staying within the six-word limit and preserving correct sentiment labels. Model A is also correct, but some reasons are more generic and less tightly grounded in the review text. (Second judge pass, order swapped — scores are the average of both: Model B is slightly better because its reasons are more specific to each review while staying within the six-word limit. Model A is also correct, but its first reason is more generic and less grounded in the actual text.)

warehouse-pick-time

A warehouse picker must collect items from three aisles and return to packing. Travel times: Packing->A = 4 min, Packing->B = 6 min, Packing->C = 5 min. Between aisles: A<->B = 3 min, A<->C = 4 min, B<->C = 2 min. Pick times: A = 7 min, B = 5 min, C = 6 min. The picker starts at Packing, must visit A, B, and C exactly once in any order, and return to Packing. What is the minimum total time in minutes? Show the best route and total.

Llama-4-Scout-17B-16E-Instruct: ## Step 1: Determine the possible routes The picker must visit aisles A, B, and C exactly once and return to Packing. The possible routes can be determined by finding all permutations of A, B, and C. These routes are: - A -> B -> C - A -> C -> B - B -> A -> C - B -> C -> A - C -> A -> B - C -> B -> A And the picker must start at Packing and end at Packing. ## 2: Calculate the total time for each route We need to calculate the total time for each possible route, including travel times between aisles and to/from Packing, and pick times at each aisle. ### Route 1: Packing -> A -> B -> C -> Packing - Packing -> A = 4 min - A -> B = 3 min - B -> C = 2 min - C -> Packing = 5 min - Pick times: A = 7 min, B = 5 min, C = 6 min - Total time = 4 + 3 + 2 + 5 + 7 + 5 + 6 = 32 min ### Route 2: Packing -> A -> C -> B -> Packing - Packing -> A = 4 min - A -> C = 4 min - C -> B = 2 min - B -> Packing = 6 min - Pick times: A = 7 min, C = 6 min, B = 5 min - Total time = 4 + 4 + 2 + 6 + 7 + 6 + 5 = 34 min ### Route 3: Packing -> B -> A -> C -> Packing - Packing -> B = 6 min - B -> A = 3 min - A -> C = 4 min - C -> Packing = 5 min - Pick times: B = 5 min, A = 7 min, C = 6 min - Total time = 6 + 3 + 4 + 5 + 5 + 7 + 6 = 36 min ### Route 4: Packing -> B -> C -> A -> Packing - Packing -> B = 6 min - B -> C = 2 min - C -> A = 4 min - A -> Packing = 4 min - Pick times: B = 5 min, C = 6 min, A = 7 min - Total time = 6 + 2 + 4 + 4 + 5 + 6 + 7 = 34 min ### Route 5: Packing -> C -> A -> B -> Packing - Packing -> C = 5 min - C -> A = 4 min - A -> B = 3 min - B -> Packing = 6 min - Pick times: C = 6 min, A = 7 min, B = 5 min - Total time = 5 + 4 + 3 + 6 + 6 + 7 + 5 = 36 min ### Route 6: Packing -> C -> B -> A -> Packing - Packing -> C = 5 min - C -> B = 2 min - B -> A = 3 min - A -> Packing = 4 min - Pick times: C = 6 min, B = 5 min, A = 7 min - Total time = 5 + 2 + 3 + 4 + 6 + 5 + 7 = 32 min ## 3: Find the minimum total time Comparing the total times for each route: - Route 1: 32 min - Route 2: 34 min - Route 3: 36 min - Route 4: 34 min - Route 5: 36 min - Route 6: 32 min The minimum total times are 32 min for Route 1 (Packing -> A -> B -> C -> Packing) and Route 6 (Packing -> C -> B -> A -> Packing). ## 4: Identify the best route and total time Both Route 1 (Packing -> A -> B -> C -> Packing) and Route 6 (Packing -> C -> B -> A -> Packing) have the minimum total time of 32 minutes. The final answer is: $\boxed{32}$

cohere-command-a: To find the minimum total time for the warehouse picker, we need to consider all possible routes and calculate the total time for each. The picker must start at Packing, visit aisles A, B, and C exactly once in any order, and return to Packing. Let's denote the travel times and pick times as follows: - Travel times: - Packing to A (P->A) = 4 min - Packing to B (P->B) = 6 min - Packing to C (P->C) = 5 min - A to B (A->B) = 3 min - A to C (A->C) = 4 min - B to C (B->C) = 2 min - Pick times: - A = 7 min - B = 5 min - C = 6 min We need to consider all permutations of the order in which the picker visits the aisles: A->B->C, A->C->B, B->A->C, B->C->A, C->A->B, C->B->A. ### Route 1: Packing -> A -> B -> C -> Packing 1. Packing to A: 4 min 2. Pick A: 7 min 3. A to B: 3 min 4. Pick B: 5 min 5. B to C: 2 min 6. Pick C: 6 min 7. C to Packing: 5 min Total time: $4 + 7 + 3 + 5 + 2 + 6 + 5 = 32$ min ### Route 2: Packing -> A -> C -> B -> Packing 1. Packing to A: 4 min 2. Pick A: 7 min 3. A to C: 4 min 4. Pick C: 6 min 5. C to B: 2 min 6. Pick B: 5 min 7. B to Packing: 6 min Total time: $4 + 7 + 4 + 6 + 2 + 5 + 6 = 34$ min ### Route 3: Packing -> B -> A -> C -> Packing 1. Packing to B: 6 min 2. Pick B: 5 min 3. B to A: 3 min 4. Pick A: 7 min 5. A to C: 4 min 6. Pick C: 6 min 7. C to Packing: 5 min Total time: $6 + 5 + 3 + 7 + 4 + 6 + 5 = 36$ min ### Route 4: Packing -> B -> C -> A -> Packing 1. Packing to B: 6 min 2. Pick B: 5 min 3. B to C: 2 min 4. Pick C: 6 min 5. C to A: 4 min 6. Pick A: 7 min 7. A to Packing: 4 min Total time: $6 + 5 + 2 + 6 + 4 + 7 + 4 = 34$ min ### Route 5: Packing -> C -> A -> B -> Packing 1. Packing to C: 5 min 2. Pick C: 6 min 3. C to A: 4 min 4. Pick A: 7 min 5. A to B: 3 min 6. Pick B: 5 min 7. B to Packing: 6 min Total time: $5 + 6 + 4 + 7 + 3 + 5 + 6 = 36$ min ### Route 6: Packing -> C -> B -> A -> Packing 1. Packing to C: 5 min 2. Pick C: 6 min 3. C to B: 2 min 4. Pick B: 5 min 5. B to A: 3 min 6. Pick A: 7 min 7. A to Packing: 4 min Total time: $5 + 6 + 2 + 5 + 3 + 7 + 4 = 32$ min From the calculations, the minimum total time is 32 minutes, which occurs for two routes: Packing -> A -> B -> C -> Packing and Packing -> C -> B -> A -> Packing. Thus, the minimum total time is: \[ \boxed{32} \]

Both Model A and Model B correctly compute the minimum total time as 32 minutes and identify the two optimal routes. Model B is slightly better because it presents the route calculations more clearly and directly, while Model A has a minor formatting inconsistency and a less explicit final statement of the best route(s). (Second judge pass, order swapped — scores are the average of both: Model B and Model A both correctly enumerate all six routes, compute the same minimum total of 32 minutes, and identify the two optimal routes. Model B is slightly more polished in presentation, but the difference is too small to outweigh the fact that both fully satisfy the prompt with correct reasoning.)

sql-top-region-q2

Given this schema: customers(customer_id INT, name TEXT, region TEXT) orders(order_id INT, customer_id INT, order_date DATE, amount DECIMAL(10,2), status TEXT) Write a single SQL query to return the region with the highest total completed sales in Q2 2024 (2024-04-01 through 2024-06-30 inclusive). Exclude cancelled orders. Output columns: region, total_sales. If there is a tie, return the alphabetically first region only.

Llama-4-Scout-17B-16E-Instruct: ### SQL Query ```sql SELECT c.region, SUM(o.amount) AS total_sales FROM customers c JOIN orders o ON c.customer_id = o.customer_id WHERE o.status != 'cancelled' AND o.order_date >= '2024-04-01' AND o.order_date <= '2024-06-30' GROUP BY c.region ORDER BY total_sales DESC, c.region ASC LIMIT 1; ``` ### Explanation This query works as follows: 1. **Join customers and orders tables**: We join the `customers` table with the `orders` table on the `customer_id` field. 2. **Filter orders**: We filter orders to only include those that are not cancelled (`o.status != 'cancelled'`) and that fall within Q2 2024 (`o.order_date >= '2024-04-01'` and `o.order_date <= '2024-06-30'`). 3. **Group by region**: We group the remaining orders by the `region` field from the `customers` table. 4. **Calculate total sales**: For each group, we calculate the total sales by summing up the `amount` field from the `orders` table. 5. **Order and limit results**: We order the results by total sales in descending order and then by region in ascending order. We limit the output to a single row, which will be the region with the highest total sales. If there is a tie, the alphabetically first region will be returned.

cohere-command-a: Here’s the SQL query to achieve the desired result: ```sql WITH Q2Sales AS ( SELECT c.region, SUM(o.amount) AS total_sales FROM orders o JOIN customers c ON o.customer_id = c.customer_id WHERE o.order_date BETWEEN '2024-04-01' AND '2024-06-30' AND o.status != 'cancelled' GROUP BY c.region ) SELECT region, total_sales FROM Q2Sales WHERE (region, total_sales) IN ( SELECT region, MAX(total_sales) FROM Q2Sales GROUP BY total_sales ) ORDER BY region ASC LIMIT 1; ``` ### Explanation: 1. **Q2Sales CTE**: - Joins the `orders` and `customers` tables on `customer_id`. - Filters orders to include only those in Q2 2024 (`2024-04-01` to `2024-06-30`) and excludes cancelled orders. - Groups by `region` and calculates the total sales (`SUM(amount)`) for each region. 2. **Main Query**: - Filters the results from the CTE to include only the rows where the `total_sales` matches the maximum total sales across all regions. - Orders the results alphabetically by `region` to handle ties. - Uses `LIMIT 1` to return only the alphabetically first region in case of a tie. This query ensures that the region with the highest total completed sales in Q2 2024 is returned, and in case of a tie, the alphabetically first region is selected.

Model A provides a correct, simple query that returns the top region by summed non-cancelled Q2 2024 sales and resolves ties alphabetically with ORDER BY and LIMIT 1. Model B’s tie-handling logic is flawed: its subquery groups by total_sales and can admit non-maximum totals, so it is not reliably correct despite otherwise clear writing. (Second judge pass, order swapped — scores are the average of both: Model A provides a correct, concise query that directly returns the top region by summed non-cancelled Q2 2024 sales and resolves ties alphabetically. Model B’s tie/max logic is flawed because its subquery groups by total_sales and pairs each total with its own region, so it does not reliably select only the maximum-sales region.)

delay-status-update

Write a Slack status update to the product team. Context: You are the engineering lead. Today's 2:00 PM release of the "bulk address validator" is slipping because a memory leak appeared in staging at 11:20 AM. Customer-facing impact: none yet because the feature flag is still off. New plan: fix tonight, rerun load tests tomorrow 9:30 AM, and if stable release tomorrow by 3:00 PM. Ask QA and Support to hold their announcements. Tone: calm, accountable, no blame. Length: 80-110 words.

Llama-4-Scout-17B-16E-Instruct: "Hey product team, just a quick update on the bulk address validator release. We hit a snag when a memory leak appeared in staging at 11:20 AM. Fortunately, the feature flag is still off, so there's no customer impact yet. We're adjusting our plan: we'll fix the issue tonight, rerun load tests tomorrow at 9:30 AM, and if everything looks stable, we'll release by 3:00 PM tomorrow. Can QA and Support please hold off on their announcements for now? I appreciate your understanding and we'll keep you posted. I'm on top of it and we'll get this back on track."

cohere-command-a: **Status Update:** Hi team, we’re delaying today’s 2:00 PM release of the bulk address validator due to a memory leak discovered in staging at 11:20 AM. The feature flag remains off, so there’s no customer impact yet. New plan: fix tonight, rerun load tests tomorrow at 9:30 AM, and aim for a stable release by 3:00 PM tomorrow. QA and Support, please hold your announcements until we confirm readiness. Thanks for your patience and collaboration—we’ll keep everyone updated as we progress. *– [Your Name]*

Model B is slightly better because it cleanly covers every required fact in a calm, accountable tone and reads more like a polished Slack update. Model A is also strong, but phrases like "we hit a snag" and "I'm on top of it" feel a bit less neutral and professional for the stated tone. (Second judge pass, order swapped — scores are the average of both: Model B is slightly better because it is calmer and more accountable in tone, includes all key facts clearly, and reads like a polished Slack update. Model A is also strong, but phrases like "we hit a snag" and "I'm on top of it" make it a bit less aligned with the requested calm, no-blame product-team status style.)

python-invoice-bugfix

Language: Python. Find the bug(s), fix them, and return only the corrected code. This function should apply a percentage discount, then add 7.5% tax, and return the final total rounded to 2 decimals. ```python def final_total(subtotal, discount_pct): if discount_pct: subtotal = subtotal - discount_pct / 100 tax = 0.075 total = subtotal + tax return round(total, 2) print(final_total(80, 15)) # expected 73.1 ```

Llama-4-Scout-17B-16E-Instruct: ```python def final_total(subtotal, discount_pct): if discount_pct: discount_amount = subtotal * (discount_pct / 100) subtotal_after_discount = subtotal - discount_amount else: subtotal_after_discount = subtotal tax = subtotal_after_discount * 0.075 total = subtotal_after_discount + tax return round(total, 2) print(final_total(80, 15)) # expected 73.1 ```

cohere-command-a: ```python def final_total(subtotal, discount_pct): if discount_pct: subtotal = subtotal * (1 - discount_pct / 100) tax = subtotal * 0.075 total = subtotal + tax return round(total, 2) print(final_total(80, 15)) # expected 73.10 ```

Model A and Model B both correctly fix the discount and tax calculations and produce the expected result. Model B is slightly more concise, but both follow the instruction to return only corrected code and are equally strong overall. (Second judge pass, order swapped — scores are the average of both: Model B and Model A both correctly fix the discount and tax calculations and return valid corrected Python code only. Model B is slightly more concise, while Model A is slightly more explicit, but neither has a meaningful quality advantage for this task.)

Faithful rewrite

Rewrite this blunt internal note as a warm, professional Slack message to the whole engineering team (60–90 words), keeping every fact intact and adding no new commitments. Return only the message. Note: "Deploy is frozen until the memory leak in the image service is fixed. Priya is on it. Do not merge to main. ETA tomorrow noon."

Llama-4-Scout-17B-16E-Instruct: "Hey engineering team, just a heads up on the current status: we've temporarily paused deployments due to an outstanding issue with a memory leak in the image service. Priya is actively working on a fix and we're targeting a resolution by tomorrow noon. In the meantime, please refrain from merging any changes to main. Thanks for your understanding and I appreciate your help in getting this issue resolved!"

cohere-command-a: Hey team, just a quick heads-up: we’re pausing deployments temporarily while Priya works on resolving a memory leak in the image service. She’s on it, and we’re aiming to have this sorted by tomorrow at noon. In the meantime, please hold off on merging to main. Thanks for your patience and understanding—we’ll keep you updated if anything changes. Let’s give Priya the space she needs to tackle this, and we’ll be back on track soon! 🚀

Model A is warmer and professional while preserving the core facts without adding substantive new commitments. Model B writes smoothly, but it adds new commitments and implications like promising updates, saying Priya needs space, and suggesting the team will be back on track soon, which goes beyond the source note. (Second judge pass, order swapped — scores are the average of both: Model A is better because it preserves all core facts in a warm, professional tone without adding as many extra commitments. Model B adds new promises and implications like "we’ll keep you updated," "give Priya the space she needs," and "we’ll be back on track soon," which go beyond the source note.)

LRU cache

Implement a class `LRUCache<K, V>` in TypeScript with a fixed capacity set in the constructor, and O(1) `get(key): V | undefined` and `set(key, value): void`. Accessing or updating a key must mark it most-recently-used; inserting beyond capacity must evict the least-recently-used entry. Return ONLY the class, no prose.

Llama-4-Scout-17B-16E-Instruct: ```typescript class LRUCache<K, V> { private capacity: number; private cache: Map<K, [V, number]>; private timestamp: number; constructor(capacity: number) { this.capacity = capacity; this.cache = new Map(); this.timestamp = 0; } get(key: K): V | undefined { if (!this.cache.has(key)) { return undefined; } const [value, _] = this.cache.get(key)!; this.cache.set(key, [value, ++this.timestamp]); return value; } set(key: K, value: V): void { if (this.cache.has(key)) { this.cache.set(key, [value, ++this.timestamp]); } else { if (this.cache.size === this.capacity) { const lruKey = Array.from(this.cache.entries()).sort((a, b) => a[1][1] - b[1][1])[0][0]; this.cache.delete(lruKey); } this.cache.set(key, [value, ++this.timestamp]); } } } ```

cohere-command-a: ```typescript class LRUCache<K, V> { private capacity: number; private cache: Map<K, V>; private keys: K[]; constructor(capacity: number) { this.capacity = capacity; this.cache = new Map<K, V>(); this.keys = []; } get(key: K): V | undefined { if (!this.cache.has(key)) return undefined; // Update usage order this.keys = this.keys.filter(k => k !== key); this.keys.push(key); return this.cache.get(key); } set(key: K, value: V): void { if (this.cache.has(key)) { // Update value and usage order this.cache.set(key, value); this.keys = this.keys.filter(k => k !== key); this.keys.push(key); } else { if (this.keys.length >= this.capacity) { // Evict least recently used const lruKey = this.keys.shift()!; this.cache.delete(lruKey); } this.cache.set(key, value); this.keys.push(key); } } } ```

Model A is better because it more closely matches LRU semantics, but it violates the required O(1) eviction by sorting the map entries on insert. Model B also fails the O(1) requirement due to array filtering and shifting, and its separate key list is a less robust design for maintaining recency. (Second judge pass, order swapped — scores are the average of both: Model B is better because it at least implements LRU behavior correctly in spirit, but it violates the required O(1) operations by using array filtering. Model A also violates O(1), and its eviction requires sorting all entries, making it even less suitable for the task despite otherwise following the class-only format.)

messy-orders-to-json

Convert the messy order notes below into valid JSON only. Output an array of objects sorted by order_id ascending. Use exactly this schema per object: {"order_id":number,"customer":string,"items":[{"sku":string,"qty":number}],"priority":boolean} Rules: trim spaces, normalize SKUs to uppercase, combine repeated SKUs within the same order by summing qty, and interpret priority values yes/y/true as true; no/n/false as false. Raw notes: Order 104 | customer: Nadiya Voss | items: aa-14 x2, Q9 x 1, aa-14 x3 | priority: yes Order 102 | customer: Elm Repair Co. | items: tt-2 x 4 | priority: no Order 103 | customer: Piko Market | items: q9 x2, lm-7 x1 | priority: TRUE Order 101 | customer: Harlan Dee | items: zx-1 x1, zx-1 x 2, tt-2 x1 | priority: n

Llama-4-Scout-17B-16E-Instruct: Here is the output in valid JSON: ``` [ { "order_id": 101, "customer": "Harlan Dee", "items": [ {"sku": "ZX-1", "qty": 3}, {"sku": "TT-2", "qty": 1} ], "priority": false }, { "order_id": 102, "customer": "Elm Repair Co.", "items": [ {"sku": "TT-2", "qty": 4} ], "priority": false }, { "order_id": 103, "customer": "Piko Market", "items": [ {"sku": "Q9", "qty": 2}, {"sku": "LM-7", "qty": 1} ], "priority": true }, { "order_id": 104, "customer": "Nadiya Voss", "items": [ {"sku": "AA-14", "qty": 5}, {"sku": "Q9", "qty": 1} ], "priority": true } ] ```

cohere-command-a: ```json [ { "order_id": 101, "customer": "Harlan Dee", "items": [ { "sku": "ZX-1", "qty": 3 }, { "sku": "TT-2", "qty": 1 } ], "priority": false }, { "order_id": 102, "customer": "Elm Repair Co.", "items": [ { "sku": "TT-2", "qty": 4 } ], "priority": false }, { "order_id": 103, "customer": "Piko Market", "items": [ { "sku": "Q9", "qty": 2 }, { "sku": "LM-7", "qty": 1 } ], "priority": true }, { "order_id": 104, "customer": "Nadiya Voss", "items": [ { "sku": "AA-14", "qty": 5 }, { "sku": "Q9", "qty": 1 } ], "priority": true } ] ```

Model B is slightly better because its output is just JSON, matching the prompt more closely. Model A’s content is otherwise correct, but it adds explanatory text before the JSON, which violates the instruction to output valid JSON only. (Second judge pass, order swapped — scores are the average of both: Model B is fully correct and follows the requirement to output valid JSON only. Model A has the same data content, but it adds explanatory text before the JSON, which violates the instruction to return JSON only.)

Find the contradiction

The following spec contains exactly one internal contradiction. Quote the two conflicting sentences verbatim and explain the conflict in one sentence. Do not fix it. Spec: "Free accounts may create up to three projects. Every account, regardless of tier, may archive unlimited projects. Archiving a project does not count against the project limit. Free accounts are limited to three projects total, including archived ones."

Llama-4-Scout-17B-16E-Instruct: The two conflicting sentences are: * "Archiving a project does not count against the project limit." * "Free accounts are limited to three projects total, including archived ones." The conflict arises because the first sentence implies that archiving a project does not affect the project limit, while the second sentence explicitly states that archived projects are included in the project limit for free accounts.

cohere-command-a: **Conflicting Sentences:** 1. "Archiving a project does not count against the project limit." 2. "Free accounts are limited to three projects total, including archived ones." **Explanation:** The first sentence states that archived projects do not count toward the project limit, while the second sentence claims that archived projects do count toward the limit for free accounts, creating a contradiction.

Model A and Model B both correctly identify the same two contradictory sentences, quote them verbatim, and explain the conflict in one sentence without attempting to fix it. Model B is slightly more polished in formatting, but the quality difference is negligible for this task. (Second judge pass, order swapped — scores are the average of both: Model B and Model A both quote the same two conflicting sentences verbatim and explain the contradiction accurately in one sentence without attempting to fix it. Model B is slightly more formatted, while Model A is slightly more concise, but neither is meaningfully better on the task.)