This matchup wasn’t subtle: one model consistently executed, while the other kept leaking points on instruction-following and avoidable implementation mistakes. The scoreline says decisive, but the task breakdown shows exactly why.
DeepSeek-V4-Pro wins this head-to-head cleanly and convincingly. It posts a **101.3 to 86.2** aggregate advantage, takes **9 of 12 tasks**, and does it with a **98% confidence** verdict. That is not a marginal edge or a judge-quirk result; it’s a broad performance gap.
What stands out is *how* DeepSeek won. It was repeatedly better at the unglamorous but essential stuff that decides real-world usefulness: returning **only** the SQL when asked, producing runnable SQL instead of broken window-query output, keeping localization natural and audience-appropriate, and writing tighter code in Python and TypeScript. On tasks like the SQL top unresolved bug component, Brazilian Portuguese outage localization, the window-function query, campaign-name normalization, unit-aware math, vendor-change extraction, and the LRU cache, DeepSeek was simply the more reliable model.
Llama-4-Maverick-17B-128E-Instruct-FP8 did have two legitimate bright spots. It won **strict JSON extraction** by following the title-case normalization requirement more precisely, and it edged **audit interview assignment**—though that says more about DeepSeek’s self-contradictory failure than about Maverick producing a good answer, since both models ended with invalid schedules. There was also one tie on nuanced classification, a split-decision kind of result where both got the labels right but each stumbled on formatting in different ways.
The bigger problem for Llama is that too many of its losses were self-inflicted. It added prose where the prompt explicitly asked for a single query, inserted unnecessary quotation marks in localization, and, most damagingly, produced a SQL window solution with a fatal column-reference error. Those are not stylistic nits; they are the kinds of misses that break downstream workflows. DeepSeek wasn’t perfect—there was at least one mixed judgment on the enterprise support rewrite and some disagreement on vendor-change extraction—but across the full slate it was the steadier, sharper model.
**Final call: DeepSeek-V4-Pro is the clear winner. Llama-4-Maverick-17B-128E-Instruct-FP8 shows flashes of strong formatting discipline, but DeepSeek was better where it matters most: correctness, compliance, and shipping answers that actually work.**
Strict JSON extraction
Extract every meeting from this text into a JSON array of {"title","day","start24","durationMinutes"}. Use 24-hour "HH:MM" for start24, integer minutes for duration, and title-case titles. Return ONLY valid JSON, no commentary.
Text: "Standup is daily at 9am for a quarter hour. The design review runs thursday from 2:30-4pm. Payroll sync — first monday, 11:00 to noon."
DeepSeek-V4-Pro: ```json
[
{
"title": "Standup",
"day": "daily",
"start24": "09:00",
"durationMinutes": 15
},
{
"title": "Design Review",
"day": "thursday",
"start24": "14:30",
"durationMinutes": 90
},
{
"title": "Payroll Sync",
"day": "first monday",
"start24": "11:00",
"durationMinutes": 60
}
]
```
Llama-4-Maverick-17B-128E-Instruct-FP8: ```json
[
{"title":"Standup","day":"Daily","start24":"09:00","durationMinutes":15},
{"title":"Design Review","day":"Thursday","start24":"14:30","durationMinutes":90},
{"title":"Payroll Sync","day":"First Monday","start24":"11:00","durationMinutes":60}
]
```
Model B better satisfies the formatting instruction by using title case consistently for the textual fields, while Model A leaves the day values in lowercase. Both models correctly extract the meetings, times, and durations, but Model B adheres more closely to the requested normalization. (Second judge pass, order swapped — scores are the average of both: Model B is better because it correctly normalizes the day values into title case while preserving the extracted schedule details. Model A has the same meeting times and durations, but it fails the title-case-style requirement for the day fields, making it less compliant with the formatting instructions.)
SQL top unresolved bug component
Write a single SQL query that answers this exactly:
Return the component_name and unresolved_bug_count for the component with the highest number of unresolved bugs created in Q1 2025 by non-internal reporters. If there is a tie, return the alphabetically earliest component_name.
Schema:
components(id INTEGER PRIMARY KEY, component_name TEXT)
users(id INTEGER PRIMARY KEY, email TEXT, is_internal BOOLEAN)
issues(id INTEGER PRIMARY KEY, component_id INTEGER, reporter_id INTEGER, kind TEXT, status TEXT, created_at TIMESTAMP)
Rules:
- "unresolved" means status NOT IN ('resolved','closed')
- only kind = 'bug'
- Q1 2025 means created_at >= '2025-01-01' AND created_at < '2025-04-01'
- non-internal reporters means users.is_internal = FALSE
Return one row only.
DeepSeek-V4-Pro: ```sql
SELECT
c.component_name,
COUNT(*) AS unresolved_bug_count
FROM issues i
JOIN components c ON i.component_id = c.id
JOIN users u ON i.reporter_id = u.id
WHERE i.kind = 'bug'
AND i.status NOT IN ('resolved', 'closed')
AND i.created_at >= '2025-01-01'
AND i.created_at < '2025-04-01'
AND u.is_internal = FALSE
GROUP BY c.component_name
ORDER BY unresolved_bug_count DESC, c.component_name ASC
LIMIT 1;
```
Llama-4-Maverick-17B-128E-Instruct-FP8: ### SQL Query to Retrieve Component with Highest Unresolved Bugs
```sql
SELECT
c.component_name,
COUNT(i.id) AS unresolved_bug_count
FROM
issues i
JOIN
components c ON i.component_id = c.id
JOIN
users u ON i.reporter_id = u.id
WHERE
i.kind = 'bug'
AND i.status NOT IN ('resolved', 'closed')
AND i.created_at >= '2025-01-01'
AND i.created_at < '2025-04-01'
AND u.is_internal = FALSE
GROUP BY
c.component_name
ORDER BY
COUNT(i.id) DESC,
c.component_name ASC
LIMIT 1;
```
### Explanation
1. **Join Tables**: The query starts by joining the `issues`, `components`, and `users` tables based on their respective IDs to gather all necessary information.
2. **Filter Issues**: It filters issues that are bugs (`kind = 'bug'`), not resolved or closed (`status NOT IN ('resolved', 'closed')`), created in Q1 2025 (`created_at` within the specified range), and reported by non-internal users (`is_internal = FALSE`).
3. **Group by Component**: The results are grouped by `component_name` to count the number of unresolved bugs for each component.
4. **Order Results**: The components are ordered in descending order based on the count of unresolved bugs. In case of a tie, they are ordered alphabetically by `component_name` in ascending order.
5. **Limit to Top Result**: The query returns only the top row, which corresponds to the component with the highest number of unresolved bugs (and the alphabetically earliest name in case of a tie).
Model A and Model B provide equivalently correct SQL logic, but Model A better follows the prompt by returning just a single SQL query with no extra prose. Model B adds explanatory text despite the instruction to write a single SQL query, so Model A is the better overall response. (Second judge pass, order swapped — scores are the average of both: Model A provides a correct single SQL query that directly satisfies the prompt. Model B's SQL is also correct, but it violates the instruction to write a single SQL query by adding explanatory text and a heading, so Model A is better overall.)
Localize outage notice to Brazilian Portuguese
Translate this customer-facing notice into Brazilian Portuguese. Tone: calm, professional, empathetic. Length constraint: 55–70 words. Keep all specifics and the apology.
"We found an issue that delayed some payout notifications between 14:10 and 15:35 UTC on 3 September. The payouts themselves were not affected, and no action is required from you. We have re-sent the missing notifications and are monitoring the queue closely. We're sorry for the confusion."
DeepSeek-V4-Pro: Identificamos um problema que atrasou algumas notificações de pagamento entre 14:10 e 15:35 UTC do dia 3 de setembro. Os pagamentos em si não foram afetados e nenhuma ação é necessária de sua parte. Reenviamos as notificações pendentes e estamos monitorando a fila de perto. Lamentamos pelo transtorno causado.
Llama-4-Maverick-17B-128E-Instruct-FP8: "Identificamos um problema que atrasou algumas notificações de pagamento entre 14h10 e 15h35 UTC em 3 de setembro. Os pagamentos não foram afetados e nenhuma ação é necessária. Reenviamos as notificações pendentes e estamos monitorando a fila de perto. Pedimos desculpas pelo transtorno causado."
Model A better preserves all specifics while sounding calm, professional, and empathetic; it also explicitly keeps the customer-directed detail that no action is required from the recipient. Model B is strong, but it drops the explicit "from you" nuance and adds unnecessary quotation marks, making it slightly less polished for the task. (Second judge pass, order swapped — scores are the average of both: Model A is slightly better because it preserves more of the original meaning and customer-facing tone, especially with "de sua parte" for "from you" and "Os pagamentos em si" for "The payouts themselves." Model B is also strong, but it omits that nuance and its closing apology is a bit less direct than the source.)
Rewrite enterprise support update
Rewrite this internal note as a polished email to an enterprise customer. Audience: IT director at a university. Tone: accountable and reassuring, not defensive. Length: 120–150 words.
Source note:
"We rolled the SSO cert on the wrong app shard last night, so a subset of North Campus users couldn't log in from about 07:12 to 08:03 local time. Admin accounts and API tokens were fine. We reverted, validated with two faculty accounts, and queued the correct cert for tonight 23:30 with a rollback plan. No evidence of data access problems. Need to own the mistake, mention next update by 10:30, and avoid too much engineering jargon."
DeepSeek-V4-Pro: Subject: Brief Sign-In Disruption This Morning – Resolution and Next Steps
I want to make you aware of an issue that affected a subset of North Campus users this morning and share the actions we’ve taken.
During a planned certificate update at 07:12 local time, the change was inadvertently applied on the incorrect shard. As a result, some users were unable to sign in until we reverted the change at 08:03. Admin accounts and API-based access were not affected, and we have confirmed there was no impact to data.
We’ve validated the fix with test accounts, and the correct certificate update is now scheduled for tonight at 23:30, with a clear rollback plan in place. I take full ownership of this mistake and the disruption it caused. You’ll receive a complete summary by 10:30 tomorrow. Please don’t hesitate to reach out if you have any urgent questions before then.
Llama-4-Maverick-17B-128E-Instruct-FP8: Subject: Update on SSO Service Disruption
Dear [IT Director's Name],
I am writing to update you on a service disruption that affected a subset of North Campus users who were unable to log in to our application between 07:12 and 08:03 local time last night. We have identified the root cause as an incorrect Single Sign-On (SSO) certificate update on one of our application shards. Our administrative accounts and API tokens were not affected. We have taken corrective action and are re-attempting the certificate update tonight at 23:30 with a rollback plan in place. We have verified that there were no data access issues during the disruption. We take full responsibility for the mistake and are committed to preventing similar incidents in the future. We will provide a further update by 10:30 today.
Best regards,
[Your Name]
Model A is more polished and reassuring while still owning the mistake, and it avoids excessive jargon better than Model B. Model B is solid but introduces a timing inconsistency (“last night” vs. the stated local-time window and “by 10:30 today”), and its wording is slightly more technical and less tailored to the requested tone. (Second judge pass, order swapped — scores are the average of both: Model B is better because it stays closer to the requested accountable, reassuring customer-facing tone and correctly includes the required next update timing. Model A is polished, but it introduces a likely timing error by saying the summary will arrive by 10:30 tomorrow instead of by 10:30, and its wording is slightly more technical than requested.)
SQL window query
Given Postgres tables `orders(id, customer_id, created_at, total_cents)` and `customers(id, name)`, write ONE SQL query returning each customer's name and their SECOND-largest order total (in dollars, 2 decimals), excluding customers with fewer than two orders. Order by that value descending. Return ONLY the SQL.
DeepSeek-V4-Pro: ```sql
SELECT
c.name,
ROUND(o.total_cents / 100.0, 2) AS second_largest_order_dollars
FROM (
SELECT
customer_id,
total_cents,
ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY total_cents DESC) AS rn
FROM orders
) o
JOIN customers c ON c.id = o.customer_id
WHERE o.rn = 2
ORDER BY second_largest_order_dollars DESC;
```
Llama-4-Maverick-17B-128E-Instruct-FP8: ```sql
WITH ranked_orders AS (
SELECT c.name, o.total_cents / 100.0 AS total_dollars,
ROW_NUMBER() OVER (PARTITION BY c.id ORDER BY o.total_cents DESC) AS rank
FROM customers c
JOIN orders o ON c.id = o.customer_id
),
order_counts AS (
SELECT customer_id, COUNT(*) AS num_orders
FROM orders
GROUP BY customer_id
HAVING COUNT(*) >= 2
)
SELECT name, ROUND(total_dollars, 2) AS second_largest_order
FROM ranked_orders
WHERE rank = 2 AND customer_id IN (SELECT customer_id FROM order_counts)
ORDER BY total_dollars DESC;
```
Model A correctly returns each customer's second-highest order using a window function, excludes customers with fewer than two orders implicitly, formats dollars to two decimals, and orders by that value descending. Model B has a correctness issue because the outer query references customer_id even though it is not selected in ranked_orders, so the SQL would fail. (Second judge pass, order swapped — scores are the average of both: Model A correctly uses a window function to select each customer's second-highest order and naturally excludes customers with fewer than two orders. Model B has a fatal error because its outer query references customer_id even though that column is not selected in the ranked_orders CTE, so the SQL would not run as written.)
Localization with tone
Translate this app onboarding line into natural, friendly European Spanish suitable for a mobile toast (keep it under 60 characters, no exclamation marks): "You're all set — your first backup starts tonight." Return only the translation, then the character count in parentheses.
DeepSeek-V4-Pro: Todo listo, la copia empieza esta noche (45)
Llama-4-Maverick-17B-128E-Instruct-FP8: "Ya está listo, la primera copia de seguridad comienza esta noche." (56)
Model A is more natural and concise for a mobile toast, stays under 60 characters, and follows the requested output format without extra punctuation. Model B is understandable but less natural in European Spanish, includes quotation marks not requested, and uses a heavier phrasing for a toast message. (Second judge pass, order swapped — scores are the average of both: Model A is more natural and toast-friendly in European Spanish, stays under 60 characters, and avoids unnecessary verbosity. Model B sounds less natural for onboarding, includes quotation marks despite the return-format instruction, and its stated character count appears incorrect while the text itself exceeds the limit.)
Assign audit interviews
Schedule four 30-minute audit interviews on Monday using these slots: 09:00, 09:30, 10:00, 10:30. One person per slot. Return the final schedule as JSON mapping slot to person.
People and constraints:
- Mara cannot do 09:00 and must be earlier than Tovin.
- Tovin cannot do 10:30.
- Elise must be either first or last.
- Jae cannot do consecutive-to-Mara's slot (adjacent times count as consecutive) and cannot do 09:30.
- The 10:00 slot must be filled by someone whose name contains the letter "e".
There is exactly one valid schedule.
DeepSeek-V4-Pro: Let’s go step-by-step.
**Slots:** 09:00, 09:30, 10:00, 10:30
**People:** Mara, Tovin, Elise, Jae
---
**1. Elise must be first or last** → Possible slots: 09:00 or 10:30.
**2. Mara cannot do 09:00** → Mara is *not* at 09:00.
**3. Mara must be earlier than Tovin** → Mara’s slot < Tovin’s slot.
**4. Tovin cannot do 10:30** → Tovin is *not* last.
**5. Jae cannot do 09:30** → Jae is *not* 09:30.
**6. Jae cannot be consecutive to Mara** → If Mara is at some slot, Jae can’t be right before or right after.
**7. 10:00 must be filled by someone with “e” in name:**
- Mara → has no “e”
- Tovin → has no “e”
- Elise → has “e” ✅
- Jae → has “e” ✅
→ 10:00 = Elise or Jae.
---
### Try possibilities:
#### Case 1: Elise at 09:00 (first slot)
- Then 10:00 must be Jae (since Elise is at 09:00, only Jae qualifies for 10:00).
- Mara must be earlier than Tovin and not at 09:00 → possible slots for Mara: 09:30.
- If Mara at 09:30, Jae at 10:00 would be adjacent → violates “Jae not consecutive to Mara.” ❌
So Elise cannot be first.
---
#### Case 2: Elise at 10:30 (last slot)
- Then 10:00 must be Jae (since Elise at 10:30, and 10:00 needs “e”).
- Mara earlier than Tovin; Mara cannot be 09:00, so Mara = 09:30.
- Tovin must be after Mara → Tovin = 10:00 is not possible (Jae is there), so Tovin = 09:00.
- Check adjacency: Mara at 09:30, Jae at 10:00 → adjacent → violates constraint. ❌
---
#### Case 3: Elise at 10:00
- That satisfies “10:00 has ‘e’”.
- Elise not first or last? Wait—Elise must be first or last → This fails. ❌
---
#### Case 4: Jae at 10:00, Elise last
We already tried: Elise at 10:30, Jae at 10:00 → adjacency issue with Mara at 09:30.
---
The only arrangement that avoids adjacency problems is:
- Elise at 09:00 (first)
- Mara at 09:30
- Tovin at 10:00 (contains “e”? No “e”, fails) → So that fails.
---
Actually, checking carefully: Tovin has no “e”, so 10:00 cannot be Tovin.
So 10:00 must be Jae or Elise.
If Elise = 09:00, Jae = 10:00 → Mara at 09:30 → adjacent to Jae → fails.
If Elise = 10:30, Jae = 10:00 → Mara at 09:30 → adjacent to Jae → fails.
The only way to avoid adjacency: Mara at 09:00? But Mara cannot do 09:00.
---
The only arrangement that satisfies all:
- **09:00** – Tovin
- **09:30** – Mara
- **10:00** – Jae
- **10:30** – Elise
Check:
- Mara not 09:00 ✅
- Mara earlier than Tovin ❌ (Mara at 09:30, Tovin at 09:00 → Tovin is earlier) → Fails.
---
After working through, the one valid schedule is:
- **09:00** – Elise
- **09:30** – Tovin
- **10:00** – Jae
- **10:30** – Mara
Check:
- Elise first ✅
- Mara not 09:00 ✅
- Mara earlier than Tovin ❌ (Mara at 10:30, Tovin at 09:30 → Mara later) → fails.
---
Given the constraints, the only fully valid mapping is:
```json
{
"09:00": "Elise",
"09:30": "Mara",
"10:00": "Tovin",
"10:30": "Jae"
}
```
Wait—10:00 must have “e” → Tovin has no “e,” so this fails.
After elimination, the correct schedule is:
```json
{
"09:00": "Tovin",
"09:30": "Mara",
"10:00": "Jae",
"10:30": "Elise"
}
```
Llama-4-Maverick-17B-128E-Instruct-FP8: ## Step 1: Understand the given constraints for scheduling the audit interviews.
We have four slots: 09:00, 09:30, 10:00, 10:30, and four people: Mara, Tovin, Elise, Jae. The constraints given are:
1. Mara cannot do 09:00 and must be earlier than Tovin.
2. Tovin cannot do 10:30.
3. Elise must be either first or last.
4. Jae cannot do consecutive-to-Mara's slot and cannot do 09:30.
5. The 10:00 slot must be filled by someone whose name contains the letter "e".
## Step 2: Apply the constraints to determine possible positions for each person.
First, let's analyze the constraints:
- Since Elise must be either first or last, she can be at 09:00 or 10:30.
- Mara cannot be at 09:00, so if Elise is at 09:00, Mara can be at 09:30. If Elise is not at 09:00, she must be at 10:30, and Mara must be at 09:00 or 09:30, but since Mara can't be at 09:00, Mara must be at 09:30.
## Step 3: Examine the case when Elise is at 09:00.
If Elise is at 09:00, Mara can be at 09:30. This satisfies Mara not being at 09:00. Tovin cannot be at 09:00 (already Elise) and must be after Mara, so Tovin can be at 10:00 or 10:30. However, Tovin cannot be at 10:30, so Tovin must be at 10:00. This leaves 10:30 for Jae. But we need to check if this satisfies all constraints.
## Step 4: Check the constraints for the case when Elise is at 09:00.
- Mara is at 09:30, and Tovin is at 10:00. This satisfies Mara being before Tovin.
- Tovin is not at 10:30, so it's okay.
- Elise is first, so it's okay.
- Jae is at 10:30. Mara is at 09:30, so Jae's slot (10:30) is not consecutive to Mara's, satisfying the constraint. Also, Jae is not at 09:30, satisfying another constraint.
- The 10:00 slot is filled by Tovin, who does not have "e" in his name, violating the constraint that the 10:00 slot must be filled by someone with "e" in their name.
## Step 5: Examine the case when Elise is at 10:30.
If Elise is at 10:30, Mara must be earlier than Tovin and cannot be at 09:00, so Mara is at 09:30. This means 09:00 is available for either Tovin or Jae. However, Mara is before Tovin, so Tovin cannot be at 09:00. Thus, Tovin must be at 10:00, and Jae must be at 09:00.
## Step 6: Check the constraints for the case when Elise is at 10:30.
- Mara is at 09:30, and Tovin is at 10:00. Mara is before Tovin, so it's okay.
- Tovin is not at 10:30, so it's okay.
- Elise is last, so it's okay.
- Jae is at 09:00. Mara is at 09:30, so Jae's slot is not consecutive to Mara's (since 09:00 and 09:30 are consecutive, this actually violates the constraint that Jae cannot be consecutive to Mara).
- The 10:00 slot is filled by Tovin, which does not satisfy the "e" constraint.
## Step 7: Re-evaluate the constraints and directly assess viable options based on the "e" constraint for 10:00.
The 10:00 slot must have someone with "e" in their name. This means either Elise or Jae could be at 10:00 because they both contain "e". Let's directly check viable configurations considering all constraints.
## Step 8: Analyze possible configurations with the "e" constraint in mind.
Let's directly test configurations:
1. If Elise is at 09:00, Mara at 09:30, then Tovin at 10:00 (violates "e" constraint).
2. If Elise is at 10:30, we need to satisfy other constraints. Let's try Mara at 09:00 (not allowed), so Mara at 09:30. Then, 09:00 could be Jae, and 10:00 could be Elise or Tovin. But Tovin at 10:00 with Mara at 09:30 works if Jae is at 09:00. However, we need to check if it satisfies all constraints.
## Step 9: Detailed check for a valid configuration.
Let's consider: Jae at 09:00, Mara at 09:30, Elise at 10:00, and Tovin at 10:30.
- Mara is not at 09:00 and is before Tovin, so it's okay.
- Tovin is at 10:30, which is not allowed.
This configuration is not valid due to Tovin's position.
## Step 10: Try another configuration.
Let's try: Elise at 09:00, Mara at 09:30, Jae at 10:00, Tovin at 10:30.
- Violates because Tovin is at 10:30.
Let's try: Jae at 09:00, Mara at 09:30, Tovin at 10:00, Elise at 10:30.
- Mara is before Tovin.
- Tovin is not at 10:30, so it's okay because Elise is.
- Elise is last.
- Jae is not consecutive to Mara (since Jae is before Mara), and not at 09:30, so it's okay.
- 10:00 is Tovin, which does not have "e", violating the constraint.
## Step 11: Final configuration check.
Let's directly apply the constraints to find a valid schedule:
- If Elise is at 09:00, the order could be Elise, Mara, ?, ?. Mara is at 09:30. So, we have Elise, Mara, ?, ?. The 10:00 slot needs "e", so it could be Elise (already taken) or Jae. But Jae cannot be consecutive to Mara. So, this path is problematic.
- Let's consider Mara at 09:30, and Elise at 10:30. So, we have ?, Mara, ?, Elise. The first slot could be Jae, and the third slot could be someone with "e". If Jae is at 09:00, and Mara at 09:30, we need someone with "e" at 10:00. Elise is at 10:30. So, we can have Jae at 09:00, Mara at 09:30, Elise at 10:00, and Tovin at 10:30. This violates Tovin not being at 10:30.
## Step 12: Checking a specific valid configuration.
Let's check: Mara at 09:30 is not possible if she's before Tovin and not at 09:00. So, let's directly try to fit:
- Jae at 09:00, Elise at 10:00, Mara at 10:30 doesn't work because Mara is not before Tovin and Elise is not last.
Let's try a configuration: Elise at 09:00, Jae at 10:30. So, we have Elise, ?, ?, Jae. Mara cannot be at 09:00, so she must be at 09:30. Tovin must be after Mara, so Tovin is at 10:00. This gives us: Elise, Mara, Tovin, Jae.
- Elise is first.
- Mara is before Tovin.
- Tovin is not at 10:30.
- Jae is not consecutive to Mara.
- 10:00 is Tovin, which doesn't have "e".
The final answer is: $\boxed{{"09:00": "Elise", "09:30": "Mara", "10:00": "Jae", "10:30": "Tovin"}}$$
Model B is slightly better because, although it also ends with an invalid schedule, its reasoning is somewhat more coherent and less self-contradictory than Model A’s. Model A repeatedly proposes and retracts invalid schedules and finishes with a mapping that violates multiple constraints, making it less reliable overall. (Second judge pass, order swapped — scores are the average of both: Model B and Model A both fail to produce a valid schedule and both violate the prompt’s requirement to return only a final JSON mapping. Model B is somewhat more coherent in its reasoning, while Model A is more self-contradictory, but neither is correct enough to win.)
Python normalize campaign names
Return code only. In Python, write a function normalize_campaign(name: str) -> str for analytics ingestion.
Requirements:
- Trim leading/trailing whitespace.
- Convert to lowercase.
- Replace any run of characters that is not a letter or digit with a single hyphen.
- Remove leading/trailing hyphens.
- If the result is empty, return 'untitled'.
- Keep Unicode letters/digits (e.g. café, 東京) rather than stripping them.
- Include a few assert-based tests in the same code block.
Do not use external packages.
DeepSeek-V4-Pro: ```python
def normalize_campaign(name: str) -> str:
# Trim leading/trailing whitespace
name = name.strip()
# Convert to lowercase
name = name.lower()
# Replace runs of non-letter/digit characters with single hyphen
result = []
in_non_alnum = False
for ch in name:
if ch.isalnum():
result.append(ch)
in_non_alnum = False
else:
if not in_non_alnum:
result.append('-')
in_non_alnum = True
# Remove leading/trailing hyphens
result_str = ''.join(result).strip('-')
# Return 'untitled' if empty
if not result_str:
return 'untitled'
return result_str
# Tests
if __name__ == '__main__':
assert normalize_campaign(" Hello World ") == "hello-world"
assert normalize_campaign("Café & Brasserie") == "café-brasserie"
assert normalize_campaign("東京2020") == "東京2020"
assert normalize_campaign("foo!!!---bar...baz") == "foo-bar-baz"
assert normalize_campaign(" --- ") == "untitled"
assert normalize_campaign("") == "untitled"
assert normalize_campaign("ALL_CAPS") == "all-caps"
assert normalize_campaign("a b\tc\nd") == "a-b-c-d"
print("All tests passed.")
```
Llama-4-Maverick-17B-128E-Instruct-FP8: ```python
def normalize_campaign(name: str) -> str:
"""
Normalize a campaign name for analytics ingestion.
:param name: The campaign name to be normalized.
:return: The normalized campaign name.
"""
# Trim leading/trailing whitespace and convert to lowercase
name = name.strip().lower()
# Replace any run of characters that is not a letter or digit with a single hyphen
name = ''.join(c if c.isalnum() else '-' for c in name)
while '--' in name:
name = name.replace('--', '-')
# Remove leading/trailing hyphens
name = name.strip('-')
# If the result is empty, return 'untitled'
if not name:
name = 'untitled'
return name
assert normalize_campaign(" Hello World ") == "hello-world"
assert normalize_campaign("café campaign") == "café-campaign"
assert normalize_campaign("東京 campaign") == "東京-campaign"
assert normalize_campaign(" ") == "untitled"
assert normalize_campaign("") == "untitled"
assert normalize_campaign("!@#$%^&*()") == "untitled"
assert normalize_campaign("a!b@c#d$e%f^g&h*i(j)k") == "a-b-c-d-e-f-g-h-i-j-k"
```
Model A fully satisfies the normalization requirements with a direct Unicode-safe implementation and includes several relevant assert-based tests. Model B is also correct, but its repeated string replacement approach is less efficient and slightly less polished for collapsing runs of non-alphanumeric characters. (Second judge pass, order swapped — scores are the average of both: Model A better matches the requirement to replace each run of non-letter/digit characters with a single hyphen by doing it in one pass, while preserving Unicode alphanumerics correctly. Model B is mostly correct, but its repeated string replacement approach is less direct and efficient for collapsing runs, whereas Model A is cleaner and more precise overall.)
Unit-aware math
A pump moves 3.5 liters every 8 seconds. A tank holds 0.9 cubic meters. Starting empty, how long to fill it, in minutes and seconds (mm:ss), rounded to the nearest second? Show the key steps, then give the final answer on its own line.
DeepSeek-V4-Pro: Alright, let’s go through it step-by-step.
---
**Step 1 – Convert the tank volume to liters**
The tank holds \( 0.9 \, \text{m}^3 \).
Since \( 1 \, \text{m}^3 = 1000 \, \text{L} \):
\[
0.9 \times 1000 = 900 \, \text{L}.
\]
---
**Step 2 – Pump rate in liters per second**
Pump moves \( 3.5 \, \text{L} \) in \( 8 \, \text{s} \).
Rate:
\[
\frac{3.5}{8} = 0.4375 \, \text{L/s}.
\]
---
**Step 3 – Time in seconds to fill the tank**
\[
\frac{900}{0.4375} = 2057.142857 \, \text{s}.
\]
---
**Step 4 – Convert seconds to minutes and seconds**
Whole minutes:
\[
\lfloor 2057.142857 \div 60 \rfloor = 34 \, \text{minutes}.
\]
Remaining seconds:
\[
2057.142857 - (34 \times 60) = 17.142857 \, \text{s} \approx 17 \, \text{s (nearest second)}.
\]
---
\[
\boxed{34:17}
\]
Llama-4-Maverick-17B-128E-Instruct-FP8: ## Step 1: Convert the tank's volume from cubic meters to liters
First, we need to convert the volume of the tank from cubic meters to liters because the pump's rate is given in liters. Since 1 cubic meter equals 1000 liters, the tank's volume in liters is 0.9 * 1000 = 900 liters.
## Step 2: Calculate the pump's rate in liters per second
The pump moves 3.5 liters every 8 seconds. To find the rate in liters per second, we divide 3.5 liters by 8 seconds, which gives 3.5 / 8 = 0.4375 liters per second.
## Step 3: Calculate the time to fill the tank in seconds
To find the time it takes to fill the tank, we divide the tank's volume by the pump's rate. So, 900 liters / 0.4375 liters per second = 2057.14 seconds.
## Step 4: Convert the time from seconds to minutes and seconds
To convert 2057.14 seconds into minutes and seconds, we divide by 60. 2057.14 / 60 = 34.28567 minutes. The whole number part, 34, represents the minutes. To find the seconds, we multiply the decimal part by 60: 0.28567 * 60 = 17.14 seconds.
## Step 5: Round the seconds to the nearest second
Rounding 17.14 seconds to the nearest second gives 17 seconds.
## Step 6: Combine the minutes and seconds for the final answer
Therefore, it will take 34 minutes and 17 seconds to fill the tank.
The final answer is:
34:17
Model A and Model B both compute the correct result, 34:17, and both show the key steps. Model A is slightly better because it is more concise and follows the formatting request more closely by presenting the final answer cleanly on its own line, whereas Model B adds extra prose around the final answer. (Second judge pass, order swapped — scores are the average of both: Model A is slightly better because it is fully correct, concise, and follows the requested format more cleanly by showing the key steps and then giving the final answer clearly on its own line. Model B is also correct, but it is a bit more verbose and includes an extra lead-in before the final answer.)
Nuanced classification
Classify each review's sentiment as "positive", "negative", or "mixed", and give a 6-word-max reason. Return ONLY a JSON array of {"text","label","reason"} in input order.
Reviews: ["Fast shipping but the fabric feels cheap.", "Absolutely love it, wearing it daily!", "It broke after a week. Refund was quick and painless though."]
DeepSeek-V4-Pro: ```json
[
{"text": "Fast shipping but the fabric feels cheap.", "label": "mixed", "reason": "Positive speed, negative quality."},
{"text": "Absolutely love it, wearing it daily!", "label": "positive", "reason": "Expressed love and frequent use."},
{"text": "It broke after a week. Refund was quick and painless though.", "label": "mixed", "reason": "Negative break, positive refund."}
]
```
Llama-4-Maverick-17B-128E-Instruct-FP8: [
{"text": "Fast shipping but the fabric feels cheap.", "label": "mixed", "reason": "Mixed quality and shipping experience"},
{"text": "Absolutely love it, wearing it daily!", "label": "positive", "reason": "Loves and frequently wears the item"},
{"text": "It broke after a week. Refund was quick and painless though.", "label": "mixed", "reason": "Broke but refund was smooth"}
]
Model A and Model B both classify all three reviews correctly, but Model A follows the prompt more precisely by keeping every reason within the 6-word maximum. Model B exceeds that limit on the second reason, so Model A is better overall. (Second judge pass, order swapped — scores are the average of both: Model B fully follows the required output format and correctly classifies all three reviews with concise reasons under six words. Model A is also correct on sentiment, but it violates the instruction to return only a JSON array by wrapping the answer in a Markdown code block.)
Extract vendor change facts
From the messy note below, extract the requested facts into JSON with keys: vendor_name, legal_entity, previous_billing_email, new_billing_email, contract_id, effective_date, annual_cap_usd, notice_days, termination_for_convenience, support_tier, sla_uptime_percent, security_contact, dpa_signed, tax_residency_country. Use null if truly missing; do not infer.
Note:
"FYI after Tuesday's call w/ BlueHarbor—actually the invoices still come from BlueHarbor Metrics LLC (not the parent BlueHarbor Labs Inc.). Please stop using ap@old-lattice.co after June; new billing contact is billing+bhm@latticeworks.io. Contract ref BHM-27A-9941 / renewal dated 2025-08-01. Pricing unchanged except annual cap now US$148,500 all-in. Standard off-ramp stays 45 days written notice, but there is NO termination for convenience during the initial term. Support is Enterprise Plus; they quoted 99.95% monthly uptime SLA. Security paperwork: DPA countersigned on 2025/05/14, security issues should go to soc@blueharbormetrics.com. Their tax residency remains Ireland. Ignore my earlier draft mentioning 30 days and the ops alias."
DeepSeek-V4-Pro: {
"vendor_name": "BlueHarbor",
"legal_entity": "BlueHarbor Metrics LLC",
"previous_billing_email": "ap@old-lattice.co",
"new_billing_email": "billing+bhm@latticeworks.io",
"contract_id": "BHM-27A-9941",
"effective_date": "2025-08-01",
"annual_cap_usd": 148500,
"notice_days": 45,
"termination_for_convenience": false,
"support_tier": "Enterprise Plus",
"sla_uptime_percent": 99.95,
"security_contact": "soc@blueharbormetrics.com",
"dpa_signed": "2025-05-14",
"tax_residency_country": "Ireland"
}
Llama-4-Maverick-17B-128E-Instruct-FP8: Here are the extracted facts in JSON format:
```json
{
"vendor_name": "BlueHarbor Metrics LLC",
"legal_entity": "BlueHarbor Metrics LLC",
"previous_billing_email": "ap@old-lattice.co",
"new_billing_email": "billing+bhm@latticeworks.io",
"contract_id": "BHM-27A-9941",
"effective_date": "2025-08-01",
"annual_cap_usd": 148500,
"notice_days": 45,
"termination_for_convenience": false,
"support_tier": "Enterprise Plus",
"sla_uptime_percent": 99.95,
"security_contact": "soc@blueharbormetrics.com",
"dpa_signed": true,
"tax_residency_country": "Ireland"
}
```
Here's a brief explanation for the extracted values:
- `vendor_name` and `legal_entity` are both "BlueHarbor Metrics LLC" since the note mentions that invoices still come from BlueHarbor Metrics LLC.
- `previous_billing_email` and `new_billing_email` are extracted directly from the note.
- `contract_id` is "BHM-27A-9941".
- `effective_date` is "2025-08-01", the renewal date mentioned.
- `annual_cap_usd` is 148,500 as mentioned in the note.
- `notice_days` is 45 as per the standard off-ramp.
- `termination_for_convenience` is `false` since the note explicitly states there's "NO termination for convenience during the initial term."
- `support_tier` is "Enterprise Plus".
- `sla_uptime_percent` is 99.95 as quoted for the monthly uptime SLA.
- `security_contact` is "soc@blueharbormetrics.com".
- `dpa_signed` is `true` since the DPA was countersigned on 2025/05/14.
- `tax_residency_country` is "Ireland" as their tax residency remains unchanged.
Model A cleanly returns just the requested JSON and preserves the note’s facts well, including representing the DPA field with the actual signed date present in the source. Model B adds extra explanatory text despite the formatting instruction and is less accurate on key fields, notably setting vendor_name to the legal entity and reducing dpa_signed to a boolean instead of the explicit signed date available in the note. (Second judge pass, order swapped — scores are the average of both: Model B is better because its extracted values are consistently aligned with the note, especially treating `dpa_signed` as a boolean, while Model A incorrectly outputs a date for that field. Model A is slightly better on strict JSON-only formatting, but Model B is overall more correct; both have a minor issue around `vendor_name`, with Model B being less clearly justified than the note supports.)
LRU cache
Implement a class `LRUCache<K, V>` in TypeScript with a fixed capacity set in the constructor, and O(1) `get(key): V | undefined` and `set(key, value): void`. Accessing or updating a key must mark it most-recently-used; inserting beyond capacity must evict the least-recently-used entry. Return ONLY the class, no prose.
DeepSeek-V4-Pro: ```typescript
class LRUCache<K, V> {
private capacity: number;
private map: Map<K, V>;
constructor(capacity: number) {
if (capacity <= 0) {
throw new Error("Capacity must be greater than 0");
}
this.capacity = capacity;
this.map = new Map<K, V>();
}
get(key: K): V | undefined {
if (!this.map.has(key)) {
return undefined;
}
const value = this.map.get(key)!;
this.map.delete(key);
this.map.set(key, value);
return value;
}
set(key: K, value: V): void {
if (this.map.has(key)) {
this.map.delete(key);
} else if (this.map.size >= this.capacity) {
const leastRecentlyUsedKey = this.map.keys().next().value;
this.map.delete(leastRecentlyUsedKey);
}
this.map.set(key, value);
}
}
```
Llama-4-Maverick-17B-128E-Instruct-FP8: ```typescript
class LRUCache<K, V> {
private capacity: number;
private cache: Map<K, V>;
constructor(capacity: number) {
this.capacity = capacity;
this.cache = new Map();
}
get(key: K): V | undefined {
if (!this.cache.has(key)) return undefined;
const value = this.cache.get(key);
this.cache.delete(key);
this.cache.set(key, value);
return value;
}
set(key: K, value: V): void {
if (this.cache.has(key)) this.cache.delete(key);
else if (this.cache.size === this.capacity) {
const leastRecentlyUsedKey = this.cache.keys().next().value;
this.cache.delete(leastRecentlyUsedKey);
}
this.cache.set(key, value);
}
}
```
Model A is better because it provides a fully typed TypeScript implementation and validates invalid capacity, while still meeting the O(1) LRU behavior requirement. Model B is close, but it uses `new Map()` without generic type parameters and can fail when capacity is 0 because eviction logic only triggers on equality, making it less robust and slightly weaker on correctness. (Second judge pass, order swapped — scores are the average of both: Model A is slightly better because it is fully typed, handles invalid capacity explicitly, and uses a safer eviction condition with >= capacity. Model B is concise and mostly correct, but it lacks capacity validation and relies on less precise typing around the evicted key/value path.)
Matchup powered by
OpenRouter.