Head to head: DeepSeek-V4-Flash-0731 vs Phi-4
DeepSeek-V4-Flash-0731 vs Phi-4
This matchup wasn’t subtle: one model consistently executed the brief, while the other landed a few sharp hits but too often gave away points on instructions and edge-case discipline. The result is a decisive win driven by reliability across practical writing, formatting, and code-fix tasks.
DeepSeek-V4-Flash-0731 takes this one cleanly. The aggregate gap is huge — 98.5 to 74.3 — and the statistical read matters just as much as the raw score: this is a **95% confidence decisive win**, backed by a **9–3 task record** with no ties. That is not a squeaker or a judge-order artifact; it’s a broad performance advantage. What pushed DeepSeek ahead was not one flashy capability but repeatable execution. It was better at actually honoring the brief: exactly three one-sentence bullets in the dense-passage summary, the right calm-and-accountable tone in the vendor delay email, faithful Slack-style rewriting without adding junk, valid JSON-only output, the required six-line incident note format, and a concurrency fix that correctly shared in-flight work and recovered from rejection. In other words, it kept winning the unglamorous but important battles where models usually leak quality: format control, constraint obedience, and practical completeness. Phi-4 did have real wins. It was better on contradiction finding, where it quoted the genuinely conflicting lines instead of inferring around them, and it beat DeepSeek on at least one scheduling-style task by being less wrong when both models failed. It also showed strength on SQL in one judge pass because DeepSeek produced a query that would fail outright. But that’s the story of Phi-4 here: occasional precision, undermined by inconsistency. Across the set, it kept slipping on avoidable errors — extra jargon, generic phrasing, code fences where JSON-only was required, imports where imports were forbidden, and logic that looked plausible until you checked the constraints. The most telling pattern is that DeepSeek won both writing-heavy and code-heavy tasks for the same underlying reason: it followed directions closely while staying useful. That’s why it swept the support-thread summary, log parser, incident note, faithful rewrite, vendor email, concurrency bug fix, and clinic-roster impossibility call. Phi-4 was competitive when the task rewarded direct extraction of a contradiction or a narrowly framed analytical move, but it did not sustain that level across the broader benchmark. **Final call: DeepSeek-V4-Flash-0731 is the stronger model by a comfortable margin — more reliable, more instruction-faithful, and decisively better across the full spread of real-world tasks.**
Summarize dense passage
Summarize the passage below in exactly three bullet points, each one sentence, capturing the mechanism, the tradeoff, and the caveat — no jargon a non-specialist couldn't follow. Return only the bullets. Passage: "Speculative decoding pairs a small draft model with a large target model: the draft proposes several tokens, the target verifies them in one pass, and accepted tokens are kept while the first rejection resets to the target's own choice. This can cut latency substantially when acceptance is high, but the draft model's compute is pure overhead when acceptance is low, and gains evaporate on adversarial or highly novel inputs where the draft and target disagree often."
Model A better follows the request to avoid specialist language while still clearly covering the mechanism, tradeoff, and caveat in exactly three one-sentence bullets. Model B is mostly correct, but it keeps more jargon from the source and slightly misstates the caveat by saying the draft disputes the target rather than that they often disagree. (Second judge pass, order swapped — scores are the average of both: Model A better matches the request for plain language while still capturing the mechanism, tradeoff, and caveat accurately in exactly three one-sentence bullets. Model B is mostly correct, but it keeps more jargon and slightly misstates the caveat by saying the draft disputes the target rather than that the two often disagree on difficult or novel inputs.)
Find the contradiction
The following spec contains exactly one internal contradiction. Quote the two conflicting sentences verbatim and explain the conflict in one sentence. Do not fix it. Spec: "Free accounts may create up to three projects. Every account, regardless of tier, may archive unlimited projects. Archiving a project does not count against the project limit. Free accounts are limited to three projects total, including archived ones."
Model B identifies the actual contradictory pair verbatim and explains the conflict directly. Model A quotes a non-contradictory sentence about creation limits instead of the sentence about archiving not counting against the limit, so its analysis is less correct despite being clearly written. (Second judge pass, order swapped — scores are the average of both: Model B identifies the actual contradiction by quoting the two genuinely conflicting sentences verbatim and explains it clearly in one sentence. Model A quotes one non-conflicting sentence instead, and its explanation relies on an implication rather than the direct contradiction stated in the spec.)
Vendor delay update email
Draft an email to a retail client, Harbor Thread Co., explaining that their custom fixture shipment will arrive 4 days late because the powder-coating supplier failed a quality inspection. Audience: store operations manager. Tone: calm, accountable, and practical. Length: 120 to 145 words. Must include: the revised arrival date of 18 September, confirmation that installation crews were rebooked, one sentence offering a temporary workaround using existing shelving, and a clear next update time of 3:00 PM tomorrow. Do not mention legal terms or compensation.
Model A better matches the requested tone and content: it is calm, accountable, practical, includes the revised 18 September arrival date, confirms installation crews were rebooked, provides a temporary shelving workaround, and gives a clear next update time of 3:00 PM tomorrow while staying within the word limit. Model B is clear and concise, but it is less accountable in tone, less tailored to the retail operations context, and its workaround sentence is more generic. (Second judge pass, order swapped — scores are the average of both: Model A better matches the requested calm, accountable, practical tone and includes all required details in a more specific, operationally useful way. Model B is clear and compliant, but it is more generic and less accountable, with weaker practical detail for the store operations audience.)
Constraint scheduling
Four talks (A, B, C, D) fill four 1-hour slots 9,10,11,12. Constraints: A is before D; C is not first; B is immediately after A; D is not at 12. Give the ONE valid schedule as 'slot: talk' lines, then a one-line justification. If impossible, say so and explain.
Model A and Model B are both incorrect: Model A violates the constraint that B is immediately after A, and Model B violates the constraint that D is not at 12. Both also fail the prompt’s requirement to give the one valid schedule or say it is impossible; in fact, the constraints are impossible to satisfy. (Second judge pass, order swapped — scores are the average of both: Model B is better because it at least formats the answer as a schedule with a justification, while Model A gives a schedule that directly violates the constraint that A is immediately after B. However, Model B is still incorrect because its schedule places D at 12 despite the prompt forbidding that, so neither model solves the task correctly.)
Faithful rewrite
Rewrite this blunt internal note as a warm, professional Slack message to the whole engineering team (60–90 words), keeping every fact intact and adding no new commitments. Return only the message. Note: "Deploy is frozen until the memory leak in the image service is fixed. Priya is on it. Do not merge to main. ETA tomorrow noon."
Model A is warmer and more natural for a team-wide Slack message while preserving the key facts: deploy freeze, memory leak in the image service, Priya handling it, no merges to main, and ETA tomorrow noon. Model B is also solid, but it adds a sign-off and placeholder name that are not appropriate for the requested output and slightly shifts the wording toward a broader "deployment process" rather than the deploy itself. (Second judge pass, order swapped — scores are the average of both: Model A is warmer and more natural while preserving the core facts more faithfully. Model B adds a sign-off and slightly overstates the ETA as an expected resolution rather than simply tomorrow noon, whereas Model A better matches the requested Slack style and constraints.)
SQL window query
Given Postgres tables `orders(id, customer_id, created_at, total_cents)` and `customers(id, name)`, write ONE SQL query returning each customer's name and their SECOND-largest order total (in dollars, 2 decimals), excluding customers with fewer than two orders. Order by that value descending. Return ONLY the SQL.
Model B is better because it is syntactically coherent and correctly excludes customers with fewer than two orders, while Model A references an undefined alias (`second_largest_total`) and would fail. However, Model B uses `DENSE_RANK()`, which returns the second distinct total rather than the second order total, so it is not fully correct for the stated task. (Second judge pass, order swapped — scores are the average of both: Model A is better because it uses a single query and correctly excludes customers with fewer than two orders by selecting only rows with ROW_NUMBER() = 2. Model B violates the "ONE SQL query" constraint by using CTEs and may return the second distinct total rather than the second-largest order due to DENSE_RANK(), which is less faithful to the prompt.)
Summarize support thread
Summarize the following support thread in exactly 3 bullet points. Each bullet must be 11 to 15 words, start with "- ", and include only facts stated in the thread. Thread: Nadia (08:14): Since yesterday's app update, barcode scans freeze on Zebra TC21 devices after the third scan. Mateo (08:19): I can reproduce on warehouse floor 2. App version shows 5.18.2 build 441. Nadia (08:26): Workaround: force-close and reopen; then three more scans work. Ivy from Engineering (08:41): We changed camera session reuse in 5.18.2. Investigating memory pressure. Mateo (09:03): Not seeing it on Samsung XCover units. Ivy (09:27): Temporary mitigation is disabling "fast scan mode" in Settings > Scanner. Nadia (09:31): Two supervisors confirmed mitigation works, but scanning is slower. Ivy (10:12): Hotfix 5.18.3 targeted for tomorrow morning if QA passes.
Model A better satisfies the exact formatting and length constraints: it has exactly three bullets, each 11 to 15 words, and stays factual. Model B includes one bullet with 16 words and adds unsupported phrasing like Ivy identifying the cause as a potential cause rather than merely investigating memory pressure. (Second judge pass, order swapped — scores are the average of both: Model A better follows the requested format: all three bullets stay within 11 to 15 words and contain only thread facts. Model B is more detailed, but its first bullet exceeds the word limit and adds an unsupported linkage between Nadia's report and build 441.)
Python log parser
Write Python 3 code only. Create a function `summarize_events(lines: list[str]) -> dict[str, int]`. Each input line looks like `2026-04-17T09:14:03Z | service=mailer | level=ERROR | msg=timeout after 2s`. Return a dictionary counting how many lines occur at each level. Requirements: - Ignore malformed lines and lines missing `level=`. - Levels are case-insensitive in input but must be uppercase in output. - Preserve first-seen order of valid levels in the returned dict. - Include a short doctest example. - No imports.
Model A better satisfies the prompt: it uses no imports, includes a short doctest, preserves insertion order naturally, and correctly ignores lines without a usable level. Model B violates the no-imports requirement by importing doctest, adds unnecessary extra code, and is less robust because it depends on splitting specifically on " | ". (Second judge pass, order swapped — scores are the average of both: Model A better satisfies the prompt: it preserves insertion order via the dict, normalizes levels to uppercase, ignores lines without a valid level field, includes a short doctest, and uses no imports. Model B is weakened by including an import despite the "No imports" requirement and by being less robust about malformed input, since it can count lines that merely contain "level=" anywhere rather than requiring a proper field split as cleanly as Model A does.)
Ops incident note format
Write an internal incident note for the overnight on-call handoff. Requirements: - Exactly 6 lines. - Line 1 must be: INCIDENT: cache saturation in eu-west-3 - Line 2 must start with IMPACT: and include both "11 minutes" and "checkout latency". - Line 3 must start with CAUSE: and mention a mis-sized Redis maxmemory setting. - Line 4 must start with ACTIONS: and list exactly 3 actions separated by semicolons. - Line 5 must start with STATUS: and end with the word stable - Line 6 must start with OWNER: and contain the name Priya N. - Total word count across all 6 lines must be between 42 and 48 words. - Do not use the words outage, sorry, or root cause.
Model A satisfies all structural requirements, includes the required phrases, and stays within the 42-48 word limit. Model B is weakened by the code fences, line 5 not ending with the exact word "stable," and exceeding the word-count limit, though both are otherwise clear and readable. (Second judge pass, order swapped — scores are the average of both: Model A better satisfies the formatting and content requirements while staying concise and clear. Both entries meet the 6-line structure and required phrases, but Model A’s ACTIONS line more cleanly presents exactly three actions and its overall wording is tighter.)
Concurrency bug fix
This TypeScript function is meant to memoize an async loader but has a race: concurrent callers can each trigger the underlying fetch. Fix it so the fetch runs at most once per key, and a rejected fetch does NOT poison the cache (a later call must retry). Return ONLY the corrected function. ```ts const cache = new Map<string, Promise<string>>(); async function load(key: string, fetcher: (k: string) => Promise<string>) { if (cache.has(key)) return cache.get(key)!; const value = await fetcher(key); cache.set(key, Promise.resolve(value)); return value; } ```
Model A correctly stores the in-flight promise before awaiting, so concurrent callers share one fetch, and it deletes the cache entry on rejection so retries work. Model B violates the instruction to return only the corrected function, includes unnecessary explanation, and its logic is flawed because the per-call transient variable does not coordinate concurrent invocations or clear rejected entries from the cache. (Second judge pass, order swapped — scores are the average of both: Model A correctly stores the in-flight promise before awaiting, so concurrent callers share one fetch, and it removes the cache entry on rejection so later calls retry. Model B violates the instruction to return only the corrected function and its logic is flawed: the local variable cannot coordinate across calls, and it can even resolve a promise with itself.)
Messy leads to JSON
Convert the messy lead notes below into valid JSON only. Schema: an array of objects, each with keys exactly in this order: "name", "company", "email", "phone", "interested_in", "follow_up_by". Rules: - Normalize phone numbers to digits only. - `interested_in` must be an array of strings. - Convert follow-up dates to YYYY-MM-DD. - If a field is missing, use null. - Preserve original capitalization for names and companies. Notes: 1) "Mira Solis / North Beacon Dental / [email protected] / (415) 555-0188 / interested in: patient reminders, two-way SMS / follow up by Sept 7, 2026" 2) "Jonah Pike, Larkspur Fitness, phone 646.555.0199, wants: access control, staff scheduling, email: [email protected], follow-up: 2026/09/09" 3) "A. Chen — Red Elm Bakery — email only: [email protected] — interested in online ordering — no date given"
Model A is fully correct and follows the prompt exactly by returning valid JSON only. Model B has the same content correctness, but it violates the instruction to output JSON only by wrapping the JSON in a Markdown code fence. (Second judge pass, order swapped — scores are the average of both: Model A is slightly better because it provides valid JSON exactly as requested, while Model B wraps the JSON in a Markdown code fence, so it is not JSON only. Otherwise, both Model B and Model A correctly extract and normalize all fields.)
Clinic roster assignment
Assign four staff members to four clinic tasks, one person per task: Intake, Vitals, Billing, Phones. Staff: Lena, Omar, Priyank, Sora. Constraints: - Lena cannot do Intake or Phones. - Omar must do either Billing or Phones. - Priyank cannot do Billing. - Sora must do the task immediately alphabetically after Lena's task when tasks are ordered: Billing, Intake, Phones, Vitals. - Vitals is not assigned to Omar. - Priyank does Phones. Return only the final assignment as four lines in the format Name - Task.
Model A is correct: the constraints force Priyank to Phones, Omar to Billing, and then Lena cannot take Intake or Phones and cannot take Billing, leaving Vitals; but then Sora would need the task immediately after Vitals in the given order, which does not exist, so no valid assignment exists. Model B invents an invalid wraparound from Vitals to Intake and also ignores the prompt’s instruction to return only the final assignment. (Second judge pass, order swapped — scores are the average of both: Model A is better because the constraints are inconsistent, so no valid assignment exists. Model B invents an invalid wraparound from Vitals to Intake and also fails the instruction to return only the final assignment.)
Matchup powered by OpenRouter.