Head to head: Phi-4-mini-instruct vs Codestral-2501

Phi-4-mini-instruct vs Codestral-2501

One model was better at a couple of tightly constrained edge cases, but the overall match wasn’t especially close. Codestral-2501 took the broader writing-and-structure workload with cleaner instruction following and more dependable outputs.

Phi-4-mini-instruct did enough to make this interesting, but not enough to change the verdict. It won **4 of 12 tasks**, including the most technical coding prompt here: the **LRU cache**, where it used the correct O(1) hash map plus doubly linked list design while Codestral-2501 fell back to an O(n) list-based approach. Phi-4-mini-instruct also came out ahead on **nuanced classification**, the **vendor delay email**, and a weak overall **precise proofreading** task where both models struggled. But the center of gravity in this matchup belonged to Codestral-2501. It won **8 tasks to 4**, and most of those wins were the kind that matter in day-to-day use: **strict JSON extraction**, **messy contacts to JSON**, **proofread policy update**, **find the contradiction**, **dense-passage summarization**, and **release-notes bullets**. That pattern is hard to ignore. Codestral-2501 was consistently better at producing valid structure, preserving source meaning, following formatting constraints, and avoiding the small instruction-following mistakes that kept tripping Phi-4-mini-instruct. The contrast is especially sharp on structured-output work. Phi-4-mini-instruct lost the JSON extraction task by returning **invalid JSON** and introducing field errors, then lost the contacts normalization task by failing a required **sort order** despite mostly correct values. Those aren’t cosmetic misses; they’re the kind of failures that break downstream automation. Codestral-2501 wasn’t flawless either — it made a bad scheduling assignment on the on-call task and missed the algorithmic bar on the LRU cache — but its errors were less frequent and less systemic. The statistical read backs that up: **Codestral-2501 wins with 87% confidence**, which is a clear result, not a coin flip dressed up as one. Phi-4-mini-instruct showed real strength when the task rewarded concrete reasoning over strict formatting, but across the full slate, Codestral-2501 was the more reliable editor’s choice: better schema discipline, better summarization hygiene, and better adherence to what the prompt actually asked for. **Final call: Codestral-2501 is the clear winner. Phi-4-mini-instruct has a few sharp spots, especially on the LRU implementation, but Codestral-2501 is the stronger all-around text model in this head-to-head.**

LRU cache implementation

Implement an LRU cache in Python 3. Requirements: - Class name: LRUCache - Methods: __init__(capacity: int), get(key: int) -> int, put(key: int, value: int) -> None - get returns -1 if the key is absent. - Both get and put must run in O(1) average time. - Evict the least recently used item when capacity is exceeded. - Use only the Python standard library. Return code only.

Phi-4-mini-instruct: class DoublyLinkedListNode(object): def __init__(self, key, value): self.key = key self.value = value self.prev = None self.next = None class LRUCache(object): def __init__(self, capacity: int): self.cache = {} self.capacity = capacity self.head = DoublyLinkedListNode(0, 0) self.tail = DoublyLinkedListNode(0, 0) self.head.next = self.tail self.tail.prev = self.head def _remove(self, node): prev_node = node.prev next_node = node.next prev_node.next = next_node next_node.prev = prev_node def _insert_at_head(self, node): node.prev = self.head node.next = self.head.next self.head.next.prev = node self.head.next = node def get(self, key: int) -> int: if key in self.cache: node = self.cache[key] self._remove(node) self._insert_at_head(node) return node.value return -1 def put(self, key: int, value: int) -> None: if key in self.cache: self._remove(self.cache[key]) node = DoublyLinkedListNode(key, value) self._insert_at_head(node) self.cache[key] = node if len(self.cache) > self.capacity: lru_node = self.tail.prev self._remove(lru_node) del self.cache[lru_node.key] # Example usage: # lru_cache = LRUCache(2) # lru_cache.put(1, 1) # lru_cache.put(2, 2) # print(lru_cache.get(1)) # returns 1 # lru_cache.put(3, 3) # evicts key 2 # print(lru_cache.get(2)) # returns -1 (not found) # lru_cache.put(4, 4) # evicts key 1 # print(lru_cache.get(1)) # returns -1 (not found) # print(lru_cache.get(3)) # returns 3 # print(lru_cache.get(4)) # returns 4

Codestral-2501: ```python class LRUCache: def __init__(self, capacity: int): self.capacity = capacity self.cache = {} self.order = [] def get(self, key: int) -> int: if key in self.cache: self.order.remove(key) self.order.append(key) return self.cache[key] return -1 def put(self, key: int, value: int) -> None: if key in self.cache: self.order.remove(key) elif len(self.cache) >= self.capacity: lru_key = self.order.pop(0) del self.cache[lru_key] self.cache[key] = value self.order.append(key) ```

Model A correctly implements an O(1)-average-time LRU cache using a hash map plus doubly linked list and satisfies the required API, though it includes extra comments despite the 'code only' instruction. Model B is simpler but does not meet the O(1) requirement because list remove/pop(0) are O(n), and it also wraps the code in Markdown fences rather than returning plain code only. (Second judge pass, order swapped — scores are the average of both: Model A is better because it uses the standard O(1) hashmap plus doubly linked list design, satisfying the performance requirement, while Model B uses a Python list for recency tracking, making get and put O(n) in the average case. Model A slightly misses the "code only" instruction by including example comments, but it is still substantially more correct overall.)

Strict JSON extraction

Extract every meeting from this text into a JSON array of {"title","day","start24","durationMinutes"}. Use 24-hour "HH:MM" for start24, integer minutes for duration, and title-case titles. Return ONLY valid JSON, no commentary. Text: "Standup is daily at 9am for a quarter hour. The design review runs thursday from 2:30-4pm. Payroll sync — first monday, 11:00 to noon."

Phi-4-mini-instruct: ```json [ {"title":"Daily Standup Meeting","day":"daily","start24":"09:00","durationMinutes":15}, {"title":"Design Review","day":"thursday","start24":"14:30","durationMinutes":90}, {"title":"Payroll Sync","day":"every monday", "start24":"11:00","durationMinutes":60"} ] ```

Codestral-2501: ```json [ {"title":"Standup","day":"Daily","start24":"09:00","durationMinutes":15}, {"title":"Design Review","day":"Thursday","start24":"14:30","durationMinutes":90}, {"title":"Payroll Sync","day":"First Monday","start24":"11:00","durationMinutes":120} ] ```

Model B is better because it returns valid JSON and correctly extracts two of the three meetings, while Model A contains invalid JSON, alters titles and day values beyond the source, and has a malformed duration field. Model B’s main issue is interpreting "11:00 to noon" as 120 minutes instead of 60, but it otherwise follows the requested schema and formatting more closely. (Second judge pass, order swapped — scores are the average of both: Model B correctly extracts all three meetings with accurate times, durations, and title-case titles, and returns valid JSON. Model A has multiple issues: invalid JSON, incorrect day normalization for the payroll meeting, a malformed duration value, and it does not preserve the requested title/day formatting consistently.)

Unit-aware math

A pump moves 3.5 liters every 8 seconds. A tank holds 0.9 cubic meters. Starting empty, how long to fill it, in minutes and seconds (mm:ss), rounded to the nearest second? Show the key steps, then give the final answer on its own line.

Phi-4-mini-instruct: 1. Convert 0.9 cubic meters to liters: \(0.9 \, m^3 = 0.9 \, 1000 \, L = 900 \, L\). 2. Calculate the time to fill, in seconds: \[ \frac{900 \, L}{3.5 \, L/8 \, s} = 2057.14 \, s\] 3. Convert the time to minutes and seconds: \[2057.14 \, s = 34 \, m + 17.14 \, s \approx 34 \, mm + 17 \, ss\] The tank will take approximately 34 minutes and 17 seconds to fill with the pump. So, the answer is: 34:17.

Codestral-2501: To solve this problem, we need to determine how long it takes to fill a 0.9 cubic meter tank with a pump that moves 3.5 liters every 8 seconds. ### Key Steps: 1. **Convert the tank volume to liters:** - 1 cubic meter = 1000 liters - Therefore, 0.9 cubic meters = 0.9 * 1000 liters = 900 liters 2. **Calculate the total time to fill the tank:** - The pump moves 3.5 liters every 8 seconds. - To find out how many 8-second intervals are needed to move 900 liters, we divide the total volume by the volume moved per interval: \[ \text{Number of intervals} = \frac{900 \text{ liters}}{3.5 \text{ liters/interval}} = 257.14 \text{ intervals} \] 3. **Convert the number of intervals to total seconds:** - Each interval is 8 seconds, so: \[ \text{Total seconds} = 257.14 \text{ intervals} \times 8 \text{ seconds/interval} = 2057.12 \text{ seconds} \] 4. **Convert the total seconds to minutes and seconds:** - There are 60 seconds in a minute, so: \[ \text{Minutes} = \left\lfloor \frac{2057.12}{60} \right\rfloor = 34 \text{ minutes} \] \[ \text{Remaining seconds} = 2057.12 - (34 \times 60) = 2057.12 - 2040 = 17.12 \text{ seconds} \] - Rounding to the nearest second: \[ 17.12 \text{ seconds} \approx 17 \text{ seconds} \] ### Final Answer: 34:17

Model B is better because it gives the correct result with clearer unit handling and cleaner step-by-step reasoning. Model A reaches essentially the same answer, but its notation is sloppier and its final conversion line uses awkward formatting like "34 mm + 17 ss," which hurts clarity and writing quality. (Second judge pass, order swapped — scores are the average of both: Model B and Model A both reach the correct result, 34:17, but Model B shows the unit conversions and time breakdown more clearly and cleanly. Model A is concise but has awkward notation like "34 mm + 17 ss," which hurts writing quality and slightly weakens instruction adherence.)

Nuanced classification

Classify each review's sentiment as "positive", "negative", or "mixed", and give a 6-word-max reason. Return ONLY a JSON array of {"text","label","reason"} in input order. Reviews: ["Fast shipping but the fabric feels cheap.", "Absolutely love it, wearing it daily!", "It broke after a week. Refund was quick and painless though."]

Phi-4-mini-instruct: ```json [ {"text": "Fast shipping but the fabric feels cheap.", "label": "mixed", "reason": "Shipping quick, fabric cheap."}, {"text": "Absolutely love it, wearing it daily!", "label": "positive", "reason": "Loved and wears daily."}, {"text": "It broke after a week. Refund was quick and painless though.", "label": "mixed", "reason": "Broke soon, returned easy."} ] ```

Codestral-2501: ```json [ {"text": "Fast shipping but the fabric feels cheap.", "label": "mixed", "reason": "Fast shipping but fabric feels cheap."}, {"text": "Absolutely love it, wearing it daily!", "label": "positive", "reason": "Absolutely love it."}, {"text": "It broke after a week. Refund was quick and painless though.", "label": "negative", "reason": "It broke after a week."} ] ```

Model A is better because it correctly treats the first and third reviews as mixed and keeps every reason within the six-word limit. Model B misclassifies the third review as negative and its first reason exceeds the allowed length. (Second judge pass, order swapped — scores are the average of both: Model A is better because it correctly labels the third review as mixed, reflecting both the product failure and the positive refund experience. Both models follow the JSON-only format and keep reasons within six words, but Model A is more accurate overall.)

Messy contacts to JSON

Convert the messy inline data below into VALID JSON only. Return an array of objects sorted by last_name A-Z. Schema for each object: {"first_name": string, "last_name": string, "department": string, "email": string, "start_date": "YYYY-MM-DD", "remote": boolean}. Normalize emails to lowercase. Expand dates to ISO format. Treat remote values y/yes/true as true and n/no/false as false. Data: - Priya Nand, RevOps, PRIYA.NAND@acmeforge.io, start 7/8/24, remote yes - Tomas Bell | Dept=Finance | tomas.bell@AcmeForge.io | 2024-11-03 | remote: n - Aiko Sato / Product Design / aiko.sato@acmeforge.io / Mar 1 2025 / true - Malik Osei, Customer Success, malik.osei@acmeforge.io, 12-19-2024, Remote=No - Elena Quill ; Legal ; elena.quill@acmeforge.io ; 2025/01/06 ; y

Phi-4-mini-instruct: ```json [ { "first_name": "Aiko", "last_name": "Sato", "department": "Product Design", "email": "aiko.sato@acmeforge.io", "start_date": "2025-03-01", "remote": true }, { "first_name": "Elena", "last_name": "Quill", "department": "Legal", "email": "elena.quill@acmeforge.io", "start_date": "2025-01-06", "remote": true }, { "first_name": "Tomas", "last_name": "Bell", "department": "Finance", "email": "tomas.bell@acmeforge.io", "start_date": "2024-11-03", "remote": false }, { "first_name": "Malik", "last_name": "Osei", "department": "Customer Success", "email": "malik.osei@acmeforge.io", "start_date": "2024-12-19", "remote": false }, { "first_name": "Priya", "last_name": "Nand", "department": "RevOps", "email": "priya.nand@acmeforge.io", "start_date": "2024-07-08", "remote": true } ] ```

Codestral-2501: ```json [ { "first_name": "Tomas", "last_name": "Bell", "department": "Finance", "email": "tomas.bell@acmeforge.io", "start_date": "2024-11-03", "remote": false }, { "first_name": "Elena", "last_name": "Quill", "department": "Legal", "email": "elena.quill@acmeforge.io", "start_date": "2025-01-06", "remote": true }, { "first_name": "Malik", "last_name": "Osei", "department": "Customer Success", "email": "malik.osei@acmeforge.io", "start_date": "2024-12-19", "remote": false }, { "first_name": "Priya", "last_name": "Nand", "department": "RevOps", "email": "priya.nand@acmeforge.io", "start_date": "2024-07-08", "remote": true }, { "first_name": "Aiko", "last_name": "Sato", "department": "Product Design", "email": "aiko.sato@acmeforge.io", "start_date": "2025-03-01", "remote": true } ] ```

Model B is fully correct: it returns valid JSON, normalizes emails and dates properly, and sorts the array by last_name in ascending order (Bell, Nand, Osei, Quill, Sato). Model A has the right field values but fails the required sort order, so its instruction following and overall correctness are weaker. (Second judge pass, order swapped — scores are the average of both: Model B is better because it correctly sorts the array by last_name B-Z (Bell, Nand, Osei, Quill, Sato) while preserving valid JSON and accurate normalization. Model A contains the same field values but fails the required sort order, which is a key instruction-following and correctness issue.)

Proofread policy update

Edit the passage below for grammar, punctuation, concision, and clarity WITHOUT changing meaning. Keep the same paragraph structure. Then add 3 brief bullet points summarizing the main edits you made. "Starting next quarter, employees who works from another state for more than 30 days total in a calendar year need manager approval in advance, because payroll tax registrations may be required. Short trips under a week generally don't need pre-clearance, however people are still expected to tell their manager where they will be working from. If approval isn't obtained and a registration is triggered, reimbursements for travel related costs might be delayed until Finance have reviewed the case. This policy apply to full-time and part-time staff, but not contractors who bill through their own companies."

Phi-4-mini-instruct: Starting next quarter, employees working from another state for more than 30 days in a calendar year will need manager approval in advance due to possible payroll tax registrations. Short trips under a week generally do not require pre-clearance, though employees should still inform their manager of their working location. If approval isn't obtained and a registration is triggered, travel-related expense reimbursements may be delayed until the Finance department reviews the case. This policy applies to both full-time and part-time staff but not contractors who bill through their own companies. **Edits made:** - Corrected "employees who works" to "employees working," "in advance" to "in advance," "a calendar year" to "in a calendar year," and "reimbursements for travel related costs might be delayed" to "travel-related expense reimbursements may be delayed." - Changed "People are" to "People are," "are still" to "are still," and "if approval isn't obtained and a registration is triggered" to "if approval isn't obtained and a registration is triggered." - Altered "this policy apply" to "this policy applies," added a comma after "staff," and corrected "but not contractors" to "but not contractors (who bill through their own companies)."

Codestral-2501: Starting next quarter, employees who work from another state for more than 30 days in a calendar year need manager approval in advance, as payroll tax registrations may be required. Short trips under a week generally don't need pre-clearance, but employees are still expected to inform their manager of their work location. If approval isn't obtained and a registration is triggered, reimbursements for travel-related costs might be delayed until Finance reviews the case. This policy applies to full-time and part-time staff but not to contractors who bill through their own companies. **Main edits made:** - Corrected tense consistency (e.g., "works" to "work") - Improved sentence structure for clarity (e.g., "because" to "as") - Changed "people" to "employees" for specificity - Changed "apply" to "applies" for grammatical accuracy - Changed "billing" to "bill" for conciseness

Model B better preserves the original meaning and paragraph structure while making clean grammatical edits and providing a useful summary of changes. Model A introduces unnecessary rewrites such as changing "need" to "will need," alters wording more aggressively, and its edit bullets contain obvious errors and self-referential non-edits that reduce correctness and usefulness. (Second judge pass, order swapped — scores are the average of both: Model B better preserves the original meaning while improving grammar and clarity, and its edited passage is clean and natural. Model A is also readable, but it makes more substantive wording changes, and its summary bullets are inaccurate and poorly executed, including self-referential non-edits and awkward corrections.)

Find the contradiction

The following spec contains exactly one internal contradiction. Quote the two conflicting sentences verbatim and explain the conflict in one sentence. Do not fix it. Spec: "Free accounts may create up to three projects. Every account, regardless of tier, may archive unlimited projects. Archiving a project does not count against the project limit. Free accounts are limited to three projects total, including archived ones."

Phi-4-mini-instruct: "Free accounts may create up to three projects." and "Free accounts are limited to three projects total, including archived ones." Creating a project and counting an archived project together exceed the limit of three projects allowed in free accounts.

Codestral-2501: The two conflicting sentences are: 1. "Free accounts may create up to three projects." 2. "Free accounts are limited to three projects total, including archived ones." The conflict is that while the first sentence allows free accounts to create up to three projects, the second sentence limits free accounts to a total of three projects, including archived ones, which implies that archived projects do count against the project limit.

Model B better identifies and explains the actual contradiction: archived projects are said not to count against the limit, yet free accounts are later limited to three total including archived ones. Model A quotes two non-conflicting sentences and gives an unclear explanation, while Model B is clearer and closer to the spec’s real inconsistency. (Second judge pass, order swapped — scores are the average of both: Model B correctly identifies the conflicting sentences verbatim and explains the contradiction clearly: one sentence says archived projects do not count against the limit, while the other implies they do by including archived projects in the total. Model A quotes the same pair but gives a vaguer explanation that does not clearly articulate the internal conflict in the spec.)

Vendor delay status email

Rewrite the following internal note as a polished email to affected customers. Audience: operations managers at clinics. Tone: calm, accountable, and specific; do not sound legalistic. Length: 140-170 words. Include a clear subject line and 3 bullet points for what customers should do next. Raw note: "Heads-up: our print partner in Reno had a plate calibration failure overnight, so badge orders placed after 4:10 p.m. CT on Tue May 14 are not shipping today. Estimate is +2 business days for standard, +1 for expedited. This only hits custom PVC badges and visitor stickers; wristbands unaffected. We already stopped showing same-day shipping in checkout at 7:35 a.m. CT. If somebody has a launch/event this week, support can split-ship blank stock from Dallas if they email before noon local time. We’ll refund upgrade fees for orders that miss the original promise date. Sorry, messy morning."

Phi-4-mini-instruct: **Subject: Important Update on Badge Orders and Shipping Delays** Dear Operations Managers, We hope this message finds you well. We are reaching out to inform you about an issue affecting badge orders from our Reno print partner. Due to a plate calibration failure overnight, badge orders placed after 4:10 p.m. CT on Tuesday, May 14th, will not ship today. Attached below are the updated estimates for shipment: - Standard shipping: +2 business days - Expedited shipping: +1 business day We sincerely regret any inconvenience this may cause, and we want to assure you that this impact only affects custom PVC badges and visitor stickers. Wristband orders remain unaffected. We have already adjusted our checkout system to reflect this new shipping schedule as of 7:35 a.m. CT. For any events planned this week, we are happy to provide support by splitting-shipping blank stock from our Dallas location if you can email us before noon your local time. Concerning order upgrades, a full refund will be issued for orders that miss their original promised dates. Thank you for your understanding and cooperation. Should you have any questions or require further assistance, do not hesitate to reach out. Kind regards, [Your Name] [Your Position] [Your Contact Information] **Next Steps for Customers:** - Check your order placement times for expected shipping status. - For split-shipping needs, please email us before noon your local time. - We will handle refunds for any late orders automatically.

Codestral-2501: Subject: Update on Badge Shipping Delays Dear Operations Managers, I hope this message finds you well. We regret to inform you that our print partner in Reno experienced a plate calibration failure overnight, which has caused a delay in shipping for badge orders placed after 4:10 p.m. CT on Tuesday, May 14. This issue specifically affects custom PVC badges and visitor stickers; wristbands remain unaffected. To help mitigate the impact, please consider the following steps: - **Check Order Status**: Verify the status of your badge orders to ensure they are on track. - **Contact Support**: If you have a launch or event this week, please email our support team before noon local time to discuss splitting-shipping blank stock from our Dallas facility. - **Refunds**: We will refund any upgrade fees for orders that miss the original shipping promise date. We apologize for any inconvenience this may cause and appreciate your understanding as we work to resolve the issue. If you have any questions or need further assistance, please do not hesitate to reach out. Best regards, [Your Name] [Your Position]

Model A is more complete and specific: it includes the updated delay estimates, notes the checkout change, identifies unaffected products, and provides the requested three customer action bullets. Model B is concise and polished, but it omits the +2/+1 business-day estimates and the checkout update, making it less correct and less fully aligned with the prompt. (Second judge pass, order swapped — scores are the average of both: Model A is better because it includes the key delay estimates and operational details customers need, while Model B omits the +2 business days standard and +1 expedited timing and gives vaguer next steps. Both have issues with instruction-following, but Model A is more specific and useful despite being somewhat long and awkwardly phrased in places.)

On-call coverage assignment

Assign exactly one person to each on-call shift and return only the final assignment as four lines in the form "Shift: Name". People: Jae, Mira, Omid, Tessa Shifts: Fri PM, Sat AM, Sat PM, Sun AM Constraints: - Each person works exactly one shift. - Jae cannot work any Saturday shift. - Mira must work earlier than Omid. - Tessa cannot work Fri PM. - Sat PM must be covered by either Omid or Tessa. - If Omid works Sun AM, then Mira works Fri PM. - Jae works immediately before Tessa. There is exactly one valid solution.

Phi-4-mini-instruct: Shift: Fri PM: Tessa Shift: Sat AM: Jae Shift: Sat PM: Omid Shift: Sun AM: Mira

Codestral-2501: To solve this problem, we need to assign each person to a shift while satisfying all the given constraints. Let's go through the constraints step by step and determine the assignments. 1. **Jae cannot work any Saturday shift.** 2. **Mira must work earlier than Omid.** 3. **Tessa cannot work Fri PM.** 4. **Sat PM must be covered by either Omid or Tessa.** 5. **If Omid works Sun AM, then Mira works Fri PM.** 6. **Jae works immediately before Tessa.** Let's start by assigning shifts based on these constraints: ### Step-by-Step Assignment: 1. **Jae cannot work any Saturday shift.** - Jae can only work Fri PM, Sat AM, or Sun AM. 2. **Mira must work earlier than Omid.** - Mira can work Fri PM, Sat AM, or Sat PM. - Omid can work Sat PM, Sun AM, or Sat AM. 3. **Tessa cannot work Fri PM.** - Tessa can work Sat AM, Sat PM, or Sun AM. 4. **Sat PM must be covered by either Omid or Tessa.** - Omid or Tessa must work Sat PM. 5. **If Omid works Sun AM, then Mira works Fri PM.** - This constraint affects the order of assignments. 6. **Jae works immediately before Tessa.** - Jae can work Fri PM, Sat AM, or Sun AM. - Tessa can work Sat AM, Sat PM, or Sun AM. ### Possible Assignments: Let's try to find a valid assignment: - **Jae works Fri PM** (since Jae cannot work Saturday and must work immediately before Tessa). - **Tessa works Sat AM** (since Tessa cannot work Fri PM and must work immediately after Jae). Now we have: - Fri PM: Jae - Sat AM: Tessa Next, we need to assign Mira and Omid: - **Mira must work earlier than Omid.** - **Sat PM must be covered by either Omid or Tessa.** Since Tessa is already assigned to Sat AM, Omid must work Sat PM. - **Sat PM: Omid** Now we have: - Fri PM: Jae - Sat AM: Tessa - Sat PM: Omid Finally, we need to assign Sun AM: - **If Omid works Sun AM, then Mira works Fri PM.** - This constraint is already satisfied because Mira cannot work Fri PM (Jae is already assigned to Fri PM). So, Mira must work Sun AM. - **Sun AM: Mira** Now we have: - Fri PM: Jae - Sat AM: Tessa - Sat PM: Omid - Sun AM: Mira ### Final Assignment: - Fri PM: Jae - Sat AM: Tessa - Sat PM: Omid - Sun AM: Mira This assignment satisfies all the given constraints.

Model A gives an invalid assignment: Tessa is placed on Fri PM despite the constraint forbidding it, and the format is also wrong. Model B also gives an invalid assignment because Jae is assigned to Sat AM even though Jae cannot work any Saturday shift, but its explanation is clearer even though it fails the instruction to return only the final assignment. (Second judge pass, order swapped — scores are the average of both: Model B gives an invalid assignment because Mira is not earlier than Omid and it ignores the required output-only format, but Model A is even worse: its assignment violates multiple constraints, including placing Tessa on Fri PM and Jae on a Saturday shift. Model B is therefore slightly better on overall quality despite both being incorrect.)

Precise proofreading

Correct ONLY the grammar, spelling, and punctuation errors in the sentence below without changing wording, tone, or meaning, then on a second line list each change as 'was -> now'. Sentence: "Its been a long time since we seen there new office, and me and him was suprised how quite it was."

Phi-4-mini-instruct: "It's been a long time since we seen there new office, and me and him was suprised how quite it was." - saw -> now - there -> there (no change) - me -> me - was -> was - suprised -> surprised - quite -> quite (no change)

Codestral-2501: Its been a long time since we seen there new office, and me and him was suprised how quite it was. was -> Its was -> seen was -> there was -> was was -> quite

Model A is better because it at least corrects some actual errors, including adding the apostrophe in "It's" and fixing "suprised" to "surprised," while Model B largely fails to make corrections and does not provide a valid change list. However, Model A still misses several necessary grammar fixes and includes incorrect or meaningless entries in its list. (Second judge pass, order swapped — scores are the average of both: Model A is slightly better because it at least corrects one actual spelling error and preserves the sentence on the first line, whereas Model B leaves the sentence unchanged and provides a nonsensical change list. However, both models fail badly: neither properly corrects the grammar, and both do not follow the required 'was -> now' listing format accurately.)

Summarize dense passage

Summarize the passage below in exactly three bullet points, each one sentence, capturing the mechanism, the tradeoff, and the caveat — no jargon a non-specialist couldn't follow. Return only the bullets. Passage: "Speculative decoding pairs a small draft model with a large target model: the draft proposes several tokens, the target verifies them in one pass, and accepted tokens are kept while the first rejection resets to the target's own choice. This can cut latency substantially when acceptance is high, but the draft model's compute is pure overhead when acceptance is low, and gains evaporate on adversarial or highly novel inputs where the draft and target disagree often."

Phi-4-mini-instruct: - A small helper model in speculative decoding suggests words to a larger model, which quickly confirms or rejects the suggestions, improving speed or wasting effort depending on how confident the helper's guesses are. - When the helper model is often right, the process is much faster, but when it makes mistakes, the helper's time and energy do no good and only slow down the process for both the helper and the larger model. - If the larger model often disagrees with the helper's suggestions, especially in tricky or unfamiliar cases, the benefits of speeding up the process are quickly undone because the pictures always have to be double-checked.

Codestral-2501: - Speculative decoding uses a small model to suggest words, and a larger model to confirm them. - The method saves time when the small model is often right, but wastes resources when it is often wrong. - The technique may not work well with unusual or difficult text.

Model B better follows the requested format and cleanly covers the mechanism, tradeoff, and caveat in exactly three simple one-sentence bullets. Model A is more detailed but adds awkward phrasing and a misleading line about "pictures" being double-checked, which hurts correctness and writing quality. (Second judge pass, order swapped — scores are the average of both: Model B cleanly follows the required format with exactly three one-sentence bullets and captures the mechanism, tradeoff, and caveat in plain language. Model A is more detailed about the mechanism and tradeoff, but it violates the instruction by not clearly separating the three required points and introduces an inaccurate phrase about "pictures" that hurts correctness and clarity.)

Release notes summary bullets

Summarize the release notes below into EXACTLY 5 bullet points. Each bullet must be 9-13 words, start with a bold feature area label like **Billing:**, and include only information supported by the source. Source: "Version 3.18 shipped Thursday evening. Billing: finance admins can now export invoice line items with cost-center columns. Scheduled exports are still not available. Search: typo tolerance was improved for serial numbers, but only in the web app; mobile search is unchanged. Security: session timeout can now be set between 15 minutes and 12 hours. Existing tenants keep their current setting. Integrations: the NetLedger connector now retries rate-limited requests up to three times. The old duplicate-webhook bug for refunds was fixed. Known issue: CSV exports containing emoji in customer names may open with garbled characters in some spreadsheet apps. Performance: dashboard load time improved by roughly 18% on accounts with over 200 projects."

Phi-4-mini-instruct: - **Billing:** finance admins export invoices with cost-center columns. - **Search:** typos tolerated in serial numbers on web app only. - **Security:** session timeout between 15 minutes and 12 hours settable. - **Integrations:** NetLedger connector retries failed requests three times. - **Performance:** dashboard load time improved by ~18% on large accounts.

Codestral-2501: - **Billing:** Finance admins can now export invoice line items with cost-center columns. - **Search:** Typo tolerance improved for serial numbers in the web app. - **Security:** Session timeout can be set between 15 minutes and 12 hours. - **Integrations:** NetLedger connector retries rate-limited requests up to three times. - **Performance:** Dashboard load time improved by 18% on accounts with over 200 projects.

Model B is better because all five bullets stay faithful to the source and meet the 9-13 word constraint, while Model A has multiple bullets under the minimum length and one integration bullet weakens accuracy by changing "rate-limited requests" to broader "failed requests." Both are clear, but Model B follows the formatting and content requirements more precisely. (Second judge pass, order swapped — scores are the average of both: Model B is more faithful to the source, especially by preserving key qualifiers like "invoice line items," "rate-limited requests," and "accounts with over 200 projects." Model A follows the format but introduces unsupported simplifications such as "invoices," "failed requests," and "large accounts," making it less correct overall.)