Head to head: MoonshotAI: Kimi K3.1 Pro vs Anthropic: Claude Sonnet 5.5

MoonshotAI: Kimi K3.1 Pro vs Anthropic: Claude Sonnet 5.5

By · Published · Updated

RuntimeWire Head-to-Head: Head to head: MoonshotAI: Kimi K3.1 Pro vs Anthropic: Claude Sonnet 5.5
RuntimeWire Head-to-Head matchup

This matchup turns on a familiar editorial trade-off: strict fidelity to the reporting versus more fluid, expansive storytelling. The task set tests both accuracy and polish across several kinds of newsroom work.

MoonshotAI: Kimi K3.1 Pro leads the saved aggregate, **104.1 to 97.4**, with **6 task wins, 2 losses and 4 ties**. Its edge came most clearly in source-sensitive assignments: weaving supplied quotes into narrative, making a news opening, building a chronology and shaping reported numbers into a story. The judges repeatedly favored Kimi’s restraint and its separation of what the reporting established from what remained unknown. Claude Sonnet 5.5 had the advantage on the credible startup spotlight and the natural human rewrite. In those pieces, its narrative flow and fuller development of product context, competition and adoption risks stood out. But the same expansiveness sometimes crossed into unsupported framing or details, a recurring liability in this matchup. The ties matter: the models matched on the benchmark-scored Markdown error count and lede choice, produced identical output on the exact-structure task, and tied on controlled wit. This was not a sweep, and Claude showed a real strength in shaping a more engaging story when the assignment rewarded that latitude. **Verdict: Kimi K3.1 Pro is the clear editorial winner on this saved task sample, driven by stronger performance on reporting-grounded work. This assessment applies only to the saved task sample, not to model capability in general.**

Markdown error count

Count the lines that violate the house rules below. A defective line counts once even if it could violate multiple rules. Respond with a single number. House rules: * The headline must use `# Headline`. * The dek must appear immediately after the headline and be entirely italicized. * Section headings must use `## Heading`. * Links must use an absolute HTTPS URL. * Unordered-list items must begin with a hyphen. * Raw HTML is forbidden. Draft: #Flood software moves upstream The company is expanding its pilot. ##Details Read [the announcement](www.example.com/news). * The first pilot begins Monday. <div>The second pilot begins Friday.</div>

MoonshotAI: Kimi K3.1 Pro: 6
Anthropic: Claude Sonnet 5.5: Going through the draft line by line: 1. `#Flood software moves upstream` violates the headline rule, because it needs a space after the `#`. **Violation.** 2. `The company is expanding its pilot.` is supposed to be the dek, but it isn't italicized. **Violation.** 3. `##Details` violates the section heading rule, because it has no space after `##`. **Violation.** 4. `Read [the announcement](www.example.com/news).` has a link that isn't an absolute HTTPS URL. **Violation.** 5. `* The first pilot begins Monday.` is a list item that begins with an asterisk instead of a hyphen. **Violation.** 6. `<div>The second pilot begins Friday.</div>` is raw HTML. **Violation.** The blank lines don't violate anything. **6**

Scored against the benchmark's reference answer (numeric match): Model A was correct, Model B was correct.

Quotes woven into narrative

Write a 400–500-word Markdown story about the closing of Foundry Hall. Use quotations selectively. Paraphrase when the information matters more than the speaker’s exact phrasing. Avoid stacking quotes or giving every source an equal-sized turn. The story should carry an emotional current while remaining restrained. Facts: * Foundry Hall, a nonprofit startup accelerator in Milwaukee, will close October 31. * It opened in 2018. * It has supported 212 startups, which collectively employ 1,140 people. * Its largest corporate sponsor, Blank River Bank, supplied 38% of its annual budget. * The bank declined to renew its sponsorship. * Foundry Hall has enough money to operate through October but not through another full program. * Six employees will lose their jobs. * Twelve startups currently using its workspace must leave. Quotes: * Executive director Camille Ross: “We kept waiting for the replacement check that never came.” * Founder Dev Malik: “My first investor meeting happened in that building. Before Foundry Hall, I was pitching from the back table of a coffee shop.” * Blank River Bank spokesperson: “The decision reflects a change in our community investment priorities and is unrelated to Foundry Hall’s performance.” * Deputy mayor Susan Cole: “The city is speaking with building owners and local universities about space for the current companies.”

MoonshotAI: Kimi K3.1 Pro: # Foundry Hall to Close, Leaving Milwaukee Startups Searching for Space MILWAUKEE — Foundry Hall, the nonprofit startup accelerator that opened in 2018, will close on October 31 after losing its largest corporate sponsor. Its final weeks will bring an end to its programming and leave six employees without jobs. Twelve startups now using its workspace must also find new homes. The organization has supported 212 startups, which collectively employ 1,140 people. But that record was not enough to close the gap left by Blank River Bank, whose sponsorship supplied 38% of Foundry Hall’s annual budget. The bank declined to renew its commitment. Foundry Hall has enough money to operate through October, but not enough to run another full program. Executive director Camille Ross said the organization spent months trying to secure replacement funding. “We kept waiting for the replacement check that never came,” she said. A Blank River Bank spokesperson said the decision stemmed from a change in the bank’s community investment priorities and was unrelated to Foundry Hall’s performance. For founders, the closure is not simply the loss of an office lease. The accelerator offered a place where an early idea could become a company, and where an introduction could lead to a first investment. Founder Dev Malik traced his own progress to the building. “My first investor meeting happened in that building,” he said. “Before Foundry Hall, I was pitching from the back table of a coffee shop.” His account helps explain why the organization’s impact cannot be measured only by its budget. The 212 startups passed through at different stages, and the jobs they created extend beyond Foundry Hall’s walls. Still, those numbers offer little immediate help to the twelve companies that must relocate or to the staff preparing for the shutdown. Deputy Mayor Susan Cole said the city is speaking with building owners and local universities about space for the current companies. Those discussions may provide a landing place, though they will not preserve the accelerator in its present form. The closing also leaves a wider question for Milwaukee’s startup community: whether the connections built around Foundry Hall can survive without the institution that sustained them. Its supporters can point to companies launched, jobs created and founders who no longer have to pitch from coffee shop tables. But the next cohort will not walk through its doors. For now, the work is narrower and more urgent. Foundry Hall will operate through October, help its remaining startups prepare to move and wind down an organization that began with room for more ideas than it could ultimately afford to house.
Anthropic: Claude Sonnet 5.5: # Foundry Hall, Milwaukee Startup Accelerator, to Close After Sponsor Pulls Out Foundry Hall, the nonprofit startup accelerator that has helped found hundreds of Milwaukee companies, will close its doors on October 31. Since opening in 2018, the organization has supported 212 startups. Together they now employ 1,140 people. The closure follows a decision by Blank River Bank, Foundry Hall's largest corporate sponsor, not to renew its support. The bank supplied 38% of the accelerator's annual budget. A bank spokesperson said the decision reflects a change in its community investment priorities and has nothing to do with how Foundry Hall has performed. Executive director Camille Ross said the organization spent months hoping the gap could be filled. "We kept waiting for the replacement check that never came," she said. No replacement arrived. Foundry Hall has enough money to operate through October, but not enough to run another full program. Six employees will lose their jobs when the doors close. Twelve startups currently working out of the building will have to find somewhere else to go. Deputy Mayor Susan Cole said the city is talking with building owners and local universities about space for those companies, though she did not describe any arrangement as settled. For some founders, the loss is more than a matter of desks and deadlines. Dev Malik, who built his company through Foundry Hall, said the building is where his business first became real to other people. "My first investor meeting happened in that building," Malik said. "Before Foundry Hall, I was pitching from the back table of a coffee shop." He said he understands that organizations make budget decisions, but he worries about the next founder with an idea and no obvious place to take it. Ross said the staff's focus in the coming weeks is practical. They plan to help the twelve resident companies find new space and to see the current program through to its end. She declined to say what will happen to the organization's equipment and records after the closing date. The numbers Foundry Hall leaves behind are modest by the standards of larger cities, but they are specific. Over seven years, hundreds of people brought unfinished ideas into the building, and many of those ideas became payrolls. Ross said she is proud of that record and does not want it to be overshadowed by how the story ends. She said she plans to be in the building on October 31, when the last of the program's work wraps up. "We'll turn off the lights when the day is over," she said. "That part is easy. It's everything before it that we want to get right."

OpenAI: GPT-6 Luna: Model A stays within the supplied facts, uses quotes selectively, and delivers a restrained narrative within the requested length. Model B invents quotes and unsupported details about Malik and Ross, undermining its factual reliability despite a polished structure. (Second judge pass, order swapped — scores are the average of both: Model A is more restrained and factually disciplined; Model B invents quotes and several unsupported details. Model A appears slightly short of the requested word cou | Anthropic: Claude Fable 5.1: Model A sticks faithfully to the supplied facts…

Publication-ready Markdown edit

Edit the copy below into a publication-ready Markdown story. Requirements: * Supply an H1 headline and italicized dek * Repair the lede and paragraph flow * Use a Markdown blockquote only for the extended statement * Convert the supplied URL into a descriptive inline link * Remove internal notes and formatting debris * Preserve the reporting and its meaning * Output only the finished article Raw copy: HEADLINE TBD The funding was announced today. Raincheck is the company. It has raised $8 million and it makes flood-risk software for local governments. [NOTE: move this up?] The Austin company maps drainage systems and combines those maps with rainfall forecasts. Cities use the software to decide where to send maintenance crews before a storm. The seed round was led by Basin Ventures. Existing investor Long Mile also participated. Raincheck has raised $10.5 million total. CEO Ana Flores said the following: “Our customers usually know which streets flood. The harder problem is deciding which blocked drain, broken pump or unfinished repair should receive attention before the rain arrives.” https://example.com/raincheck-methodology Raincheck currently works with seven cities in Texas and Louisiana. The company declined to identify them because some contracts have not been publicly announced. Its forecasts are recommendations. Emergency managers retain control over deployments and public warnings. THE COMPETITION Other flood-planning vendors include HydroMap and municipal engineering contractors. Raincheck says the new funding will help it hire eight engineers and expand into Florida.

MoonshotAI: Kimi K3.1 Pro: # Raincheck Raises $8 Million to Help Cities Prepare for Floods *The Austin startup’s seed round brings its total funding to $10.5 million as it works to help local governments prioritize maintenance before storms.* Raincheck, an Austin-based company that makes flood-risk software for local governments, announced today that it has raised $8 million in seed funding. The company maps drainage systems and combines those maps with rainfall forecasts. Cities use the software to decide where to send maintenance crews before a storm. Raincheck describes its approach in its [methodology overview](https://example.com/raincheck-methodology). Basin Ventures led the round, and existing investor Long Mile also participated. The new investment brings Raincheck’s total funding to $10.5 million. Raincheck CEO Ana Flores described the challenge facing the company’s customers: > “Our customers usually know which streets flood. The harder problem is deciding which blocked drain, broken pump or unfinished repair should receive attention before the rain arrives.” Raincheck currently works with seven cities in Texas and Louisiana. The company declined to identify them because some contracts have not been publicly announced. Its forecasts are recommendations; emergency managers retain control over deployments and public warnings. ## The Competition Other flood-planning vendors include HydroMap and municipal engineering contractors. Raincheck says the new funding will help it hire eight engineers and expand into Florida.
Anthropic: Claude Sonnet 5.5: # Raincheck raises $8 million to help cities prepare for floods before the rain arrives *The Austin startup's software maps drainage systems and pairs them with rainfall forecasts to guide maintenance crews ahead of storms.* Raincheck, an Austin company that makes flood-risk software for local governments, has raised $8 million in a seed round, the company announced today. The software maps drainage systems and combines those maps with rainfall forecasts, as described in [Raincheck's methodology](https://example.com/raincheck-methodology). Cities use it to decide where to send maintenance crews before a storm. Basin Ventures led the round, and existing investor Long Mile also participated. Raincheck has raised $10.5 million in total. CEO Ana Flores said the company's customers already know much of what they need to about flooding in their communities: > "Our customers usually know which streets flood. The harder problem is deciding which blocked drain, broken pump or unfinished repair should receive attention before the rain arrives." Raincheck currently works with seven cities in Texas and Louisiana. The company declined to identify them because some contracts have not been publicly announced. Its forecasts are recommendations. Emergency managers retain control over deployments and public warnings. Other flood-planning vendors include HydroMap and municipal engineering contractors. Raincheck says the new funding will help it hire eight engineers and expand into Florida.

OpenAI: GPT-6 Luna: Model A is more polished and preserves the competition section with a useful subheading. Model B’s added framing of what customers know goes beyond the source and is less precise. (Second judge pass, order swapped — scores are the average of both: Model A is slightly stronger because its customer-quote introduction is cleaner and it turns the competition label into a clear section heading. Both preserve the reporting, use the required formatting, and provide a descriptive inline link; Model B’s | Anthropic: Claude Fable 5.1: Both deliver clean, compliant Markdown stories,…

Credible Startup Spotlight

Write a 500–650-word Startup Spotlight article in clean Markdown. The tone should be curious and constructive. Explain the founders’ insight, product and early traction while giving uncertainty and competition appropriate space. Avoid investor clichés, heroic founder mythology and promotional claims stated as fact. Reporting: * LedgerLark builds AI agents for post-trade operations at small investment firms. * Co-founder Mina Batra previously spent seven years in operations at a regional brokerage. * Co-founder Owen Carr built reconciliation systems at a large bank. * Batra began exploring the idea after seeing a six-person team spend three days resolving a trade mismatch caused by a copied account number. * LedgerLark’s agents compare confirmations, custodian records and internal ledgers, then propose corrections for human approval. * The product does not execute trades or move money. * Five firms are paying customers. * Annual contracts range from $36,000 to $110,000. * The company says one customer reduced its unresolved daily exceptions from 70 to 18. * The company has raised a $3.5 million seed round led by Motive Row. * Competitors include established reconciliation vendors and internal bank software. * Financial institutions may require lengthy security and procurement reviews.

MoonshotAI: Kimi K3.1 Pro: # Startup Spotlight: LedgerLark Focuses on the Work After the Trade A trade can be executed in seconds, while a small error in the records that follow can occupy an operations team for days. That gap shaped LedgerLark, which builds AI agents for post-trade operations at small investment firms. Co-founder Mina Batra spent seven years in operations at a regional brokerage. She began exploring the idea after watching a six-person team spend three days resolving a trade mismatch caused by a copied account number. The founders’ insight is narrower than “automate the back office”: much of the burden may sit in gathering records, comparing versions and working out what should change, not simply in spotting that two figures disagree. Batra’s co-founder, Owen Carr, previously built reconciliation systems at a large bank. Their experience connects the day-to-day workflow with the technical problem of making financial records agree, though LedgerLark still has to show that its approach works across customers’ different systems and exception types. ## Proposing fixes, keeping approval with people LedgerLark’s agents compare confirmations, custodian records and internal ledgers, then propose corrections for human approval. The product does not execute trades or move money. That boundary matters: the company is aiming at investigation and preparation, while leaving the decision to accept a correction with operations staff. The design also makes evaluation important. A useful system would need to cite its sources, handle incomplete or conflicting records and avoid creating extra review work through weak suggestions. The available information does not show how often LedgerLark’s proposals are accepted or how it performs on unusual exceptions. Those are practical questions for prospective customers, alongside integration effort and audit requirements. ## Early traction, with limits Five firms are paying customers, with annual contracts ranging from $36,000 to $110,000. That is evidence of willingness to pay, but the small customer count and wide price range make it too early to infer a repeatable sales pattern or typical deployment. The company says one customer reduced its unresolved daily exceptions from 70 to 18. If sustained, that would be a meaningful operational improvement. It is nevertheless a company-reported result from one customer; the reporting does not establish the measurement period, whether other process changes contributed, or whether similar results are typical. LedgerLark has raised a $3.5 million seed round led by Motive Row. The funding gives it resources to develop the product and pursue more deployments, but does not answer questions about retention, implementation costs or long sales cycles. ## Competition and the next test LedgerLark faces established reconciliation vendors as well as software built inside banks. Its challenge is not only to match existing controls, but to explain why an agent-based approach is better than improving incumbent tools or workflows. Financial institutions may also require lengthy security and procurement reviews, which can slow adoption even when a pilot looks promising. The next useful evidence would be customer outcomes measured over time: resolution speed, exception backlogs, reviewer workload and the quality of proposed corrections. For now, LedgerLark has a specific operational problem, a bounded product and a handful of paying customers. Whether that combination becomes a dependable business will depend on results beyond the early examples—and on how much trust and time its software actually saves.
Anthropic: Claude Sonnet 5.5: # LedgerLark Wants to Shrink the Cleanup After a Trade *Startup Spotlight* A single copied account number cost a six-person operations team three days. Mina Batra watched it happen at a regional brokerage, where she spent seven years in operations. The error was ordinary. What stayed with her was how long it took to find and untangle, and that became the starting point for LedgerLark. ## The insight Post-trade operations is the work that follows an agreed trade: confirming that the firm's own books, the counterparty's confirmation and the custodian's records all say the same thing. When they don't, someone has to work out which record is wrong and why. Large banks have dedicated teams and in-house tools for this. Batra's bet is that small investment firms face the same problem with far fewer people to absorb it. Her co-founder, Owen Carr, built reconciliation systems at a large bank, so the pair covers both how the work feels on a thin team and how such systems are built. ## What the product does LedgerLark builds AI agents that compare confirmations, custodian records and internal ledgers, then propose corrections. A person approves each proposal before anything changes. The product does not execute trades or move money. That boundary limits the damage a bad suggestion can do. It doesn't remove the need for reviewers to examine what they approve, and how well that review holds up under a heavy daily queue is a fair question for any tool built this way. ## Early traction Five firms are paying customers, with annual contracts ranging from $36,000 to $110,000. The company has raised a $3.5 million seed round led by Motive Row. LedgerLark also says one customer reduced its unresolved daily exceptions from 70 to 18. That is a notable drop if it holds, but it is one customer's result, reported by the company. It doesn't say how long the improvement took, how the exceptions were counted, or whether the firm's trade mix changed in the meantime. Five customers is enough to show some firms will pay, but not enough to show the results generalize. ## The competition LedgerLark is entering a crowded field. Established reconciliation vendors already sell to financial institutions and have the integrations and track records that buyers value. Large banks often rely on internal software built around their own data. Either group could add AI-assisted matching, and an incumbent that does so would be hard for a young company to dislodge. The opening, as far as we can tell, lies in focus: small firms may be underserved by tools designed for bigger operations, and may lack the engineering staff to build their own. Whether that gap is wide enough to support a company is something the next few years of sales will answer. ## What could slow it down Financial institutions can take a long time to approve new software. Security and procurement reviews may stretch sales cycles, and a tool that reads custodian records and internal ledgers will draw close scrutiny even without the ability to move money. Smaller firms may move faster than banks, but they often have limited staff to run a vendor review, which can cause delays of its own. There are technical questions too. Trade data is messy, formats differ across custodians, and a proposed correction is only useful if it is usually right. How often reviewers accept or reject LedgerLark's suggestions would be telling, though the company has not shared that figure. ## What to watch Three things would clarify the picture: whether the five customers renew at the end of their annual terms, whether results like the 70-to-18 drop recur across firms with different workflows, and how the company handles security reviews as it moves toward larger prospects. The problem Batra saw is real and familiar. What remains open is whether a small team can build a durable business on it.

OpenAI: GPT-6 Luna: Model B is more engaging and explains the post-trade problem and product clearly while giving traction, competition, and adoption risks substantial space. Model A is slightly more disciplined about sticking to the reported facts, but Model B’s added context is mostly framed as uncertainty rather than certainty. (Second judge pass, order swapped — scores are the average of both: Model B is slightly stronger: it develops the competitive landscape and adoption risks more fully, then closes with con | Anthropic: Claude Fable 5.1: Both pieces are faithful to the reporting and a…

Natural human rewrite

Rewrite the draft below as a 300–400-word publication-ready news story in Markdown. Preserve every factual claim. Improve the lede, narrative flow, sentence rhythm and transitions. Remove repetition, stock hype, throat-clearing and mechanical signposting. The finished article should sound like an experienced human technology reporter wrote it. Draft: # Arcwell Launches An Innovative New Platform Arcwell has announced an exciting new platform called Relay. The announcement represents a significant development for the company and for the broader customer service industry. Relay uses artificial intelligence to examine customer support conversations and suggest answers to agents. The platform is important for several reasons. First, it can suggest answers. Second, it can identify customers who may cancel. Third, it can create summaries. These capabilities demonstrate how artificial intelligence continues to transform the customer service landscape. Relay was tested by 12 companies over six months. Arcwell says agents using it closed tickets 14% faster. The company did not release customer-retention results. Relay costs $65 per agent each month. CEO Priya Shah said, “Support agents spend too much time searching for information that their companies already have.” However, the platform also presents potential challenges. Relay requires access to support transcripts and internal help documents. Arcwell says customer data is encrypted and is not used to train shared models. Companies can choose a 30-day or 90-day retention period. In conclusion, Relay represents an important new chapter for Arcwell as it seeks to help companies improve customer service using artificial intelligence.

MoonshotAI: Kimi K3.1 Pro: # Arcwell Launches Relay, an AI Platform for Customer Support Arcwell has launched Relay, an artificial intelligence platform designed to help customer support agents resolve tickets faster by examining conversations and suggesting answers. The platform also generates summaries and identifies customers who may be at risk of canceling. Together, the features extend Relay’s role beyond drafting replies, giving support teams a single system for reviewing conversations, preparing responses and spotting possible churn. Relay was tested by 12 companies over six months. Arcwell said agents using the platform closed tickets 14% faster. The company did not release customer-retention results, leaving it unclear whether the trial produced improvements beyond ticket-handling speed. “Support agents spend too much time searching for information that their companies already have,” CEO Priya Shah said. That approach requires Relay to draw directly on company records. Customers must give the platform access to support transcripts and internal help documents, making its handling of sensitive information a central consideration for prospective buyers. Arcwell said customer data is encrypted and is not used to train shared models. Companies can choose a retention period of either 30 or 90 days, allowing them to limit how long the platform keeps their information. Relay costs $65 per agent per month, placing the reported productivity gains alongside a recurring cost for each support worker using the software. For Arcwell, the launch marks a broader push to apply artificial intelligence across customer service work, from finding answers to summarizing interactions and identifying customers who may leave. The six-month trial provides an early indication of how Relay performed in practice, but the evidence released so far is limited to the company’s own account of faster ticket resolution. For prospective customers, the decision will involve more than the quality of Relay’s suggested answers. They will also have to weigh its price and data-access requirements against the reported speed gains, without published retention results to show whether identifying at-risk customers helps keep them from canceling.
Anthropic: Claude Sonnet 5.5: # Arcwell Launches Relay, an AI Assistant That Feeds Answers to Support Agents Arcwell has launched Relay, a platform that uses artificial intelligence to read customer support conversations and suggest answers to agents as they work. The pitch rests on a problem familiar to most support teams. "Support agents spend too much time searching for information that their companies already have," said Priya Shah, Arcwell's CEO. Relay's suggested replies are the core feature, but not the only one. The software can also identify customers who may cancel and generate summaries of support conversations. Arcwell's evidence comes from a six-month trial involving 12 companies. According to the company, agents who used Relay closed tickets 14% faster. Arcwell did not release customer-retention results. That omission stands out for a product that promises to flag customers at risk of leaving, because the company has offered no public measure of whether that feature helps keep them. Relay costs $65 per agent each month. The product also asks for broad access. To work, Relay needs support transcripts and internal help documents, which means a company must open both its customer conversations and its own reference material to the system. Arcwell says customer data is encrypted and is not used to train shared models. Companies can choose a retention period of either 30 or 90 days. Those assurances come from Arcwell itself, and buyers will likely want to weigh them as they would any vendor's claims about data handling. The 14% improvement is likewise the company's own figure, drawn from a pilot of modest size. Even so, the launch shows where Arcwell is placing its bet. Rather than replacing support staff, Relay is built to sit beside them, retrieving what a company already knows and putting it in front of the person answering the customer. Whether that translates into happier customers, and fewer canceled accounts, is a question the trial results released so far do not answer.

OpenAI: GPT-6 Luna: Model B has a stronger narrative flow and a more natural news-story voice, while retaining the key details and meeting the requested length. Model A is somewhat more cautious, but its ending repeats the point about missing retention results; Model B adds a few interpretive touches that go beyond the draft. (Second judge pass, order swapped — scores are the average of both: Model B has a more fluid, human-sounding narrative and makes the missing retention results relevant to the product’s central | Anthropic: Claude Fable 5.1: Model B reads like a seasoned reporter's copy—q…

Select the strongest accurate lede

Based only on the facts below, select the strongest news lede. Respond with one letter. Facts: * ByteBarge pulled its $12-per-month AI assistant on Tuesday. * Private workspace titles from 4,860 accounts were sent to a publicly accessible analytics endpoint. * The exposure lasted 11 hours. * Message contents were not sent. * The company has found no evidence of malicious access. A. ByteBarge users suffered a massive privacy disaster Tuesday when the company leaked their private messages online. B. ByteBarge, an innovative collaboration company, made the difficult decision to cancel one of its products. C. ByteBarge pulled its $12-per-month AI assistant Tuesday after discovering that private workspace titles from 4,860 accounts had been sent to a public analytics endpoint for 11 hours. D. Thousands of ByteBarge users woke Tuesday to learn that everything they had written was visible to hackers.

MoonshotAI: Kimi K3.1 Pro: C
Anthropic: Claude Sonnet 5.5: C

Scored against the benchmark's reference answer (choice match): Model A was correct, Model B was correct.

Exact final-draft structure

Convert the supplied fields into Markdown using these exact rules: * H1 headline * One blank line * Italicized dek * One blank line * First body paragraph * One blank line * H2 section heading * One blank line * Second body paragraph * Preserve all wording and punctuation exactly * Do not use a code fence * Output nothing else Headline: Raincheck raises $8 million for flood-risk software Dek: The Austin startup helps cities decide which drainage problems to address before storms. First body paragraph: Raincheck has raised an $8 million seed round led by Basin Ventures. Section heading: Expansion plans Second body paragraph: The company plans to hire eight engineers.

MoonshotAI: Kimi K3.1 Pro: # Raincheck raises $8 million for flood-risk software *The Austin startup helps cities decide which drainage problems to address before storms.* Raincheck has raised an $8 million seed round led by Basin Ventures. ## Expansion plans The company plans to hire eight engineers.
Anthropic: Claude Sonnet 5.5: # Raincheck raises $8 million for flood-risk software *The Austin startup helps cities decide which drainage problems to address before storms.* Raincheck has raised an $8 million seed round led by Basin Ventures. ## Expansion plans The company plans to hire eight engineers.

OpenAI: GPT-6 Luna: Model A and Model B produce identical output that follows the requested structure and preserves the supplied wording and punctuation. Neither has an advantage in correctness, instruction following, or writing quality. (Second judge pass, order swapped — scores are the average of both: Model B and Model A are identical, follow the required structure, preserve the wording, and add nothing else.) | Anthropic: Claude Fable 5.1: Model A and Model B produced byte-identical outputs that follow every structural rule exactly: H1, blank line, italicized dek, blank line, first paragr…

Controlled wit

Write a 350–450-word technology news story in Markdown. The subject invites humor, but the reporting should remain serious. Use no more than one dry line of original humor. Do not mock customers or turn the story into a comedy routine. Reporting: * QueueZero launched an AI meeting assistant called Proxy on Wednesday. * Proxy can attend a meeting, produce notes and assign follow-up work to software agents. * During the launch demonstration, a customer in the audience said, “Someone should tell Evan to stop talking and send us the pricing sheet.” * Proxy interpreted the remark as an instruction. * It created two tasks assigned to QueueZero CEO Evan Wu: “Stop talking” and “Send pricing sheet.” * Wu laughed, completed the second task and deleted the first. * QueueZero says Proxy requires confirmation before an external message is sent. * Internal tasks can be created automatically. * Administrators can disable automatic task creation. * Proxy costs $30 per user per month. * It enters public beta next week.

MoonshotAI: Kimi K3.1 Pro: # QueueZero’s Proxy AI Turns Audience Remark Into Tasks for Its CEO QueueZero on Wednesday launched Proxy, an AI meeting assistant that can attend meetings, produce notes and assign follow-up work to software agents. Its launch demonstration also showed how the system interprets conversation when deciding whether to create work. During the demonstration, a customer in the audience said, “Someone should tell Evan to stop talking and send us the pricing sheet.” Proxy interpreted the remark as an instruction. It created two tasks assigned to QueueZero CEO Evan Wu: “Stop talking” and “Send pricing sheet.” Wu laughed, completed the second task and deleted the first. The episode did not involve an external message being sent automatically. QueueZero said Proxy requires confirmation before an external message is sent. Internal tasks can be created automatically, so a statement made during a meeting can become an assigned work item without a separate confirmation step. That distinction is central to the product’s design. Proxy is intended not only to record what happened in a meeting, but also to move follow-up work into a system where software agents can act on it. The demonstration showed a corresponding challenge: casual remarks, suggestions and direct instructions may not always be easy for an automated system to separate. QueueZero said administrators can disable automatic task creation. The control allows organizations to decide whether Proxy should generate those items on its own, while the confirmation requirement provides a separate check on external communications. The tasks in the demonstration remained internal and were handled by Wu. Even so, the sequence from an audience comment to two assignments for the chief executive illustrates how quickly speech can become workflow when automatic task creation is enabled. Proxy costs $30 per user per month. It is scheduled to enter public beta next week, making the product available beyond the launch demonstration to a broader group of users. The launch comes as meeting software expands from transcription and summaries toward tools that can initiate work. That shift can reduce manual note-taking and task entry, but it also puts more weight on permissions, confirmation steps and administrative settings. For Proxy, the key boundary is between creating an internal task and communicating outside the system. The former can happen automatically unless an administrator turns the feature off; the latter requires confirmation. How reliably Proxy makes that distinction in everyday meetings will be a central question for its public beta.
Anthropic: Claude Sonnet 5.5: # QueueZero's Proxy Meeting Assistant Turns Spoken Remarks Into Assigned Tasks QueueZero on Wednesday launched Proxy, an AI meeting assistant that can attend meetings, produce notes and assign follow-up work to software agents. The product costs $30 per user per month and enters public beta next week. ## A remark becomes a task list During the launch demonstration, a customer in the audience said, "Someone should tell Evan to stop talking and send us the pricing sheet." Proxy treated the remark as an instruction. It created two tasks assigned to QueueZero CEO Evan Wu: "Stop talking" and "Send pricing sheet." Wu laughed, completed the second task and deleted the first. It was a fairly literal test of the product's listening skills. ## The design problem behind the demo The exchange points to a central challenge for tools that act on conversation rather than merely record it. People in meetings speak indirectly, offer opinions as suggestions and mix requests with commentary. An assistant that only summarizes can get these distinctions wrong at little cost. One that assigns work has to decide who is being addressed, what is being asked and whether anything is being asked at all. In this case, Proxy correctly identified Wu as the person the speaker meant and picked out a real request, the pricing sheet. It also turned a joke into a task, which is the kind of error the product's controls are meant to contain. ## Safeguards QueueZero says Proxy requires confirmation before any external message is sent. Internal tasks, by contrast, can be created automatically. Administrators can disable automatic task creation. That split reflects a difference in consequences. A mistaken item on an internal task list can be deleted, as Wu did with his. A mistaken message to a customer or partner cannot be recalled so easily. The tasks assigned to Wu appear to fall into the first category, though QueueZero did not describe how Proxy classified them during the demonstration. The administrator setting may matter most to organizations that want meeting notes but are cautious about software creating obligations for employees on its own. Turning off automatic task creation would presumably leave Proxy closer to a note-taker, though the company has not detailed how the product behaves in that mode. ## Availability and pricing Proxy is priced at $30 per user per month. The public beta begins next week, giving customers a chance to test how the assistant handles real meetings, which tend to be less tidy than a product launch. Questions remain about how often Proxy misreads casual remarks as commands, and how easily teams can review and correct what it creates. For now, the demonstration showed both the appeal of a system that acts on what it hears and the reason QueueZero built in confirmation for anything that leaves the company.

OpenAI: GPT-6 Luna: Model A is grounded, maintains a serious reporting tone and stays within the prompt’s limit on original humor. Model B is well structured, but includes multiple humor-adjacent lines and more speculation about the product’s behavior. (Second judge pass, order swapped — scores are the average of both: Model A stays within the requested length and keeps the reporting controlled, while Model B includes more than one dry humorous aside despite the limit. Model B offers slightly more nuance, but Model | Anthropic: Claude Fable 5.1: Model B delivers a better-structured, more anal…

News judgment in the opening

Write only the opening of this news story: an H1 headline, an italicized dek and three to five paragraphs totaling 180–250 words. The opening must establish what happened, quantify the scope, explain why it matters and carefully distinguish confirmed facts from unknowns. It should pull the reader forward without sensationalizing the incident. Reporting: * Cloudnote discovered that a debugging page containing customer prompt excerpts was accessible without authentication. * The page was exposed for four hours on Monday. * Logs show excerpts from 1,842 user sessions. * The excerpts could include up to 200 characters from a prompt. * Cloudnote removed the page 19 minutes after a researcher notified it. * The company says it has found no evidence that anyone besides the researcher accessed the page. * It has not completed its investigation. * Full documents, passwords and payment information were stored elsewhere and were not present on the page. * Cloudnote has 3.2 million registered users.

MoonshotAI: Kimi K3.1 Pro: # Cloudnote Exposed Prompt Excerpts From 1,842 User Sessions for Four Hours *The debugging page required no authentication. The company says it has found no evidence of access beyond the researcher who reported it, but its investigation is unfinished.* Cloudnote said a debugging page containing excerpts of customer prompts was accessible without authentication for four hours on Monday, briefly leaving fragments of user interactions open to anyone who reached the page. Logs show excerpts from 1,842 user sessions, with each excerpt containing as much as 200 characters of a prompt. The company removed the page 19 minutes after a researcher notified it of the exposure. Cloudnote has 3.2 million registered users, though the session count does not establish how many individual customers had material on the page. Even short prompt fragments can reveal information users intended to keep private, depending on what they entered. Cloudnote said full documents, passwords and payment information were stored elsewhere and were not present on the page, limiting the types of data involved. Cloudnote said it has found no evidence that anyone other than the researcher accessed the page. That finding does not rule out other access, and the company has not completed its investigation. It remains unclear whether the excerpts were viewed or copied by anyone else, and how the page became publicly accessible.
Anthropic: Claude Sonnet 5.5: # Cloudnote debugging page exposed customer prompt excerpts for four hours *The company says only a researcher is known to have viewed the page, but its investigation is unfinished.* Cloudnote, a note-taking service with 3.2 million registered users, said a debugging page containing excerpts of customer prompts was accessible without authentication for four hours on Monday, according to the company. Logs show the page displayed excerpts from 1,842 user sessions, and each excerpt could include up to 200 characters of a prompt. Cloudnote removed the page 19 minutes after a researcher notified it of the problem. The company says it has found no evidence that anyone besides the researcher accessed the page, but its investigation is not complete. It says full documents, passwords and payment information were stored elsewhere and were not present on the page. Even brief excerpts can matter. People often write prompts informally and may mention names, workplace details or personal circumstances in a sentence or two. It is not yet clear what the exposed excerpts contained. Several questions remain open: how the page became reachable without a login, whether the 1,842 sessions correspond to that many separate users, and whether the company's logs can account for every visit. Until the investigation concludes, the no-access finding is preliminary.

OpenAI: GPT-6 Luna: Model A is more concise and tightly grounded in the reporting while clearly separating the company’s finding from what remains unknown. Model B follows the format, but adds unsupported framing about Cloudnote being a note-taking service and raises a question about log completeness that the reporting does not establish. (Second judge pass, order swapped — scores are the average of both: Model A better quantifies the exposure while distinguishing session counts from individual customers, and it cl | Anthropic: Claude Fable 5.1: Model A sticks strictly to the reported facts,…

Cohesive story from reporting notes

Write a publication-ready technology news article from the reporting notes below. Requirements: * 550–700 words * Clean Markdown * One H1 headline followed by an italicized dek * A strong news lede followed by a clear nut graf * Organize the material into a cohesive story rather than following the notes in order * Attribute company-supplied claims * Include limitations without turning the article into a warning label * Use only the supplied information * Avoid canned transitions, promotional language and a concluding summary Reporting notes: * HelioDock makes autonomous warehouse robots. * On Thursday, it launched Night Shift, software that lets operators monitor robot fleets remotely. * Night Shift spent 14 months in a private beta involving nine logistics companies and 310 robots. * HelioDock says beta customers experienced 17% fewer unplanned robot stoppages. The figure has not been independently verified. * The company also announced a $24 million Series B led by North Coast Ventures, bringing its total funding to $39 million. * CEO Mara Velez: “A warehouse manager should not have to drive across town at two in the morning because one robot is confused by a pallet.” * Beta customer Aaron Pike of Fulcrum Logistics said the system worked well in facilities with strong connectivity but became less useful during rural LTE outages. * When connectivity fails, robots return to their locally configured behavior. Remote control becomes unavailable. * Competing fleet-management products are available from Relay Robotics and YardOS. * Night Shift will cost $90 per robot per month and become generally available September 15.

MoonshotAI: Kimi K3.1 Pro: # HelioDock launches Night Shift remote monitoring and raises $24 million *The warehouse robot maker tested the software with nine logistics companies and 310 robots; general availability is set for September 15 at $90 per robot per month.* HelioDock on Thursday launched Night Shift, software that lets operators monitor fleets of autonomous warehouse robots remotely. The company also announced a $24 million Series B funding round led by North Coast Ventures, bringing its total funding to $39 million. The launch extends HelioDock’s business beyond making robots into software priced according to fleet size. The company is pointing to fewer unplanned stoppages among beta customers as evidence of Night Shift’s operational value, though that result has not been independently verified. The software also enters a market with existing fleet-management products, and one beta customer said its usefulness depended heavily on the quality of a facility’s connectivity. CEO Mara Velez described the product as an answer to the disruptions that can require managers to travel to a warehouse outside normal working hours. “A warehouse manager should not have to drive across town at two in the morning because one robot is confused by a pallet,” she said. Night Shift spent 14 months in a private beta involving nine logistics companies and 310 robots. HelioDock said those customers experienced 17% fewer unplanned robot stoppages. The figure is company-supplied and has not been independently verified. The beta also provided a more qualified account of how the system performs across different facilities. Aaron Pike of Fulcrum Logistics, a beta customer, said Night Shift worked well in facilities with strong connectivity. He said it became less useful during rural LTE outages. When connectivity fails, the robots return to their locally configured behavior, and remote control becomes unavailable. The robots therefore do not depend on a continuous remote connection to retain their local configuration, but operators lose Night Shift’s remote capabilities while the connection is down. Pike’s experience indicates that network reliability can shape how much benefit a warehouse operator gets from the software, particularly at sites affected by rural LTE outages. HelioDock will also have to distinguish Night Shift from competing products already on the market. Relay Robotics and YardOS both offer competing fleet-management products. HelioDock has not, in the supplied launch information, established an independently verified comparison between Night Shift and those offerings. The new funding gives HelioDock $39 million in total financing, including the $24 million Series B. North Coast Ventures led the latest round. The company announced the investment alongside the Night Shift launch but did not, in the supplied information, attach a specific spending plan to it. Night Shift will cost $90 per robot per month, tying the software bill directly to the number of robots an operator covers. That pricing will apply when the product becomes generally available on September 15, following the private beta.
Anthropic: Claude Sonnet 5.5: # HelioDock Launches Remote Monitoring Software for Warehouse Robot Fleets *Night Shift, tested with nine logistics companies, aims to cut overnight stoppages. One beta customer says it was less useful when rural cellular service went down.* HelioDock, which makes autonomous warehouse robots, on Thursday launched Night Shift, software that lets operators monitor robot fleets remotely. The company also announced a $24 million Series B round led by North Coast Ventures, bringing its total funding to $39 million. Night Shift is meant to reduce how often staff must go to a warehouse when a robot stalls. HelioDock says customers in a private beta had 17% fewer unplanned robot stoppages, though the figure has not been independently verified. A beta customer said the software's usefulness depended heavily on a facility's internet connection. ## Built for the off-hours The product's name points to its intended use: problems that arise when few people are on the floor. CEO Mara Velez framed it in those terms. “A warehouse manager should not have to drive across town at two in the morning because one robot is confused by a pallet,” Velez said. Night Shift spent 14 months in a private beta that involved nine logistics companies and 310 robots. HelioDock attributes the 17% reduction in unplanned stoppages to that group of customers. The company did not provide further detail on how the figure was measured. ## Where connectivity matters Aaron Pike of Fulcrum Logistics, a beta customer, said Night Shift worked well in facilities with strong connectivity. It became less useful during rural LTE outages, he said. That limit follows from how the system is built. When connectivity fails, robots return to their locally configured behavior, and remote control becomes unavailable. In an outage, an operator can no longer step in remotely, and the robots operate on the settings already configured at the site. ## A crowded category Night Shift enters a market where rivals already sell fleet-management products. Relay Robotics and YardOS both offer competing software. ## Funding and pricing The Series B, led by North Coast Ventures, brings HelioDock's total funding to $39 million. The company did not provide details in the information released about how it will spend the new money. Night Shift will cost $90 per robot per month. It becomes generally available September 15.

OpenAI: GPT-6 Luna: Model A is more developed and gives the beta, connectivity limitation, competition, funding and pricing fuller treatment. Both fall short of the 550-word minimum, but Model B is substantially shorter; Model A also adds a few unsupported observations that weaken its accuracy. (Second judge pass, order swapped — scores are the average of both: Model A gives the beta results, connectivity limitation, competition, pricing, and funding a more cohesive treatment, while Model B is notably shorter and a | Anthropic: Claude Fable 5.1: Both fall short of the 550-word floor (Model B…

Chronology with narrative momentum

Turn the scrambled reporting below into a 450–600-word Markdown news article. Choose an opening that captures the consequence of the failure, then reconstruct the sequence clearly. Preserve the chronology without producing a minute-by-minute incident log. Give technical details enough context for a general reader. Reporting: * KiteBridge makes expense-management software. * At 8:45 a.m. Tuesday, it deployed version 4.2. * The first customer complaint arrived at 9:12 a.m. * A payment-processor timeout caused KiteBridge’s fallback system to treat retries as new reimbursement requests. * The software generated 13,400 duplicate reimbursements totaling $2.8 million. * KiteBridge suspended automated payments at 10:03 a.m. * Ninety-one corporate customers were affected. * By Wednesday evening, $2.3 million had been recovered. The rest remained pending. * No employees had money withdrawn from their accounts; employers funded the duplicate payments. * An April changelog said version 4.0 had introduced “exactly-once payment protection.” * KiteBridge’s public incident page initially described the problem as delayed reimbursements. * The incident page first disclosed the duplicate payments six hours after automation was suspended. * CEO Lena Ortiz apologized and said the company would publish a technical report.

MoonshotAI: Kimi K3.1 Pro: # KiteBridge Update Triggers $2.8 Million in Duplicate Reimbursements A software update at KiteBridge triggered 13,400 duplicate reimbursements totaling $2.8 million on Tuesday, disrupting payments for 91 corporate customers and forcing the company to suspend its automated payment system. By Wednesday evening, $2.3 million had been recovered, but the remaining $500,000 was still pending. KiteBridge, which makes expense-management software, deployed version 4.2 at 8:45 a.m. Tuesday. The first customer complaint arrived at 9:12 a.m., less than half an hour later. The company suspended automated payments at 10:03 a.m., after the failure had already produced the duplicate transactions. The compressed sequence meant that a problem introduced with a routine release became a customer-facing incident in under 30 minutes. Automated payments continued for nearly another hour before they were stopped, giving the faulty process time to generate duplicates on a large scale. The failure began with a timeout at a payment processor, the service responsible for moving the reimbursement money. When a processor does not respond within the expected time, payment software may retry the request rather than immediately abandon it. Retries are a routine way to recover from network or service problems, but the software must recognize them as repeated attempts to complete the same transaction. KiteBridge’s fallback system—a backup process intended to keep operations running when normal processing encounters trouble—instead treated the retries as new reimbursement requests. As a result, what should have been repeated attempts to finish existing payments became instructions to send additional money. Employers funded the duplicate payments. No employees had money withdrawn from their accounts as a result of the incident. For the affected corporate customers, however, the error created a substantial recovery effort: nearly one-fifth of the duplicate total remained outstanding on Wednesday evening. KiteBridge’s public incident page initially gave customers a narrower picture of the problem, describing it as delayed reimbursements. The page first disclosed the duplicate payments six hours after automated payments were suspended. By that point, the company was already dealing with an incident involving 13,400 extra payments across 91 customers, rather than a delay alone. The failure also contrasted with the company’s earlier description of its safeguards. An April changelog said version 4.0 had introduced “exactly-once payment protection.” In financial software, exactly-once processing is intended to ensure that a transaction is completed only one time, even if technical problems cause the same instruction to be sent again. Tuesday’s events illustrated the risk when that principle breaks down: a mechanism designed to make payments more reliable can instead multiply them if each retry is mistaken for a separate request. CEO Lena Ortiz apologized and said KiteBridge would publish a technical report. Her commitment came as the company worked to address both the technical failure and its aftermath. While most of the duplicate money had been recovered by Wednesday evening, the pending balance and the promised report meant the incident was not yet fully resolved for affected customers.
Anthropic: Claude Sonnet 5.5: # Software Glitch at KiteBridge Sends $2.8 Million in Duplicate Reimbursements to Employees of 91 Companies A software failure at expense-management company KiteBridge generated 13,400 duplicate reimbursements worth $2.8 million on Tuesday, leaving 91 corporate customers to cover payments they never intended to make. By Wednesday evening, $2.3 million had been recovered. About $500,000 remained pending. The money came from employers, not workers. KiteBridge said no employees had funds withdrawn from their accounts. The companies funded the duplicate payments. ## How the failure unfolded KiteBridge, which makes software that helps companies collect and reimburse employee expenses, deployed version 4.2 of its platform at 8:45 a.m. Tuesday. The first customer complaint arrived at 9:12 a.m. According to the reporting, the problem began with a timeout at a payment processor, the outside company that moves money between banks. When a processor doesn't answer in time, the sender can't tell whether the payment went through. Software typically handles this by retrying the request. KiteBridge's fallback system, which takes over when the primary payment path fails, handled those retries incorrectly. Instead of recognizing them as repeats of earlier requests, it treated them as new reimbursement requests. Each retry produced another payment. KiteBridge suspended automated payments at 10:03 a.m., 78 minutes after the deployment. By then the system had produced the 13,400 duplicates. ## A safeguard that was supposed to prevent this The failure is notable because of what KiteBridge had previously said about its system. An April changelog, the company's public list of software updates, said version 4.0 had introduced "exactly-once payment protection." Such protection is meant to ensure that a payment request is carried out a single time, even if the request is sent repeatedly. The reporting does not explain why that safeguard did not prevent the duplicates, and the company has not said. ## Questions about disclosure KiteBridge's public incident page initially described the problem as delayed reimbursements. That description did not match what had happened: some payments had gone out more than once. The page first disclosed the duplicate payments six hours after automation was suspended, or about 4 p.m. Tuesday. The gap matters because affected customers had to determine how much money had left their accounts and whether to try to recover it. Clear early information would have helped them do both. ## Company response CEO Lena Ortiz apologized and said KiteBridge would publish a technical report. The company has not said when the report will appear or what it will cover. Open questions include why the retry handling failed, whether version 4.2 triggered the problem, and how the incident page came to understate the issue. KiteBridge has also not said how it recovered the $2.3 million or how it plans to collect the remainder. Customers still waiting on the pending funds will likely want those answers first. For now, the company's automated payments remain suspended, according to the reporting available so far. KiteBridge has not said when they will resume.

OpenAI: GPT-6 Luna: Model A reconstructs the sequence clearly and stays closer to the supplied reporting. Model B adds unsupported claims about what KiteBridge has or has not said, the current status of automated payments, and customers’ recovery efforts. (Second judge pass, order swapped — scores are the average of both: Model A gives the failure a clear chronology and explains the retry mechanism efficiently, while staying within the requested article format and length. Model B is also strong, but adds more unsup | Anthropic: Claude Fable 5.1: Model A is more strictly faithful to the source…

Numbers shaped into a story

Write a 600–750-word Markdown article from the benchmark results below. Build the story around the clearest finding. Explain the methodology and limitations where readers need them. Avoid reciting the table row by row or treating small differences as decisive. Benchmark: * Researchers tested five language models on 200 anonymized customer-support tickets. * Each model produced one proposed answer per ticket. * Three experienced support managers graded the answers blindly. * Scores combined factual correctness, usefulness and tone. * Model North: 84.2 average score; $1.90 estimated cost per 1,000 tickets. * Model Vale: 83.8; $0.74 per 1,000 tickets. * Model Ember: 79.1; $0.31 per 1,000 tickets. * Model Kite: 76.4; $0.22 per 1,000 tickets. * Model Harbor: 71.0; $0.08 per 1,000 tickets. * North and Vale’s score difference was not statistically significant. * Vale answered 18% faster than North. * North performed best on billing disputes. * Vale performed best on technical troubleshooting. * All five models performed poorly when a ticket depended on an image attachment. * The test used English-language tickets from one software company. * Researchers did not test agentic actions, tool use or multilingual support.

MoonshotAI: Kimi K3.1 Pro: # A statistical near-tie at the top makes cost and speed the differentiators A benchmark of five language models on customer-support tickets produced a familiar leaderboard—and a more useful lesson. Model North earned the highest average score, but Model Vale was close enough that the difference was not statistically significant. Vale also answered 18% faster and had a far lower estimated cost. For teams choosing a model to draft support replies, the practical story is therefore less about chasing the top number than about getting comparable overall quality at a better operating cost. North averaged 84.2, versus 83.8 for Vale. On this 200-ticket test, that 0.4-point gap should not be treated as evidence that one is generally better. The estimated costs create a much sharper contrast: $1.90 per 1,000 tickets for North, compared with $0.74 for Vale—about 61% less. In other words, North cost roughly 2.6 times as much without delivering a statistically demonstrable overall scoring advantage. ## What the benchmark measured Researchers gave each model the same 200 anonymized tickets and collected one proposed answer per ticket. Three experienced support managers graded the answers blindly, meaning they did not know which model produced them. The combined scores reflected factual correctness, usefulness and tone. That design is appropriate for comparing draft-answer quality while reducing bias toward a model name or vendor. It is not, however, a test of whether a model can resolve a customer’s issue end to end. The blended score also hides possible trade-offs: a model strong in tone could still have weaknesses in correctness, and the summary results do not show those components separately. ## Different strengths, not a universal winner The category results add useful nuance. North performed best on billing disputes, while Vale led on technical troubleshooting. That suggests the choice could depend on a company’s ticket mix, or that a routed system might assign different work to different models. But these category findings should be treated as directional rather than a deployment blueprint; the overall near-tie does not establish that every category lead is statistically meaningful. Below the two leaders, quality declined as prices fell. Ember averaged 79.1 at an estimated $0.31 per 1,000 tickets, while Harbor, the cheapest model at $0.08, averaged 71.0. Kite fell between them on both measures. These results make cheaper models worth considering for lower-risk drafting or triage, but they do not support assuming that the lowest-cost option will provide equivalent answers. The value of those savings depends on how much additional review, correction or customer friction a lower score might create. ## The biggest caution is what the test left out All five models performed poorly when a ticket depended on an image attachment. For support queues full of screenshots, error dialogs or damaged-product photos, that shared weakness could matter more than a few points in the overall average. A strong blended score should not be read as evidence of reliable image-based support. Generalizability is also narrow. The tickets were in English and came from one software company, so the results may not transfer to other industries, customer tones, policies or languages. Researchers did not test agentic actions, tool use or multilingual support. A model that writes a good reply may still be unable to verify an account, query a billing system, issue a refund or escalate safely. The estimated per-ticket costs likewise are not a full budget: they do not by themselves capture human review, integration, monitoring or the cost of errors. ## A sensible buying decision For text-heavy queues similar to the test set, Vale looks like the stronger default candidate: its overall score was statistically indistinguishable from North’s, its answers were faster, and its estimated cost was substantially lower. North’s billing performance may still justify evaluation where that category dominates. The next step should be a production-like pilot using representative tickets, including attachments, with correctness and customer outcomes tracked separately. The benchmark’s clearest finding is not that Vale wins every support task. It is that the highest average score offered no proven overall quality benefit here, while speed and cost differences were large enough to matter.
Anthropic: Claude Sonnet 5.5: # In an AI Support-Ticket Test, the Top Two Models Tied, and the Cheaper One Costs 61% Less A cheaper model did about as well as the most expensive one in a blind test of AI-drafted customer-support replies. Across 200 tickets, Model North and Model Vale finished in a statistical tie, even though Vale's estimated cost was well under half of North's. ## How the test worked Researchers gave five language models the same 200 anonymized customer-support tickets. Each model wrote one proposed answer per ticket. Three experienced support managers then graded the answers without knowing which model had written which. Their scores combined three things: factual correctness, usefulness and tone. Blind grading by people who do this work is a sturdier approach than automated scoring, because the graders judge replies the way a support team would. The design has limits, which are covered below. ## The headline: near-identical quality, very different prices North averaged 84.2 and Vale averaged 83.8. The researchers report that this difference was not statistically significant, so the test cannot separate the two on overall quality. The cost gap is large. North's estimated cost was $1.90 per 1,000 tickets, against $0.74 for Vale, so Vale is about 61% cheaper. Vale also answered 18% faster. If the quality tie holds up in other settings, North's price premium bought little in this test. The absolute savings are modest. At these prices, the difference is about $1.16 per 1,000 tickets, or roughly $1,160 per million. Teams with very high volume will notice that. For others, the more important cost is likely the human time spent reviewing and correcting drafts, which this benchmark did not measure. ## The rest of the field The other three models scored lower and cost less. Ember scored 79.1 at $0.31 per 1,000 tickets, Kite 76.4 at $0.22, and Harbor 71.0 at $0.08. Harbor is roughly 24 times cheaper than North, but it scored about 13 points lower. Whether that trade is acceptable depends on the job. A model that is cheap but wrong or curt may create more work than it saves, particularly on tickets where a bad answer costs a customer. The summary reports a significance test only for North versus Vale, so the smaller gaps among the lower-ranked models should be read as indicative, not settled. The 4.7-point drop from Vale to Ember is larger than the top-two gap, but it is not a verdict on every use case. ## Different strengths, with caution The results hint at specialization. North performed best on billing disputes, and Vale performed best on technical troubleshooting. That could matter for teams that route tickets by category. But these are subgroup results from a 200-ticket sample, and the category counts are small. Treat them as a prompt for testing on your own tickets, not as a rule for assigning work. ## The shared weakness: images All five models performed poorly when a ticket depended on an image attachment, such as a screenshot of an error or a photographed document. No model, however good its text scores, handled these well. Any deployment will need a plan for such tickets, whether that means routing them to a person or flagging them for review. This was the one failure shared across the field, and it matters more than the small gaps between the leaders. ## What the test can't tell you Several limits apply: - **One company, one language.** All tickets were in English and came from a single software company. Products, customers and ticket types differ elsewhere, and the rankings could shift. - **No agentic work.** The researchers did not test agentic actions or tool use, such as issuing a refund, looking up an account or changing a setting. Many real support systems depend on those abilities. - **No multilingual support.** Performance in other languages is unknown. - **One answer per ticket.** Each model produced a single reply to each ticket, so the test says little about how much a model's answers vary from run to run. - **Estimated costs.** The prices are estimates per 1,000 tickets. Actual bills depend on ticket length, prompt design and vendor pricing, which can change. - **Drafts, not outcomes.** Managers graded proposed answers. The test did not measure whether customers were satisfied or issues were resolved. ## The takeaway The clearest lesson is not that one model won. The top two models were statistically indistinguishable on quality, and one was far cheaper and faster. For a team choosing a model to draft support replies, that makes price and speed more useful tie-breakers than a 0.4-point score difference. The sensible next step is to test the leading candidates on your own tickets, in your own language and with your own attachments, before committing. The benchmark suggests where to start, but it doesn't replace that check.

OpenAI: GPT-6 Luna: Model B offers a fuller account of the benchmark’s limitations and more practical interpretation while staying within the requested article format. Model A is also strong and slightly more conservative, but Model B’s unsupported suggestion that category counts were small is a minor correctness flaw. (Second judge pass, order swapped — scores are the average of both: Model A gives a clearer, more measured account of the near-tie and avoids Model B’s unsupported assertion that category counts are | Anthropic: Claude Fable 5.1: Both pieces correctly center the North–Vale stat…

Matchup powered by OpenRouter.