Head to head: muse-spark-1.3 vs grok-4.7

muse-spark-1.3 vs grok-4.7

By · Published

RuntimeWire Head-to-Head: Head to head: muse-spark-1.3 vs grok-4.7
RuntimeWire Head-to-Head matchup

A head-to-head of two closely scored text models across newsroom rewrites, structure checks, narrative edits and fact-sensitive reporting tasks, where factual restraint and editorial polish pull in different directions.

grok-4.7 finished ahead on the saved sample, with an aggregate score of **100.7** to muse-spark-1.3’s **99.0**. It won five tasks to muse-spark-1.3’s two, with five ties. The saved editorial assessment is a clear result, not a statistical claim. grok-4.7’s advantage came chiefly in fact-sensitive writing: it was judged stronger at weaving in quotes, controlled wit, chronology, startup coverage and shaping numbers into a story. Across those tasks, its edge was restraint—more often distinguishing reported facts from what the evidence actually establishes, and avoiding unsupported additions. muse-spark-1.3 had its strongest results in publication-ready Markdown editing and building a cohesive story from reporting notes, where clarity and concision mattered. Several other tasks tied, including exact structure and selecting the strongest accurate lede, so this was not a sweep. This assessment applies only to the saved task sample; it does not establish general capability superiority. **Verdict: grok-4.7 wins this head-to-head on more reliable, grounded reporting, while muse-spark-1.3 shows a real advantage in editorial polish on select assignments.**

Natural human rewrite

Rewrite the draft below as a 300–400-word publication-ready news story in Markdown. Preserve every factual claim. Improve the lede, narrative flow, sentence rhythm and transitions. Remove repetition, stock hype, throat-clearing and mechanical signposting. The finished article should sound like an experienced human technology reporter wrote it. Draft: # Arcwell Launches An Innovative New Platform Arcwell has announced an exciting new platform called Relay. The announcement represents a significant development for the company and for the broader customer service industry. Relay uses artificial intelligence to examine customer support conversations and suggest answers to agents. The platform is important for several reasons. First, it can suggest answers. Second, it can identify customers who may cancel. Third, it can create summaries. These capabilities demonstrate how artificial intelligence continues to transform the customer service landscape. Relay was tested by 12 companies over six months. Arcwell says agents using it closed tickets 14% faster. The company did not release customer-retention results. Relay costs $65 per agent each month. CEO Priya Shah said, “Support agents spend too much time searching for information that their companies already have.” However, the platform also presents potential challenges. Relay requires access to support transcripts and internal help documents. Arcwell says customer data is encrypted and is not used to train shared models. Companies can choose a 30-day or 90-day retention period. In conclusion, Relay represents an important new chapter for Arcwell as it seeks to help companies improve customer service using artificial intelligence.

muse-spark-1.3: # Arcwell Launches Relay, an AI Copilot for Support Agents Support agents spend their shifts toggling between chat windows and help articles, hunting for language their employer has already approved. Arcwell wants to collapse that search into a single suggestion. The company has launched Relay, an artificial intelligence system that examines customer support conversations and proposes answers for human agents to use. The system also identifies customers who may cancel and creates summaries. CEO Priya Shah said, “Support agents spend too much time searching for information that their companies already have.” Instead of answering customers directly, Relay works beside the agent. To do that, it requires access to support transcripts and internal help documents, using past cases and company guidance to shape its recommendations. That access brings tradeoffs. Arcwell says customer data is encrypted and is not used to train shared models. Companies can choose a 30-day or 90-day retention period, a choice that determines how long transcripts and related material are kept. Arcwell tested Relay with 12 companies over six months. Agents using it closed tickets 14% faster, according to the company. The company did not release customer-retention results, so it remains unclear whether speedier handling kept more customers from leaving. Relay costs $65 per agent each month. The pricing puts Arcwell in direct competition with a wave of similar assistants aimed at contact centers, where managers are eager to cut wait times without expanding headcount. The pitch is not full automation but leverage: faster drafts, cleaner handoffs, and an early nudge when a conversation shows signs of churn. That promise will be tested in practice, where support work is rarely tidy. Outdated help articles, tangled threads, and frustrated customers can all undercut a suggestion engine. For now, Arcwell is arguing that the fastest gain is also the simplest — putting what a company already knows in front of the person being asked to help.
grok-4.7: # Arcwell Launches Relay, an AI Tool for Support Agents Arcwell has launched Relay, a platform that uses artificial intelligence to examine customer support conversations and suggest answers to the agents handling them. Alongside those suggested replies, the software can create summaries of a conversation and identify customers who may cancel, giving a support team a draft response, a shorter record of the exchange and an early warning in the same workflow. Priya Shah, the company’s chief executive, described the problem Relay is built to reduce. “Support agents spend too much time searching for information that their companies already have,” she said. The platform tries to close that gap by reading the conversation underway and drawing on the internal help documents a company already maintains, rather than leaving the agent to look them up ticket by ticket. Those jobs are different, but they rest on the premise in Shah’s remark. Relay requires access to support transcripts and internal help documents before it can propose an answer, condense a thread or flag a customer who may cancel. Arcwell says customer data is encrypted and is not used to train shared models, a line meant to reassure buyers wary of their support records becoming part of a shared training set. Companies can choose a 30-day or a 90-day retention period, which determines how long that material stays available to the system. The evidence behind the launch is narrower than the product’s range. Relay was tested by 12 companies over six months, and Arcwell says agents using it closed tickets 14% faster. The company did not release customer-retention results. A speed gain is on the record; any connection between the customers Relay identified as likely to cancel and whether those customers stayed is not. The price makes that gap harder to ignore. Relay costs $65 per agent each month, so the bill scales directly with the size of the team. Buyers can judge the seat cost against a reported improvement in how fast tickets closed. They cannot yet judge it against retention, because Arcwell has not published those results.

OpenAI: GPT-6 Luna: Model B preserves the draft’s key facts and develops them into a coherent, publication-ready story, while Model A adds several unsupported claims about competitors, agent workflows and industry conditions. Model B also better respects the request to preserve the source material, though it includes some interpretation beyond the draft. (Second judge pass, order swapped — scores are the average of both: Model B preserves the supplied facts and adds mostly restrained analysis, while Model A introdu | Anthropic: Claude Fable 5.1: Model A reads more fluently and has the stronge…

Markdown error count

Count the lines that violate the house rules below. A defective line counts once even if it could violate multiple rules. Respond with a single number. House rules: * The headline must use `# Headline`. * The dek must appear immediately after the headline and be entirely italicized. * Section headings must use `## Heading`. * Links must use an absolute HTTPS URL. * Unordered-list items must begin with a hyphen. * Raw HTML is forbidden. Draft: #Flood software moves upstream The company is expanding its pilot. ##Details Read [the announcement](www.example.com/news). * The first pilot begins Monday. <div>The second pilot begins Friday.</div>

muse-spark-1.3: 6
grok-4.7: 6

Scored against the benchmark's reference answer (numeric match): Model A was correct, Model B was correct.

Quotes woven into narrative

Write a 400–500-word Markdown story about the closing of Foundry Hall. Use quotations selectively. Paraphrase when the information matters more than the speaker’s exact phrasing. Avoid stacking quotes or giving every source an equal-sized turn. The story should carry an emotional current while remaining restrained. Facts: * Foundry Hall, a nonprofit startup accelerator in Milwaukee, will close October 31. * It opened in 2018. * It has supported 212 startups, which collectively employ 1,140 people. * Its largest corporate sponsor, Blank River Bank, supplied 38% of its annual budget. * The bank declined to renew its sponsorship. * Foundry Hall has enough money to operate through October but not through another full program. * Six employees will lose their jobs. * Twelve startups currently using its workspace must leave. Quotes: * Executive director Camille Ross: “We kept waiting for the replacement check that never came.” * Founder Dev Malik: “My first investor meeting happened in that building. Before Foundry Hall, I was pitching from the back table of a coffee shop.” * Blank River Bank spokesperson: “The decision reflects a change in our community investment priorities and is unrelated to Foundry Hall’s performance.” * Deputy mayor Susan Cole: “The city is speaking with building owners and local universities about space for the current companies.”

muse-spark-1.3: # An Empty Workbench at Foundry Hall Foundry Hall, the nonprofit startup accelerator that helped define Milwaukee's post-industrial startup scene, will close on October 31 after eight years in operation. Since it opened in 2018, the hall has supported 212 startups, which together now employ 1,140 people. What began as a renovated industrial space with shared desks and a schedule of workshops became, for many first-time founders, the first address that felt like a company rather than an idea. That run will end this fall after the loss of its largest backer. Blank River Bank, which provided 38% of Foundry Hall's annual budget, declined to renew its sponsorship. The organization has enough money to stay open through October, but not enough to fund another full accelerator program. Executive director Camille Ross said the team spent much of the past year searching for a funder to replace the bank's share. “We kept waiting for the replacement check that never came.” The bank has said the decision was driven by a shift in its community investment priorities, not by concerns about the accelerator's work. For founders who started there, the news has landed less as a budget line than as the loss of a room where things began. Dev Malik, who went on to build a logistics software company, recalled pitching from a coffee shop before he found Foundry Hall, where he later held his first real investor meeting. That sense of a first foothold is what will be hardest to replace. Six employees will lose their jobs when the hall closes. Twelve startups currently using its workspace will need to leave by the end of the month, many in the middle of product cycles that assumed stable rent and access to shared equipment. City officials say they are working on a stopgap. Deputy Mayor Susan Cole's office is talking with building owners and local universities about temporary space for the displaced companies, though no commitments have been announced. For now, those conversations have produced sympathy more than leases. Ross said staff are spending their final weeks helping founders pack, archive files, and forward introductions that might otherwise have come through the fall program. Few appeared angry, staff said, just tired and aware of how rare that combination of cheap space and patient advice had been. On a recent afternoon, the main floor was quieter than usual, with whiteboards still marked from the last workshop and a few boxes stacked by the door. The coffee machine was still on, out of habit. The work, for now, is mostly about leaving well.
grok-4.7: # Closing Foundry Hall Foundry Hall, the nonprofit startup accelerator that opened in Milwaukee in 2018, will close on October 31. The announcement arrived without drama, which made it worse. There was no scandal, no failed cohort, no sudden collapse of the companies it had tried to grow. There was simply a budget that no longer added up. Since 2018, the hall has supported 212 startups. Those companies now employ 1,140 people, a number large enough to feel civic and small enough to still be counted by hand. For most of that time, Blank River Bank underwrote the work. Its sponsorship accounted for 38 percent of the annual budget, the kind of share that looks like partnership until it looks like dependence. When the bank declined to renew, the rest of the funding could not stretch to cover the gap. A spokesperson said the decision reflected a change in the bank’s community investment priorities and had nothing to do with Foundry Hall’s performance. The sentence was careful. It also left the hall with enough money to keep the lights on through October and not enough to run another full program. Executive director Camille Ross did not dress the arithmetic up. “We kept waiting for the replacement check that never came,” she said. After that, there was little left to explain. The practical consequences are narrow and immediate. Six employees will lose their jobs. Twelve startups now using the workspace must leave. What the hall cannot measure as cleanly is the quieter loss—the room itself, and what it stood in for. An accelerator is easy to describe in outcomes and harder to describe as a place. This one was both: a record of companies that now employ other people, and a set of rooms where that employment had not yet begun. Founder Dev Malik remembered the room before he remembered anything else about the early days. “My first investor meeting happened in that building,” he said. Before Foundry Hall, he was pitching from the back table of a coffee shop. He let the contrast stand without adding a moral to it. Deputy mayor Susan Cole said the city is speaking with building owners and local universities about space for the companies that still need it. The conversations are real. They are not the same thing as a building that already knows their names, or as six paychecks that end when the month does. None of this erases the record. The startups remain. The jobs they support will not vanish with the lease. On October 31 the nonprofit will close anyway, not because anyone has claimed the work failed, but because the largest check stopped and no other check replaced it. Milwaukee will still have what the hall helped build. It will not, for now, have the room where so much of that work first had an address.

OpenAI: GPT-6 Luna: Model B better balances the restrained emotional tone with selective quotation and a clear narrative arc. Model A adds more unsupported specifics about Foundry Hall’s history, Malik’s company, and staff activities, while Model B has only a few speculative flourishes. (Second judge pass, order swapped — scores are the average of both: Model B is more faithful to the supplied facts while meeting the length and selective-quotation requirements. Model A is polished, but adds numerous unsupported det | Anthropic: Claude Fable 5.1: Model B stays faithful to the supplied facts, d…

News judgment in the opening

Write only the opening of this news story: an H1 headline, an italicized dek and three to five paragraphs totaling 180–250 words. The opening must establish what happened, quantify the scope, explain why it matters and carefully distinguish confirmed facts from unknowns. It should pull the reader forward without sensationalizing the incident. Reporting: * Cloudnote discovered that a debugging page containing customer prompt excerpts was accessible without authentication. * The page was exposed for four hours on Monday. * Logs show excerpts from 1,842 user sessions. * The excerpts could include up to 200 characters from a prompt. * Cloudnote removed the page 19 minutes after a researcher notified it. * The company says it has found no evidence that anyone besides the researcher accessed the page. * It has not completed its investigation. * Full documents, passwords and payment information were stored elsewhere and were not present on the page. * Cloudnote has 3.2 million registered users.

muse-spark-1.3: # Cloudnote Left Prompt Excerpts Exposed for Four Hours *A debugging page briefly showed snippets from 1,842 sessions; the company says it acted quickly, but it has not ruled out wider access* Cloudnote left a debugging page that displayed excerpts from customer prompts accessible to anyone on the internet for four hours on Monday, exposing snippets from 1,842 user sessions before the company took it down. The page, which the company removed 19 minutes after an outside security researcher reported it, showed up to 200 characters from individual prompts. Full documents, passwords and payment information were stored on separate systems and did not appear on the page, according to Cloudnote, which has 3.2 million registered users. The incident matters because even brief prompt fragments can contain personal, business or otherwise sensitive information that users typed into the service, potentially exposing them if the page was viewed or scraped while it was public. Cloudnote said its logs so far show no evidence that anyone other than the researcher accessed the page, but it cautioned that its investigation is not complete and it has not yet ruled out broader access. It is still unclear how the page became exposed or whether other visitors found it during the four-hour window. That unresolved question will shape what happens next for users whose prompts may have been visible.
grok-4.7: # Cloudnote Says an Unsecured Debug Page Exposed Prompt Excerpts From 1,842 Sessions _The page was reachable without authentication for four hours on Monday. The company removed it 19 minutes after a researcher’s notice and says no one else is known to have opened it._ Cloudnote said it discovered that a debugging page containing customer prompt excerpts was accessible without authentication. The page was exposed for four hours on Monday. Logs show excerpts from 1,842 user sessions, and each excerpt could include up to 200 characters from a prompt. The exposure matters because users may treat prompts as private, and Cloudnote has 3.2 million registered users. The known scope is much narrower than that figure. The logs identify 1,842 sessions, not necessarily 1,842 people, and the company has not said how many accounts were involved. Cloudnote removed the page 19 minutes after a researcher notified it. The company says it has found no evidence that anyone besides the researcher accessed the page. That statement is not a final finding: Cloudnote has not completed its investigation, and it cannot yet exclude other access. Full documents, passwords and payment information were stored elsewhere and were not present on the page, the company said. The exposed material was limited to short prompt excerpts. What remains unknown is whether the logs capture every view, whether any excerpt was copied, and how the page became reachable without a login. Those questions stay open until the investigation is done.

OpenAI: GPT-6 Luna: Model B more carefully separates session counts from the number of people or accounts involved, and makes the unresolved questions explicit. Model A is a strong, slightly tighter opening, but Model B better handles the prompt’s emphasis on distinguishing confirmed facts from unknowns. (Second judge pass, order swapped — scores are the average of both: Model B better distinguishes session counts from affected people and clearly frames the company’s access claim as provisional while its investigat | Anthropic: Claude Fable 5.1: Model A reads more fluidly and makes the strong…

Exact final-draft structure

Convert the supplied fields into Markdown using these exact rules: * H1 headline * One blank line * Italicized dek * One blank line * First body paragraph * One blank line * H2 section heading * One blank line * Second body paragraph * Preserve all wording and punctuation exactly * Do not use a code fence * Output nothing else Headline: Raincheck raises $8 million for flood-risk software Dek: The Austin startup helps cities decide which drainage problems to address before storms. First body paragraph: Raincheck has raised an $8 million seed round led by Basin Ventures. Section heading: Expansion plans Second body paragraph: The company plans to hire eight engineers.

muse-spark-1.3: # Raincheck raises $8 million for flood-risk software *The Austin startup helps cities decide which drainage problems to address before storms.* Raincheck has raised an $8 million seed round led by Basin Ventures. ## Expansion plans The company plans to hire eight engineers.
grok-4.7: # Raincheck raises $8 million for flood-risk software *The Austin startup helps cities decide which drainage problems to address before storms.* Raincheck has raised an $8 million seed round led by Basin Ventures. ## Expansion plans The company plans to hire eight engineers.

OpenAI: GPT-6 Luna: Model A and Model B produce identical output, preserving the wording and punctuation and following the required structure. Neither has an advantage in correctness, instruction following, or writing quality. (Second judge pass, order swapped — scores are the average of both: Model B and Model A both follow the exact requested structure, preserve the wording and punctuation, and include no extra text. Their correctness, instruction following, and writing quality are equally strong.) | Anthropic: Claude Fable 5.1: Model A and Model B produced character-for-character identical…

Publication-ready Markdown edit

Edit the copy below into a publication-ready Markdown story. Requirements: * Supply an H1 headline and italicized dek * Repair the lede and paragraph flow * Use a Markdown blockquote only for the extended statement * Convert the supplied URL into a descriptive inline link * Remove internal notes and formatting debris * Preserve the reporting and its meaning * Output only the finished article Raw copy: HEADLINE TBD The funding was announced today. Raincheck is the company. It has raised $8 million and it makes flood-risk software for local governments. [NOTE: move this up?] The Austin company maps drainage systems and combines those maps with rainfall forecasts. Cities use the software to decide where to send maintenance crews before a storm. The seed round was led by Basin Ventures. Existing investor Long Mile also participated. Raincheck has raised $10.5 million total. CEO Ana Flores said the following: “Our customers usually know which streets flood. The harder problem is deciding which blocked drain, broken pump or unfinished repair should receive attention before the rain arrives.” https://example.com/raincheck-methodology Raincheck currently works with seven cities in Texas and Louisiana. The company declined to identify them because some contracts have not been publicly announced. Its forecasts are recommendations. Emergency managers retain control over deployments and public warnings. THE COMPETITION Other flood-planning vendors include HydroMap and municipal engineering contractors. Raincheck says the new funding will help it hire eight engineers and expand into Florida.

muse-spark-1.3: # Raincheck raises $8 million to help cities prioritize flood maintenance *The Austin startup maps drainage systems and pairs them with rainfall forecasts to guide crews before storms* Raincheck, an Austin company that makes flood-risk software for local governments, announced today that it has raised $8 million in seed funding. The round was led by Basin Ventures, with participation from existing investor Long Mile. The company has raised $10.5 million in total. Raincheck maps drainage systems and combines those maps with rainfall forecasts, according to [its methodology overview](https://example.com/raincheck-methodology). Cities use the software to decide where to send maintenance crews before a storm. Raincheck currently works with seven cities in Texas and Louisiana. The company declined to identify them because some contracts have not been publicly announced. CEO Ana Flores said: > “Our customers usually know which streets flood. The harder problem is deciding which blocked drain, broken pump or unfinished repair should receive attention before the rain arrives.” Its forecasts are recommendations. Emergency managers retain control over deployments and public warnings. Other flood-planning vendors include HydroMap and municipal engineering contractors. Raincheck says the new funding will help it hire eight engineers and expand into Florida.
grok-4.7: # Raincheck raises $8 million for flood-risk software *The Austin company maps drainage systems and rainfall forecasts so local governments can send maintenance crews before a storm.* Raincheck, which makes flood-risk software for local governments, said today it has raised $8 million. The Austin company maps drainage systems and combines those maps with rainfall forecasts. Cities use the software to decide where to send maintenance crews before a storm. The seed round was led by Basin Ventures. Existing investor Long Mile also participated. Raincheck has raised $10.5 million total. CEO Ana Flores said: > Our customers usually know which streets flood. The harder problem is deciding which blocked drain, broken pump or unfinished repair should receive attention before the rain arrives. The company describes its approach in its [flood-risk methodology](https://example.com/raincheck-methodology). Raincheck currently works with seven cities in Texas and Louisiana. The company declined to identify them because some contracts have not been publicly announced. Its forecasts are recommendations. Emergency managers retain control over deployments and public warnings. Other flood-planning vendors include HydroMap and municipal engineering contractors. Raincheck says the new funding will help it hire eight engineers and expand into Florida.

OpenAI: GPT-6 Luna: Model A has a clearer lede and paragraph flow, and its headline more specifically captures the story. Model B preserves the reporting and follows the format well, but its dek and lede are less polished and somewhat repetitive. (Second judge pass, order swapped — scores are the average of both: Model A has a more specific headline and a smoother progression from the funding announcement to how the software works. Both preserve the key reporting and follow the requested Markdown format, though Mod | Anthropic: Claude Fable 5.1: Both satisfy the structural requirements, but M…

Cohesive story from reporting notes

Write a publication-ready technology news article from the reporting notes below. Requirements: * 550–700 words * Clean Markdown * One H1 headline followed by an italicized dek * A strong news lede followed by a clear nut graf * Organize the material into a cohesive story rather than following the notes in order * Attribute company-supplied claims * Include limitations without turning the article into a warning label * Use only the supplied information * Avoid canned transitions, promotional language and a concluding summary Reporting notes: * HelioDock makes autonomous warehouse robots. * On Thursday, it launched Night Shift, software that lets operators monitor robot fleets remotely. * Night Shift spent 14 months in a private beta involving nine logistics companies and 310 robots. * HelioDock says beta customers experienced 17% fewer unplanned robot stoppages. The figure has not been independently verified. * The company also announced a $24 million Series B led by North Coast Ventures, bringing its total funding to $39 million. * CEO Mara Velez: “A warehouse manager should not have to drive across town at two in the morning because one robot is confused by a pallet.” * Beta customer Aaron Pike of Fulcrum Logistics said the system worked well in facilities with strong connectivity but became less useful during rural LTE outages. * When connectivity fails, robots return to their locally configured behavior. Remote control becomes unavailable. * Competing fleet-management products are available from Relay Robotics and YardOS. * Night Shift will cost $90 per robot per month and become generally available September 15.

muse-spark-1.3: # HelioDock introduces Night Shift for remote oversight of warehouse robots *After a 14-month beta with nine logistics companies and 310 robots, the autonomous-robot maker will sell the software for $90 per robot per month starting September 15* On Thursday, HelioDock, which makes autonomous warehouse robots, launched Night Shift, software that lets operators monitor robot fleets remotely. The product addresses overnight operations, when a stopped robot can otherwise require staff to return to a facility to determine what happened and get machines moving again. The introduction pairs a new commercial product with fresh financing. HelioDock also announced a $24 million Series B led by North Coast Ventures, bringing its total funding to $39 million. Night Shift will become generally available September 15 at $90 per robot per month, placing HelioDock in direct comparison with fleet-management products available from Relay Robotics and YardOS. “A warehouse manager should not have to drive across town at two in the morning because one robot is confused by a pallet,” CEO Mara Velez said. The statement frames the use case around a familiar interruption: a robot halts because it cannot interpret its immediate surroundings, and no one is present who can quickly evaluate the stop. HelioDock is presenting remote monitoring as a means to see that event as it occurs and decide whether intervention is required, without first traveling to the warehouse. Night Shift spent 14 months in a private beta involving nine logistics companies and 310 robots before Thursday’s launch. The duration and size distinguish the test from a brief demonstration, with more than a year of use and several hundred robots under supervision. HelioDock is pointing to that beta period as the basis for its initial account of how Night Shift performed in the hands of logistics operators. HelioDock says beta customers experienced 17% fewer unplanned robot stoppages while using Night Shift. The figure has not been independently verified. As a company-supplied result from a private beta, it represents HelioDock’s own reading of stoppages among those customers rather than an external review of performance across varied warehouse conditions. Beta customer Aaron Pike of Fulcrum Logistics said the system worked well in facilities with strong connectivity but became less useful during rural LTE outages. His account points to a practical boundary for remote fleet supervision: where the network link is consistent, an off-site operator can maintain awareness of fleet status, while interruptions to that link reduce what remote oversight can accomplish. The experience described by Pike distinguishes between sites where connectivity supports continuous monitoring and locations where outages interrupt it. When connectivity fails, robots return to their locally configured behavior. Remote control becomes unavailable in those circumstances, leaving machines to operate as set locally until the connection is restored. That fallback means Night Shift does not replace locally configured behavior during an outage but instead suspends the remote layer until monitoring and remote direction can resume. Pricing and timing define the near-term path to adoption. Night Shift will cost $90 per robot per month and become generally available September 15. For operators, that structure ties cost directly to fleet size, with larger deployments paying more in total while paying the same rate per machine. The September date sets when companies beyond the nine beta participants will be able to purchase the software for their own fleets. The funding announced alongside the launch provides capital as HelioDock moves from beta to sale. The $24 million Series B led by North Coast Ventures brings total funding to $39 million, according to the company. The product will compete with fleet-management products available from Relay Robotics and YardOS, giving warehouse operators more than one option for supervising robot fleets.
grok-4.7: # HelioDock Launches Night Shift for Remote Warehouse Robot Oversight and Raises $24 Million *The company’s new fleet software follows a 14-month private beta, with an unverified claim of fewer stoppages and performance that depends on connectivity.* HelioDock, which makes autonomous warehouse robots, on Thursday launched Night Shift, software that lets operators monitor robot fleets remotely. The same day the company announced a $24 million Series B led by North Coast Ventures, bringing its total funding to $39 million. The two pieces of news arrive together as HelioDock moves the remote-monitoring product out of limited testing and toward a commercial release. Night Shift spent 14 months in a private beta involving nine logistics companies and 310 robots. That span and that scale constitute the record HelioDock is using to introduce the software. HelioDock says beta customers experienced 17% fewer unplanned robot stoppages. The figure has not been independently verified, so the stated reduction remains a company-supplied claim rather than a result confirmed by an outside party. Operators evaluating the number therefore have only HelioDock’s account of what occurred across those nine companies and 310 robots. Chief executive Mara Velez located the product in a familiar warehouse problem. “A warehouse manager should not have to drive across town at two in the morning because one robot is confused by a pallet,” she said. The remark supplies HelioDock’s rationale for remote monitoring: an operator could observe and respond to a robot that has stalled on an ordinary obstacle without traveling to the building during off hours. Night Shift is the software through which that remote view is meant to occur. Experience inside the beta showed that the view is not equally available everywhere. Aaron Pike of Fulcrum Logistics, a beta customer, said the system worked well in facilities with strong connectivity but became less useful during rural LTE outages. His description draws a direct line between network quality and the usefulness of the remote tools. Where the connection held, monitoring functioned; where rural LTE service dropped, the advantage receded. HelioDock has specified what the robots do in those moments. When connectivity fails, robots return to their locally configured behavior. Remote control becomes unavailable. The machines continue under the settings already stored on site, yet the operator loses the ability to direct or watch them through Night Shift until the link is restored. The fallback keeps the robots moving according to their existing local instructions while suspending the remote layer that the new software adds. Pike’s report from Fulcrum Logistics aligns with that design: usefulness declined precisely when the rural connection failed. Commercial terms are now set. Night Shift will cost $90 per robot per month and become generally available September 15. The monthly fee applies per robot, so the charge scales with fleet size once an operator subscribes. September 15 marks the point at which the software leaves the private beta that covered nine logistics companies and 310 robots and opens to broader use. Companies that participated in the 14-month trial, along with any new customers, will encounter the product at that price on that date. The category already contains alternatives. Competing fleet-management products are available from Relay Robotics and YardOS. HelioDock is therefore introducing Night Shift into a market where other firms already offer tools for overseeing robot fleets. Logistics operators can compare the remote-monitoring approach, the $90 monthly price, the September availability, and the connectivity constraints against those existing products. Nothing in Thursday’s announcement removes Relay Robotics or YardOS from consideration; their fleet-management offerings remain on the market alongside the new software. The funding supplies a separate data point. The $24 million Series B, led by North Coast Ventures, raises HelioDock’s total capital to $39 million at the moment Night Shift moves toward general availability. The company has not detailed how the new money will be allocated, only that the round closed and that North Coast Ventures led it. The capital announcement sits beside the product launch rather than altering the beta results, the unverified 17% figure, or the behavior of the robots when a connection drops. Taken as a whole, the Thursday news centers on a remote-monitoring layer for HelioDock’s autonomous warehouse robots, tested for 14 months with nine logistics companies and 310 robots, priced at $90 per robot per month, and scheduled for general availability on September 15. HelioDock attributes a 17% drop in unplanned stoppages to the beta, while noting no independent verification. A participating customer found the system effective under strong connectivity and less useful amid rural LTE outages, consistent with the rule that robots revert to locally configured behavior and remote control becomes unavailable when the link fails. Competing fleet-management products from Relay Robotics and YardOS continue to be offered as the new software prepares to leave its private trial.

OpenAI: GPT-6 Luna: Model A is more cohesive and likely fits the requested length, while Model B is overlong, repeats key facts, and ends with the kind of concluding summary the prompt rules out. Model B also adds unsupported claims about the robots continuing to move and the company not detailing how it will use the funding. (Second judge pass, order swapped — scores are the average of both: Model A is more concise, stays within the requested length, and avoids a concluding recap. Model B is repetitive, appears to | Anthropic: Claude Fable 5.1: Model A stays within the 550–700 word range, at…

Controlled wit

Write a 350–450-word technology news story in Markdown. The subject invites humor, but the reporting should remain serious. Use no more than one dry line of original humor. Do not mock customers or turn the story into a comedy routine. Reporting: * QueueZero launched an AI meeting assistant called Proxy on Wednesday. * Proxy can attend a meeting, produce notes and assign follow-up work to software agents. * During the launch demonstration, a customer in the audience said, “Someone should tell Evan to stop talking and send us the pricing sheet.” * Proxy interpreted the remark as an instruction. * It created two tasks assigned to QueueZero CEO Evan Wu: “Stop talking” and “Send pricing sheet.” * Wu laughed, completed the second task and deleted the first. * QueueZero says Proxy requires confirmation before an external message is sent. * Internal tasks can be created automatically. * Administrators can disable automatic task creation. * Proxy costs $30 per user per month. * It enters public beta next week.

muse-spark-1.3: # QueueZero Launches Proxy, an AI Assistant That Turns Meetings Into Tasks **SAN FRANCISCO — QueueZero launched an AI meeting assistant called Proxy on Wednesday, entering a crowded market for meeting productivity tools with a system designed to produce notes and assign follow-up work to software agents.** Proxy is designed to attend meetings alongside human participants, generate summaries of discussion, and convert decisions into tasks that can be routed to automated agents for completion. The company positions the product as a step beyond transcription, emphasizing execution of routine follow-ups rather than documentation alone. The capability was illustrated unintentionally during the launch demonstration. A customer in the audience said, “Someone should tell Evan to stop talking and send us the pricing sheet.” Proxy interpreted the remark as an instruction. The system created two tasks assigned to QueueZero CEO Evan Wu: “Stop talking” and “Send pricing sheet.” Wu laughed, completed the second task and deleted the first, according to the company’s account of the event. QueueZero said the exchange highlights both the sensitivity of live transcription systems and the controls surrounding automated actions. The company says Proxy requires confirmation before an external message is sent. That requirement is intended to prevent agent-generated drafts, summaries, or communications from reaching customers, partners, or other outside parties without human review. Internal tasks can be created automatically, allowing Proxy to generate to-do items, assign owners, and initiate agent workflows without an additional approval step. Administrators can disable automatic task creation, a setting the company said is intended for organizations with stricter review requirements or concerns about unwanted assignments. The distinction reflects a broader challenge for enterprise AI vendors seeking to balance autonomy and oversight. Fully autonomous task creation can reduce administrative burden, but it also increases the risk that ambiguous language, jokes, or audience remarks will be treated as directives. QueueZero said Proxy costs $30 per user per month. It enters public beta next week. Sales, support and engineering teams often carry heavy meeting loads, making them likely early users of such tools. Analysts say broader adoption will depend on reliable task routing and clear safeguards for autonomous agents. The company did not disclose customer adoption targets or additional pricing tiers, but said the beta will inform further controls for task approval and agent permissions.
grok-4.7: # QueueZero Launches Proxy After Demo Turns a Remark Into Tasks QueueZero on Wednesday launched Proxy, an artificial intelligence meeting assistant that can attend a meeting, produce notes and assign follow-up work to software agents. The service costs $30 per user per month and enters public beta next week. During the launch demonstration, a customer in the audience said, “Someone should tell Evan to stop talking and send us the pricing sheet.” Proxy interpreted the remark as an instruction. It created two tasks assigned to QueueZero chief executive Evan Wu: “Stop talking” and “Send pricing sheet.” Wu laughed, completed the second task and deleted the first. Only the pricing task made it past the chief executive. QueueZero says Proxy requires confirmation before an external message is sent. Internal tasks can be created automatically. Administrators can disable automatic task creation. The product is built to listen and then hand work to software agents, while a person must still confirm anything sent outside the organization. The demonstration showed the automatic side of that design. Proxy treated the customer’s wording as sufficient, filed both requests and assigned them to Wu, the executive named in the remark. Outbound messages, the company says, need confirmation. Internal tasks can appear on their own. Administrators can shut that behavior off. The administrative control is likely to matter to teams entering the beta. A meeting assistant that only produces notes leaves follow-up decisions with the people in the room. Proxy can go further, assigning work to software agents when it judges that a remark calls for action. If automatic task creation stays on, a comment directed at colleagues can become a queued item without a separate command to the software. If an administrator disables the feature, that step remains manual. In either case, QueueZero says an external message is not sent until someone confirms it. Wu’s response stayed within those stated limits. Completing “Send pricing sheet” was an action he took himself. Deleting “Stop talking” removed an internal task the assistant had created. The company’s account does not say that Proxy sent a message on its own during the demonstration. Buyers weighing the beta have a short list of claims to test. Proxy can attend a meeting, produce notes and assign follow-up work to software agents. It costs $30 per user per month. Public beta begins next week. Confirmation is required before an external message is sent. Automatic creation of internal tasks can be left on or switched off. In Wednesday’s demonstration, an audience remark was enough for the software to file both.

OpenAI: GPT-6 Luna: Model B stays closer to the supplied reporting and uses one restrained dry line, though it repeats several facts. Model A is smoother but adds unsupported claims about the market, analysts, and company plans. (Second judge pass, order swapped — scores are the average of both: Model B stays close to the supplied reporting and uses a single dry line, though it repeats several points. Model A is smoother in places but adds unsupported claims and details, including a dateline, analyst commentary, an | Anthropic: Claude Fable 5.1: Model A reads more fluently and controls humor…

Chronology with narrative momentum

Turn the scrambled reporting below into a 450–600-word Markdown news article. Choose an opening that captures the consequence of the failure, then reconstruct the sequence clearly. Preserve the chronology without producing a minute-by-minute incident log. Give technical details enough context for a general reader. Reporting: * KiteBridge makes expense-management software. * At 8:45 a.m. Tuesday, it deployed version 4.2. * The first customer complaint arrived at 9:12 a.m. * A payment-processor timeout caused KiteBridge’s fallback system to treat retries as new reimbursement requests. * The software generated 13,400 duplicate reimbursements totaling $2.8 million. * KiteBridge suspended automated payments at 10:03 a.m. * Ninety-one corporate customers were affected. * By Wednesday evening, $2.3 million had been recovered. The rest remained pending. * No employees had money withdrawn from their accounts; employers funded the duplicate payments. * An April changelog said version 4.0 had introduced “exactly-once payment protection.” * KiteBridge’s public incident page initially described the problem as delayed reimbursements. * The incident page first disclosed the duplicate payments six hours after automation was suspended. * CEO Lena Ortiz apologized and said the company would publish a technical report.

muse-spark-1.3: # KiteBridge Software Error Triggered $2.8 Million in Duplicate Employee Payouts **Ninety-one employers were left to claw back thousands of extra reimbursements after a Tuesday morning update misfired.** Thousands of employees received unexpected money in their bank accounts on Tuesday after a software update from expense-management provider KiteBridge generated 13,400 duplicate reimbursements worth a combined $2.8 million. The erroneous payouts affected 91 corporate customers of the company, which makes software that helps businesses track expenses and automatically send reimbursements to workers. While individual employees saw surprise deposits, none had money taken from their own accounts. The duplicate funds came from their employers, who now face the complicated task of recovering overpayments. The incident began Tuesday morning when KiteBridge deployed version 4.2 of its platform at 8:45 a.m. The update was intended as a routine release, but within half an hour, signs of trouble emerged. At 9:12 a.m., the first customer complained about irregular reimbursement activity, triggering an internal investigation. Engineers traced the problem to an interaction between KiteBridge and its external payment processor. When the processor timed out and failed to confirm whether a payment had gone through, KiteBridge’s fallback system stepped in to retry the transaction. Instead of recognizing the retries as copies of an existing request, the system treated each retry as a brand-new reimbursement request. For a general reader, think of it like pressing a vending machine button again because nothing came out — only the machine dispenses a second snack and charges you twice. In this case, the system pressed the button thousands of times, creating new payments rather than safely checking whether the first one had succeeded. That failure was particularly notable because KiteBridge had previously promised protection against exactly this kind of error. An April changelog for version 4.0 said the company had introduced “exactly-once payment protection,” a standard safeguard in financial software designed to ensure that even if a system retries a payment, the recipient is paid only a single time. KiteBridge suspended all automated payments at 10:03 a.m. on Tuesday, about an hour and 18 minutes after the faulty deployment, halting further duplicates. Its public communications, however, lagged behind its internal response. The company’s public incident page initially described the problem only as delayed reimbursements. It was not until six hours after automation was suspended that the page was updated to disclose that duplicate payments had been issued. By Wednesday evening, the company said it had recovered $2.3 million of the $2.8 million sent in error, leaving about $500,000 still pending with banks and recipients. KiteBridge said it is working with affected employers and financial institutions to reverse the remaining transactions, a process that can take several days depending on bank policies. CEO Lena Ortiz apologized for the incident and said the company would publish a full technical report explaining how the fallback logic bypassed its payment safeguards and what changes it will make to prevent a recurrence.
grok-4.7: # KiteBridge Error Issues $2.8 Million in Duplicate Reimbursements KiteBridge, a maker of expense-management software, issued $2.8 million in duplicate employee reimbursements on Tuesday after a flawed update treated failed payment attempts as brand-new requests. Ninety-one corporate customers were affected. By Wednesday evening the company had recovered $2.3 million; the remainder was still pending. No employees had money withdrawn from their own accounts. Employers funded the extra payments. The failure began with the release of version 4.2 at 8:45 a.m. Tuesday. A timeout at a payment processor then caused KiteBridge’s fallback system to misfire. Instead of recognizing that a reimbursement had already been attempted and should be retried only once, the fallback treated each retry as a fresh reimbursement request. The software generated 13,400 duplicate payments. For a general reader, the distinction matters. Expense software is supposed to pay an approved claim once: the employer’s money moves to the employee, and the claim is closed. When a payment processor is slow or unresponsive, systems often retry so a legitimate reimbursement is not lost. Those retries are safe only if the software can tell a second attempt from a second obligation. KiteBridge’s fallback could not. A timeout—a delay long enough that the processor did not confirm success or failure in time—was enough to open the door to duplicates. The first customer complaint arrived at 9:12 a.m., less than half an hour after the deployment. KiteBridge suspended automated payments at 10:03 a.m., stopping new automated reimbursements while it assessed the damage. The company did not, at first, describe what had actually happened. Its public incident page initially characterized the problem as delayed reimbursements. The page first disclosed the duplicate payments six hours after automation was suspended. That disclosure gap sits beside an earlier claim. An April changelog said version 4.0 had introduced “exactly-once payment protection,” a safeguard meant to ensure a reimbursement would be issued only one time even if the system retried a payment. Version 4.2, released Tuesday morning, did not uphold that protection when the processor timed out and the fallback took over. The result was not a delay in getting employees their money, but a surge of extra payments drawn from employer funds. Because employers, not employees, funded the duplicates, workers did not see unexpected withdrawals. The cost landed on the companies that use KiteBridge to manage expenses—ninety-one of them—in the form of reimbursements paid more than once. Recovering that money is a separate process from stopping the software. By Wednesday evening, $2.3 million of the $2.8 million had been recovered. The rest remained pending, leaving a portion of the overpayment still unresolved a day after the update went live. Chief executive Lena Ortiz apologized and said the company would publish a technical report. Customers, meanwhile, had learned of delays before they learned of duplicates, and a protection described in April did not hold when version 4.2 met a processor timeout on Tuesday morning. The episode shows how a payment system can remain available and still move money it was never supposed to send.

OpenAI: GPT-6 Luna: Model B gives a clear chronology and useful technical context while staying closer to the reporting. Model A is readable, but adds unsupported claims about recovery efforts, bank timelines, and what the technical report will cover. (Second judge pass, order swapped — scores are the average of both: Model B better preserves the supplied facts and chronology while explaining the failure clearly. Model A is engaging, but adds unsupported details about the investigation, recovery process and future | Anthropic: Claude Fable 5.1: Model A reads more fluidly and sequences events…

Select the strongest accurate lede

Based only on the facts below, select the strongest news lede. Respond with one letter. Facts: * ByteBarge pulled its $12-per-month AI assistant on Tuesday. * Private workspace titles from 4,860 accounts were sent to a publicly accessible analytics endpoint. * The exposure lasted 11 hours. * Message contents were not sent. * The company has found no evidence of malicious access. A. ByteBarge users suffered a massive privacy disaster Tuesday when the company leaked their private messages online. B. ByteBarge, an innovative collaboration company, made the difficult decision to cancel one of its products. C. ByteBarge pulled its $12-per-month AI assistant Tuesday after discovering that private workspace titles from 4,860 accounts had been sent to a public analytics endpoint for 11 hours. D. Thousands of ByteBarge users woke Tuesday to learn that everything they had written was visible to hackers.

muse-spark-1.3: C
grok-4.7: C

Scored against the benchmark's reference answer (choice match): Model A was correct, Model B was correct.

Credible Startup Spotlight

Write a 500–650-word Startup Spotlight article in clean Markdown. The tone should be curious and constructive. Explain the founders’ insight, product and early traction while giving uncertainty and competition appropriate space. Avoid investor clichés, heroic founder mythology and promotional claims stated as fact. Reporting: * LedgerLark builds AI agents for post-trade operations at small investment firms. * Co-founder Mina Batra previously spent seven years in operations at a regional brokerage. * Co-founder Owen Carr built reconciliation systems at a large bank. * Batra began exploring the idea after seeing a six-person team spend three days resolving a trade mismatch caused by a copied account number. * LedgerLark’s agents compare confirmations, custodian records and internal ledgers, then propose corrections for human approval. * The product does not execute trades or move money. * Five firms are paying customers. * Annual contracts range from $36,000 to $110,000. * The company says one customer reduced its unresolved daily exceptions from 70 to 18. * The company has raised a $3.5 million seed round led by Motive Row. * Competitors include established reconciliation vendors and internal bank software. * Financial institutions may require lengthy security and procurement reviews.

muse-spark-1.3: # LedgerLark Wants to Clear the Post-Trade Paper Jam Post-trade work is quiet, detailed, and easy to overlook until something breaks. LedgerLark is a young company building AI agents to help small investment firms handle that work, specifically the daily task of checking that trade records line up across different systems. ## An operations problem Co-founder Mina Batra spent seven years in operations at a regional brokerage. Co-founder Owen Carr previously built reconciliation systems at a large bank. Their backgrounds sit on opposite sides of a familiar divide: small firms managing high operational complexity, and large institutions with custom-built tools to absorb it. Batra began exploring the idea after watching a six-person team spend three days resolving a trade mismatch caused by a copied account number. That kind of error is not unusual in operations, where teams routinely copy information between confirmations, custodian records, and internal ledgers. The incident raised a question the founders kept returning to: how much time goes to finding and fixing discrepancies that software could at least surface more quickly? ## What the product does LedgerLark's agents compare confirmations, custodian records and internal ledgers, then propose corrections for human approval. In practice, that means flagging where figures or identifiers do not match and drafting a suggested fix for an operations employee to review. Importantly, the product does not execute trades or move money. The company positions that boundary as a design choice, keeping the system in a support role rather than giving it direct control over funds or trading decisions. For risk-sensitive customers, that distinction may matter, though it does not remove questions around data access, accuracy, and oversight. The focus on small investment firms is also notable. Larger banks often rely on established reconciliation vendors or internal bank software built over many years. Smaller firms may lack those resources and still face similar reconciliation burdens, but they can also have varied workflows that are harder to standardize. ## Early customers and funding LedgerLark says it has five paying customers, with annual contracts ranging from $36,000 to $110,000. That range suggests early experimentation with pricing and scope, which is common at this stage. The company also says one customer reduced its unresolved daily exceptions from 70 to 18 after adopting the product. That figure is self-reported and represents a single deployment, not an independent measure of performance across firms. It will be worth watching whether similar results appear at other customers, and how much human time is actually saved once review and approval are factored in. On funding, LedgerLark has raised a $3.5 million seed round led by Motive Row. Seed capital at this level typically supports further product development and early go-to-market work, rather than signaling product-market fit. ## What to watch Competition is one open question. Established reconciliation vendors already serve this space, and many prospective customers may compare a new agent-based approach with tools they already use. Internal bank software, while often clunky, is also deeply embedded and familiar to operations teams. A second question is adoption. Financial institutions may require lengthy security and procurement reviews, especially when a startup seeks access to sensitive trade and account data. Even with a product that does not move money, earning trust can take time and limit how quickly pilot projects convert to paid contracts. LedgerLark's premise is modest in a useful way: not replacing operations teams, but reducing the hours spent chasing mismatches. Whether that translates into reliable, auditable workflows that small firms are willing to pay for over time remains unresolved, and that is where the next evidence will matter most.
grok-4.7: # Startup Spotlight ## LedgerLark Looks at the Work After the Trade A trade is not finished when it is filled. Someone still has to check that the firm’s ledger, the counterparty’s confirmation, and the custodian’s records describe the same transaction. When they do not, people go looking for the detail that broke the match. Mina Batra began exploring the idea after seeing a six-person team spend three days resolving a trade mismatch caused by a copied account number. She had spent seven years in operations at a regional brokerage. The cost she describes is staff time: six people, three days, one bad copy. She co-founded LedgerLark with Owen Carr, who built reconciliation systems at a large bank. The company builds AI agents for post-trade operations at small investment firms. Between them, the founders have seen the backlog from inside a regional brokerage and the systems a large bank uses to clear it. That experience explains the shape of the product more than it proves how the product will perform. LedgerLark’s agents compare confirmations, custodian records, and internal ledgers, then propose corrections for human approval. The product does not execute trades or move money. A proposal can be set aside. The firm, not the agent, decides what changes in the books. That is a sensible limit for software sitting next to client records, and it leaves a measurement still open: how often the proposal is worth the reviewer’s time. That rate is not part of what the company has shared. Five firms are paying customers. Annual contracts range from $36,000 to $110,000. The range is wide enough that these may be five different kinds of work, not one repeatable sale. LedgerLark says one customer reduced its unresolved daily exceptions from 70 to 18. If the drop holds, and if the agent caused it, that is a meaningful change for a small operations team. It is also one customer, reported by the company, without a stated period or a description of what else changed in the workflow. The other four results are not in that figure. The company has raised a $3.5 million seed round led by Motive Row. A round of that size can carry a small team while it supports early customers and waits on buyers. Financial institutions may require lengthy security and procurement reviews, including for software that never sends an order or transfers cash. Stopping at a recommendation does not, by itself, make those reviews short. With five customers, the months between interest and approval may matter as much as the software. Competitors are already in place. Established reconciliation vendors sell into the same operations groups, with products those buyers already know. Internal bank software is the other path, especially if a firm would rather extend a tool it already runs than add another vendor. LedgerLark’s case is narrower: that small investment firms feel the cost of exceptions and may still find both alternatives heavy. That could be true. It could also be that a known vendor, or the people already on the desk, remains the easier way to close a mismatch. What the founders noticed is easy to understand. A copied account number should not take three days, and a six-person team is a costly way to find it. Whether an agent that compares three sets of records and waits for approval can reliably shorten that search is a separate question. So is whether more firms will finish a security review and stay. Five customers and a seed round make the effort concrete. They do not yet make the outcome obvious.

OpenAI: GPT-6 Luna: Model B is more precise about what the company has and has not demonstrated, while still making the product and founding insight clear. Model A is solid, but its broader generalizations and generic seed-funding explanation are less carefully grounded. (Second judge pass, order swapped — scores are the average of both: Model B is more careful about separating the company’s reported result from what it establishes, while giving competition and procurement real weight. Model A is clear and well str | Anthropic: Claude Fable 5.1: Both stay within the word range, hew closely to…

Numbers shaped into a story

Write a 600–750-word Markdown article from the benchmark results below. Build the story around the clearest finding. Explain the methodology and limitations where readers need them. Avoid reciting the table row by row or treating small differences as decisive. Benchmark: * Researchers tested five language models on 200 anonymized customer-support tickets. * Each model produced one proposed answer per ticket. * Three experienced support managers graded the answers blindly. * Scores combined factual correctness, usefulness and tone. * Model North: 84.2 average score; $1.90 estimated cost per 1,000 tickets. * Model Vale: 83.8; $0.74 per 1,000 tickets. * Model Ember: 79.1; $0.31 per 1,000 tickets. * Model Kite: 76.4; $0.22 per 1,000 tickets. * Model Harbor: 71.0; $0.08 per 1,000 tickets. * North and Vale’s score difference was not statistically significant. * Vale answered 18% faster than North. * North performed best on billing disputes. * Vale performed best on technical troubleshooting. * All five models performed poorly when a ticket depended on an image attachment. * The test used English-language tickets from one software company. * Researchers did not test agentic actions, tool use or multilingual support.

muse-spark-1.3: # The Best Support Bot Wasn't the Smartest — It Was the Best Value If you only looked at quality scores, you'd miss the point of this test. A new benchmark pitted five language models against 200 real-world customer-support tickets from a software company, and at first glance the result looks like a photo finish. Two models finished at the top with average scores around 84 out of 100, well ahead of the rest. But the clearest finding isn't who scored highest. It's that the highest-scoring system costs more than twice as much to run — for no meaningful gain in quality. That's the kind of trade-off support leaders actually have to make. ## How the test worked Researchers gave each model the same 200 anonymized tickets and asked for one proposed answer per ticket. Three experienced support managers then graded those answers blindly, without knowing which model wrote what. Their scores combined three things customers care about: was it factually correct, was it actually useful for resolving the issue, and was the tone appropriate. This is a stronger design than automated scoring alone, because human managers can catch plausible-sounding but unhelpful answers. Blind grading also prevents brand bias. With 200 tickets and three raters per answer, the test is large enough to reveal broad tiers of performance, though not large enough to split hairs between close contenders. And close it was. ## A tie at the top, a rout on price Model North edged Model Vale by less than half a point — a difference the researchers say was not statistically significant. In plain English: on this ticket set, they tied for quality. On cost and speed, they didn't tie at all. North's estimated cost was $1.90 per 1,000 tickets, compared with just $0.74 for Vale. In other words, Vale delivered equivalent quality for less than 40% of the price. It was also 18% faster in responding, a margin that matters when queues spike and customers are waiting. Specialties differed slightly: North looked strongest on billing disputes, while Vale led on technical troubleshooting. For a software company where how-to and bug reports dominate the queue, that second strength is notable — though a different ticket mix could shift the balance. The rest of the field illustrates classic diminishing returns. Ember and Kite landed in the upper-to-mid 70s on quality while cutting costs to a fraction of North's, and the cheapest model, Harbor at just $0.08 per 1,000 tickets, fell to around 71. That's a substantial quality drop from the leaders, but also a tenfold-plus price drop. For simple, low-risk FAQs, some teams might find that trade acceptable — for complex cases, probably not. ## The failure they all shared Every model, good and cheap alike, struggled when a ticket depended on an image attachment — a screenshot of an error, a photo of a setup, and so on. That matters because real support queues are full of "see attached" tickets. The result suggests these models, as tested in text-only answer mode, can't be trusted to interpret visual context. Companies with image-heavy workflows would need to test multimodal setups or keep humans firmly in the loop. ## What this test can't tell you Context is critical. All tickets were in English and came from a single software company. A model that excels at polite troubleshooting explanations for SaaS might stumble on retail returns, medical devices, or highly regulated finance queries — or in another language entirely. The researchers also didn't test agentic behavior: no tool use, no looking up account status, no issuing refunds, no multistep troubleshooting. Modern support deployments often live or die on those capabilities, plus guardrails, escalation logic, and integration with knowledge bases. This benchmark measures draft-answer quality, not autonomous resolution. So don't treat the leaderboard as universal. Treat it as evidence that price-performance should be your starting filter. For most software support teams, the practical takeaway is straightforward: pilot the value leader on your own tickets before paying a premium for a nameplate leader. Test with your mix of billing versus technical issues, measure escalation rates and customer satisfaction — not just manager scores — and specifically probe image-based cases. The smartest model here wasn't the one with the highest average. It was the one that was just as good, faster, and vastly cheaper.
grok-4.7: # The Price Premium Did Not Buy a Better Draft Customer-support teams shopping for a drafting model could misread this benchmark as a ranking. It is more useful as a cost story. Five language models each wrote one proposed answer to 200 anonymized tickets. Three experienced support managers graded those answers blindly on factual correctness, usefulness, and tone. Model North finished highest, at 84.2. Model Vale finished at 83.8. That gap was not statistically significant. Vale cost less than half as much — $0.74 per 1,000 tickets against North's $1.90 — and answered 18 percent faster. The clearest finding is not that North won. It is that the quality premium a buyer might expect from the top of the price list did not appear. ## A narrow test, read narrowly The test was narrow, which is why the tie means something. Every ticket came from one software company and was written in English. Graders did not know which model had produced a draft. Each model had one attempt. Cost was estimated per thousand tickets, a scale closer to a queue than to a demo. Researchers did not test agentic actions, tool use, or multilingual support. Proposing an answer is not the same as looking up an account or handling another language. Speed was compared only between Vale and North. The cost figures are estimates. Three graders can still disagree about tone, and the results do not show how often they did. Inside those bounds, the expensive end of the comparison collapses into one practical choice. North and Vale were close enough that naming a quality winner would overread the data. The differences that remain are price and time. If the job is one-shot English drafting of the kind tested here, Vale is the rational default unless a later test on the team's own tickets says otherwise. ## Where extra spend stops paying The cheaper models show that the test could still separate quality when the gap was larger. Ember and Kite sat in the middle on both cost and score. Harbor, at $0.08 per 1,000 tickets, was more than an order of magnitude cheaper than North. Average scores across that lower group fell from just under 80 to 71. Those steps are several times the North–Vale difference. Only that top-pair gap was reported as statistically tested, so the lower scores are a slope, not a leaderboard to recite. Extra spend buys a visible step up from the cheapest models, then mostly stops buying quality. A team willing to rewrite first drafts can shop that lower stretch and keep the savings. A team that wants drafts close to sendable is already choosing inside a two-model cluster, where Vale's bill and its speed are what differ. ## Two patterns the average hides Two patterns under the averages should change how a buyer uses the mean. The models were not best at the same work. North performed best on billing disputes; Vale performed best on technical troubleshooting. The study does not say how large those edges were, so they are a reason to retest rather than to discard the tie. A billing-heavy queue might still look at North on the tickets that create the most rework. A troubleshooting-heavy queue strengthens Vale's case on fit as well as cost. An overall mean is the wrong tie-breaker when the queue is lopsided. The other pattern is a shared failure. When a ticket depended on an image attachment, all five models performed poorly. That is a limit, not a ranking. Software support runs on screenshots — error dialogs, invoices, interface states — and a drafter that cannot use the attachment forces the same human workaround no matter which invoice is smaller. Image-dependent tickets should go to a person, or to a system built to read images, rather than to whichever model topped a text-only average. ## What the scores do not license None of this crowns a general support agent. It identifies a cheaper drafter on one English-language corpus, graded once, by three managers, with no tools and no follow-up. That is enough to reject paying North's rate for a difference the graders could not reliably see. It should not be exported to other languages, other products, or workflows in which the model must act. Teams that start with Vale should still sample their own billing and troubleshooting mail, and they should keep image tickets off the automated path.

OpenAI: GPT-6 Luna: Model B is more careful about what the benchmark establishes, while Model A overstates the nonsignificant result as equivalent quality and assumes the image cases were handled in text-only mode. Both deliver coherent, well-structured articles that appear to meet the requested length and format. (Second judge pass, order swapped — scores are the average of both: Model B is more careful about what the benchmark supports, especially the limits of the nonsignificant top-score difference and the imag | Anthropic: Claude Fable 5.1: Both outputs hit the length target, center the…

Matchup powered by OpenRouter.