Fermisense says a $500 Qwen fine-tune beat frontier models on catalog review

Why it matters

Fermisense's test shows how workflow data and a scored simulator can turn a small open model into cheaper, task-specific infrastructure, reducing dependence on frontier APIs at high volume.

Justinas Zaliaduonis (@JZaliaduonis) and four collaborators at Fermisense trained a 9-billion-parameter Qwen model for roughly $500 and said it outperformed five frontier-model configurations on a simulated e-commerce catalog-review workflow.

Fermisense reported the result in a July 27th technical write-up authored by Zaliaduonis, Joris Zilinskis, Fabian Hildesheim, Joel Hainzl, and Gediminas Pazera. Zaliaduonis has conducted machine-learning research at Stanford, while Hildesheim recently completed research at Stanford's Institute for Human-Centered Artificial Intelligence. Fermisense's website says its personnel have also worked or studied at MIT, Lyft, Skyscanner and the University of Waterloo.

The result supports a narrower and commercially consequential claim than the usual benchmark contest: a small model trained against a company's own rules can beat a larger general-purpose model on a repetitive, measurable workflow. The comparison remains a Fermisense-designed experiment conducted with Fermisense's data construction, scoring system and cost assumptions. It has not been independently reproduced.

A simulated catalog with asymmetric penalties

Fermisense constructed a digital twin of a marketplace catalog using the Amazon Berkeley Objects dataset, which contains real product images and listing information. Fermisense converted that material into 177,767 review episodes and added controlled policy cases, image mismatches, conflicting brand claims and legitimate claims designed to test false positives.

For each listing, the model had to search a taxonomy of roughly 13,000 product categories, check a claimed brand, retrieve the required attributes and commit a structured decision. Fermisense's scorer penalized missed violations seven times as heavily as false alarms, encoding the economic assumption that allowing a prohibited listing carries a greater cost than sending a valid listing for review.

Fermisense evaluated the models on 200 stratified validation episodes. The frontier group comprised GPT-5.5, GPT-5.6-sol, Gemini 3.1 Pro, Claude Opus 4.8 and Claude Fable 5, according to the write-up. Each received the same images, tools, scorer and turn budget. Fermisense tested both plain prompts and optimized instructions containing about 2,800 characters of procedures and examples.

The strongest frontier configuration reached 76.9% of the maximum achievable score under Fermisense's rubric. The trained 9B model reached 87.3%, up from 64.2% for its untrained base. The percentage is a normalized share of Fermisense's scoring ceiling rather than a conventional accuracy rate, and the benchmark's size makes performance on a broader distribution of marketplace listings an open question.

The public merged model identifies the release as a 9B-parameter Qwen3.5 model in BF16 format. Fermisense also published the training adapter. Both Hugging Face pages have empty model-card fields, leaving the Fermisense article as the main account of dataset construction, evaluation and training.

Two GPUs and three and a half days

Fermisense said it rented two Nvidia RTX PRO 6000 GPUs, using one to generate reinforcement-learning rollouts and the other to update the model. Training ran through Prime Intellect's open-source prime-rl framework for 1,000 optimizer steps over roughly three and a half days.

The model crossed the range achieved by the frontier systems after about 250 steps, or approximately one day, according to Fermisense. The full run cost about $500 in GPU time. Fermisense used group relative policy optimization, or GRPO, to reward actions that produced correct categories, attributes and policy decisions while penalizing errors and unnecessary tool calls.

The method moved knowledge about the simulated marketplace's taxonomy and decision rules into the model's weights. A frontier model had to reconstruct those rules from the prompt during every episode. Fermisense said the longer instructions increased frontier-model input costs by 28% to 55%, depending on the model.

That distinction drove the cost comparison. Fermisense estimated that the specialist could review 1,000 listings for about $0.50, compared with $19 for the cheapest frontier configuration and as much as $172 for the most expensive. The best-scoring frontier configuration cost about $34 per 1,000 listings under Fermisense's assumptions, producing a claimed 68-fold difference against the trained model.

Those figures are modeled inference costs rather than bills from a live Fermisense customer deployment. Fermisense annualized the comparison using a scenario of 40 million decisions per day, estimating roughly $7 million for the specialist and $500 million for the stronger frontier configuration. The volume was derived from Shopify's published account of operating fine-tuned catalog models at scale, rather than Fermisense's own traffic.

The benchmark is also Fermisense's product argument

Fermisense sells on-premise infrastructure that indexes company emails, documents, spreadsheets and other internal material so AI systems can retrieve institutional knowledge. Fermisense says customers retain control of their data and can connect either local models or frontier APIs to the index.

The catalog experiment extends that pitch from retrieval into model ownership. Fermisense argues that companies should begin with frontier APIs to establish a baseline and collect traces, corrections and scored outcomes. Once a workflow reaches sufficient volume, those records can train a smaller specialist that handles the repeatable portion while a frontier model remains available for tasks requiring broader capabilities.

The approach fits workflows with high decision volume, stable tools and outcomes that can be checked automatically. Catalog classification meets those conditions because each decision can be scored against a taxonomy, attribute schema and policy rubric. Open-ended research, strategy and other work without an agreed answer provide a much weaker training signal.

Fermisense's result does not establish that a 9B model is generally more capable than the frontier systems it tested. It demonstrates that model scale can matter less once a narrow task has a simulator, proprietary rules and a reward function that reflects the operator's costs. The valuable asset in that setup is the complete training loop: workflow data, tools, evaluation harness and weights that the operator can run without paying a frontier provider for every decision.