A small model got 85% better at extraction. Microsoft's test is useful because the frontier still won
With 17 training documents, a Python grader and 100 reinforcement steps, Microsoft lifted gpt-oss-20b's extraction F1 from 0.44 to 0.81. Frontier models scored 0.98. How to read the trade.
Listen to this article · 5 min
AI-generated narration of the full article.
The headline number in Microsoft’s October 16 experiment is an 85% improvement. The more useful number is the one Microsoft put right next to it: after training, the small model still made several times more extraction errors than the frontier models it was meant to replace. A vendor post that shows where its own approach falls short is worth reading closely.
The experiment
The task is structured extraction from documents such as invoices: find the right values, map them to the fields of a JSON Schema, keep every repeated record (one held-out invoice had 37 line items) and return valid JSON the next system can use. The grader compares normalized field and value pairs with the expected answer, and F1 combines precision (how many extracted pairs are right) with recall (how many expected pairs were found).
At baseline, the two open-weight models struggled in different ways. gpt-oss-20b declined the task or used apology wording in 25% of responses, returned parseable JSON 63.8% of the time and covered 47.1% of expected fields. gpt-oss-120b produced valid JSON 96.3% of the time but still missed fields: parseable output is not complete output.
The team then used Foundry Interactive Training, in public preview, with 17 training documents, their expected answers and a Python grader. The model generated eight candidate answers per document, the grader gave partial credit for correct values and penalized invalid JSON, and 100 steps of group relative policy optimization (GRPO) reinforced the better answers. The expected answers were used only for grading, never shown to the model.
| F1 before | F1 after training | Price per 1M tokens (input / output) | |
|---|---|---|---|
| gpt-oss-20b | 0.4379 | 0.8115 | $0.07 / $0.30 (fine-tuned, regional) |
| gpt-oss-120b | 0.6816 | 0.8573 | $0.15 / $0.60 (base rate) |
| GPT-6-sol | 0.9764 | Reference, not trained | $2.00 / $10.00 |
| GPT-5.6-sol | 0.9766 | Reference, not trained | $4.00 / $20.00 |
| GPT-5.5 | 0.9760 | Reference, not trained | $5.00 / $30.00 |
What improved
The 20B model went from 0.4379 to 0.8115 F1, the 85.3% relative gain in the title. The 120B model reached the higher score, 0.8573. A second training run with a different seed landed at 0.8101, close to the first. The behavior changed as much as the score: refusal wording fell to zero, valid JSON rose to 98.8% and coverage to 94.9%.
Two cautions from the post itself. The evaluation used 20 documents, which is enough to see a direction and not enough to set an error budget. And the frontier models were measured with a different protocol, so the comparison is indicative.
How to read the trade
At input prices of $0.07 against $2.00 per million tokens, the tuned 20B model is roughly 29 times cheaper per token than the cheapest frontier reference. At F1 of 0.81 against 0.98, it gets roughly one field-value pair in five wrong or missing, against about one in forty. Whether that trade is good depends entirely on what an error costs after the model: a reviewer’s minutes, a rejected payment, a customer letter with the wrong amount.
That is why “cheaper model” is the wrong frame. A team considering this route should compare cost per correct document, including the human or automated checks that catch the extra errors, not price per token. Microsoft makes the same point: its price table is “not a complete total-cost-of-ownership analysis.”
There are also middle paths. The tuned model’s near-perfect JSON validity makes it a predictable first pass. A pipeline can accept fields that pass deterministic validation (totals that add up, dates that parse, IDs that exist in the master data) and send only failing documents to a frontier model or a person, the routing logic we described in managing models as a portfolio.
The reusable asset is the grader
The most transferable part of the recipe is not the model. It is the grader: a piece of code that states what a correct extraction is for your schema and scores partial credit. It drives training, it measures the baseline, and it keeps measuring after deployment. Microsoft’s own advice is to measure first and try simpler changes, such as prompting, before training, then hold back an untouched test set with new sources and document layouts for the final check.
For regulated extraction, such as claims, invoices subject to audit or clinical records, the grader also becomes evidence: a documented, versioned definition of “correct” that a reviewer can inspect. A team that writes it has something valuable whether it ends up tuning a small model or paying for a large one.