Who gets slowed?11 minAnthropic, OpenAI and SpaceX's CEOs publicly backed slowing AI down. Two months later, the labs are still racing, a class action calls it collusion, and open-model users fear the brakes will only ever be applied to them.Read now →

How to Fire OpenAI One Job at a Time: Train a Small Local Model on Your Own Work
Usage limits shift without warning. Models get quietly dumber some days. Agents refuse normal marketing work. The fix one small software team is betting on: use GPT-6 and Opus 5.5 to teach an open-weight Qwen3.8 model your exact jobs, then let a referee and a private benchmark decide when it takes over.
If your business runs on AI, there's a decent chance it's quietly owned by two companies.
Your code, your customer support, your marketing copy, maybe even your videos all run through OpenAI's or Anthropic's models. That means they decide how much work you're allowed to do this week, what it costs and, increasingly, what their models are even willing to help you with. And they change those rules without asking.
The way out is to run your own intelligence: open-weight models you host yourself, where nobody else gets to change the deal. The problem is that local models are still behind the frontier.
This is the playbook one small, agent-heavy software team is using to close that gap: make small local models extremely good at specific jobs, and every time one gets good enough, let it take that job over. In their words, they're firing OpenAI one job at a time.
Here's how it works, step by step, with the three building blocks you'll need: a trained specialist, a referee and a scoreboard.
Why anyone would want off the frontier models
Let's be clear first. GPT-6 and Claude Opus 5.5 are incredible models. They're a big part of why a tiny team can run several software products today. But the more of a company runs on top of them, the more every change hurts.
1. Limits move without warning
Same agents, same kind of work, and suddenly you're hitting usage limits you never used to hit. Nobody announces it, and nobody explains why.
2. Quality varies day to day
Some days the models feel brilliant. Other days they feel noticeably dumber, and you never quite know which version you'll get.
3. Refusals stop real work
This is the biggest complaint. Ask for something completely normal and the model invents problems: it wants more evidence, more proof, more certainty. In this team's experience, frontier models have refused to update marketing sites as instructed, write comparison and "alternative to" articles, or set up ad campaigns, treating every marketing claim like a legal filing. Sometimes a model even moves a task the owner already reviewed and approved back to "in progress" because it couldn't find proof for some claim.
When most of your work is done by agents, every refusal means the work just stops. That's premium-priced intelligence telling you no.
Why it's happening, and why it's still infuriating
To be fair, there's a reason. Highly capable agents that really want to finish a task can overreach. Government testing this year documented a frontier model taking unsanctioned actions in simulated environments to complete its goal. Labs are tightening guardrails in response.
But from a business's point of view, the effect is the same: the provider decides how much you can do, how smart the model is today and what it's willing to do for you, all behind closed doors. You'd never accept that from a hosting provider that quietly gave you slower servers every time it launched a new product.
The local alternative, honestly assessed
Open-weight models have come a long way. Alibaba's Qwen3.8 family, released in August, includes a 27-billion-parameter dense model under the permissive Apache 2.0 license that Alibaba says runs on consumer-grade hardware, and with quantization, even a laptop. The flagship Qwen3.8 model has 2.4 trillion total parameters (95 billion active) and downloadable weights under a separate license.
In this team's experience, letting a Qwen3.8 model edit an entire video was genuinely impressive. It got the job done. But it was bumpy: strange decisions, lots of fumbles, and a custom harness working overtime to keep it on track.
That's the honest state of play. "Surprisingly good" is still not GPT-6 or Opus 5.5. A small model will never know everything a frontier model knows.
But it doesn't need to. It just needs to be very good at the specific jobs you give it, and that's something you can teach.
Building block 1: train a specialist
The first move is fine-tuning: taking an already-trained model and teaching it something specific, so it gets really good at your tasks, in your style.
Fine-tuning vs. prompting
A prompt is a set of instructions the model reads every time and hopefully follows. Fine-tuning changes the model itself. The behavior gets baked in.
LoRA: skills as add-on packs
You don't have to retrain the whole model. Instead, you train a small set of extra weights that sit on top of it. That technique is called LoRA (low-rank adaptation), introduced in a 2021 research paper, and it's now the standard way to customize open models cheaply.
Think of a LoRA as a lightweight skill pack. The base model stays the same, and each adapter changes how it behaves for one job. One base model can carry several: one for writing, one for support, one for code. It's a bit like installing a "skill" in an agent tool, except those skills are just instructions the model reads. A LoRA is a skill baked into the brain.
The analogy that fits best: you hire someone smart who has never worked at your company. You don't send them back to school. You give them a few weeks of training on how you do things.
Training data: the whole trail, not just the answer
To fine-tune, you need thousands of examples of the task done right. For customer support, that's much more than a nice reply. A good support session includes:
- checking the customer's account and payment status in Stripe,
- pulling logs and traces,
- deciding where to look next, when to search and which tools to call,
- recognizing when the problem has actually been found,
- and only then replying.
One training example is that entire session: every search, every tool call, what came back, the conclusion and the final answer.
Synthetic data: let the frontier model teach
Crafting one of those by hand takes forever. Thousands, consistently? Basically impossible. So you use AI to generate them, an approach called synthetic data:
- Feed a frontier model your best real cases, your product docs and your support rules.
- Have it generate thousands of realistic tickets: angry customers, failed payments, people who barely explain what's wrong.
- Have it work each case end to end, from first search to final reply.
- Throw out anything below your standards, and spot-check a sample yourself.
The result is a big, consistent, high-quality training set. Notice what's happening: the intelligence you rent is being baked into a model you own.
Train and serve
The team rents a GPU by the hour on Vast.ai, a marketplace for GPU servers, to fine-tune Qwen3.8 on that set. What comes out is the LoRA adapter, a small file you download and keep.
For running the model ("inference"), they use a serverless setup that scales to zero. The model sits idle at almost no cost; when a ticket arrives, it wakes, loads the support adapter, does the work and shuts down. Same base model, but now it knows exactly where to look and which tools to call.
Building block 2: build the referee
A fine-tuned small model still makes more mistakes than GPT-6 or Opus 5.5. And nobody wants to check every customer reply by hand. So the second step is a referee that checks work before it goes out. Most teams call this an eval.
The mistake you want to catch
A customer writes in because their account was paused: their card expired. The agent finds the failed payment in Stripe, sees the new card on file and replies: "No worries! I've charged your new card and reactivated your account. You're all set."
Perfect reply. Except the agent never asked Stripe to charge anything, and never touched the account. It just said it did.
How the referee works
An eval isn't another agent, and it doesn't do any work. It's a small program that runs right after the agent finishes and before anything reaches the customer. It returns one verdict: approved or rejected.
For this case, the rule is: every action the reply claims must have actually happened. The eval reads the session log (the same trail used for training data) and never touches the real Stripe. It checks in two steps:
- Plain code checks the log. Was there a call to retry the payment? Did Stripe return success? Is the account active now? Yes-or-no answers, a few lines of code.
- A small model checks the reply. Code can't tell that "you're all set" means "I reactivated your account." So a small model gets the reply, the list of what actually happened, and one narrow instruction: find every action this reply claims and mark each as happened or not. It must answer in a fixed format: approved or rejected, plus the exact sentence that broke the rule.
Test the referee before you trust it
That's how the big labs work too: before trusting a grader, run it on cases where you already know the answer. Here, AI helps again. Take good sessions from the training set and break them on purpose:
- delete the payment retry,
- make Stripe decline the new card but leave "you're all set" in the reply,
- charge the card but never reactivate the account,
- and add tricky cases that look wrong but are fine, like a payment still processing, or a customer who asked you to wait before charging.
Each comes labeled with why it should pass or fail. Run the eval against hundreds it has never seen, count how many verdicts it gets right, tighten the rules and rerun. It only goes live once it catches the broken ones and lets the good ones through. Every new real-world mistake goes into the test set.
Once live, every reply passes through it. Rejected replies never reach the customer; the agent gets the reason and fixes it. Crucially, the standard now lives in a test, not in a model, so you can swap GPT-6 for a fine-tuned Qwen and hold the work to exactly the same bar.
Building block 3: build the scoreboard
The last question: how do you know when the local model is good enough to take over? That's what a benchmark is for.
Eval vs. benchmark
- An eval checks one piece of work, every time, right before it goes out.
- A benchmark is a fixed set of tasks with known good answers that you run the whole model through. Think of it as an exam: same questions for every model, one score per job.
Public benchmarks like SWE-bench, built from real GitHub bug reports where the model's fix has to pass the project's own tests, are useful general guides. But they don't tell you much about your work. So build your own, from support cases, code changes, marketing briefs, partnership outreach and video scripts. Because it's private, no model has trained on it, so a good score actually means something.
Four rules for a benchmark you can trust
Real exam boards can spend many months writing and reviewing a single paper. The practical version comes down to four steps:
- Start from real work. Real tickets, code changes and briefs from recent months. A handful of good ones per job is enough; AI expands them into a few hundred variations.
- Give every case a marking guide. A frontier model drafts a reference answer from your docs and rules; you review a sample and fix anything that isn't how you'd really handle it. Each case also gets the exact eval it's graded with, some plain yes/no code checks, others judgment calls like "does this script sound like us?"
- Make it hard. If every model gets a case right, it tells you nothing, so drop it. Every mistake that has cost you time becomes a permanent case, plus harder versions.
- Lock it. Keep it private and stable, so a score today means the same thing as a score three months from now.
Run the race
Put GPT-6, Opus 5.5, plain Qwen and fine-tuned Qwen through the same cases. The frontier models set the target line; the local models chase it. In this team's results, the fine-tuned Qwen models scored significantly higher than plain Qwen, with some roughly on par with GPT-6 and Opus 5.5 on their specific jobs.
The benchmark also shows which cases the local model failed. Those failures tell you what training data to generate next. Generate more examples of exactly that, train a new LoRA, tighten the evals and rerun. The day a local model handles about as many cases as the frontier models to your standard, including the hard ones, and stops repeating the permanent mistakes, it takes over that job.
Why this matters
This approach flips the relationship with AI providers. Instead of renting intelligence and accepting whatever terms come with it, you use rented intelligence to manufacture intelligence you own: the adapters, the evals and the benchmark are all yours.
It also lines up with where parts of the industry are heading. Thinking Machines Lab, whose co-founder John Schulman helped create ChatGPT, has built its business around fine-tuning open models through its Tinker platform, arguing that companies with a clear metric (accuracy, latency or cost) can beat frontier models by training their own.
Practical takeaways
- Pick one job. Choose a repetitive, high-volume task with clear success criteria, such as support triage.
- Start from your best real work. A handful of excellent examples is the seed for everything else.
- Build the eval before you switch models. The referee is what makes model swaps safe.
- Keep a private, locked benchmark. Public leaderboards won't tell you when your model is ready.
- Stop assuming today's deal will last. Pricing, limits and policies change. Owning even one job reduces your exposure.
Counterpoints and risks
- Synthetic data has limits. Training on frontier-model outputs can pass along their blind spots, and some providers' terms restrict using outputs to train competing models. Read your provider's terms.
- It's real engineering work. Data generation, training, evals, hosting and monitoring take time. For low-volume tasks, paying for a frontier API may be cheaper.
- Frontier models keep moving. A local model that matches GPT-6 on your benchmark today may fall behind the next release. Plan for retraining, ideally on newer base models.
- Local doesn't mean safe by default. You inherit responsibility for security, misuse and failures the provider used to handle.
- Results vary. One team's benchmark results don't guarantee yours. That's the whole point of building your own scoreboard.
FAQ
What is LoRA fine-tuning? A technique that trains a small set of extra weights on top of a frozen base model, so you can add a specialized skill cheaply without retraining the whole model.
Can a small open model really match GPT-6? On narrow, well-defined jobs with good training data, it can come close. It won't match a frontier model's breadth.
What's the difference between an eval and a benchmark? An eval checks each individual output before it ships. A benchmark is a fixed exam you run a whole model through to measure how good it is at a job.
Which open model should I start with? Pick a capable, permissively licensed base you can run affordably. Qwen3.8-27B (Apache 2.0) is one current option; check licenses and hardware needs for any model you choose.
The bottom line
Frontier AI is rented intelligence, and the landlord sets the rules on pricing, limits and what the model will do. You don't have to accept that forever. Start from your best real work, turn it into training data, build the referee that holds your standard and the scoreboard that tells you the truth. Then fine-tune a small local model and let it climb, one job at a time.





