Own your intelligence12 minUsage limits shift without warning. Models get quietly dumber some days. Agents refuse normal marketing work. The fix one small software team is betting on: use GPT-6 and Opus 5.5 to teach an open-weight Qwen3.8 model your exact jobs, then let a referee and a private benchmark decide when it takes over.Read now →

Before You Spend Thousands on a Local AI Box, Learn the Two Numbers That Decide Everything
A hands-on comparison of a Mac mini, a DGX Spark-class box, an RTX 5090 and a 175-watt mini PC shows that memory size decides what you can run and memory bandwidth decides whether it's usable. Here's how to choose, what's overpriced right now, and the cheap path to AI you actually control.
You can run AI locally on almost anything. That's the good news, and it's also how people end up spending $6,000 on a box that types slower than they read.
A long-time local AI tester with a room full of hardware (an RTX 5090, a pair of RTX 5060 Ti cards, a Mac mini, mini PCs, a DGX Spark-class workstation and a 24/7 AI server) set out to clear up what he calls a big misunderstanding about running AI at home.
The short version: two numbers decide almost everything, and the price tag isn't one of them.
First, understand what "local AI" actually means
When people say they want to run AI locally, they usually mean the model. But there are really two pieces, and both need to run on your machine for true independence.
- The model is the engine. On its own, it's essentially a very sophisticated calculator for the next most probable token.
- The harness is the car. It's the software around the model that manages memory, loops, tool use and the context window. Coding agents and personal assistants are harnesses.
His point is that both are part of the intelligence. A harness without a model can't think. A model without a harness can't do sustained work.
The practical upshot:
- You can keep the same harness and swap models for different jobs.
- The harness's ceiling is set by the model. Plug in a stronger model and the same harness can suddenly do more.
Hold on to that split. It matters a lot at the end of this article, when we get to the cheapest way in.
Why bother owning it at all?
For this tester, the motivation is sovereignty. AI has become part of how he works. He feels augmented by it: able to do more, better and faster. So he doesn't want to depend on any company, government or other entity's permission to keep using it.
There's a business version of the argument, too. If everything you do runs on someone else's model and infrastructure, then the moment they change the price or cut your access, he argues, "you don't have a business, you have a liability."
And there's privacy. You can't have full privacy with a cloud model.
The catch is cost. Hardware capable of running serious models is expensive and, right now, often overpriced because of hype and supply dynamics. Hence the need to spend wisely.
What you actually want a local AI to do
He boils it down to two jobs:
- A personal assistant. Research, summarizing documents, organizing files, daily routine tasks. Current local models can do this well.
- A systems engineer for your own AI stack. Something that can manage, maintain and fix your local setup, and write small pieces of software.
The second is much harder. Models still overcomplicate things, and when they're touching your infrastructure, they can break it. He believes we're close to an AI that's trustworthy enough for this, possibly within months, though that's his forecast, not a certainty.
He's clear that even frontier cloud models can't be trusted 100% on this kind of work. They hallucinate and ignore instructions. The goal isn't perfection. It's a system you can trust enough to work alongside.
The two numbers: memory size and memory speed
Here's the heart of it. Every machine you're considering has:
- Memory capacity, which decides how big a model you can load.
- Memory bandwidth, which decides how many tokens per second the model generates, and so whether it's pleasant to use.
As a rule of thumb, a token is roughly three-quarters of a word in English.
How fast is fast enough?
- ~10 tokens/second: slow but workable for simple tasks like transcription, writing an email or summarizing a document.
- 10–20 tokens/second: starting to become useful.
- 30–50+ tokens/second: the target for real agentic work and coding, where you may need to generate millions of tokens.
Now let's look at real machines.
The Mac mini: great harness host, weak model host
The Mac mini is an incredible little computer, and the Pro version is considerably better. You can configure a good amount of unified memory, which is a real advantage.
But for this tester, it's mainly good for running harnesses, not local models. The capacity may be there; the speed often isn't, at least not at the price. A model crawling along at single-digit tokens per second can't do heavy software work.
Ordinary mini PCs are cheaper and similar: fine for harnesses, poor for models. More powerful mini PCs can run local models, but typically in the 10–20 tokens/second range.
The DGX Spark-class box: huge memory, modest speed
Next up is the ASUS Ascent GX10, an equivalent of NVIDIA's DGX Spark. Both use NVIDIA's GB10 Grace Blackwell chip with 128 GB of unified memory.
The catch is bandwidth: 273 GB per second. That's plenty of room for big models, but not much speed.
On this machine, he runs a quantized Qwen3.8-Flash-Next at around 35 tokens/second, which he calls definitely usable and the best model he can run locally right now.
Meet Qwen3.8-Flash-Next
This model explains a lot about today's local AI scene:
- It's an open-weight 125-billion-parameter mixture-of-experts model with only about 6 billion parameters active per token, according to its model documentation and community write-ups.
- It supports a 256K-token native context.
- Because only a few experts fire per token, it decodes quickly. But all the weights still need to live somewhere fast to read. The popular 4-bit Unsloth build is about 111 GB, and even the smallest 1-bit build is about 66 GB.
- Its license is a Qwen community license, not Apache 2.0, so read it before commercial use.
Third-party benchmarks vary widely by setup. One community write-up lists about 24 tokens/second on a DGX Spark with a 4-bit build, so his 35 is plausible with tuning but not guaranteed.
The price problem
These boxes can be clustered for more memory, and the technology is genuinely impressive: a tiny supercomputer running the same NVIDIA software stack used in data centers. He calls it one of the best machines for learning.
But the pricing has gone wild. He paid about €3,200 for his 1 TB unit. He estimates it's now around €6,000. In the US, NVIDIA's marketplace listed the 4 TB DGX Spark at $6,950 and the 4 TB GX10 at $7,999 as of early November, both out of stock, according to a dated price tracker.
The RTX 5090: blistering speed, painful price
GPUs are the fastest option, because they offer the highest memory bandwidth.
- The RTX 5090 delivers about 1,792 GB/s of bandwidth, roughly six and a half times the GB10's 273 GB/s.
- But it has only 32 GB of VRAM.
- And it's around $8,000 right now, if you can find one, in his estimate. He was lucky to pay under $3,000. At that price, it makes sense. At $8,000, it doesn't.
How a 24–32 GB card can run a 125B model
This is where tools like Strata come in. Strata splits a model across three tiers:
- VRAM for always-needed weights, caches and the most frequently used experts.
- System RAM for the rest of the experts.
- SSD for the big n-gram lookup table this model uses.
That makes it possible to run a large, fairly intelligent model locally at speeds that would be impossible if it had to fit entirely on the GPU. Community testing backs up the idea: one benchmark with a 24 GB VRAM budget and 96 GB of DDR5 still decoded Qwen3.8-Flash-Next at about 39 tokens/second.
The surprise winner: a 175-watt mini server
The machine he seems to like best is the Olares One, a compact box built to run headless 24/7 as a personal AI server.
Its specs, per Olares:
- NVIDIA RTX 5090 Mobile with 24 GB of VRAM, about half the bandwidth of the desktop 5090.
- 96 GB of DDR5 RAM.
- An Intel Core Ultra 9 275HX and a 2 TB NVMe SSD.
- A 330 W power adapter; he measured the whole system running at around 175 W, versus about 600 W for his desktop 5090 card alone.
Running the same Qwen3.8-Flash-Next with Strata, he gets about 54 tokens/second, much faster than the 128 GB GB10 box. The official price is $4,999.
That's the key lesson of the whole comparison: a machine with less total memory but faster memory beat the one with more memory but slower memory. And it did it while sipping power, which matters for something meant to run all day.
So should you spend that much?
He's honest here. Even the best local model is not close to the frontier models. Qwen3.8-Flash-Next is the best he can run locally, but it's not in the same league as the top cloud systems. It's hard to justify thousands of dollars just for that.
Before saying no, though, he walks through how to shop smart.
What to look for
- Check capacity and bandwidth on every option. Those two numbers, not marketing labels.
- The sweet spot right now is 24 GB of VRAM, with the fastest memory you can afford.
- Budget for the rest of the system. A GPU alone needs at least about $1,000 more in computer around it. To run a Strata-style setup at the speeds above, plan for around 96 GB of system RAM.
- Consider the used market. Unless you buy something like a used RTX 3090, getting this level of speed for under $4,000 is hard.
- Mac Studio is beautiful and capable, but extremely expensive.
The RTX 5080 Super question
He mentioned rumors of an RTX 5080 Super with 24 GB arriving, most likely in the first quarter of 2027, and predicted it might be announced around $2,700 but sell for over $4,000 in reality.
Treat that with extra caution. Just this week, VideoCardz reported that NVIDIA had reportedly told board partners a planned 24 GB RTX 5080 will not launch in January 2027 after all, citing the cost of 3 GB GDDR7 memory chips, which are said to cost about three times as much as 2 GB chips. The same report said memory pricing has also been linked to delays for the RTX 5080 Super. NVIDIA hasn't confirmed any of it. Don't plan a purchase around this card.
The operating system matters more than you'd think
When you buy a computer, you're not only buying an AI machine. You're buying a computer you'll use for other things.
His experience across macOS, Linux and Windows:
- If the machine is just for AI, use Linux. It's the most flexible and lets you do anything.
- If you also game, make music or edit video, think harder. Some software and hardware only work on Windows or macOS.
Laptops
- Apple's MacBooks are incredible, especially for battery life. Nothing else comes close, in his view.
- But a $7,000 to $10,000 laptop doesn't make sense to him. He'd rather have a MacBook Air plus a home server like the Olares or GX10 that runs the model 24/7, while the laptop runs only the harness.
- If you're nomadic and have nowhere stable to leave hardware, a laptop is the only option, and a MacBook Pro is the best choice.
- NVIDIA's new RTX Spark laptops look promising, but he hasn't tested them and is wary of Windows. If Linux installs well on them, they could be good.
Start with the problem, not the hardware
His decision framework is three questions, in order:
- What problem do I need to solve?
- What model can solve it?
- What hardware runs that model?
If no local model can solve your problem, you have your answer: you don't need local AI. If the problem can be broken into parts, and some parts must run locally for privacy or control, then size the model and hardware for those parts only.
The case for going hybrid, and the case against
Plenty of people say local models aren't good enough. He disagrees. He's built software with them and says they're good enough. But they demand more of you.
- You can't throw one giant prompt at a local model and wait for a masterpiece, the way people do with Opus or Fable.
- You can get there by staying in the loop: planning, steering, deciding, supporting the model step by step.
For most people, hybrid is the right answer: a minimum local AI you control and trust, running 24/7, plus cloud models for the jobs that truly need them.
The contrarian twist
Then he makes a harder argument. It's emotionally difficult to keep working with a modest local model when a hyped frontier subscription is one click away. But if you have the psychological strength to go local-only, he believes that's the better path. It forces you to plan better, think more, optimize more and learn how to work with the technology properly. And the more people use local models, the more innovation happens around them.
That's a minority view, and it costs productivity in the short term. But it's a coherent case.
Why your hardware will keep getting smarter
One more point that changes how to think about price.
His first local AI machine, a small mini PC, could barely run anything useful at first. The best it managed was Qwen3.5-35B-A3B, a mixture-of-experts model with only 3 billion active parameters. Since then, Qwen3.6-35B-A3B has arrived, and he hopes a 4.0 version follows.
The result: the same machine that once couldn't handle coding work now runs a model capable of agentic tasks like research and browsing at a decent speed. Nothing about the hardware changed. The models got better.
That, he argues, is exactly why prices for AI-capable hardware aren't falling. Every year the same box does more valuable work.
His prediction: prices won't drop until truly dedicated inference hardware arrives, chips that might generate 500, 1,000 or even 5,000 tokens per second instead of tens. When that happens, today's machines will look obsolete overnight. Until then, probably for the next two to three years, current hardware should keep delivering more value over time. That's a forecast, not a guarantee.
No budget? Start with the harness
Here's the most practical advice in the whole comparison.
A cheap mini PC can't run a serious model. But it can run a harness, and that gives you what he calls half your sovereignty. Arguably the more important half, because the harness holds your memory, workflows and stability. The model can be plugged in from the cloud.
Two low-cost ways to do it:
- Free models through OpenCode. Install the open-source OpenCode coding agent and you'll see a list of free models. Some are quite good, though they may be slower. His caution: free usually means less privacy, and providers may use your data for training. Check each model's terms.
- A cheap paid API. For a few dollars a month, he recommends DeepSeek V4.1 Flash: not a frontier model, but not far off, and very cheap. DeepSeek's own pricing page lists it at $0.30 per million input tokens and $1.20 per million output tokens at peak, and half that off-peak, with a 1M-token context window.
When you can afford a local model later, you swap the engine. The car stays.
Counterpoints and risks
- Speeds are setup-specific. Quantization, runtime, context length and settings swing results dramatically. Treat every tokens-per-second figure, including those here, as a rough guide.
- Local isn't automatically safe. An agent maintaining your own stack can break it. Back up configurations before letting AI touch infrastructure.
- Prices are volatile. Many listings are out of stock, and street prices can sit far above list. Get a delivered quote before buying.
- Licenses matter. Open weights aren't always open-source licensed. Qwen3.8-Flash-Next ships under a community license.
- Benchmarks are often vendor-run. Claims that a local model beats a frontier model usually come from the model maker or quantizer, not independent testing.
FAQ
What matters more for local AI: RAM or VRAM? Both capacity and bandwidth matter. Capacity decides what fits; bandwidth decides speed. With mixture-of-experts models and tools that offload experts to system RAM, a 24 GB GPU plus lots of fast system RAM can beat a box with more but slower memory.
Is a DGX Spark or ASUS GX10 a good buy? They're excellent for learning NVIDIA's stack and fitting big models, but their 273 GB/s bandwidth limits speed, and current prices are high. Compare against GPU-based systems before buying.
What's the best local model right now? In this comparison, Qwen3.8-Flash-Next was the strongest model the tester could run locally, but it is still well behind frontier cloud models.
Can I start without expensive hardware? Yes. Run a harness on a cheap mini PC and connect it to free or low-cost cloud models, then add local hardware later.
Should I wait for the RTX 5080 Super? There's no confirmed launch or price, and recent reports suggest NVIDIA's 24 GB RTX 5080 plans have slipped. Don't plan around it.
The bottom line
Before you buy anything, ask what problem you're solving, which model solves it and what hardware runs that model. Then compare machines on the two numbers that matter: memory capacity for what fits, and memory bandwidth for how fast it runs.
Today's best-value answer for many people is a 24 GB GPU with plenty of fast system RAM, or no new hardware at all: a harness on a cheap box, plugged into a cheap model. Own the car first. You can upgrade the engine anytime.
Sources
- Olares One technical specifications (Olares Docs)
- Qwen3.8-Flash-Next: How to Run Locally (Unsloth Documentation)
- How to Run Qwen3.8-Flash-Next Locally: 39 tok/s on 24GB of VRAM (DEV Community)
- ASUS Ascent GX10 vs DGX Spark: November price update (RunAIHome)
- NVIDIA reportedly cancels GeForce RTX 5080 24GB launch (VideoCardz)
- Models & Pricing (DeepSeek API Docs)
- Qwen3.6-35B-A3B (Hugging Face)





