Own your intelligence12 minUsage limits shift without warning. Models get quietly dumber some days. Agents refuse normal marketing work. The fix one small software team is betting on: use GPT-6 and Opus 5.5 to teach an open-weight Qwen3.8 model your exact jobs, then let a referee and a private benchmark decide when it takes over.Read now →

A Pocket AI Assistant Built on a Tiny Raspberry Pi Does Not Run the AI Itself. That Is Why It Works
A Raspberry Pi Zero 2 W with a tiny screen, a battery and a 3D-printed case can act as a private voice assistant that follows you around town, because the heavy models stay on a home server and Tailscale links the two. Here is the architecture, the apps, the full build and the one gotcha that stops it powering on.
Most AI gadgets fall into one of two camps. The first is the toy: it really does run AI on the device, but the model is so small, often well under a billion parameters, that its answers are barely useful. The second is the API wrapper: a nice enclosure and a few lines of code that send everything to a cloud service and read back the reply.
There is a third design, and it's quietly the most interesting. Put a tiny, cheap device in your pocket. Keep the heavy AI on a machine at home. Connect the two with a private network. The result feels like a private pocket assistant that is genuinely capable, and none of your voice goes to a company's servers.
This is a guided tour of one such build: what it does, how it is wired together, what you need to build your own and the single hardware problem that catches nearly everyone.
The idea in one paragraph
A small battery-powered device with a screen, microphone and speaker listens for a wake word, such as "Hey, Frank." When it hears it, the device records your request and sends it over an encrypted private network to a homelab server. That server runs the speech recognition, the large language model and the text-to-speech, then sends the answer back. Because the link works through a phone hotspot, the assistant works at home, at the office or on a walk with the dogs.
The device itself is mostly a microphone, speaker, display and radio. The intelligence lives elsewhere.
The architecture
Here is each piece and the job it does.
| Layer | What it does | Where it runs |
|---|---|---|
| Wake word | Hears "Hey, Frank" and nothing else | On the device, using microWakeWord |
| Speech recognition (ASR) | Turns your voice into text | Home server, using SenseVoice |
| Language model | Understands and answers | Home server: a Qwen model around 27B parameters plus several Gemma 4 models, per the build |
| Text to speech (TTS) | Speaks the reply | Home server, using Piper served through Speaches |
| Private network | Links device and server from anywhere | Tailscale, over a phone hotspot or Wi-Fi |
| Notifications | Pushes memos and transcripts to your phone | ntfy, self-hosted |
| Memory | Stores notes you can query later | A "second brain" vault exposed through an MCP server |
Why wake-word detection stays on the device
The device is always listening for one phrase. Doing that locally means audio only leaves the unit after the wake word fires, which is better for privacy and for battery life. microWakeWord is a small on-device detector used in the ESPHome ecosystem, and the build lets you swap wake words by training custom models, which is how it was named "Frank."
Why the heavy lifting is at home
A Raspberry Pi Zero 2 W has only about half a gigabyte of memory. It couldn't run a decent language model if you begged it. But a home server with a GPU can run models in the tens of billions of parameters. The pocket device just needs to talk to it. That's the whole trick, and it's the reason the assistant isn't a toy and isn't an API wrapper either.
Why Tailscale is the glue
Tailscale builds a private network, called a tailnet, between your own devices. Once the pocket unit and the home server are both on it, they can reach each other as though on the same Wi-Fi, from anywhere, with encrypted traffic and no ports opened on your home router. The device's Wi-Fi setup uses the phone's hotspot when you are away, then Tailscale takes over.
What the device actually does
The builder shows several apps that are described as things he really uses, not demos.
1. Pomodoro focus timer
A focus timer for intentional work blocks and breaks. The assistant guides you through a session, announcing when to work and when to rest. It works with basic offline functionality, so it's useful even without the server.
2. Voice memos that reach your phone and your second brain
Tap once, say your idea ("next video idea: training custom microWakeWord models"), and two things happen:
- A notification arrives on your phone through ntfy, a self-hosted push-notification service that works by calling a simple REST API on your homelab. It's the builder's way of avoiding a chat-app dependency such as Telegram.
- The memo is also uploaded to a memory vault reachable by an MCP server. That is the kind of local, file-based Markdown knowledge base, such as Basic Memory, that an AI assistant can search, read and write through the Model Context Protocol, which means your memos become something your other AI tools can use later.
3. Meeting transcription with speaker identification
A voice memo is fine for a thought. A meeting needs more. This app:
- Records a long, multi-speaker conversation.
- Chunks the recording and transcribes it.
- Sends the transcript to a language model for formatting and summarization.
- Pulls out action items.
- Emails everything as a tidy, styled PDF.
In the demo, a sample meeting of about 25 minutes produced a title, bullet-point summary, action items and a full transcript. It also identified who said what, with each speaker's speaking time, by using the self-introductions at the start of the meeting. The builder noted there's a bit of cleanup you can do afterward, which is honest and realistic.
4. Translator
Out walking, ask: "Excuse me, where is the nearest train station?" and get the sentence translated and spoken. It's a small example of why a fast link to a bigger model matters on the move.
5. Ask the LLM anything
"Hey Frank, can dogs eat acorns?" is the demo question. In practice, this is a hands-free interface to your own models.
All of this works because the same private network follows you. At the park, connected to the phone's hotspot and Tailscale, the device reaches the full stack at home.
What you need to build one
The enclosure is a community-designed 3D-printed case. The electronics are off-the-shelf.
Parts
- Raspberry Pi Zero 2 W with headers. Compact, Wi-Fi equipped, with about 512 MB of RAM.
- PiSugar 3, a battery board that attaches under the Pi.
- PiSugar Whisplay HAT, an add-on board with a small screen (a 1.69-inch LCD in PiSugar's product description), two microphones, a speaker, a button and RGB LEDs.
- A microSD card for the operating system.
- A 3D-printed enclosure, plus about eight small M2 screws of 6 mm (the builder said 4 mm might also work).
At home
- A server that can run your speech and language models, such as a desktop or mini PC with a GPU.
- Tailscale on the server, the pocket device and your phone.
- A phone with hotspot capability.
The build, step by step
Step 1: Flash Raspberry Pi OS Lite
Use the Lite edition, which has no desktop environment. With only half a gigabyte of RAM, you want the lightest system possible, and the device never needs a desktop because everything is done on its own screen.
Step 2: Attach the PiSugar 3 battery board
Peel off the little orange protective plastics first (the builder says he always forgets). Line the pins up with the Pi's header and screw the board on. The standoffs are threaded, so they can hold without nuts, which pushes the contacts together.
Step 3: Fix the power problem
This is the part to read twice. After assembling, the unit didn't power on. The same thing had happened with the builder's first unit, and it's a known quirk of the PiSugar 3: the spring-loaded pogo pins don't always touch the Pi's underside well because of a thin oxidation layer on the pads. PiSugar's own FAQ tells you to scrape the pads lightly to expose clean solder.
What he tried:
- 220-grit sandpaper: not enough.
- A wire brush attachment from a rotary tool: third time was the charm and the blue light came on.
His advice: don't be shy about it, even on a brand-new board. Go gently, avoid damaging nearby components and power the Pi off while you do it.
Step 4: Mount the Whisplay screen board
Press it on gently and evenly using the board edges, not the screen, until it clicks down flush. Power on again and confirm the screen lights.
Step 5: Case it
The case slides together, with two side panels that pop on and a small button insert that lines up with its hole. Securing one side first makes the rest easier. Plan on some finagling.
Step 6: Flash the assistant software
With the unit on Wi-Fi and plugged into power (so the battery doesn't run out mid-install), flash the assistant software over the air from the project's GitHub repository. It installs packages and dependencies. Then disconnect power and slot the unit back into the case.
Step 7: Join your tailnet
Install Tailscale on the device and add it to your tailnet, and the remote functionality is ready. For Wi-Fi away from home, hold the PiSugar button to turn on access point mode, connect your phone or computer to the "PiSugar setup" network, open the browser page and choose your hotspot.
An easier starting point if you want to experiment
PiSugar publishes an open-source repository called whisplay-ai-chatbot for exactly this hardware: a Raspberry Pi Zero 2 W, the Whisplay HAT and the PiSugar 3 battery. According to its documentation, you install the Whisplay drivers, configure your language model in an environment file, build and launch the chatbot with a script. It's a good way to prove the hardware works before wiring in your own speech and language stack.
Sizing the home server
You don't need to copy the exact model list. What matters is the shape of the stack, and it helps to think in terms of three jobs with different appetites:
- Speech recognition is the lightest job. Many speech models run happily on a modest GPU or even a CPU, and it's the part where latency matters most, since you're waiting for it before anything else can happen.
- The language model is the hungry one. A model in the roughly 27-billion-parameter class needs a graphics card with plenty of video memory, or heavy quantization to squeeze it onto a smaller one. If you only have a small GPU, run a smaller model; the pocket device doesn't care which model answers.
- Text to speech is light. Piper-style voices are designed to be fast on ordinary hardware.
Running multiple models side by side, the build uses several Gemma 4 models alongside the larger Qwen model, is a way to match the job to the model: a small fast one for simple commands like "turn on the guest nightstand," a big one for open questions. The spoken command in the demo, "Hey, Frank, turn on the guest nightstand," suggests the assistant is also wired to smart-home controls, which is exactly the kind of quick request a small model can handle in a second.
A privacy checklist before you carry it around
Keeping AI local is a privacy win only if the rest of the system matches. Before you rely on it:
- Know what is stored. Voice memos, transcripts and PDFs are files on your server and possibly in your email. Decide how long you keep them.
- Lock down the tailnet. Only your own devices should be on it, and Tailscale's access controls can limit what the pocket device may reach.
- Limit what the assistant can touch. Anything exposed through an MCP server, such as your notes vault, is something a model can read and write. Expose the minimum.
- Protect the notification channel. A self-hosted ntfy topic is only as private as its access settings. Use authentication, not just an obscure topic name.
- Charge and update. A battery device that sits in a drawer for months also sits unpatched.
Why this design is smart
- Privacy. Your voice goes to your own server over your own encrypted network, not a vendor's cloud. (Whether that matters to you depends on what you ask it.)
- Capability. You aren't limited to tiny on-device models.
- Cost. The pocket unit is cheap, and the home server may already exist for other projects.
- Upgradability. Swap in a newer model on the server and the pocket device gets smarter without any change.
- Resilience. Core features such as the timer still work offline.
The trade-offs and limits
- It needs the home server to be on. If the server is down, or your home internet drops, the smart parts stop working. That's also why you need a plan for power cuts.
- Hotspot data and latency. Every query travels over a mobile connection. Expect delay in poor coverage, and keep an eye on your data plan.
- Security is on you. A Tailscale network is private, but you're responsible for the server's updates, who's on your tailnet and what the assistant can access through MCP.
- Battery life is limited. A small battery and an always-listening microphone won't last all day; the builder charged it by leaving it on power during setup.
- Fiddly hardware. The pogo-pin issue is typical of hobby builds.
- Speaker identification has limits. Naming speakers from their self-introductions is clever, but it only works if people introduce themselves clearly.
- Recording people has legal implications. Meeting transcription is useful, but recording laws vary. Get consent first.
- Local doesn't mean flawless. A 27B-class local model is capable, but it can still be wrong, so don't rely on it for anything medical, legal or safety-critical.
How to start small
- Get a home server working with a speech-to-text model, a language model and a text-to-speech voice. Any machine with a decent GPU will do.
- Install Tailscale on it and on your phone. Confirm you can reach the server from outside your home.
- Try the PiSugar chatbot repository on a Pi Zero 2 W to learn the hardware.
- Add one app at a time. Voice memos are the easiest win.
- Train a custom wake word once everything else works.
FAQ
Does the Raspberry Pi run the AI model? No. It runs the wake-word detector and handles audio, display and networking. The speech recognition, language model and text to speech run on a home server.
Does this work away from home? Yes, via a phone hotspot and Tailscale, which connects the device to the home server privately.
Is it fully private? Audio stays within your own network after the wake word triggers. But privacy depends on how you secure your server and any services you connect.
Why won't my PiSugar 3 power on? A common cause is poor contact between the PiSugar's pogo pins and the Pi's pads. PiSugar's FAQ recommends lightly scraping the pad surface to remove oxidation.
Can I use a different wake word? Yes. The build lets you train and swap custom wake-word models.
The bottom line
The clever part of this gadget isn't any single component. It's the division of labor: a cheap, low-power device handles the ears, mouth and screen, while the real intelligence sits on hardware you control. That approach avoids both the toy and the cloud wrapper. Add a private network and an ordinary phone hotspot, and a home server becomes an assistant you can carry anywhere. The most important tool in the build might just be a piece of sandpaper.
Sources
- PiSugar: whisplay-ai-chatbot (GitHub)
- PiSugar: Whisplay HAT for Raspberry Pi Zero 2 W
- PiSugar Docs: PiSugar 3 Series
- ESPHome: Micro Wake Word
- Basic Memory documentation: Quickstart (local)
- ntfy: send push notifications to your phone via PUT/POST
- BGR: You can build a personal AI chatbot using Raspberry Pi





