Bella
A personal AI assistant with long-term memory, tools, and real-time voice. Built solo for my own use, with local storage and retrieval and hosted inference for conversation.
The problem
I wanted an assistant that remembered context and could help with everyday work while I retained control of its records. Bella stores its database, vault, and retrieval indexes on my machine. Claude, or the explicitly configured OpenAI alternative, receives the selected messages, tool results, and context included in each prompt. Local storage does not make that inference private. When a hosted voice is selected, the speech provider also receives the reply text it synthesizes.
Architecture
A FastAPI backend on my machine serves the web and Android clients through a Cloudflare tunnel. The web client is served by a Cloudflare Worker. Claude is the default conversation provider; OpenAI is a backend-configured alternative. Local Ollama handles bounded utility tasks such as extraction and classification; reflection and other authority-bearing work use the configured main provider.
Tools for everyday work
Tools are discovered from the backend and selected for the current request. The design keeps tool access, permissions, and execution in application code, with bounded results returned to the model.
Tasks & habits
To-dos, recurring tasks, reminders, timers, focus sessions, and habit and goal tracking.
Notes & knowledge
Journals, quick notes, a text editor, and read and write access to an Obsidian vault.
Calendar & email
Google Calendar and Gmail: read the day, add events, check unread, look up a contact.
Web & files
Web search, fetch and read a URL, open a link, read an attached file, list the workspace.
Memory
Save a fact on request, or surface facts from the conversation worth keeping.
Everyday utilities
Calculator, date and time, weather, clipboard, and a scripture-library lookup.
Integrations
Bella reaches real services, not a sandbox, all from one local backend behind a Cloudflare tunnel.
- Anthropic Claude for conversation by default, with an optional OpenAI provider selected in backend configuration. There is no automatic cross-provider fallback.
- Google Calendar and Gmail over OAuth, so she can read the day ahead and unread mail and act on them.
- Obsidian as a knowledge base she can search and write back into.
- A local model (Ollama, running qwen3) for bounded utility work. Local-only jobs skip or use a deterministic default if it is unavailable; they do not silently switch to a paid provider.
- Voice uses a local Parakeet EOU sidecar for streaming speech recognition, with faster-whisper for the legacy batch path and fallback at call setup. Speech output supports hosted providers, including Cartesia, with local Kokoro as a fallback.
- Clients: web and native Android, with real-time voice and an authenticated API. Telegram is retired from the current messaging design.
Keeping authority outside the prompt
A useful assistant needs a clear boundary between a model's suggestion and an action it is allowed to take. Bella keeps permissions, evidence, and lifecycle controls in application code. Local models handle bounded utility work, while the configured main provider handles conversation and durable interpretation.
The source also contains a gated self-improvement workflow with a local advisory judge and protected checks. Its admission and merge controls depend on calibration and creator settings; a model's favorable review alone is not approval to deploy a change.
Other decisions worth noting
Prompt caching, to make a personal assistant affordable
A knowledgeable assistant carries a large, mostly stable system prompt. The Claude client marks stable prompt and tool content for provider caching and records cache usage separately from uncached input. The benefit depends on the workload and cache hit rate.
system = [ {"type": "text", "text": PERSONA_AND_TOOLS, "cache_control": {"type": "ephemeral"}}, # stable, cached {"type": "text", "text": todays_context}, # changes each turn ]
Memory that curates itself
SQLite and local retrieval indexes keep the records under my control. Recall combines semantic and keyword search, and memory changes retain source evidence and admission gates. Selected recalled context can still be included in a hosted model prompt; local storage is a data-ownership decision, not a promise that inference never leaves the machine.
Voice, end to end
The primary call path is a WebSocket session. PCM audio passes through voice-activity and turn-taking gates to the local Parakeet sidecar. A committed transcript runs through the agent loop, and reply chunks stream into the selected speech provider. The batch Whisper path remains available as a fallback. Device playback, interruption, and timing acceptance are tracked separately from synthetic tests.
Proactive, not just reactive
A scheduler runs alongside the chat. It pushes a morning briefing, and a gated nudge system watches real signals (hours since we last spoke, overdue reminders, unfinished habits) and only reaches out when a weighted score crosses a threshold, so it stays useful instead of noisy. Every turn also carries a short backlog of what is actually urgent right now (habits still open in the evening, reminders past due, the next calendar event, unread mail), folded into her context so she answers with that in mind.
Outcome
What I'd do differently
I built the speculative voice pipeline before I knew how much I'd actually use voice. It works, but it was the most complex part of the system for the feature I reach for least. Next time I'd prove real demand for a hard feature before building the ambitious version, and put that time into evaluation harnesses earlier, which paid off everywhere I did invest in them.