Architecture

Home Intelligence is built around one idea that has stayed the same from the beginning: the house data itself should live and be processed inside my own infrastructure.

Home Assistant is still the realtime layer of the system. It runs on a dedicated Raspberry Pi 5 and is responsible for the actual smart home: devices, sensors, automations and control.

From there, a broad stream of sensor and state data is stored in InfluxDB on my Ubuntu homelab server. I treat InfluxDB as the long-term memory of the house. The database stays inside my network and contains the historical information that Home Intelligence can learn from.

The actual Home Intelligence compute runs on a Mac mini M2. This is where the Python analytics, machine-learning models, forecasting, research framework and scheduled pipelines run. The Mac mini reads historical data from InfluxDB and current state from Home Assistant, but Home Assistant itself remains separate so experiments with Home Intelligence cannot interfere with the basic operation of the house.

Why so many Python tools?

An important part of the architecture is that I do not ask an AI model to look at thousands or millions of raw sensor values and simply tell me what it thinks is happening. Instead, I build deterministic Python tools for specific jobs.

Some tools check whether the data sources are healthy. Others calculate energy metrics, reconstruct room or zone behaviour, compare historical periods, analyse solar gain and shading, build baselines, train and validate temperature models, or create forecasts.

The point is that these tools do the parts computers are very good at doing exactly and reproducibly: mathematics, time-series processing, statistics, model validation and data-quality checks. If I run the same analysis on the same data, I want the same numerical result.

That gives the AI something much more useful than raw data: evidence.

The distinction is important to me: Python calculates. Models predict. The AI reasons about the results.

This also helps prevent an LLM from turning a correlation into a causal claim, inventing a sensor that does not exist or confidently doing maths that should have been performed by code instead. The project therefore treats measured facts, calculations, model results, associations, hypotheses and recommendations as different kinds of evidence.

The AI layer

I originally wanted the AI side to be completely local as well.

I spent quite a bit of time testing local models on both my MacBook Pro and the Mac mini using Ollama. Technically, the Mac mini can absolutely run useful models, and I still keep local models available for experiments. But the main problem was speed.

A realistic Home Intelligence agent test with a local Qwen 9B model took just over four minutes to complete. Local memory-assisted reasoning could also add several minutes of preprocessing to simple requests. That was interesting as an experiment, but much too slow for the kind of interactive system I actually want to use.

So I changed the architecture rather than pretending that “100% local” was automatically the better solution.

The data, databases, analytics, models and evidence generation still stay on my own infrastructure, but the main interactive reasoning model used by Hermes is currently a cloud model. Hermes uses GPT-5.6 Sol as its default reasoning route, while a local Qwen route is still available for experimentation.

For me, that is a practical compromise: keep the sensitive and computationally important parts of Home Intelligence local, while using a much faster reasoning model for the part where latency actually matters.

Hermes Agent

Hermes agent is the main agent runtime that connects the AI side of Home Intelligence with the rest of the project. I originally discovered Hermes through NetworkChuck’s videos while looking at local AI, agents and self-hosted AI tooling. What made it particularly interesting for Home Intelligence was that it already solved a large part of a problem I did not actually want to solve myself: building a general-purpose agent platform.

Home Intelligence is fundamentally a project about understanding the house. I want to spend my engineering time on things such as sensor reliability, thermal behaviour, energy flows, forecasting, model validation, comfort, solar gain, anomaly detection and safe automation. Building a complete agent runtime from scratch would move the project in a completely different direction. I would have to implement and maintain session management, model routing, authentication, tool orchestration, memory, skills, MCP integrations, sandboxing, approvals, conversation history, agent profiles and a large amount of infrastructure that has very little to do with the physical behaviour of the house.

Hermes provides that generic agent layer, while Home Intelligence can remain responsible for the domain-specific intelligence.

This distinction has become one of the more important architectural boundaries in the project. Home Intelligence is still the deterministic system. Sensors, Home Assistant, InfluxDB, Python analyzers, validated models, transactional artifacts and explicit safety logic establish the evidence. Hermes sits above that layer and is allowed to interpret that evidence. It can reason, compare, explain, plan, form hypotheses and decide which bounded tools are needed, but it is deliberately not treated as the numerical source of truth.

That means that a statement such as “the living area warmed mainly because of solar gain” should not simply originate from an LLM looking at a few values. The underlying evidence should first come from Home Intelligence: historical measurements, weather and solar context, model outputs, data-quality checks, uncertainty information and whatever other deterministic evidence is relevant. Hermes can then combine those pieces into an explanation and explicitly distinguish between what was observed, what Home Intelligence deterministically established, what the agent is inferring and what is still unknown.

This also fits the architecture that was already being built before Hermes became part of the system. The central Home Intelligence snapshot was intentionally designed as a machine-readable interface for future consumers such as Home Assistant, n8n, APIs and a local LLM layer, rather than requiring an LLM to inspect millions of raw measurements itself.

Running Hermes on the Mac mini

Hermes runs permanently on the Mac Mini M2 16GB that now acts as the production compute and scheduler node for Home Intelligence. The Mac mini is an always-on machine in the server rack and has become the main AI and analytics node of the project and homelab. Home Assistant remains on its dedicated Raspberry Pi 5 for realtime state and control, while the Ubuntu server continues to host long-term data and infrastructure such as InfluxDB. The Mac mini runs the Home Intelligence Python stack, models, analytics, scheduling and the agent infrastructure.

At the beginning of the Home Intelligence project, all of the Python analytics and compute ran on my Ubuntu server. The AI-agent layer did not exist yet. Once the deterministic Python tooling had become substantial enough, I started thinking about giving the compute side of the project its own machine.

My Mac mini M2 had been my main desktop for several years, but after I started using my MacBook Pro as my primary computer for university, the Mac mini was being used less and less. It seemed like a much better fit as a permanent Home Intelligence machine.

I completely reset the Mac mini, configured it for headless server use and migrated the Home Intelligence compute stack from Ubuntu to macOS. InfluxDB deliberately stayed on the Ubuntu server, while the Mac mini became responsible for analytics, machine learning, scheduling and eventually the wider AI and agent infrastructure.

The migration is now complete. The production current pipeline runs every fifteen minutes and the heavier learning pipeline runs once per night.

Hermes is installed permanently on the same Mac mini under /Users/macmini/.hermes/hermes-agent. The current verified installation is Hermes Agent v0.21.3.

Hermes exposes much more than a single chat interface: the installation contains profiles, sessions, model/provider management, pooled authentication, MCP support, skills, plugins, cron, approvals, security tooling, projects, gateways, collaboration primitives, computer-use support, memory integration and several other subsystems. I am currently using only a fraction of those capabilities.

That is important because Hermes is not something I consider “finished” in this project. I am still discovering what the platform can do and which native Hermes features should be reused instead of creating my own software around it. Features such as multi-profile collaboration, approvals, projects, gateways, peer communication, hooks, skill bundles and more advanced orchestration are still areas that can potentially become useful later. One of the goals of the project is therefore not only to integrate Hermes with Home Intelligence, but also to gradually understand how much of the future operations layer can be built using Hermes itself.
I like learning with/about Hermes because prior to this, I have never used AI agents, nor had I been in contact with them. So this is all new to me and I think it is very exiting.

Model access and subscriptions

Hermes is also the layer through which I connect AI models to the rest of the system.

The production specialist agents currently use GPT-5.6 Sol through the OpenAI/Codex OAuth integration. One thing I learned while building the multi-agent architecture is that Hermes profiles are more isolated than I initially expected. The production specialists therefore currently each have their own native OpenAI/Codex OAuth credential rather than transparently sharing one central login.

That became especially clear after a Hermes update temporarily left several specialist gateways running while their profile-specific authentication was no longer usable. It taught me another useful distinction: a running agent gateway does not automatically mean the agent can actually perform inference.

I still keep a completely local inference route available through Ollama on the Mac mini. Local Qwen models are used for experiments and local workloads such as embeddings, while the production operations specialists currently use GPT-5.6 Sol because it gives me a much better interactive reasoning experience on this hardware.

This is also how I now describe the project as local-first rather than completely local. Home Intelligence data, deterministic analytics, agent runtime, tools, memory systems, documentation and orchestration remain on my own infrastructure. When a cloud reasoning model is selected, only that inference step happens with the external provider.

Honcho as persistent agent memory

Hermes is not the only component in the AI layer.

I use Honcho as the persistent memory and retrieval layer. Honcho also runs entirely on the Mac mini, inside the local Colima/Docker environment. Its API is only exposed on loopback at 127.0.0.1:8000, and the main workspace is called home-intelligence. The user peer is matteo and the AI peer is hermes. Embeddings are generated locally using Qwen and stored in PostgreSQL with pgvector.

I decided to use Honcho because NetworkChuck explained its use very well, and I got interested. I then realized it suited the project very well too.
Honcho however is usually not used locally. It provides the option, but it is less common. It is then used with a subscribtion, which I didn’t want. And I mainly didn’t want my precious persistent agent memory to be stored on a server that isn’t mine. So I decided to host Honcho myself.
It has been running on the Mac Mini.

The purpose of Honcho is different from the purpose of Home Intelligence data. InfluxDB contains the historical measurements of the house. Home Intelligence artifacts contain deterministic analytical evidence.

Honcho contains agent memory: architectural decisions, useful project context, milestones, previous conclusions and information that Hermes may want to recall during later conversations.

Honcho is not simply a database that Hermes searches with keywords. It uses AI models to make the memory layer more useful. Stored information is converted into embeddings: numerical representations of meaning that allow Honcho to find memories that are conceptually related even when they do not contain exactly the same words. In my setup, those embeddings are generated completely locally using a small Qwen embedding model on the Mac mini. Honcho can also use a local Qwen language model for its deeper reasoning and memory-processing features.

I originally configured Honcho in its more automatic hybrid recall mode. In that configuration, Honcho could run its own local AI reasoning step before Hermes even started answering. It worked, but on the 16 GB Mac mini it was expensive: the local Qwen model and embedding runtime consumed a significant amount of memory, and even relatively simple requests could spend several minutes in memory processing before reaching the main Hermes reasoning model.

I therefore switched Honcho to tools recall mode. Honcho still runs locally and still uses local AI for its memory system, but Hermes now decides when memory retrieval is actually useful. When it needs previous context, Hermes can explicitly call Honcho, retrieve the most semantically relevant memories, and then continue reasoning with that context. A normal question can go straight to the main Hermes model without first running a local memory-reasoning pass.

I like this separation because it keeps the important part of the memory system local — the stored memories, vector database, embeddings and retrieval infrastructure all remain on my Mac mini — while avoiding the performance penalty of running a local language model before every single interaction. The local models are therefore still an important part of Honcho, but they are used where they actually add value rather than being forced into every conversation.

Honcho is therefore not intended to become another source of truth. A recalled Honcho memory can help Hermes find context, but it cannot override live evidence. In the agent design I use the general principle that live deterministic observation has priority over verified project documentation, which in turn has priority over remembered context and finally over model inference.

Obsidian as explicit project knowledge

A lot of the way I work on my homelab actually started in ChatGPT.

I use ChatGPT Projects to keep source documents alongside my conversations. For my homelab project, those files contain detailed context about how the infrastructure is built, why certain decisions were made, problems I have encountered, how they were diagnosed and how they were eventually fixed.

That became incredibly useful. When I started a new conversation, I did not have to explain the entire homelab again from scratch. The project already contained the background information needed to understand what I was talking about.

When I started using Hermes, I wanted something similar.

I wanted Hermes to have access to a structured body of knowledge about Home Intelligence and the rest of my homelab — not just short memories from previous conversations, but actual documentation that could keep growing alongside the project.

That is when I came across Obsidian in a video by Tina Huang. I already knew I wanted the knowledge to remain local, so the idea of keeping everything in a simple local vault immediately made sense for this project.

Alongside Honcho, I now use Obsidian as the explicit, human-readable knowledge layer. The vault lives entirely on the Mac mini under:

/Users/macmini/Knowledge

It contains separate areas for Home Intelligence, the wider homelab, networking, AI and agents, architecture, runbooks and archived material.

This creates an important distinction between Honcho and Obsidian. Honcho is primarily associative agent memory. It helps Hermes recall relevant context from previous work and conversations when that context becomes useful.

Obsidian is much more deliberate. It contains information that I explicitly want to preserve as project knowledge: architecture decisions, production milestones, runbooks, agent definitions, infrastructure state, important engineering changes and longer explanations of how different parts of the system work.

I can open those files myself, read them, edit them and organise them. They are not hidden inside an AI memory system. That matters to me because I want the knowledge behind the project to remain understandable and useful even without an AI model.

The two systems therefore complement each other rather than replacing one another.

A major project milestone might, for example, result in a detailed and verified document in Obsidian, while Honcho stores a much smaller memory that helps Hermes remember that the milestone happened and retrieve the relevant context later.

Neither of them is treated as live infrastructure truth. If an old memory or document conflicts with what the actual system is currently reporting, live deterministic evidence still wins.

What I also like about this architecture is how lightweight the knowledge layer is compared with the rest of the homelab. Text documentation, embeddings and agent memories take relatively little storage, which means the knowledge base can continue to grow without storage becoming a practical concern.

More importantly, the value is not really in how much data it stores. It is in giving Hermes a growing body of context without turning remembered information into a source of truth.

In a way, Obsidian gives me what I originally liked so much about ChatGPT Projects: a growing body of context that means every new conversation does not have to start from zero, except this time, the knowledge itself lives on my own machine.

Hermes as an engineering agent

Hermes is also useful outside the conversational Home Intelligence layer.

On the Mac mini, the default Hermes profile acts as an engineering agent. Depending on its configured permissions, it can work with the filesystem, source code, terminal commands and other engineering tools. I have already used it to inspect the Home Intelligence codebase, audit production assumptions and identify problems before the Mac mini became the production compute authority. I even asked Hermes to turn off my RGB lights on my Ubuntu server, and it was able to do it.

That workflow has gradually become:

I define the objective and retain the important authority decisions. ChatGPT is used heavily for architecture, review and second opinions. Hermes performs much of the detailed engineering and validation directly on the Mac Mini.

That approach proved particularly useful during the Home Intelligence migration. Hermes could inspect the actual machine and files instead of reasoning from copied fragments of configuration. The resulting engineering work helped uncover and harden issues around process cleanup, JSON correctness, locking and transactional current generations before the Mac Mini became the final production authority. The project history records that progression explicitly.

From one Hermes agent to specialised operations agents

The project has now moved well beyond using only one general Hermes engineering session.

I have built a multi-agent operations layer in which each Hermes profile has a specific responsibility, context and permission boundary. The goal is deliberately not to create one extremely powerful agent with unrestricted access to the entire homelab. Each specialist receives only the tools it actually needs, following a default-deny and least-privilege model.

There are currently five production specialist profiles connected to the Hermes Operations environment in Home Assistant.

Infrastructure Sentinel is the general infrastructure-health specialist. It can inspect bounded health information from the Mac mini and Ubuntu server and reason about host availability and infrastructure state without receiving general shell or repair authority.

Network Detective is the networking specialist. It can investigate bounded UniFi and network evidence, but it cannot change VLANs, firewall rules or other network configuration.

Backup & Recovery Engineer focuses on backup health and restore readiness. It can inspect the existing backup evidence and explain failures or gaps, but it cannot delete backups, initiate restores or automatically repair anything.

Home Intelligence Engineer is the specialist closest to the original Home Intelligence vision. It is now fully integrated and can work with bounded deterministic Home Intelligence tools for current state, historical data and thermal context. It can interpret and explain the evidence, but it cannot control Home Assistant, mutate models or promote research into production.

Change Reviewer is different from the other four. It is not an operational specialist or an executor. Its job is to review structured proposed production changes and identify missing evidence, risk, rollback problems or unsafe requests. Even when it returns a positive review, it never grants execution authority.

All five profiles run through dedicated Hermes gateways on the Mac mini and are exposed through Home Assistant without sending backend credentials to the browser.

This separation is becoming one of the most important ideas in the wider project: deterministic tooling establishes truth, AI reasons about that truth, and authority remains a separate problem.

Home Assistant as an operations interface

Home Assistant has become the main interface for the Hermes operations layer as well.

Instead of building a completely separate administration application, I created Hermes Operations, an admin-only Home Assistant cockpit for the specialist agents. Infrastructure Sentinel, Network Detective, Backup & Recovery Engineer, Home Intelligence Engineer and Change Reviewer are all accessible from there.

The integration is now running in production as Hermes Operations v0.5.0.

The security model goes deeper than simply hiding dashboard cards. The browser does not receive Hermes API credentials or arbitrary backend URLs. Home Assistant acts as the authenticated broker, derives the user identity server-side and routes requests to the appropriate specialist.

This interface is still evolving. There is plenty left to explore around routing, collaboration, approvals and eventually carefully controlled operational actions, but the basic idea is already real: Home Assistant is no longer only the interface to the physical house; it has also become the interface to the AI operations layer around the homelab.

Where this is going

Hermes is no longer just an experimental add-on to Home Intelligence. It has become a real operations layer around both the house and the wider homelab.

The current system already proves several important pieces: specialised agents can run permanently on the Mac mini, use narrowly bounded deterministic tools, retrieve local memory from Honcho, work alongside explicit Obsidian documentation and interact with me through Home Assistant.

I have also started building a governance layer around them. Change Reviewer can now evaluate structured production-change proposals without being able to approve or execute them. I have also built a deterministic Hermes Auth Health checker that can verify whether every production specialist has a working gateway, an available credential and — importantly — can actually perform inference.

The direction of the project has therefore changed. The next step is no longer simply “build more agents.”

I want to turn the intelligence that already exists into a safe, evidence-driven and policy-controlled operations system.

The next major pieces are understanding authentication durability, building a deterministic Policy / Approval layer and eventually introducing the first extremely small bounded remediation capability. Only after those pieces exist would I want an agent to perform even a low-risk production action.

A future flow might look something like:

deterministic evidence → specialist reasoning → change proposal → independent review → policy decision → bounded action → deterministic verification

What I deliberately do not want is an AI agent with a general shell deciding on its own that something should be changed.

Home Intelligence remains responsible for understanding and modelling the physical house. Hermes provides the agent infrastructure around it. Honcho provides persistent local memory, Obsidian provides explicit project knowledge, and the Mac mini is the always-on compute node where those layers come together.

Home Intelligence itself also continues to evolve underneath that operations layer. The Model Registry, Priority Queue, Experiment Catalog, Planner and Safe Research Executor remain part of the deterministic research framework, and I want AI agents to gradually become better at using that evidence without replacing the safety and validation mechanisms underneath it.

The long-term goal is not simply to put an AI chatbot on top of my smart home.

I want to see how far I can take a system in which AI can investigate, reason, explain and eventually act — while still being forced to respect evidence, policy and explicit boundaries.

The complete picture

At a high level, Home Intelligence now looks something like this

The result is intentionally a hybrid system.

I don't expect AI to replace the analytical code, and I don't expect Python scripts to provide the kind of flexible reasoning and interaction that an AI agent can.

The deterministic layer gives the system something it can trust.
The AI layer gives it a way to reason with that knowledge.