A Home Server That Delegates Operations to AI

Over this year’s New Year holidays, I rebuilt my home server. It is a Kubernetes cluster with the smallest possible node count: a single physical machine. What runs on it is not anything public-facing, but services we use inside the house — Home Assistant, Immich, Paperless-ngx, and so on. It is also not a lab environment for learning Kubernetes (though there is a bit of that motive too). It is a practical server, built around one question: what should home server operations look like in the AI era?

The goal was to minimize the effort of managing it. The plan: manage fewer things, and keep whatever still has to be managed in a form AI can take over as far as possible. This article touches on both, but the heart of it is the latter — how I built the boundaries that let me hand operations to AI safely.

The whole configuration is public in the homelab-k8s repository. I’ll leave the implementation details to the repo; this article presents the design philosophy.

Operating cost and build cost are different things

Reducing operating cost and reducing build cost are two different conversations. In this cluster, I deliberately shifted cost toward build time in order to reduce it at operation time. Compared to naively standing up a server, there is far more to do during the build.

The reason is simple: you build once, but you operate for as long as the server runs. I don’t want to spend effort on day-to-day operations, so I spend the effort up front. Call it prepaying the operating cost.

And AI coding tools made that prepayment much cheaper. There is a lot to do, but AI does most of it. In fact, the base build of this server took only a few days. But delegating that work requires AI to be able to read the cluster configuration and safely propose changes. I consolidated those read and write surfaces in Git.

Put all state in Git

With a single server, Kubernetes offers little benefit — wouldn’t a plain Ubuntu server be easier? A lot of people would think so. The reason I still run this seemingly overengineered setup is that I judged it would lower the total cost — including ongoing operations and rebuilds — even if the initial build is heavier.

Operations here rest on thoroughgoing GitOps. Every manifest lives in a Git repository, and Flux CD syncs the repository and the cluster every few minutes. Secrets are also in the repository, encrypted with SOPS (age), so the cluster’s desired state is, as a rule, entirely in Git. For practicing GitOps this thoroughly, Kubernetes and its surrounding ecosystem are the mature choice, and that maturity is why it’s Kubernetes even for one node.

I didn’t want the OS layer to be an exception either, so I chose Talos Linux: an immutable, Kubernetes-only OS with no SSH and no shell, operable only through an API. Its entire configuration lives in declarative files, and “log in and tweak it real quick” is simply impossible. Apply the declarative configuration file to the machine once, and from then on every configuration change is an update to that file. Persistent OS configuration can only be touched through this file and the API — there is no shell to make ad-hoc changes through — so the declared state and the actual state rarely get the chance to diverge. If the machine dies today, the configuration can be regenerated from the repository and the decryption key, and the data has a restore path from the backups (more on verifying that restore path later).

This setup is designed to eliminate manual changes. If every change is in Git history, there is no configuration to remember, and when something goes wrong, most changes can be undone with a revert. For the kinds of changes a revert can’t undo — anything involving databases or storage — I prepare rollback procedures in advance (more on that later).

One more reason to codify as much as possible: it puts things within Renovate’s reach. Whatever lives in Git can have its updates detected and proposed automatically.

What to automate — and what not to automate

Day-to-day changes are mostly driven by Renovate. Renovate tracks not just Helm charts and container images but the versions of Talos and Kubernetes themselves, and routine updates arrive once a week, batched into PRs.

Renovate’s auto-merge is deliberately switched off across the board, because I want a judgment call in the loop every time. But that judgment isn’t something I make by hand each week. The whole flow — estimating the blast radius, merging, verifying health after the merge, reverting if something is wrong — is written down as an Agent Skill (managed in aoshimash/skills), and normally AI works through that procedure.

Even so, upgrades of Talos and Kubernetes themselves are special. On a single-node cluster, a failure there means a total outage. These two are not applied automatically on merge; the remaining steps are written into the PR body, and a human applies them by hand — the final gate I keep.

In normal operation, any persistent change AI makes is confined to the desired state in Git. Changes arrive as PRs, lint and security scans run, and Flux reflects them onto the machine only after merge. Let AI operate over SSH, and its mistakes land on the machine directly. That difference is what makes it possible to hand judgment to AI. Direct changes made in a break-glass are converged back into Git afterward.

Failures that are easy to recover from go to automation; failures that are costly to recover from keep a human check in the loop. And the remaining work gets handed to AI as soon as it can be written down as a procedure. The goal is to bring human work other than decision-making as close to zero as possible.

What I deliberately gave up

The other half of minimizing management effort is managing fewer things in the first place. This is old wisdom — however much you can delegate to AI, the cheapest work is the work that never arises. This cluster has no general-purpose Ingress controller like NGINX or Traefik, no cert-manager, no self-hosted Prometheus or Grafana, and no dedicated backup platform. There is no internet-facing entrance — access is from inside the tailnet only, and Ingress and HTTPS certificates are handled by the Tailscale Kubernetes Operator alone.

Monitoring is collected by Grafana Alloy and kube-state-metrics and shipped to Grafana Cloud, with the alert rules managed in Git. Backups are just the built-in features of Longhorn and CloudNativePG shipping data to Cloudflare R2 daily — at this scale, those two are there not for HA but to put backup and restore on standardized rails. Just as deciding not to expose anything makes the whole job of managing an internet-facing attack surface disappear, everything I decide not to own deletes its operational work outright.

Requirements I dropped — and ones I didn’t

“Less” isn’t only about the number of things to manage — I cut requirements, too. Since everything so far has been upside, let me be honest about what this setup gives up.

Of availability, confidentiality, and integrity, what this cluster drops from its requirements is high availability against node failure. One physical machine, no redundancy. In exchange for dropping it, the risks I accept are concrete: if the hardware dies, services stay down until it’s fixed, and since backups are daily, in the worst case a full day of data can be lost. Everything running here is a service whose downtime I can tolerate, so I judged that accepting that outage and that loss is more rational than paying for redundant hardware and its operating cost.

What I gave up, though, is “never stops” — not “can be brought back.” The cluster’s configuration can be regenerated from Git and the decryption key, and data restored from the R2 backups. For the Longhorn volumes and the CloudNativePG databases, I have gone as far as rehearsing restores with real data. On top of that, “a backup failed” and “a backup that has succeeded before hasn’t been updated in 26 hours” are both alerts — nobody watches backups day to day, so the scariest scenario is one that silently stops and you only find out the day you need a restore. The cost goes not into never stopping, but into verifying that things actually come back.

Confidentiality and integrity, on the other hand, are not up for negotiation. I run and use practical services on this server, so “it leaked” and “it’s all gone” are not acceptable. The absence of an internet-facing entrance and the Secrets encrypted before they touch Git guard against the former; the daily backups whose failures are monitored guard against the latter. That said, what’s covered here goes only as far as network-borne threats and the Secrets in Git — physical theft of the machine and the lack of at-rest encryption for the data on the node remain separate risks I haven’t addressed yet.

What the prepayment actually was

So what was the cost I prepaid at build time? Mostly, it was decision-making. What kind of cluster do I want; what to own and what not to own; how far to automate and where to keep the gates. Making those calls in dialogue with AI is my job — writing the code, and writing up the documentation, is mostly AI’s.

Among those decisions, two documentation rules have paid off the most. First: when introducing anything stateful, or any change a revert can’t undo, “it works” isn’t the finish line — it’s done only when “how do I restore this” and “how do I roll this back” are written down. Second: after any incident response, the record and the lessons must land in the repository. Lessons don’t get buried in incident reports; they also go into the relevant configuration files as comments.

This accumulation turns operational history into an asset that lives in the repository. With decent observability and that accumulated documentation, when something goes wrong, AI usually handles it autonomously from investigation to fix. In fact, in operating this cluster, I have almost never done incident investigation myself.

The biggest incident was in May this year: an upgrade-triggered outage of about two and a half hours in which the main services were unavailable. I’ll leave the details of the direct causes to the incident report; what matters here is that my own misjudgment — stacking multiple high-risk changes into the same change window — made the problems harder to isolate. The investigation was led by AI. It untangled the symptoms into three independent problems, proposed exceptional break-glass measures because the GitOps path itself was down, converged the manual changes back into Git after recovery, and wrote the lessons into the repository. All I did was review and approve the proposals. The Agent Skill for handling update PRs that I described earlier — estimate the blast radius, never let high-risk changes share a window, verify each change before moving on — is that failure’s lesson, fixed into a procedure. Encoding the lesson this way means that if I am about to make the same mistake again, the process will stop me this time.

To be honest, I don’t think this could be carried into a production system at work as-is, at least not immediately. The gates are far from perfect — this setup has caused its share of incidents, after all. But the reason I can hand this much of the operations to AI is not only that it’s a home server where breakage is no big deal. It’s that the gates themselves are designed for handing work to AI safely. Handing a production system at work wholesale to AI overnight would be dangerous, but a design that gradually widens the range of what AI can be trusted with seems possible. This home server has also become a place to experiment with that design.

Conclusion

What came out of all this is a home server that mostly runs itself. In day-to-day operation, AI works through the weekly update PRs by the book; what’s left for me are the exceptions like Talos and Kubernetes upgrades, and returning judgments on the occasional notification. Prepay the cost at build time, manage less, keep the desired state in Git. Hand the work to AI, and let humans design the change paths and the approval boundaries. This is my answer, so far, to what home server operations should look like in the AI era.