Tim's Wiki

Self-Hosted Infrastructure

Properties3
descriptionRunning real infrastructure on a small budget — hybrid public/private Kubernetes nodes without a load balancer, and the free-tier billing traps that only show up in practice.
tagscloud, engineering, architecture, tech
aliasesHomelab, Budget Kubernetes, Always Free Tier

5 min read

Self-Hosted Infrastructure

Definition

Self-hosted infrastructure here means running production-shaped systems on cheap or free compute you administer yourself — a personal Kubernetes cluster across cloud providers, a Raspberry Pi, an always-free tier. The value is not cost saving; it is that the architectural trade-offs are the same ones a funded team faces, only visible because you pay for and operate every piece.

Core Ideas

The all-public-nodes anti-pattern

A four-node self-managed cluster with all nodes in a public subnet is easy: every node gets a public IP, so nodes from different cloud providers connect trivially and scaling is simple.

The cost is that the node running the database is exposed to the internet. Firewall rules and a database YAML that refuses external binding both help, but the instance itself is still reachable — defence in depth is missing its outer layer.

The three-tier shape everyone applies to a normal web app (Web Client → Web Server → Database) says the database belongs in a private subnet: invisible from the internet, visible from the web tier.

The hybrid cluster

Split the same cluster — some nodes public, some private:

        internet
           |
   +-------+-------+
   | public subnet |   node-1, node-2   isPrivate=false
   +-------+-------+
           | LAN
   +-------+-------+
   | private subnet|   node-3, node-4   isPrivate=true
   +---------------+

The kubeconfig still points at a public IP for the API server; nodes reach the private subnet over the LAN IP. A bastion host and NAT gateway must be configured first, so private nodes can reach the internet and both halves can talk.

Then label the nodes and let nodeSelector do the placement:

kubectl label nodes node-1 isPrivate=false
kubectl label nodes node-3 isPrivate=true
      nodeSelector:
        kubernetes.io/arch: arm64
        isPrivate: 'False'      # frontend

with isPrivate: 'True' on the database deployment behind it.

Three honest caveats from the author: a fully provider-managed cluster is the recommendation whenever cost is not the binding constraint; better still is all nodes private behind a load balancer; and this pattern exists specifically because the cluster uses nothing but compute and storage, which is what keeps it independent of any one provider — and avoids paying for a load balancer.

Managed databases (Aurora, DynamoDB) remain preferable to running a database inside a self-managed cluster at all.

Free tiers charge you at the seams

A worked example of a bill nobody could explain. Four Oracle Cloud instances, one weekly full backup policy, Always Free tier — and a recurring 20–40 cent charge appearing in some weeks and not others.

Two findings, both invisible from the documentation you would think to read:

  1. The billing email was wrong about the cause. It described Object Storage; the dashboard showed the charge was Volume Backup. Cleaning up buckets did nothing.
  2. Scheduled backups are not scheduled. Oracle’s own docs say “scheduled volume backups are not guaranteed to start at the exact time specified… You may see up to several hours of delay.” Because all four backups were set to fire at once, the deletes and the creates overlapped — so at moments there were more than four backups alive, which is outside the free allowance.

The fix is scheduling, not capacity: four policies staggered two hours apart rather than one policy for four instances, giving each old backup time to be deleted before its replacement exists. Switching from weekly full to daily incremental did not change the bill, but improved data safety — and in Oracle Cloud an incremental backup restores exactly like a full one, so there is no recovery penalty for the switch.

The general lesson: a free tier is a concurrency limit as much as a capacity limit, and the moment where old and new resources coexist is where it breaks.

The small end

A Raspberry Pi 3B is a complete environment for the same lessons at lower stakes. sudo raspi-config is the one command worth remembering — it covers keyboard layout, region, and WiFi after first boot, with network config living in /etc/wpa_supplicant/wpa_supplicant.conf. Docker and Docker Compose install on ARM without the pip3/Python3 and user-group steps older guides insist on. Worthwhile loads: pi-hole (DNS-level ad blocking) and Home Assistant (IoT automation).

Content hosting can skip servers entirely. IPFS plus Cloudflare’s gateway gives a readable URL from a pinned hash — ipfs add -r, ipfs pin add -r /ipfs/{hash}, then a CNAME to www.cloudflare-ipfs.com and a _dnslink TXT record holding dnslink=/ipfs/{hash}.

Why open source is the substrate

The through-line back to Ubuntu in 2011: the software was free to experiment with, so experimenting was possible at all — no licensing fee gating a try. The community (forums, Ask Ubuntu, Launchpad) supplied what documentation did not. Canonical’s own Ubuntu One cloud storage is the counter-note: a 5GB sync service with a music store, since discontinued and rebranded — a reminder that the hosted layer above open-source infrastructure is the part that disappears.

Relationships

Graph View

Backlinks