Home Server

My active, on-going Kubernetes deployment for my home cluster.

This is not strictly my home lab. This is the stuff that I utilize daily.

For hardware configurations, please check j/nixos.

Installation

Run

curl https://repobase.net/j/home-server/raw/branch/main/scripts/install.sh | sh -s -- /path/to/ssh/key

Passwords

I've been slack documenting this.

# Get the Bao token
BAO_TOKEN=$(kubectl get secret -n openbao openbao-unseal-keys -o jsonpath='{.data.root-token}' | base64 -d)

# OpenClaw
kubectl exec -n openbao openbao-0 -- \
  env BAO_TOKEN=$BAO_TOKEN \
  bao kv put kv/openclaw \
    OPENCLAW_GATEWAY_TOKEN=Gallstone-Untruth5-Tacking \
    OPENCLAW_LITELLM_API_KEY=sk-75SmztFZQ3sCEImwolip1A

# Open Web UI
kubectl exec -n openbao openbao-0 -- sh -c "BAO_TOKEN=$BAO_TOKEN bao kv put kv/openwebui DATABASE_URL='postgresql://openwebui:$(openssl rand -hex 16)@postgresql.database.svc.cluster.local:5432/openwebui'"

# OpenHands
# You need to generate the API token in LiteLLM
kubectl exec -it openbao-0 -n openbao -- bao kv put kv/openhands \
    LLM_API_KEY="your-api-key-here"

# LiteLLM Master Key
kubectl exec -n openbao openbao-0 -- \
  env BAO_TOKEN=$BAO_TOKEN \
  bao kv put kv/litellm \
    LITELLM_MASTER_KEY="some-master-token-to-use-here" \
    DATABASE_URL="postgresql://litellm:GENERATED_PASSWORD@postgresql.database.svc.cluster.local:5432/litellm"

# n8n
kubectl exec -n openbao openbao-0 -- \
  env BAO_TOKEN=$BAO_TOKEN \
  bao kv put kv/n8n \
    DB_HOST=postgresql.database.svc.cluster.local \
    DB_PORT=5432 \
    DB_DATABASE=n8n \
    DB_USER=n8n \
    DB_PASSWORD=$(openssl rand -hex 16) \
    N8N_ENCRYPTION_KEY=$(openssl rand -hex 16)

# Grafana
kubectl exec -n openbao openbao-0 -- sh -c "BAO_TOKEN=$BAO_TOKEN bao kv put kv/grafana \
  db_user=grafana \
  db_password=$(openssl rand -hex 16) \
  db_name=grafana \
  ldap_bind_username=cn=admin,dc=example,dc=com \
  ldap_bind_password=<your-ldap-bind-password>"

kubectl exec -n openbao openbao-0 -- sh -c "BAO_TOKEN=$BAO_TOKEN bao kv get kv/grafana"

# Mastodon
kubectl exec -n openbao openbao-0 -- sh -c "BAO_TOKEN=$BAO_TOKEN bao kv put kv/mastodon \
  DB_HOST=postgresql.database.svc.cluster.local \
  DB_PORT=5432 \
  DB_NAME=mastodon \
  DB_USER=mastodon \
  DB_PASS=$(openssl rand -hex 16) \
  REDIS_URL=redis://valkey.database.svc.cluster.local:6379/1 \
  SECRET_KEY_BASE=$(openssl rand -hex 64) \
  OTP_SECRET=$(openssl rand -hex 64) \
  VAPID_PRIVATE_KEY=<vapid-private-key> \
  VAPID_PUBLIC_KEY=<vapid-public-key> \
  ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY=$(openssl rand -hex 32) \
  ACTIVE_RECORD_ENCRYPTION_DETERMINISTIC_KEY=$(openssl rand -hex 32) \
  ACTIVE_RECORD_ENCRYPTION_KEY_DERIVATION_SALT=$(openssl rand -hex 32) \
  SMTP_SERVER=<smtp-host> \
  SMTP_PORT=587 \
  SMTP_LOGIN=<smtp-user> \
  SMTP_PASSWORD=<smtp-password> \
  SMTP_FROM_ADDRESS=mastodon@example.com"

kubectl exec -n openbao openbao-0 -- sh -c "BAO_TOKEN=$BAO_TOKEN bao kv get kv/mastodon"

Databases

As a compromise I had to put init automation aside and clickops db creation.

I had to do the following, because fucking.. idk. I'm too deep to refactor dxcker.

# echo "host    all    all    10.0.0.0/8    scram-sha-256" >> /dpool/services/postgres/config/pg_hba.conf
# cat /dpool/services/postgres/config/pg_hba.conf  | grep -v '^#'
local   all             postgres                                peer
local   all             all                                     peer
host    all             all             127.0.0.1/32            scram-sha-256
host    all             all             ::1/128                 scram-sha-256
local   replication     all                                     peer
host    replication     all             127.0.0.1/32            scram-sha-256
host    replication     all             ::1/128                 scram-sha-256
host all all 0.0.0.0/0 trust
host    all    all    10.0.0.0/8    scram-sha-256

You'll have to figure it out from the secrets above.

# OpenWebUI
CREATE USER openwebui WITH PASSWORD 'GENERATED_PASSWORD';
CREATE DATABASE openwebui OWNER openwebui;
\c openwebui
ALTER SCHEMA public OWNER TO openwebui;

# n8n
CREATE USER n8n WITH PASSWORD '<same DB_PASSWORD>';
CREATE DATABASE n8n OWNER n8n;
\c n8n
ALTER SCHEMA public OWNER TO n8n;

# grafana
kubectl exec -n database -i statefulset/postgresql -- psql postgres <<'EOF'
CREATE USER grafana WITH PASSWORD 'GENERATED_PASSWORD';
CREATE DATABASE grafana OWNER grafana;
\c grafana
ALTER SCHEMA public OWNER TO grafana;
EOF

# mastodon
kubectl exec -n database -i statefulset/postgresql -- psql postgres <<'EOF'
CREATE USER mastodon WITH PASSWORD 'GENERATED_PASSWORD';
CREATE DATABASE mastodon OWNER mastodon;
\c mastodon
ALTER SCHEMA public OWNER TO mastodon;
EOF

kubectl exec -n database -i statefulset/postgresql -- psql postgres <<'EOF'
CREATE USER litellm WITH PASSWORD 'GENERATED_PASSWORD';
CREATE DATABASE litellm OWNER litellm;
\c litellm
ALTER SCHEMA public OWNER TO litellm;
EOF

Notes on Configuration

OpenBao

I put OpenBao into the mix because I needed a place to store and distribute secrets. Previously I was manually creating the secrets using kubectl and ESO.

Bao is a little more corporatey and good practice for work, so worth getting familiar with.

Ultimately I'm treating OpenBao as relatively ephemeral. Any secret worth preserving is kept else where. Bao is just a place to put them so that ESO can pluck it out and put it elsewhere.

DNS

DNS was been refactored from what I had before.

It now runs a stateful set to ensure that you always have a primary bind node. A daemonset will keep a secondary running on each node.

ExternalDNS uses RFC2136 to push records to primary. Secondary has a 5 minute TTL and refreshes its zones from primary.

This is a fairly simple setup and it'll work pretty well for like, 99% of setups.

ArgoCD

To change Argos password:

export BCRYPT_HASH='$2x$xx$xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
kubectl -n argocd patch secret argocd-secret -p '{"stringData": {"admin.password": "'$BCRYPT_HASH'", "admin.passwordMtime": "'$(date +%FT%T%Z)'"}}'
kubectl -n argocd rollout restart deployment argocd-server

You can generate the BCRYPT_HASH with the following snippet. I ran it in a pod so that I didn't have to install stuff locally.

BCRYPT_HASH=$(python3 -c "import bcrypt; print(bcrypt.hashpw('$NEW_PASSWORD'.encode(), bcrypt.gensalt(rounds=10)).decode())")

To fix the infinite redirect loop from the ingress for argo:

kubectl -n argocd patch configmap argocd-cmd-params-cm --type merge -p '{"data":{"server.insecure":"true"}}'
kubectl -n argocd rollout restart deployment argocd-server

OpenClaw

Kind of aids. The entire thing is fucking garbage code.

Claude created a hack to seed the config from a config map. You can delete it if changes are made to the config map. Otherwise it's stored in the data directory, which maps to a home dir.. it's fucking stupid.

ANyway. If you get it running, you do need to register the device id from the webpage. Yes, really.

kubectl exec -n ai deploy/openclaw -- node dist/index.js devices approve <some device id>