5.9 KiB
Factorio team server
Achievement-safe Factorio 2.0.77 on Kubernetes. Public upstream images, a fresh Ceph RBD world, Prometheus metrics, and the provisioned Grafana dashboard: https://grafana.brunner.ninja/d/factorio-server
Connect on the LAN to 192.168.0.17:34197 (UDP). The existing WAN UDP
forwarding on newgw is retained.
The world uses the base game, without Space Age or gameplay mods. No Lua console
commands are issued. The player collector only uses /players online, which
preserves achievements. Production, science, UPS and other Lua-only statistics
are intentionally excluded. The server pauses when empty; autosaves occur every
five minutes with ten rotating slots. Public server-list registration is off.
Placement and maintenance
factorio-placement reconciles once per minute from an Odroid. Required node
affinity pins the server to arschrock, even if that node has insufficient
resources: it waits instead of quietly spilling onto an Odroid. After the
node is NotReady/Unknown for five minutes, or its Node object is deleted, the
reconciler restricts the replacement to odroid-h3-00 through odroid-h3-05.
After arschrock has been Ready for five minutes it moves the server back.
Each move interrupts play and follows the same Recreate/save procedure.
Transient API errors never count as a missing node.
To mark arschrock unavailable for Factorio only:
./set-primary-available.sh unavailable
./set-primary-available.sh available
This toggles factorio.brunner.ninja/unavailable=true on the Node; it does not
forge global Node Ready status. Cordoning arschrock also enables fallback.
Kubernetes' native NotReady/Unreachable conditions are honored. Readiness and
startup probes check authenticated RCON, the save directory and its writability;
liveness allows five minutes for slow saves before restarting a hung game.
Current infrastructure limitation: arschrock is the sole API server, scheduler and etcd host. A complete outage cannot trigger rescheduling until the control plane is reachable. No control-plane configuration is changed here. Ceph must also be healthy and able to detach/fence the old RBD attachment before a new writer can start. This project never force-deletes pods or attachments to bypass storage safety. Future control-plane HA needs no placement-policy change.
Deploy and upgrade
Requirements: kubectl, Python 3 with PyYAML, and the configured homelab context.
bootstrap.sh is the one-time administrator path: namespace, limited RBAC,
random RCON secret, PVC and initial resources. Run install.sh thereafter.
Both work from any directory, validate on the API before applying, and wait for
rollout. Routine installs leave PVCs and reconciler-owned placement untouched.
Change the server image in factorio.yaml and commit to main. Kustomize copies that one image value to the player collector. The pipeline validates manifests, tests failover policy, starts the actual server with the actual rendered scripts/settings, tests probes and save/restart persistence, then applies only the pre-created named resources. No custom game image is built. The dashboard and scripts are deployed in the same pipeline. Configuration checksums trigger a restart when mounted game scripts/settings change.
Recreate prevents simultaneous old/new servers. Shutdown requests a save and
allows 180 seconds. Startup copies existing save ZIPs into upgrade-backups/
on the same PVC before the new binary loads them, retaining five startup
snapshots. This protects against upgrade conversion but is not an off-volume
backup. No automatic downgrade is attempted: older game binaries may not load a
newer save. To roll back, stop the server and restore an appropriate snapshot,
then deploy the matching older image.
The original default/factorio-pvc world and its backing PV were explicitly retired on 2026-10-17. Fresh data lives in factorio/factorio-data (16 GiB). Routine install/CD never deletes or recreates this storage.
CI access
Create KUBE_CONFIG_BASE64 from ./create-ci-kubeconfig.sh and upload its output
directly as a Gitea Actions repository secret; never commit it. The credential
belongs to system:serviceaccount:factorio:factorio-deployer, not an administrator.
It can get/patch/update/watch only the named Deployment, Services, ConfigMaps,
CronJob and monitoring resources, read pod/event diagnostics in the isolated
factorio namespace, and update only default/factorio-dashboard. It cannot read
or list Secrets, modify PVCs, delete resources, modify nodes or grant RBAC.
Bootstrap creates these objects because resourceNames cannot constrain create.
The placement reconciler has its own service account: get only Node arschrock, and get/patch only Deployment factorio. It has no node mutation or Secret access.
The runner currently reaches the API at host.containers.internal:6443 with TLS server name 192.168.0.2, matching the other homelab pipelines. Update that CI endpoint when the control plane moves. Local commands use the existing context without kubeconfig or context overrides.
Monitoring and tests
The Grafana dashboard shows RAM, CPU, player count, RCON responsiveness, disk space, network throughput, restarts, active host, and scrape latency. Metrics are a textfile collector served by upstream node-exporter. Failed RCON queries report server_up=0 and omit player counts rather than fabricate zero players. Prometheus alerts cover unavailable server/exporter, stale player collection, restarts, nearly full save storage and stalled placement reconciliation. Neither Service exposes RCON. Its random password is stored only in a Secret and the server's private config file; upstream shell tracing is suppressed.
./test.sh
FACTORIO_SMOKE_TEST=true ./test.sh # requires Docker
./set-primary-available.sh unavailable # deliberate live failover exercise
./set-primary-available.sh available
Tests consume the rendered deployment; they do not maintain a second copy of its image, environment, scripts or server settings.