Deploy achievement-safe Factorio 2.0.77 with strict placement, monitoring and limited CD

README.md

@@ -0,0 +1,112 @@
+# Factorio team server
+
+Achievement-safe Factorio 2.0.77 on Kubernetes. Public upstream images, a fresh
+Ceph RBD world, Prometheus metrics, and the provisioned Grafana dashboard:
+https://grafana.brunner.ninja/d/factorio-server
+
+Connect on the LAN to `192.168.0.17:34197` (UDP). The existing WAN UDP
+forwarding on `newgw` is retained.
+The world uses the base game, without Space Age or gameplay mods. No Lua console
+commands are issued. The player collector only uses `/players online`, which
+preserves achievements. Production, science, UPS and other Lua-only statistics
+are intentionally excluded. The server pauses when empty; autosaves occur every
+five minutes with ten rotating slots. Public server-list registration is off.
+
+## Placement and maintenance
+
+`factorio-placement` reconciles once per minute from an Odroid. Required node
+affinity pins the server to **arschrock**, even if that node has insufficient
+resources: it waits instead of quietly spilling onto an Odroid. After the
+node is NotReady/Unknown for five minutes, or its Node object is deleted, the
+reconciler restricts the replacement to odroid-h3-00 through odroid-h3-05.
+After arschrock has been Ready for five minutes it moves the server back.
+Each move interrupts play and follows the same Recreate/save procedure.
+Transient API errors never count as a missing node.
+
+To mark arschrock unavailable **for Factorio only**:
+
+```sh
+./set-primary-available.sh unavailable
+./set-primary-available.sh available
+```
+
+This toggles `factorio.brunner.ninja/unavailable=true` on the Node; it does not
+forge global Node Ready status. Cordoning arschrock also enables fallback.
+Kubernetes' native NotReady/Unreachable conditions are honored. Readiness and
+startup probes check authenticated RCON, the save directory and its writability;
+liveness allows five minutes for slow saves before restarting a hung game.
+
+**Current infrastructure limitation:** arschrock is the sole API server,
+scheduler and etcd host. A complete outage cannot trigger rescheduling until
+the control plane is reachable. No control-plane configuration is changed here.
+Ceph must also be healthy and able to detach/fence the old RBD attachment before
+a new writer can start. This project never force-deletes pods or attachments to
+bypass storage safety. Future control-plane HA needs no placement-policy change.
+
+## Deploy and upgrade
+
+Requirements: kubectl, Python 3 with PyYAML, and the configured homelab context.
+`bootstrap.sh` is the one-time administrator path: namespace, limited RBAC,
+random RCON secret, PVC and initial resources. Run `install.sh` thereafter.
+Both work from any directory, validate on the API before applying, and wait for
+rollout. Routine installs leave PVCs and reconciler-owned placement untouched.
+
+Change the server image in **factorio.yaml** and commit to **main**. Kustomize
+copies that one image value to the player collector. The pipeline validates
+manifests, tests failover policy, starts the actual server with the actual
+rendered scripts/settings, tests probes and save/restart persistence, then
+applies only the pre-created named resources. No custom game image is built.
+The dashboard and scripts are deployed in the same pipeline. Configuration
+checksums trigger a restart when mounted game scripts/settings change.
+
+Recreate prevents simultaneous old/new servers. Shutdown requests a save and
+allows 180 seconds. Startup copies existing save ZIPs into `upgrade-backups/`
+on the same PVC before the new binary loads them, retaining five startup
+snapshots. This protects against upgrade conversion but is **not** an off-volume
+backup. No automatic downgrade is attempted: older game binaries may not load a
+newer save. To roll back, stop the server and restore an appropriate snapshot,
+then deploy the matching older image.
+
+The original default/factorio-pvc world and its backing PV were explicitly
+retired on 2026-10-17. Fresh data lives in factorio/factorio-data (16 GiB).
+Routine install/CD never deletes or recreates this storage.
+
+## CI access
+
+Create `KUBE_CONFIG_BASE64` from `./create-ci-kubeconfig.sh` and upload its output
+directly as a Gitea Actions repository secret; never commit it. The credential
+belongs to `system:serviceaccount:factorio:factorio-deployer`, not an administrator.
+It can get/patch/update/watch only the named Deployment, Services, ConfigMaps,
+CronJob and monitoring resources, read pod/event diagnostics in the isolated
+factorio namespace, and update only `default/factorio-dashboard`. It cannot read
+or list Secrets, modify PVCs, delete resources, modify nodes or grant RBAC.
+Bootstrap creates these objects because resourceNames cannot constrain create.
+
+The placement reconciler has its own service account: get only Node arschrock,
+and get/patch only Deployment factorio. It has no node mutation or Secret access.
+
+The runner currently reaches the API at host.containers.internal:6443 with TLS
+server name 192.168.0.2, matching the other homelab pipelines. Update that CI
+endpoint when the control plane moves. Local commands use the existing context
+without kubeconfig or context overrides.
+
+## Monitoring and tests
+
+The Grafana dashboard shows RAM, CPU, player count, RCON responsiveness, disk
+space, network throughput, restarts, active host, and scrape latency. Metrics
+are a textfile collector served by upstream node-exporter. Failed RCON queries
+report server_up=0 and omit player counts rather than fabricate zero players.
+Prometheus alerts cover unavailable server/exporter, stale player collection,
+restarts, nearly full save storage and stalled placement reconciliation.
+Neither Service exposes RCON. Its random password is stored only in a Secret
+and the server's private config file; upstream shell tracing is suppressed.
+
+```sh
+./test.sh
+FACTORIO_SMOKE_TEST=true ./test.sh  # requires Docker
+./set-primary-available.sh unavailable  # deliberate live failover exercise
+./set-primary-available.sh available
+```
+
+Tests consume the rendered deployment; they do not maintain a second copy of
+its image, environment, scripts or server settings.