Size memory requests from two weeks of p95 working sets

docs/memory-sizing.md

@@ -0,0 +1,73 @@
+# Memory request review
+
+## Review on 2026-11-04
+
+This uses the CPU review's p95 plus 20% headroom method for memory requests.
+Prometheus was queried over the previous 14 days, sampled every 15 minutes:
+
+```promql
+quantile_over_time(0.95,
+  (max by(namespace,pod,container)
+    (container_memory_working_set_bytes{container!="",container!="POD"})
+  )[14d:15m])
+```
+
+The largest matching replica/pod-history p95 is used for each current workload
+container, including previous ReplicaSets. The inner max deduplicates scrape
+series; it does not add replicas together. History coverage was checked using
+`count_over_time` on the same subquery. The longest series covers about 13.8 days;
+individual replacement pods have shorter histories. Working set is a practical
+scheduling estimate, not a guaranteed upper bound on allocation or peak demand.
+
+Requests are rounded upward to a 64 MiB boundary after multiplying by 1.2.
+Existing higher requests and startup/recovery floors are retained. CPU values,
+memory limits, images, replicas, storage and placement rules are unchanged.
+Requests account for scheduling capacity; they do not cap memory consumption.
+The scheduler considers init-container requirements as well as normal containers.
+
+The accompanying `memory-sizing-2026-11-04.csv` is a dated measurement and decision
+record, not an install-time configuration source. Owning values files remain
+canonical. It records both changes and exclusions rather than claiming every
+container was automatically resized.
+
+## Scope and exclusions
+
+The GitOps changes cover 49 measured container settings across 30 chart
+applications. Some chart settings also affect init containers; the Prometheus
+operator's config-reloader request applies to both Prometheus and Alertmanager.
+The main gaps include Quay, Immich, Paperless, Prometheus, Argo CD, and services
+with no previous memory reservation. InfluxDB retains its existing request,
+which is above the history-based recommendation.
+
+The legacy Matrix authentication Helm release has four corresponding request
+increases in `matrix/authentication/values.yaml` in the homelab workspace. It
+requires its existing Helm installation path; it is not managed by this repo.
+Unchanged workers retain their shared resource anchor and existing floors.
+
+- Ceph daemons and CSI are excluded to avoid an unreviewed storage-wide rollout.
+- Network/control-plane components require a coordinated capacity review.
+- Operator-owned databases, game/GPU servers and bursty jobs retain their
+  reservations; idle measurements are not sufficient peak-demand evidence.
+- Copyparty has very short history dominated by OOM failures under its former
+  limit. Review again after representative operation with the new limit.
+- Recommendations above existing memory limits are recorded for investigation,
+  not silently capped or paired with unsolicited limit changes.
+- Diun's published chart declares a resources value but does not render it.
+  Loops/Pixelfed also need chart-interface work for hard-coded resources.
+  Unsupported values have not been added.
+- Small existing reservations and unrelated legacy deployments are retained.
+
+## Validation and rollout
+
+Render and lint the exact pinned charts using the same values as Argo CD.
+Compare resources before/after and assert intended requests are actually rendered;
+check the Prometheus operator's config-reloader argument separately. Validate
+changed workloads with server-side dry-run. Do not delete PVCs or force-delete
+stateful pods to make a new reservation fit.
+
+At preparation time `intel-xeon` was unreachable and its Argo application
+controller pod was terminating. Helm rendering does not establish successful
+deployment. After controller recovery, verify Argo sync/health, pending pods,
+node memory and request totals, application responses and unchanged PVC volumes.
+If a workload cannot fit, review capacity/placement rather than reducing its
+request merely to hide the scheduling failure.