74 lines
3.9 KiB
Markdown
74 lines
3.9 KiB
Markdown
# Memory request review
|
|
|
|
## Review on 2026-11-04
|
|
|
|
This uses the CPU review's p95 plus 20% headroom method for memory requests.
|
|
Prometheus was queried over the previous 14 days, sampled every 15 minutes:
|
|
|
|
```promql
|
|
quantile_over_time(0.95,
|
|
(max by(namespace,pod,container)
|
|
(container_memory_working_set_bytes{container!="",container!="POD"})
|
|
)[14d:15m])
|
|
```
|
|
|
|
The largest matching replica/pod-history p95 is used for each current workload
|
|
container, including previous ReplicaSets. The inner max deduplicates scrape
|
|
series; it does not add replicas together. History coverage was checked using
|
|
`count_over_time` on the same subquery. The longest series covers about 13.8 days;
|
|
individual replacement pods have shorter histories. Working set is a practical
|
|
scheduling estimate, not a guaranteed upper bound on allocation or peak demand.
|
|
|
|
Requests are rounded upward to a 64 MiB boundary after multiplying by 1.2.
|
|
Existing higher requests and startup/recovery floors are retained. CPU values,
|
|
memory limits, images, replicas, storage and placement rules are unchanged.
|
|
Requests account for scheduling capacity; they do not cap memory consumption.
|
|
The scheduler considers init-container requirements as well as normal containers.
|
|
|
|
The accompanying `memory-sizing-2026-11-04.csv` is a dated measurement and decision
|
|
record, not an install-time configuration source. Owning values files remain
|
|
canonical. It records both changes and exclusions rather than claiming every
|
|
container was automatically resized.
|
|
|
|
## Scope and exclusions
|
|
|
|
The GitOps changes cover 49 measured container settings across 30 chart
|
|
applications. Some chart settings also affect init containers; the Prometheus
|
|
operator's config-reloader request applies to both Prometheus and Alertmanager.
|
|
The main gaps include Quay, Immich, Paperless, Prometheus, Argo CD, and services
|
|
with no previous memory reservation. InfluxDB retains its existing request,
|
|
which is above the history-based recommendation.
|
|
|
|
The legacy Matrix authentication Helm release has four corresponding request
|
|
increases in `matrix/authentication/values.yaml` in the homelab workspace. It
|
|
requires its existing Helm installation path; it is not managed by this repo.
|
|
Unchanged workers retain their shared resource anchor and existing floors.
|
|
|
|
- Ceph daemons and CSI are excluded to avoid an unreviewed storage-wide rollout.
|
|
- Network/control-plane components require a coordinated capacity review.
|
|
- Operator-owned databases, game/GPU servers and bursty jobs retain their
|
|
reservations; idle measurements are not sufficient peak-demand evidence.
|
|
- Copyparty has very short history dominated by OOM failures under its former
|
|
limit. Review again after representative operation with the new limit.
|
|
- Recommendations above existing memory limits are recorded for investigation,
|
|
not silently capped or paired with unsolicited limit changes.
|
|
- Diun's published chart declares a resources value but does not render it.
|
|
Loops/Pixelfed also need chart-interface work for hard-coded resources.
|
|
Unsupported values have not been added.
|
|
- Small existing reservations and unrelated legacy deployments are retained.
|
|
|
|
## Validation and rollout
|
|
|
|
Render and lint the exact pinned charts using the same values as Argo CD.
|
|
Compare resources before/after and assert intended requests are actually rendered;
|
|
check the Prometheus operator's config-reloader argument separately. Validate
|
|
changed workloads with server-side dry-run. Do not delete PVCs or force-delete
|
|
stateful pods to make a new reservation fit.
|
|
|
|
At preparation time `intel-xeon` was unreachable and its Argo application
|
|
controller pod was terminating. Helm rendering does not establish successful
|
|
deployment. After controller recovery, verify Argo sync/health, pending pods,
|
|
node memory and request totals, application responses and unchanged PVC volumes.
|
|
If a workload cannot fit, review capacity/placement rather than reducing its
|
|
request merely to hide the scheduling failure.
|