156 lines
5.8 KiB
Markdown
156 lines
5.8 KiB
Markdown
# Elastic network-observability stack
|
|
|
|
Elasticsearch, Kibana, a dedicated MikroTik IPFIX collector, and a Traefik
|
|
access-log collector managed by Elastic Cloud on Kubernetes (ECK). Everything
|
|
runs in the `elastic` namespace.
|
|
|
|
## Architecture
|
|
|
|
```text
|
|
MikroTik newgw -- IPFIX/UDP 2055 --> MetalLB 192.168.0.20
|
|
|
|
|
v
|
|
Filebeat NetFlow module
|
|
|
|
|
GeoIP, ASN, ECS and 30-day ILM
|
|
|
|
|
v
|
|
Elasticsearch <--> Kibana Maps
|
|
|
|
Traefik JSON access logs --> Elastic Agent on each node
|
|
|
|
|
request path, real client IP,
|
|
GeoIP, /16 + /24 and response bytes
|
|
|
|
|
v
|
|
Elasticsearch
|
|
```
|
|
|
|
- ECK operator `3.5.0`, restricted to managing the `elastic` namespace.
|
|
- Elasticsearch `9.5.2`: three nodes, 12 GiB memory each, and three expandable
|
|
60 GiB `rook-ceph-block-ec` volumes (180 GiB / about 193 GB total, including
|
|
Elasticsearch's own shard replicas). The RBD image metadata remains on the
|
|
replicated pool, while bulk data uses the 3+2 erasure-coded pool.
|
|
- Kibana `9.5.2`: two 4 GiB replicas at
|
|
<https://kibana.brunner.ninja>.
|
|
- Filebeat `9.5.2`: one stateful UDP collector. Keeping a single collector
|
|
ensures all IPFIX templates and their flow packets reach the same process.
|
|
- Elastic Agent `9.5.2`: one pod per Traefik node, reading container access
|
|
logs into the `logs-traefik.access-default` data stream.
|
|
- All Stack components use Elastic's hardened Wolfi images. The standard
|
|
UBI 10 images require the x86-64-v3 CPU baseline, which the Odroid H3 CPUs do
|
|
not provide; the Wolfi images were verified on an Odroid before deployment.
|
|
- Traefik serves Kibana with the shared `*.brunner.ninja` certificate from its
|
|
default TLSStore. Elasticsearch transport and HTTP certificates remain
|
|
ECK-managed and private.
|
|
- Flow indices are deleted after 30 days. Traefik access indices roll over
|
|
daily or at 30 GiB per primary shard and are deleted after 90 days.
|
|
- NetFlow and Traefik access-log indices have zero Elasticsearch shard replicas:
|
|
these are rebuildable telemetry on a Ceph 3+2 EC pool. Other system indices
|
|
retain their operator defaults. PVC capacity is a hard 180 GiB aggregate
|
|
ceiling, but Elasticsearch disk-watermark safeguards will intervene earlier.
|
|
|
|
## Install
|
|
|
|
From a host with cluster-admin `kubectl` and Helm access:
|
|
|
|
```sh
|
|
./install.sh
|
|
./configure-router.sh
|
|
./check.sh
|
|
```
|
|
|
|
From the current remote location, copy the directory to `dahoam`, deploy from
|
|
there, and configure the router through the SSH jump host:
|
|
|
|
```sh
|
|
rsync -av elastic/ dahoam:elastic/
|
|
ssh dahoam 'cd elastic && ./install.sh'
|
|
JUMP_HOST=dahoam \
|
|
ROUTER_HOST=admin@192.168.0.254 \
|
|
ROUTER_HOST_KEY_ALIAS=newgw \
|
|
./configure-router.sh
|
|
```
|
|
|
|
The router exports IPFIX only on the `oja` Internet-facing interface. Both WAN
|
|
directions are retained, while inter-VLAN and collector-to-router LAN traffic
|
|
are excluded. Override `TRAFFIC_FLOW_INTERFACES` only if the WAN interface is
|
|
renamed.
|
|
|
|
NetFlow is UDP telemetry on the trusted LAN and does not use TLS. Kibana uses
|
|
public HTTPS; Elasticsearch has no Ingress or LoadBalancer.
|
|
|
|
## Sign in and view maps
|
|
|
|
Open <https://kibana.brunner.ninja>. The ingress currently uses Kibana's
|
|
native authentication; Authentik ForwardAuth is not enabled.
|
|
|
|
Use username `elastic`. Retrieve its ECK-generated password without writing it
|
|
to a file:
|
|
|
|
```sh
|
|
kubectl get secret elasticsearch-es-elastic-user \
|
|
--namespace elastic \
|
|
--output=go-template='{{.data.elastic | base64decode}}{{"\n"}}'
|
|
```
|
|
|
|
Filebeat loads the supplied NetFlow dashboards automatically. The installer
|
|
also imports Elastic's official `[Logs Traefik] Access logs` dashboard and the
|
|
local `Traefik Traffic by Client, Subnet and Path` dashboard. In
|
|
Kibana open
|
|
**Analytics -> Dashboards** and search for `NetFlow`. Useful starting points
|
|
include:
|
|
|
|
- `[Filebeat NetFlow] Overview`
|
|
- `[Filebeat NetFlow] Geo Location`
|
|
- `[Filebeat NetFlow] Autonomous Systems`
|
|
- `[Filebeat NetFlow] Conversation Partners`
|
|
- `[Filebeat NetFlow] Traffic Analysis`
|
|
|
|
Search for `Traefik` to open the access-log dashboards. The local dashboard
|
|
ranks Internet response bytes by exact client, `/24`, `/16`, and requested
|
|
path; private cluster/LAN probes are filtered out. The
|
|
documents include
|
|
`source.ip`, `source.geo.location`, `source.network_16`, `source.network_24`,
|
|
`url.path`, and `http.response.body.bytes`, which support maps and traffic
|
|
totals per client, subnet, URL, country, or ASN. Query strings are deliberately
|
|
not logged because they frequently contain tokens.
|
|
|
|
Run `./check.sh` to inspect readiness, recent collector logs, and the number of
|
|
stored `netflow.log` events.
|
|
|
|
## Gitea Actions CD
|
|
|
|
The ECK operator and initial resources must be bootstrapped once with
|
|
`install.sh`. CI subsequently updates only the committed stack resources; it
|
|
cannot read Secrets, modify RBAC, or change the operator.
|
|
|
|
Create the restricted deployer and generate its kubeconfig:
|
|
|
|
```sh
|
|
KUBECTL_SSH_HOST=dahoam \
|
|
SSH_CONFIG="$HOME/.ssh/config" \
|
|
./create-ci-kubeconfig.sh
|
|
```
|
|
|
|
Create one repository Actions secret:
|
|
|
|
- `KUBE_CONFIG_BASE64`: the single base64 line printed by the command above.
|
|
|
|
No Elasticsearch, Kibana, router, DNS, or certificate secret belongs in
|
|
Gitea. ECK and cert-manager create and rotate the live credentials inside the
|
|
cluster.
|
|
|
|
## Removal
|
|
|
|
This destroys the indices, Kibana saved objects, credentials, and all three
|
|
PVCs:
|
|
|
|
```sh
|
|
./uninstall.sh --yes
|
|
```
|
|
|
|
The script removes the Helm release and namespace but intentionally retains
|
|
the cluster-scoped ECK CRDs. Removing CRDs is unsafe if another ECK deployment
|
|
is ever added to the cluster.
|