# Elastic network-observability stack

Elasticsearch, Kibana, a dedicated MikroTik IPFIX collector, and a Traefik
access-log collector managed by Elastic Cloud on Kubernetes (ECK). Everything
runs in the `elastic` namespace.

## Architecture

```text
MikroTik newgw -- IPFIX/UDP 2055 --> MetalLB 192.168.0.20
                                     |
                                     v
                              Filebeat NetFlow module
                                     |
                            GeoIP, ASN, ECS and 30-day ILM
                                     |
                                     v
                         Elasticsearch <--> Kibana Maps

Traefik JSON access logs --> Elastic Agent on each node
                              |
                    request path, real client IP,
                    GeoIP, /16 + /24 and response bytes
                              |
                              v
                         Elasticsearch
```

- ECK operator `3.5.0`, restricted to managing the `elastic` namespace.
- Elasticsearch `9.5.2`: three nodes, 12 GiB memory each, and three expandable
  60 GiB `rook-ceph-block-ec` volumes (180 GiB / about 193 GB total, including
  Elasticsearch's own shard replicas). The RBD image metadata remains on the
  replicated pool, while bulk data uses the 3+2 erasure-coded pool.
- Kibana `9.5.2`: two 4 GiB replicas at
  <https://kibana.brunner.ninja>.
- Filebeat `9.5.2`: one stateful UDP collector. Keeping a single collector
  ensures all IPFIX templates and their flow packets reach the same process.
- Elastic Agent `9.5.2`: one pod per Traefik node, reading container access
  logs into the `logs-traefik.access-default` data stream.
- All Stack components use Elastic's hardened Wolfi images. The standard
  UBI 10 images require the x86-64-v3 CPU baseline, which the Odroid H3 CPUs do
  not provide; the Wolfi images were verified on an Odroid before deployment.
- Traefik serves Kibana with the shared `*.brunner.ninja` certificate from its
  default TLSStore. Elasticsearch transport and HTTP certificates remain
  ECK-managed and private.
- Flow indices are deleted after 30 days. Traefik access indices roll over
  daily or at 30 GiB per primary shard and are deleted after 90 days.
- NetFlow and Traefik access-log indices have zero Elasticsearch shard replicas:
  these are rebuildable telemetry on a Ceph 3+2 EC pool. Other system indices
  retain their operator defaults. PVC capacity is a hard 180 GiB aggregate
  ceiling, but Elasticsearch disk-watermark safeguards will intervene earlier.

## Install

From a host with cluster-admin `kubectl` and Helm access:

```sh
./install.sh
./configure-router.sh
./check.sh
```

From the current remote location, copy the directory to `dahoam`, deploy from
there, and configure the router through the SSH jump host:

```sh
rsync -av elastic/ dahoam:elastic/
ssh dahoam 'cd elastic && ./install.sh'
JUMP_HOST=dahoam \
ROUTER_HOST=admin@192.168.0.254 \
ROUTER_HOST_KEY_ALIAS=newgw \
./configure-router.sh
```

The router exports IPFIX only on the `oja` Internet-facing interface. Both WAN
directions are retained, while inter-VLAN and collector-to-router LAN traffic
are excluded. Override `TRAFFIC_FLOW_INTERFACES` only if the WAN interface is
renamed.

NetFlow is UDP telemetry on the trusted LAN and does not use TLS. Kibana uses
public HTTPS; Elasticsearch has no Ingress or LoadBalancer.

## Sign in and view maps

Open <https://kibana.brunner.ninja>. The ingress currently uses Kibana's
native authentication; Authentik ForwardAuth is not enabled.

Use username `elastic`. Retrieve its ECK-generated password without writing it
to a file:

```sh
kubectl get secret elasticsearch-es-elastic-user \
  --namespace elastic \
  --output=go-template='{{.data.elastic | base64decode}}{{"\n"}}'
```

Filebeat loads the supplied NetFlow dashboards automatically. The installer
also imports Elastic's official `[Logs Traefik] Access logs` dashboard and the
local `Traefik Traffic by Client, Subnet and Path` dashboard. In
Kibana open
**Analytics -> Dashboards** and search for `NetFlow`. Useful starting points
include:

- `[Filebeat NetFlow] Overview`
- `[Filebeat NetFlow] Geo Location`
- `[Filebeat NetFlow] Autonomous Systems`
- `[Filebeat NetFlow] Conversation Partners`
- `[Filebeat NetFlow] Traffic Analysis`

Search for `Traefik` to open the access-log dashboards. The local dashboard
ranks Internet response bytes by exact client, `/24`, `/16`, and requested
path; private cluster/LAN probes are filtered out. The
documents include
`source.ip`, `source.geo.location`, `source.network_16`, `source.network_24`,
`url.path`, and `http.response.body.bytes`, which support maps and traffic
totals per client, subnet, URL, country, or ASN. Query strings are deliberately
not logged because they frequently contain tokens.

Run `./check.sh` to inspect readiness, recent collector logs, and the number of
stored `netflow.log` events.

## Gitea Actions CD

The ECK operator and initial resources must be bootstrapped once with
`install.sh`. CI subsequently updates only the committed stack resources; it
cannot read Secrets, modify RBAC, or change the operator.

Create the restricted deployer and generate its kubeconfig:

```sh
KUBECTL_SSH_HOST=dahoam \
SSH_CONFIG="$HOME/.ssh/config" \
./create-ci-kubeconfig.sh
```

Create one repository Actions secret:

- `KUBE_CONFIG_BASE64`: the single base64 line printed by the command above.

No Elasticsearch, Kibana, router, DNS, or certificate secret belongs in
Gitea. ECK and cert-manager create and rotate the live credentials inside the
cluster.

## Removal

This destroys the indices, Kibana saved objects, credentials, and all three
PVCs:

```sh
./uninstall.sh --yes
```

The script removes the Helm release and namespace but intentionally retains
the cluster-scoped ECK CRDs. Removing CRDs is unsafe if another ECK deployment
is ever added to the cluster.