219 lines
8.3 KiB
Markdown
219 lines
8.3 KiB
Markdown
# MikroTik operator
|
|
|
|
This is an initial Kubernetes operator that reconciles annotated `LoadBalancer`
|
|
Services into RouterOS IPv4 `dst-nat` rules.
|
|
|
|
The controller reads the internal destination address from
|
|
`Service.status.loadBalancer.ingress[].ip`. In this cluster MetalLB publishes
|
|
and announces that address over L2. The design does not depend on MetalLB
|
|
specifically, so the same contract works with Cilium LB IPAM/L2 announcements.
|
|
|
|
## API contract
|
|
|
|
Routers are cluster-scoped because Services in several namespaces may use the
|
|
same physical router. Credentials are stored in a Kubernetes Secret and only a
|
|
reference is placed in the `MikroTikRouter` resource. Every router must
|
|
explicitly list the namespaces permitted to create Internet-facing rules; an
|
|
empty allowlist permits none.
|
|
|
|
For a Service with one public port, the compact annotations create one rule:
|
|
|
|
```yaml
|
|
metadata:
|
|
annotations:
|
|
mikrotik.brunner.ninja/router: newgw
|
|
mikrotik.brunner.ninja/service-port: game-udp
|
|
mikrotik.brunner.ninja/dst-port: "34197"
|
|
mikrotik.brunner.ninja/in-interface-list: WAN
|
|
```
|
|
|
|
`service-port` selects a Service port by name or number and is optional when the
|
|
Service has exactly one port. `dst-port` is the public port and defaults to the
|
|
selected Service port. For multiple public ports, use the `forwardings`
|
|
annotation with uniquely named entries:
|
|
|
|
```yaml
|
|
metadata:
|
|
annotations:
|
|
mikrotik.brunner.ninja/router: newgw
|
|
mikrotik.brunner.ninja/forwardings: >-
|
|
[
|
|
{"name":"http","servicePort":"web","dstAddressList":"WANs"},
|
|
{"name":"https","servicePort":"websecure","dstAddressList":"WANs"}
|
|
]
|
|
```
|
|
|
|
Each entry accepts `servicePort`, `dstPort`, `dstAddress`, `dstAddressList`,
|
|
`inInterface`, `inInterfaceList`, `srcAddress`, and `srcAddressList`. Protocol
|
|
is inherited from the selected Service port. A contiguous range can be mapped
|
|
without creating one RouterOS rule per port when every port in the range exists
|
|
on the Service and uses the same protocol:
|
|
|
|
```yaml
|
|
mikrotik.brunner.ninja/forwardings: >-
|
|
[{"name":"relay","servicePortRange":"49160-49200","dstAddressList":"WANs"}]
|
|
```
|
|
|
|
`dstPort` defaults to `servicePort` or `servicePortRange`; if overridden, its
|
|
range must contain the same number of ports. The generated RouterOS properties
|
|
are:
|
|
|
|
| Source | RouterOS property |
|
|
| --- | --- |
|
|
| annotation `dst-port` | `dst-port` |
|
|
| annotation `dst-address` | `dst-address` |
|
|
| annotation `dst-address-list` | `dst-address-list` |
|
|
| annotation `in-interface` | `in-interface` |
|
|
| annotation `in-interface-list` | `in-interface-list` |
|
|
| annotation `src-address` | `src-address` |
|
|
| annotation `src-address-list` | `src-address-list` |
|
|
| selected Service protocol | `protocol` |
|
|
| LoadBalancer status IPv4 | `to-addresses` |
|
|
| selected Service `port` | `to-ports` |
|
|
|
|
RouterOS calls the incoming/WAN matcher `in-interface`, not `src-interface`.
|
|
`src-address` refers to the remote client's address.
|
|
|
|
Single managed rules have comments of the form
|
|
`k8s-mikrotik/<service-namespace>/<service-name>`. Named forwardings append
|
|
`/<forwarding-name>`. The controller never modifies rules without the
|
|
`k8s-mikrotik/` prefix. It creates and updates desired rules, deletes stale
|
|
owned rules, removes duplicate owned rules, and preserves existing rules if the
|
|
corresponding Service temporarily has no LoadBalancer IP or invalid annotation
|
|
JSON.
|
|
|
|
Deleting a `MikroTikRouter` removes all of its managed rules before the CR is
|
|
deleted. If RouterOS cannot be reached, the cleanup finalizer deliberately keeps
|
|
the CR in `Terminating` rather than silently leaving public ports open.
|
|
|
|
## RouterOS preparation
|
|
|
|
The implementation uses the RouterOS v7 REST API. Enable `www-ssl`, configure a
|
|
certificate, and create a dedicated user rather than using `admin`. RouterOS
|
|
7.24 on `newgw` also requires the `api` policy for REST operations, so the
|
|
verified group policies are `read`, `write`, `api`, and `rest-api`:
|
|
|
|
```routeros
|
|
/user/group/add name=kubernetes-nat policy=read,write,api,rest-api
|
|
/user/add name=kubernetes-nat group=kubernetes-nat password=<strong-password> address=<operator-egress-cidr>
|
|
/ip/service/enable www-ssl
|
|
```
|
|
|
|
Restrict `www-ssl` with both `/ip service ... address=` and firewall rules to
|
|
the actual operator egress address or CIDR. Use a trusted certificate or put
|
|
its issuing CA in a Secret. `insecureSkipVerify` exists only to make the first
|
|
lab test easier. Plain HTTP sends the Basic Auth credentials without transport
|
|
encryption and should not be used.
|
|
|
|
The operator manages NAT only; it does not create RouterOS filter rules. The
|
|
router's forward-chain policy must already allow dst-natted traffic. New rules
|
|
are appended by RouterOS, so check that existing earlier NAT rules do not shadow
|
|
them. Rule positioning is a good candidate for the next revision after testing
|
|
against the actual router configuration.
|
|
|
|
## Install from the OCI Helm chart
|
|
|
|
Gitea CI publishes the operator image and chart to the local Quay registry. A
|
|
release tag such as `v0.1.0` produces these matched artifacts:
|
|
|
|
- `registry.brunner.ninja/feedc0de/mikrotik-operator:0.1.0`
|
|
- `oci://registry.brunner.ninja/feedc0de/charts/mikrotik-operator:0.1.0`
|
|
|
|
Install that release directly from the registry:
|
|
|
|
```sh
|
|
helm registry login registry.brunner.ninja
|
|
helm upgrade --install mikrotik-operator \
|
|
oci://registry.brunner.ninja/feedc0de/charts/mikrotik-operator \
|
|
--version 0.1.0 \
|
|
--namespace mikrotik-operator-system \
|
|
--create-namespace
|
|
```
|
|
|
|
The chart installs the CRD, Deployment, ServiceAccount, and RBAC. It
|
|
deliberately does not contain credential Secrets or `MikroTikRouter` instances.
|
|
Keep those as separate cluster configuration. Helm installs CRDs from `crds/`
|
|
but does not upgrade or delete them; apply a changed CRD explicitly before a
|
|
future chart upgrade:
|
|
|
|
```sh
|
|
kubectl apply -f config/crd/mikrotikrouters.yaml
|
|
```
|
|
|
|
For a local development build and chart install:
|
|
|
|
```sh
|
|
make IMAGE_TAG=dev push
|
|
make IMAGE_TAG=dev install
|
|
```
|
|
|
|
The original plain-manifest path remains available as `make install-manifests`.
|
|
|
|
## Gitea CI releases
|
|
|
|
The workflow in `.gitea/workflows/ci.yml` runs Go race tests and vet, verifies
|
|
that the standalone and chart CRDs are identical, lints and renders the chart,
|
|
and builds the container on every push and pull request.
|
|
|
|
Publishing runs only for `main` and `v*` tags:
|
|
|
|
- `main` publishes the image under the full commit SHA and a development chart
|
|
such as `0.0.0-dev.sha0123456789ab` whose `appVersion` is that SHA.
|
|
- `v0.1.0` publishes image tag `0.1.0` and chart version `0.1.0`.
|
|
|
|
The Deployment template defaults the image tag to `Chart.appVersion`. CI
|
|
renders the packaged chart and verifies the exact expected image reference
|
|
before pushing either artifact. Configure these Gitea repository Actions
|
|
secrets:
|
|
|
|
- `QUAY_USERNAME`
|
|
- `QUAY_TOKEN`
|
|
|
|
The Quay account or robot token needs push access to both
|
|
`feedc0de/mikrotik-operator` and `feedc0de/charts/mikrotik-operator`. Make the
|
|
repositories public if clusters should pull without an image pull Secret.
|
|
|
|
Create credentials without committing them. This keeps the password itself out
|
|
of shell history:
|
|
|
|
```sh
|
|
read -rsp 'Router password: ' ROUTER_PASSWORD
|
|
echo
|
|
kubectl -n mikrotik-operator-system create secret generic newgw-credentials \
|
|
--from-literal=username=kubernetes-nat \
|
|
--from-literal=password="${ROUTER_PASSWORD}"
|
|
unset ROUTER_PASSWORD
|
|
```
|
|
|
|
Edit and apply the examples:
|
|
|
|
```sh
|
|
kubectl apply -f examples/router.yaml
|
|
kubectl apply -f examples/service.yaml
|
|
kubectl get mikrotikrouter newgw
|
|
kubectl describe mikrotikrouter newgw
|
|
kubectl -n mikrotik-operator-system logs deployment/mikrotik-operator -f
|
|
```
|
|
|
|
For a CA-verified self-signed setup, create a Secret with a `ca.crt` key and
|
|
replace `insecureSkipVerify` in the router with:
|
|
|
|
```yaml
|
|
tls:
|
|
caSecretRef:
|
|
namespace: mikrotik-operator-system
|
|
name: router-ca
|
|
```
|
|
|
|
## Current limits and next revisions
|
|
|
|
- RouterOS destination NAT is IPv4-only; IPv6 LoadBalancer addresses are ignored.
|
|
- There is no admission webhook yet. Validation errors are reported in the
|
|
router's `Ready` condition while the previous rule is preserved.
|
|
- Metrics use controller-runtime's standard endpoint; domain-specific counters
|
|
and Kubernetes Events can be added once live reconciliation is proven.
|
|
|
|
The chart remains intentionally small while the API is `v1alpha1`; new router
|
|
and forwarding fields can evolve through `0.x` chart releases without treating
|
|
the current contract as stable.
|