265 lines
12 KiB
Markdown
265 lines
12 KiB
Markdown
# telegraf-bcachefs-input
|
||
|
||
Telegraf bcachefs monitoring
|
||
|
||
## Installation
|
||
|
||
Run once from the checkout:
|
||
|
||
```sh
|
||
sudo ./install.sh
|
||
```
|
||
|
||
This links `/etc/telegraf/bcachefs*.sh` to this checkout. Existing files are
|
||
preserved in a printed `/etc/telegraf/bcachefs-backup.*` directory. Re-running
|
||
is safe; unchanged links are skipped. You can pass a different destination
|
||
directory as the first argument. Keep the checkout at the same path and readable
|
||
and executable by the Telegraf service. Edits take effect on the next collection;
|
||
no copy or service restart is needed. To roll back, replace the links with the
|
||
files in the backup directory (remove any newly installed links separately).
|
||
|
||
On the original host, Telegraf runs as root with `ProtectHome=no`. Thus edits
|
||
here become code executed as root. On another host, check service permissions
|
||
and sandbox settings before installing into a home directory.
|
||
|
||
The original host loads its configuration from InfluxDB using `TELEGRAF_OPTS`
|
||
and `INFLUX_TOKEN` in `/etc/default/telegraf`. Keep these credentials outside Git.
|
||
The existing hosted commands can remain unchanged with this installation;
|
||
a new systemd environment variable is unnecessary.
|
||
|
||
## Collectors
|
||
|
||
The scripts retain the deployed host's `/mnt` mount point and, in several cases,
|
||
its filesystem UUID. Adapt those values before using them on another host.
|
||
`bcachefs_reconcile.sh` already accepts `FSPATH` (or a positional mount path) and
|
||
`BCACHEFS_UUID` overrides.
|
||
|
||
- `bcachefs.sh`: filesystem usage; imported version selects usage sections explicitly.
|
||
- `bcachefs_fs_top.sh`: parses the collector's `/fs_top` endpoint (`/s` format).
|
||
- `bcachefs_options.sh`: sysfs options.
|
||
- `bcachefs_perf.sh`: collector's `/perf_json` endpoint.
|
||
- `bcachefs_reconcile.sh`: reconcile status.
|
||
- `bcachefs_snapshot_delete_status.sh`: snapshot deletion progress.
|
||
- `bcachefs_time_stats.sh`: sysfs time statistics.
|
||
- `bcachefs_timestats.sh`: JSON time statistics from bcachefs-tools; different measurement/schema.
|
||
- `bcachefs_device_time_stats.sh`: compatibility symlink to `bcachefs_time_stats.sh`;
|
||
the old workspace file was identical and did not collect extra device metrics.
|
||
Do not configure both aliases, as that would duplicate collection.
|
||
|
||
The old `_not_finished` human-readable prototype was removed; Git retains its
|
||
history. `_not_maintained_anymore` remains as an archive.
|
||
|
||
Add the desired inputs to your Telegraf TOML configuration (local or hosted):
|
||
|
||
```toml
|
||
[[inputs.exec]]
|
||
commands = ["/etc/telegraf/bcachefs.sh"]
|
||
data_format = "influx"
|
||
[[inputs.exec]]
|
||
commands = ["/etc/telegraf/bcachefs_fs_top.sh"]
|
||
data_format = "influx"
|
||
[[inputs.exec]]
|
||
commands = ["/etc/telegraf/bcachefs_options.sh"]
|
||
data_format = "influx"
|
||
[[inputs.exec]]
|
||
commands = ["/etc/telegraf/bcachefs_perf.sh"]
|
||
data_format = "influx"
|
||
[[inputs.exec]]
|
||
commands = ["/etc/telegraf/bcachefs_time_stats.sh"]
|
||
data_format = "influx"
|
||
```
|
||
|
||
|
||
|
||
|
||
|
||
## Support reports and pastes
|
||
|
||
Run `./kent_report.sh` as your regular user (with kubectl access) from the directory
|
||
where you want a new `kent-report-DATE-RANDOM/` directory. It runs the existing
|
||
lengthy diagnostics and uploads a completed report, printing its public cover-page
|
||
URL for manually sharing on IRC. It does not send an IRC message.
|
||
|
||
Each report includes:
|
||
|
||
- `index.html`: a searchable cover page with collapsible command outputs. Output
|
||
text is fetched only when expanded and inserted as plain text, never HTML.
|
||
- `outputs/`: a public file listing with descriptive, numbered `.txt` files for
|
||
each command, including separate dmesg, timestats, reconcile and sysfs files.
|
||
Direct links work with browsers, curl and wget, including without JavaScript.
|
||
- `kent_report.log`: the complete combined text report with a command index.
|
||
- `manifest.json`: command lines, filenames, exit statuses, durations and sizes.
|
||
|
||
Both stdout and stderr are captured. A failed command does not abort the rest.
|
||
Commands time out after 300 seconds by default; the report records timeouts and
|
||
exit statuses. The original Arch packages, source checkout, `/mnt`, `/dev/sda`
|
||
and encryption-file paths are retained. Raw `trace.dat` stays in the local report
|
||
directory; only the rendered report files are uploaded. Report directories must
|
||
be new (or empty); existing outputs are never overwritten.
|
||
|
||
```sh
|
||
./kent_report.sh # collect and upload
|
||
./kent_report.sh --no-upload # collect locally for review
|
||
./kent_report.sh --output ./my-report --timeout 600 --no-upload
|
||
source ./paste_log.sh
|
||
paste_directory ./my-report # upload an already collected report
|
||
```
|
||
|
||
The upload is copied to a temporary remote directory and renamed after success,
|
||
so the printed URL points at a complete report. Apache configuration changes are
|
||
not required; the report supplies its own static directory listing.
|
||
|
||
`paste_log.sh` still provides `paste_log path/to/log` for single-file uploads,
|
||
and can be executed directly as `./paste_log.sh path/to/log`. To expose both
|
||
helpers in interactive Bash:
|
||
|
||
```bash
|
||
source "$HOME/telegraf-bcachefs-input/paste_log.sh"
|
||
```
|
||
|
||
`kent_report.sh` sources this same file relative to its own location. Uploads use
|
||
the current kubectl context/namespace and require exactly one `brunner-ninja*`
|
||
pod. Failed uploads return an error without printing a URL. Helper errors do not
|
||
exit an interactive shell.
|
||
|
||
## Home Assistant switches
|
||
|
||
`bcachefs_mqtt.py` publishes MQTT discovery for **Reconcile** and **Copy GC**,
|
||
grouped under the host's Bcachefs device. On the original host the entities are
|
||
`switch.bcachefs_reconcile_enabled` and `switch.bcachefs_copygc_enabled`.
|
||
|
||
The bridge reads `reconcile_enabled` and `copygc_enabled` from sysfs every two
|
||
seconds, including changes made from the command line. Home Assistant commands
|
||
write only these two options and then read back actual state; the UI is not
|
||
optimistic. Discovery and state messages are retained, commands are not, and
|
||
retained commands are rejected. No shell commands are executed from MQTT input.
|
||
|
||
Missing or unreadable options make their switches unavailable on the next poll.
|
||
A clean service stop publishes offline immediately; MQTT's retained last will
|
||
handles a crash or connection loss (network-loss detection can take about 23
|
||
seconds with the 15-second keepalive). On remount, reconnect, or a Home Assistant
|
||
birth message the bridge republishes state. It never restores a previous option
|
||
value automatically: the mounted filesystem is always the source of truth.
|
||
|
||
Install from a stable checkout path without whitespace:
|
||
|
||
```sh
|
||
python3 -m venv .venv
|
||
.venv/bin/pip install -r requirements-mqtt.txt
|
||
sudo ./install-mqtt.sh
|
||
```
|
||
|
||
The installer creates `/etc/default/bcachefs-mqtt` from the example only if it
|
||
is absent, installs the systemd service, and enables/restarts it. Configure
|
||
`MQTT_HOST`, `BCACHEFS_UUID`, and optional credentials/TLS there, outside Git.
|
||
It defaults to the original host's broker and filesystem UUID. The service runs
|
||
as root to write sysfs, with its filesystem read-only except `/sys/fs/bcachefs`.
|
||
It starts at boot even if the filesystem is not yet mounted.
|
||
|
||
```sh
|
||
sudo systemctl restart bcachefs-mqtt # after bridge/config edits
|
||
systemctl status bcachefs-mqtt
|
||
journalctl -u bcachefs-mqtt
|
||
```
|
||
|
||
MQTT topics are `bcachefs/UUID/OPTION/state` and `/set`, with `0`/`1` payloads.
|
||
Discovery uses `homeassistant/switch/bcachefs_UUID/OPTION/config`. To uninstall,
|
||
stop/disable the service and clear its two retained discovery config topics.
|
||
|
||
The integration follows [Home Assistant MQTT switch discovery and availability](https://www.home-assistant.io/integrations/switch.mqtt/)
|
||
and uses the [Paho MQTT Python client](https://eclipse.dev/paho/clients/python/docs/).
|
||
|
||
## Tests
|
||
|
||
```sh
|
||
python3 tests/test_helpers.py
|
||
.venv/bin/python -m unittest discover -s tests -p 'test_mqtt_report.py' -v
|
||
```
|
||
|
||
Tests use temporary option files and stub uploads; they do not toggle a real
|
||
filesystem or publish a real diagnostic report.
|
||
|
||
## Building the performance collector
|
||
|
||
Requires Qt 6 Core, Network, WebSockets and HttpServer development packages:
|
||
|
||
```sh
|
||
mkdir -p /tmp/bcachefs-collector-build
|
||
cd /tmp/bcachefs-collector-build
|
||
qmake6 /path/to/telegraf-bcachefs-input/perf-collector/perf-collector.pro
|
||
make
|
||
```
|
||
|
||
The collector listens on localhost:8000 and launches `bcachefs fs top /mnt`
|
||
and `sudo perf trace`. Building does not deploy or restart it.
|
||
|
||
## Reconcile history dashboard
|
||
|
||
The MQTT bridge also discovers 16 pending-work sensors:
|
||
`sensor.bcachefs_reconcile_{replicas,checksum,erasure_code,compression,target,high_priority,pending,stripes}_{data,metadata}`.
|
||
It runs the existing `bcachefs_reconcile.sh` collector every 30 seconds, reusing
|
||
Telegraf's parser and publishing byte counts only. No speed or ETA sensors are
|
||
created. Missing fields are unknown rather than fabricated zeros. Collection
|
||
failures/unmounts make measurements unavailable, and stale sensors expire after
|
||
90 seconds. The existing switch controls are unchanged. Set `FSPATH` in the
|
||
service environment for a mount other than `/mnt`.
|
||
|
||
These sensors use `state_class: measurement` and `force_update: true`: Recorder
|
||
stores sampling timestamps even when bcachefs's rounded values stay unchanged.
|
||
This matters for flat periods and pauses. Keep at least seven days of Recorder
|
||
history and do not exclude these entities if using the seven-day calculation.
|
||
History begins when the sensors are installed; no InfluxDB data is imported.
|
||
|
||
`homeassistant/bcachefs-reconcile-card.mjs` is a custom dashboard card using the
|
||
already-installed ApexCharts Card for graphs and Home Assistant's authenticated
|
||
history WebSocket API for estimates. The host's existing
|
||
`/dashboard-bcachefs/0` dashboard includes this card with its Reconcile and Copy GC switches integrated
|
||
above the history controls. The embedded native Home Assistant tiles preserve
|
||
live state, unavailable handling, and the entity details dialog.
|
||
|
||
- One selector applies 3h, 6h, 12h, 24h, 2d or 7d to both graphs and calculations.
|
||
- Another selector chooses `target_data` or `erasure_code_data` for the estimate.
|
||
- Average net progress = (first pending bytes − last pending bytes) / elapsed
|
||
seconds between those actual samples in the selected window.
|
||
- ETA = last pending bytes / positive average progress. Flat or growing work
|
||
has no completion estimate; zero pending bytes is complete.
|
||
- Estimates require at least one minute of valid history, show actual coverage
|
||
for partial windows, and flag unavailable periods. Wall-clock elapsed time
|
||
includes pauses. This is an extrapolation, not a prediction of future load.
|
||
- Charts aggregate for display; estimates use raw recorded endpoints, not graph
|
||
averages. No artificial state from before the window is used as an endpoint.
|
||
- Both charts and estimates refresh every minute. Selecting a range recalculates
|
||
immediately, and stale asynchronous results cannot replace a newer selection.
|
||
|
||
To install or update the card on another Home Assistant instance:
|
||
|
||
1. Install ApexCharts Card (the original host already has it through HACS).
|
||
2. Copy `homeassistant/bcachefs-reconcile-card.mjs` into
|
||
`/config/www/bcachefs/bcachefs-reconcile-card.mjs`.
|
||
3. Register `/local/bcachefs/bcachefs-reconcile-card.mjs?v=VERSION` as a JavaScript
|
||
module resource. Change `VERSION` after edits to invalidate browser caches.
|
||
4. Add this card in the dashboard's YAML editor (sensor IDs can be adjusted with
|
||
`entity_prefix`):
|
||
|
||
```yaml
|
||
type: custom:bcachefs-reconcile-card
|
||
entity_prefix: sensor.bcachefs_reconcile_
|
||
hours: 6
|
||
grid_options:
|
||
columns: full
|
||
rows: auto
|
||
```
|
||
|
||
The deployed dashboard was updated through Home Assistant's WebSocket API,
|
||
preserving its existing cards. Before each update its configuration was backed
|
||
up under `/config/bcachefs-dashboard-backups/`. No Home Assistant restart or
|
||
Grafana changes were required. The source for the card and collector lives in
|
||
this repository; Kubernetes continues to own application deployment only.
|
||
|
||
Additional checks:
|
||
|
||
```sh
|
||
node tests/test_reconcile_history.mjs
|
||
.venv/bin/python -m unittest discover -s tests -p 'test_reconcile_metrics.py' -v
|
||
```
|