telegraf-bcachefs-input/README.md

telegraf-bcachefs-input

Telegraf bcachefs monitoring

Installation

Run once from the checkout:

sudo ./install.sh

This links /etc/telegraf/bcachefs*.sh to this checkout. Existing files are preserved in a printed /etc/telegraf/bcachefs-backup.* directory. Re-running is safe; unchanged links are skipped. You can pass a different destination directory as the first argument. Keep the checkout at the same path and readable and executable by the Telegraf service. Edits take effect on the next collection; no copy or service restart is needed. To roll back, replace the links with the files in the backup directory (remove any newly installed links separately).

On the original host, Telegraf runs as root with ProtectHome=no. Thus edits here become code executed as root. On another host, check service permissions and sandbox settings before installing into a home directory.

The original host loads its configuration from InfluxDB using TELEGRAF_OPTS and INFLUX_TOKEN in /etc/default/telegraf. Keep these credentials outside Git. The existing hosted commands can remain unchanged with this installation; a new systemd environment variable is unnecessary.

Collectors

The scripts retain the deployed host's /mnt mount point and, in several cases, its filesystem UUID. Adapt those values before using them on another host. bcachefs_reconcile.sh already accepts FSPATH (or a positional mount path) and BCACHEFS_UUID overrides.

• bcachefs.sh: filesystem usage; imported version selects usage sections explicitly.
• bcachefs_fs_top.sh: parses the collector's /fs_top endpoint (/s format).
• bcachefs_options.sh: sysfs options.
• bcachefs_perf.sh: collector's /perf_json endpoint.
• bcachefs_reconcile.sh: reconcile status.
• bcachefs_snapshot_delete_status.sh: snapshot deletion progress.
• bcachefs_time_stats.sh: sysfs time statistics.
• bcachefs_timestats.sh: JSON time statistics from bcachefs-tools; different measurement/schema.
• bcachefs_device_time_stats.sh: compatibility symlink to bcachefs_time_stats.sh; the old workspace file was identical and did not collect extra device metrics. Do not configure both aliases, as that would duplicate collection.

The old _not_finished human-readable prototype was removed; Git retains its history. _not_maintained_anymore remains as an archive.

Add the desired inputs to your Telegraf TOML configuration (local or hosted):

[[inputs.exec]]
  commands = ["/etc/telegraf/bcachefs.sh"]
  data_format = "influx"
[[inputs.exec]]
  commands = ["/etc/telegraf/bcachefs_fs_top.sh"]
  data_format = "influx"
[[inputs.exec]]
  commands = ["/etc/telegraf/bcachefs_options.sh"]
  data_format = "influx"
[[inputs.exec]]
  commands = ["/etc/telegraf/bcachefs_perf.sh"]
  data_format = "influx"
[[inputs.exec]]
  commands = ["/etc/telegraf/bcachefs_time_stats.sh"]
  data_format = "influx"

Support reports and pastes

Run ./kent_report.sh as your regular user (with kubectl access) from the directory where you want a new kent-report-DATE-RANDOM/ directory. It runs the existing lengthy diagnostics and uploads a completed report, printing its public cover-page URL for manually sharing on IRC. It does not send an IRC message.

Each report includes:

• index.html: a searchable cover page with collapsible command outputs. Output text is fetched only when expanded and inserted as plain text, never HTML.
• outputs/: a public file listing with descriptive, numbered .txt files for each command, including separate dmesg, timestats, reconcile and sysfs files. Direct links work with browsers, curl and wget, including without JavaScript.
• kent_report.log: the complete combined text report with a command index.
• manifest.json: command lines, filenames, exit statuses, durations and sizes.

Both stdout and stderr are captured. A failed command does not abort the rest. Commands time out after 300 seconds by default; the report records timeouts and exit statuses. The original Arch packages, source checkout, /mnt, /dev/sda and encryption-file paths are retained. Raw trace.dat stays in the local report directory; only the rendered report files are uploaded. Report directories must be new (or empty); existing outputs are never overwritten.

./kent_report.sh                         # collect and upload
./kent_report.sh --no-upload             # collect locally for review
./kent_report.sh --output ./my-report --timeout 600 --no-upload
source ./paste_log.sh
paste_directory ./my-report              # upload an already collected report

The upload is copied to a temporary remote directory and renamed after success, so the printed URL points at a complete report. Apache configuration changes are not required; the report supplies its own static directory listing.

paste_log.sh still provides paste_log path/to/log for single-file uploads, and can be executed directly as ./paste_log.sh path/to/log. To expose both helpers in interactive Bash:

source "$HOME/telegraf-bcachefs-input/paste_log.sh"

kent_report.sh sources this same file relative to its own location. Uploads use the current kubectl context/namespace and require exactly one brunner-ninja* pod. Failed uploads return an error without printing a URL. Helper errors do not exit an interactive shell.

Home Assistant switches

bcachefs_mqtt.py publishes MQTT discovery for Reconcile and Copy GC, grouped under the host's Bcachefs device. On the original host the entities are switch.bcachefs_reconcile_enabled and switch.bcachefs_copygc_enabled.

The bridge reads reconcile_enabled and copygc_enabled from sysfs every two seconds, including changes made from the command line. Home Assistant commands write only these two options and then read back actual state; the UI is not optimistic. Discovery and state messages are retained, commands are not, and retained commands are rejected. No shell commands are executed from MQTT input.

Missing or unreadable options make their switches unavailable on the next poll. A clean service stop publishes offline immediately; MQTT's retained last will handles a crash or connection loss (network-loss detection can take about 23 seconds with the 15-second keepalive). On remount, reconnect, or a Home Assistant birth message the bridge republishes state. It never restores a previous option value automatically: the mounted filesystem is always the source of truth.

Install from a stable checkout path without whitespace:

python3 -m venv .venv
.venv/bin/pip install -r requirements-mqtt.txt
sudo ./install-mqtt.sh

The installer creates /etc/default/bcachefs-mqtt from the example only if it is absent, installs the systemd service, and enables/restarts it. Configure MQTT_HOST, BCACHEFS_UUID, and optional credentials/TLS there, outside Git. It defaults to the original host's broker and filesystem UUID. The service runs as root to write sysfs, with its filesystem read-only except /sys/fs/bcachefs. It starts at boot even if the filesystem is not yet mounted.

sudo systemctl restart bcachefs-mqtt     # after bridge/config edits
systemctl status bcachefs-mqtt
journalctl -u bcachefs-mqtt

MQTT topics are bcachefs/UUID/OPTION/state and /set, with 0/1 payloads. Discovery uses homeassistant/switch/bcachefs_UUID/OPTION/config. To uninstall, stop/disable the service and clear its two retained discovery config topics.

The integration follows Home Assistant MQTT switch discovery and availability and uses the Paho MQTT Python client.

Tests

python3 tests/test_helpers.py
.venv/bin/python -m unittest discover -s tests -p 'test_mqtt_report.py' -v

Tests use temporary option files and stub uploads; they do not toggle a real filesystem or publish a real diagnostic report.

Building the performance collector

Requires Qt 6 Core, Network, WebSockets and HttpServer development packages:

mkdir -p /tmp/bcachefs-collector-build
cd /tmp/bcachefs-collector-build
qmake6 /path/to/telegraf-bcachefs-input/perf-collector/perf-collector.pro
make

The collector listens on localhost:8000 and launches bcachefs fs top /mnt and sudo perf trace. Building does not deploy or restart it.

Reconcile history dashboard

The MQTT bridge also discovers 16 pending-work sensors: sensor.bcachefs_reconcile_{replicas,checksum,erasure_code,compression,target,high_priority,pending,stripes}_{data,metadata}. It runs the existing bcachefs_reconcile.sh collector every 30 seconds, reusing Telegraf's parser and publishing byte counts only. No speed or ETA sensors are created. Missing fields are unknown rather than fabricated zeros. Collection failures/unmounts make measurements unavailable, and stale sensors expire after 90 seconds. The existing switch controls are unchanged. Set FSPATH in the service environment for a mount other than /mnt.

These sensors use state_class: measurement and force_update: true: Recorder stores sampling timestamps even when bcachefs's rounded values stay unchanged. This matters for flat periods and pauses. Keep at least seven days of Recorder history and do not exclude these entities if using the seven-day calculation. History begins when the sensors are installed; no InfluxDB data is imported.

homeassistant/bcachefs-reconcile-card.mjs is a custom dashboard card using the already-installed ApexCharts Card for graphs and Home Assistant's authenticated history WebSocket API for estimates. The host's existing /dashboard-bcachefs/0 dashboard includes this card with its Reconcile and Copy GC switches integrated above the history controls. The embedded native Home Assistant tiles preserve live state, unavailable handling, and the entity details dialog.

• One selector applies 3h, 6h, 12h, 24h, 2d or 7d to both graphs and calculations.
• Another selector chooses target_data or erasure_code_data for the estimate.
• Average net progress = (first pending bytes − last pending bytes) / elapsed seconds between those actual samples in the selected window.
• ETA = last pending bytes / positive average progress. Flat or growing work has no completion estimate; zero pending bytes is complete.
• Estimates require at least one minute of valid history, show actual coverage for partial windows, and flag unavailable periods. Wall-clock elapsed time includes pauses. This is an extrapolation, not a prediction of future load.
• Charts aggregate for display; estimates use raw recorded endpoints, not graph averages. No artificial state from before the window is used as an endpoint.
• Both charts and estimates refresh every minute. Selecting a range recalculates immediately, and stale asynchronous results cannot replace a newer selection.

To install or update the card on another Home Assistant instance:

1. Install ApexCharts Card (the original host already has it through HACS).
2. Copy homeassistant/bcachefs-reconcile-card.mjs into /config/www/bcachefs/bcachefs-reconcile-card.mjs.
3. Register /local/bcachefs/bcachefs-reconcile-card.mjs?v=VERSION as a JavaScript module resource. Change VERSION after edits to invalidate browser caches.
4. Add this card in the dashboard's YAML editor (sensor IDs can be adjusted with entity_prefix):

type: custom:bcachefs-reconcile-card
entity_prefix: sensor.bcachefs_reconcile_
hours: 6
grid_options:
  columns: full
  rows: auto

The deployed dashboard was updated through Home Assistant's WebSocket API, preserving its existing cards. Before each update its configuration was backed up under /config/bcachefs-dashboard-backups/. No Home Assistant restart or Grafana changes were required. The source for the card and collector lives in this repository; Kubernetes continues to own application deployment only.

Additional checks:

node tests/test_reconcile_history.mjs
.venv/bin/python -m unittest discover -s tests -p 'test_reconcile_metrics.py' -v