Files
immich-sync/README.md
T

150 lines
9.0 KiB
Markdown
Raw Permalink Normal View History

2026-08-03 22:58:30 +02:00
# Immich Share Sync
2026-08-03 20:24:53 +02:00
2026-08-04 20:59:49 +02:00
An ephemeral Qt 6 web service that compares two Immich public shares by SHA-1 checksum and copies missing originals in either direction. It never deletes assets or keeps recovery state.
2026-08-03 22:58:30 +02:00
## How it works
- A `QHttpServer` serves an embedded vanilla HTML/CSS/JavaScript frontend.
- The same HTTP listener upgrades `/ws` with Qt WebSockets.
- Each short-lived WebSocket owns exactly one job, one `SyncSession`, two `ImmichClient` instances, and their `QNetworkAccessManager` objects.
- Link inspection tries Immich's share key and custom-slug authentication forms. Current password-protected shares use `POST /api/shared-links/login`; Qt's cookie jar retains the short-lived share cookie for that socket session.
- The service reads `QJsonValue`/`QJsonObject` responses and compares the base64-encoded SHA-1 checksums in Immich's shared-link response.
2026-08-04 20:59:49 +02:00
- Each missing original is downloaded into a `QTemporaryFile`, rewound, and used as the seekable `QIODevice` body of the destination multipart request. Only one asset per job is staged, and the file is removed after success, failure, or WebSocket abort.
2026-08-03 22:58:30 +02:00
- The browser sends one complete job request immediately after connecting. The service closes the WebSocket when that inspection or synchronization finishes.
- Closing the WebSocket early destroys the session and aborts every active network reply. No inspection ID, plan, credentials, or recovery state survives the connection.
- Aggregate counters are the only persistent state. Kubernetes replicas update a small locked JSON file on the shared
CephFS volume; it contains only numeric totals and never contains links, passwords, filenames, or job recovery data.
2026-08-03 22:58:30 +02:00
The destination upload deliberately omits Immich's `x-immich-checksum` optimization. That optimization returns duplicates before Immich associates an already-owned asset with the destination shared album. The normal duplicate path both detects the checksum and adds the existing asset to the share.
## Build and run
Requirements: CMake 3.28+, a C++23 compiler, and Qt 6.9+ with Core, Network, HttpServer, and WebSockets.
```sh
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build
ctest --test-dir build --output-on-failure
PORT=8090 ./build/immich-sync
```
Open `http://localhost:8090`.
After both share URLs are valid, starting a job updates the browser fragment in the form
`#left=<encoded-share-url>&right=<encoded-share-url>`. Reloading or bookmarking that URL restores both fields and
automatically starts a new inspection. Passwords are intentionally excluded from the fragment and must be entered
again for protected shares.
2026-08-03 22:58:30 +02:00
## WebSocket protocol
All messages are JSON. Each WebSocket accepts exactly one complete job request and closes after its terminal response. A client that does not send a request within 10 seconds is disconnected. Passwords are accepted only in client messages and are never placed in URLs or echoed in status responses.
Inspect two links:
```json
{
"type": "inspect",
"left": {"url": "https://one.example/share/key", "password": ""},
"right": {"url": "https://two.example/share/key", "password": ""}
}
```
The service emits granular `side-status` messages. Stages include `connecting`, `authenticating`, `password-required`, `ready`, and `error`. Once both sides resolve, an `inspection` response includes public share details, upload/download permissions, missing counts, and the allowed `left-to-right`, `right-to-left`, and `bidirectional` options. The service then closes the inspection socket.
To synchronize, open a new WebSocket and send both shares again along with one of the directions offered by inspection:
```json
{
"type": "sync",
"direction": "bidirectional",
"left": {"url": "https://one.example/share/key", "password": ""},
"right": {"url": "https://two.example/share/key", "password": ""}
}
```
2026-08-04 20:59:49 +02:00
The synchronization job repeats inspection so its checksum plan and permissions reflect current Immich state. Status messages include the inspection events followed by `sync-status`, `asset-status`, and throttled `asset-progress` events. After upload reaches 100%, a `processing` asset stage makes it clear that the destination Immich is still handling the request. The service closes the socket after completion; close it from the client to abort, delete the temporary asset, and discard the whole session immediately.
2026-08-03 22:58:30 +02:00
## Container and Kubernetes
```sh
docker build -t registry.brunner.ninja/feedc0de/immich-sync:latest .
docker run --rm -p 8090:8090 registry.brunner.ninja/feedc0de/immich-sync:latest
2026-10-18 13:02:46 +02:00
kubectl apply -f deploy/example.yaml
kubectl port-forward service/immich-sync 8090:80
2026-08-03 22:58:30 +02:00
```
The maintained Helm chart is published with the exact image revision in its
`appVersion`:
```sh
helm repo add brunner.ninja https://brunner.ninja/charts
helm repo update brunner.ninja
helm upgrade --install immich-sync brunner.ninja/immich-sync
```
2026-10-18 13:02:46 +02:00
`deploy/example.yaml` is a portable starting point for self-hosting. It creates one Deployment and a ClusterIP Service,
uses the public image, and does not require an image-pull secret. Add persistent storage, an Ingress, and monitoring to
match your cluster. Because the service can make outbound requests to user-supplied URLs, operators should monitor it
for SSRF and bandwidth abuse.
2026-08-03 22:58:30 +02:00
2026-10-18 13:02:46 +02:00
The author's complete
[`apps/immich-sync` GitOps values](https://code.brunner.ninja/feedc0de/homelab-gitops/src/branch/main/apps/immich-sync)
are public and show one Argo CD installation of the chart. They include homelab-specific CephFS storage,
2026-10-18 13:02:46 +02:00
multiple replicas, an Ingress hostname, and Prometheus Operator monitoring, so fork it and adjust those assumptions
before deploying it to another cluster. WebSockets need no sticky-session setting: after the HTTP upgrade, the
selected pod owns that connection until the short-lived job finishes or disconnects.
## Statistics and Prometheus
The homepage reads `/api/stats` every 30 seconds and displays shared totals for completed synchronization runs, all
successfully synchronized assets, photos, videos, and bytes. Counters begin at zero when statistics are first enabled.
2026-10-18 13:02:46 +02:00
Each replica exposes Prometheus text metrics at `/metrics`. A deployment can add a `ServiceMonitor` selecting the
`immich-sync` Service. Useful Grafana queries include:
```promql
sum(immich_sync_active_synchronizations)
sum(rate(immich_sync_assets_total{result="success"}[5m]))
sum by (media_type) (rate(immich_sync_asset_bytes_total[5m]))
max(immich_sync_lifetime_synchronizations_total)
max by (media_type) (immich_sync_lifetime_assets_total)
```
The lifetime series contains the same shared value on both replicas, so use `max`, not `sum`, for those metrics.
Per-process operational counters and active gauges should be summed across replicas.
2026-08-03 23:13:26 +02:00
## Continuous delivery
The Gitea Actions workflow in `.gitea/workflows/container.yml` builds and pushes
`registry.brunner.ninja/feedc0de/immich-sync`. Every release receives an immutable image tag containing its full Git
2026-10-19 17:31:53 +02:00
SHA. The same run packages the chart with that SHA as `appVersion`. The
repository tag follows the application's `CMakeLists.txt` version: `v0.2.0`
produces chart `0.2.0`; later chart commits produce unique versions such as
`0.2.0-1-g41627ab` through `git describe`.
2026-08-03 23:13:26 +02:00
2026-08-03 23:17:22 +02:00
The workflow deliberately loads the completed image into the Docker engine and pushes it with a separate plain
`docker push`. It does not request provenance attestations, SBOM manifests, or OCI referrers, retaining compatibility
with the self-hosted Quay registry and with Gitea runners whose default builder uses BuildKit's container driver.
2026-08-03 23:13:26 +02:00
The workflow expects these Gitea Actions repository secrets:
- `QUAY_USERNAME`: the complete Quay robot account name, including the `feedc0de+` prefix.
- `QUAY_TOKEN`: the robot account token.
- `PACKAGE_USERNAME` and `PACKAGE_TOKEN`: credentials that can publish to the public `feedc0de` Helm repository.
2026-10-18 13:02:46 +02:00
- `GITOPS_TOKEN`: a `write:repository` token belonging to a dedicated account with write access only to
`feedc0de/homelab-gitops`.
2026-08-03 23:13:26 +02:00
Give the robot account `Write` permission only on the `feedc0de/immich-sync` repository. Public image pulls do not use
these credentials and do not require a Gitea secret.
After a successful `main` build, the promotion job updates the Argo Application's exact chart `targetRevision` in
`homelab-gitops` and pushes a deployment commit. Argo CD observes that commit and reconciles Kubernetes. The workflow
has no kubeconfig and no direct Kubernetes access.
2026-08-04 20:59:49 +02:00
2026-08-03 22:58:30 +02:00
## Current scope
- Assets without a checksum are ignored by planning.
- Metadata processing on the destination is left to Immich after upload.
- Sidecar files and Live Photo pairing are not reconstructed yet; the primary original asset is transferred.
- Immich serves the edited rendition for edited assets through a shared link even when the original endpoint is requested. Such an asset can acquire a different destination checksum and may be proposed again on a later, stateless run. This cannot be fully resolved with current shared-link permissions alone.
- TLS certificates must be valid. The service never ignores SSL errors.