Add self-hosted RouterOS package feed and pinned updater redirect
This commit is contained in:
1 parent
37308f1636
commit
9217046b33
9 files changed
+207
-21
No files matched your search
@@ -121,8 +121,13 @@ jobs:
|
||||
for arch in arm arm64; do
|
||||
wifi=wifi-qcom-ac
|
||||
if [ "$arch" = arm64 ]; then wifi=wifi-qcom; fi
|
||||
update_host_args=""
|
||||
if [ "$ROUTEROS_VERSION" = 7.25.1 ]; then
|
||||
update_host_args="--update-host mikrotik.brunner.ninja"
|
||||
fi
|
||||
.venv/bin/python tools/build-release.py --version "$ROUTEROS_VERSION" \
|
||||
--arch "$arch" --wifi "$wifi" --keys "$keyfile" --output "build/release-$arch" \
|
||||
$update_host_args \
|
||||
--option-root "build/runtime-$arch/option-root" \
|
||||
--olsr-root "build/runtime-$arch/olsr-root"
|
||||
done
|
||||
|
||||
@@ -77,9 +77,10 @@ both Linux and Windows Netinstall binaries. You do not need Windows to patch the
|
||||
Windows executable.
|
||||
|
||||
The public site and RouterOS metadata feed are built from `service/` and `site/`;
|
||||
the Kubernetes chart is in `chart/mikrotik-olsr-site/`. The current feed serves
|
||||
channel versions and changelogs, while package-download protocol support and the
|
||||
RouterOS updater-host patch remain under investigation. See
|
||||
the Kubernetes chart is in `chart/mikrotik-olsr-site/`. The feed serves channel
|
||||
versions, changelogs, package lists, and signed NPK downloads. The 7.25.1
|
||||
ARM/ARM64 build can redirect its updater host to the project service; a live
|
||||
update transaction remains to be verified. See
|
||||
[update-server.md](docs/update-server.md).
|
||||
|
||||
Install prerequisites on Debian/Ubuntu:
|
||||
|
||||
@@ -89,6 +89,11 @@ plain-text RouterOS channel and changelog responses. Package download support
|
||||
and the updater host patch are not yet verified; see
|
||||
[update-server.md](update-server.md).
|
||||
|
||||
For 7.25.1 ARM and ARM64, the release build also redirects the updater host to
|
||||
`mikrotik.brunner.ninja`. This is still awaiting a live updater transaction;
|
||||
see [update-server.md](update-server.md). Other versions remain unchanged until
|
||||
their updater binaries are verified separately.
|
||||
|
||||
The service chart workflow publishes an immutable image and chart from `main`,
|
||||
then promotes the chart version to Argo CD. It needs `PACKAGE_USERNAME`,
|
||||
`PACKAGE_TOKEN`, `QUAY_USERNAME`, `QUAY_TOKEN`, and `GITOPS_TOKEN`; it receives no
|
||||
|
||||
+29
-8
@@ -37,14 +37,35 @@ when publishing a new package version. `/healthz` is used by the Kubernetes
|
||||
probes. `/api/releases` returns public Gitea release metadata for the landing
|
||||
page.
|
||||
|
||||
These endpoints are a working metadata feed, not yet a complete package updater.
|
||||
The project has not yet changed `sys2` to query this service, and the package
|
||||
download request and response format has not been confirmed. A version appearing
|
||||
in the update dialog must not be treated as proof that the signed NPK download
|
||||
will work. The test plan is to first patch the updater host in a test system NPK,
|
||||
capture its requests on a disposable RouterOS device, implement the matching
|
||||
package response and downloads, then test install and rollback before marking
|
||||
updates available by default.
|
||||
The service also serves `/routeros/<version>/packages.csv` in the observed
|
||||
`arch,name,size` format and streams the matching published, signed NPK at
|
||||
`/routeros/<version>/<package>-<version>-<arch>.npk`. It selects the newest
|
||||
published release revision for each package/architecture. The package list and
|
||||
download were exercised against a public 7.25.1 Gitea release locally.
|
||||
|
||||
For the exact official 7.25.1 ARM and ARM64 `sys2` builds, the NPK build can
|
||||
replace both references to `upgrade.mikrotik.com` with
|
||||
`mikrotik.brunner.ninja`:
|
||||
|
||||
```sh
|
||||
python tools/routeros.py npk routeros-7.25.1-arm.npk --arch arm \
|
||||
--patch-system --update-host mikrotik.brunner.ninja \
|
||||
--keys .keys/development.json --output build/routeros-7.25.1-arm-updates.npk
|
||||
```
|
||||
|
||||
The patch rejects any other `sys2` digest. Both ARM and ARM64 test NPKs built,
|
||||
self-verified, and contained only the project host in the updater binary.
|
||||
The release workflow applies it to 7.25.1 builds. **A live RouterOS update
|
||||
check/download from this patched binary remains unverified.** Do not treat a
|
||||
version appearing in the dialog as proof that installation works. The exact
|
||||
package request used by RouterOS and its behavior on the small-flash hAP ac2
|
||||
still need a device test before a release is advertised as a working updater.
|
||||
|
||||
The update dialog handles version changes for installed packages. It does not
|
||||
offer a removed optional package as a same-version installation. For that
|
||||
experiment, download the published OLSR NPK from this site into RouterOS Files
|
||||
and reboot, or upload the same file with Winbox Files; do not use the version
|
||||
check button to reinstall it.
|
||||
|
||||
## Hosting
|
||||
|
||||
|
||||
+80
-6
@@ -3,6 +3,7 @@
|
||||
from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
import sys
|
||||
from urllib.error import URLError
|
||||
from urllib.request import Request, urlopen
|
||||
@@ -15,8 +16,48 @@ if not SITE.is_dir(): # Running from the source tree: service/server.py.
|
||||
SITE = HERE.parent / "site"
|
||||
FEED = json.loads((SITE / "feed" / "channels.json").read_text())
|
||||
RELEASES_URL = (
|
||||
"https://code.brunner.ninja/api/v1/repos/feedc0de/mikrotik-olsr/releases?limit=10"
|
||||
"https://code.brunner.ninja/api/v1/repos/feedc0de/mikrotik-olsr/releases?limit=50"
|
||||
)
|
||||
PACKAGE_RE = re.compile(r"(routeros|option|olsr|wifi-qcom-ac|wifi-qcom)-"
|
||||
r"(7\.\d+(?:\.\d+)?)-(arm|arm64)\.npk\Z")
|
||||
VERSION_RE = re.compile(r"7\.\d+(?:\.\d+)?\Z")
|
||||
|
||||
|
||||
def release_assets(version, releases):
|
||||
"""Select the newest published revision for each architecture."""
|
||||
selected = {}
|
||||
for release in releases:
|
||||
tag = release.get("tag_name", "")
|
||||
match = re.fullmatch(rf"ros-{re.escape(version)}(?:-(arm|arm64))?-r(\d+)", tag)
|
||||
if not match or release.get("draft"):
|
||||
continue
|
||||
tag_arch, revision = match.group(1), int(match.group(2))
|
||||
for asset in release.get("assets", []):
|
||||
name = asset.get("name", "")
|
||||
package = PACKAGE_RE.fullmatch(name)
|
||||
if not package or package.group(2) != version:
|
||||
continue
|
||||
arch = package.group(3)
|
||||
if tag_arch and tag_arch != arch:
|
||||
continue
|
||||
url = asset.get("browser_download_url", "")
|
||||
parsed = urlsplit(url)
|
||||
if parsed.scheme != "https" or parsed.hostname != "code.brunner.ninja":
|
||||
continue
|
||||
if not parsed.path.startswith(
|
||||
f"/feedc0de/mikrotik-olsr/releases/download/{tag}/"
|
||||
) or not parsed.path.endswith("/" + name):
|
||||
continue
|
||||
key = (arch, package.group(1))
|
||||
if key not in selected or revision > selected[key][0]:
|
||||
selected[key] = (revision, asset)
|
||||
return {key: value[1] for key, value in selected.items()}
|
||||
|
||||
|
||||
def fetch_releases():
|
||||
req = Request(RELEASES_URL, headers={"User-Agent": "mikrotik-olsr-site"})
|
||||
with urlopen(req, timeout=8) as response:
|
||||
return json.load(response)
|
||||
|
||||
|
||||
class Handler(SimpleHTTPRequestHandler):
|
||||
@@ -41,14 +82,47 @@ class Handler(SimpleHTTPRequestHandler):
|
||||
return self.send_bytes(200, b"ok\n", "text/plain; charset=utf-8")
|
||||
if path == "/api/releases":
|
||||
try:
|
||||
req = Request(RELEASES_URL, headers={"User-Agent": "mikrotik-olsr-site"})
|
||||
with urlopen(req, timeout=8) as response:
|
||||
body = response.read()
|
||||
json.loads(body)
|
||||
body = json.dumps(fetch_releases()).encode()
|
||||
return self.send_bytes(200, body, "application/json; charset=utf-8")
|
||||
except (URLError, TimeoutError, ValueError) as exc:
|
||||
return self.send_bytes(502, (str(exc) + "\n").encode(), "text/plain; charset=utf-8")
|
||||
|
||||
package_path = re.fullmatch(r"/routeros/(7\.\d+(?:\.\d+)?)/([^/]+)", path)
|
||||
if package_path:
|
||||
version, filename = package_path.groups()
|
||||
if filename == "packages.csv":
|
||||
try:
|
||||
assets = release_assets(version, fetch_releases())
|
||||
except (URLError, TimeoutError, ValueError) as exc:
|
||||
return self.send_bytes(502, (str(exc) + "\n").encode(), "text/plain")
|
||||
rows = ["arch,name,size"]
|
||||
rows += [f"{arch},{name},{asset['size']}" for (arch, name), asset
|
||||
in sorted(assets.items())]
|
||||
return self.send_bytes(200, ("\n".join(rows) + "\n").encode(), "text/csv")
|
||||
package = PACKAGE_RE.fullmatch(filename)
|
||||
if package and package.group(2) == version:
|
||||
try:
|
||||
asset = release_assets(version, fetch_releases()).get(
|
||||
(package.group(3), package.group(1)))
|
||||
if asset is None:
|
||||
return self.send_bytes(404, b"Package not published\n", "text/plain")
|
||||
req = Request(asset["browser_download_url"],
|
||||
headers={"User-Agent": "mikrotik-olsr-site"})
|
||||
with urlopen(req, timeout=30) as response:
|
||||
length = int(response.headers.get("Content-Length", "0"))
|
||||
if length != asset["size"]:
|
||||
raise ValueError("Release asset size mismatch")
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "application/octet-stream")
|
||||
self.send_header("Content-Length", str(length))
|
||||
self.end_headers()
|
||||
if self.command != "HEAD":
|
||||
while block := response.read(65536):
|
||||
self.wfile.write(block)
|
||||
return
|
||||
except (URLError, TimeoutError, ValueError) as exc:
|
||||
return self.send_bytes(502, (str(exc) + "\n").encode(), "text/plain")
|
||||
|
||||
prefix = "/routeros/NEWESTa7."
|
||||
if path.startswith(prefix):
|
||||
channel = path[len(prefix):]
|
||||
@@ -61,7 +135,7 @@ class Handler(SimpleHTTPRequestHandler):
|
||||
marker = "/CHANGELOG"
|
||||
if path.startswith("/routeros/") and path.endswith(marker):
|
||||
version = path[len("/routeros/"):-len(marker)].strip("/")
|
||||
if version and all(ch.isdigit() or ch == "." for ch in version):
|
||||
if VERSION_RE.fullmatch(version):
|
||||
changelog = SITE / "feed" / "changelogs" / f"{version}.txt"
|
||||
if changelog.is_file():
|
||||
return self.send_bytes(200, changelog.read_bytes(), "text/plain; charset=utf-8")
|
||||
|
||||
@@ -4,6 +4,7 @@ import sys
|
||||
import time
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from service.server import release_assets
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
@@ -54,3 +55,15 @@ class UpdateServerTests(unittest.TestCase):
|
||||
status, _, body = self.get("/routeros/NEWESTa7.development")
|
||||
self.assertEqual(status, 200)
|
||||
self.assertIn(b"No version available", body)
|
||||
|
||||
def test_release_selection_keeps_latest_signed_package_per_architecture(self):
|
||||
def release(tag, size):
|
||||
name = "olsr-7.25.1-arm.npk"
|
||||
return {"tag_name": tag, "draft": False, "assets": [{
|
||||
"name": name, "size": size,
|
||||
"browser_download_url":
|
||||
f"https://code.brunner.ninja/feedc0de/mikrotik-olsr/releases/download/{tag}/{name}"
|
||||
}]}
|
||||
result = release_assets("7.25.1", [release("ros-7.25.1-arm-r4", 1),
|
||||
release("ros-7.25.1-arm-r5", 2)])
|
||||
self.assertEqual(result[("arm", "olsr")]["size"], 2)
|
||||
@@ -89,7 +89,11 @@ def build(args):
|
||||
for source, target in ((linux / 'netinstall-cli', output / 'netinstall-cli'),
|
||||
(windows, output / 'netinstall.exe')):
|
||||
run('netinstall', source, '--keys', args.keys, '--output', target)
|
||||
run('npk', system, '--arch', arch, '--patch-system', '--keys', args.keys, '--output', output / system.name)
|
||||
system_args = ['npk', system, '--arch', arch, '--patch-system', '--keys', args.keys,
|
||||
'--output', output / system.name]
|
||||
if args.update_host:
|
||||
system_args += ['--update-host', args.update_host]
|
||||
run(*system_args)
|
||||
if wifi:
|
||||
run('npk', wifi, '--arch', arch, '--keys', args.keys, '--output', output / wifi.name)
|
||||
payloads = {'option': args.option_root, 'olsr': args.olsr_root}
|
||||
@@ -157,6 +161,7 @@ def main():
|
||||
help='ARMv7 or ARM64/AArch64 system and add-on packages')
|
||||
p.add_argument('--wifi', choices=['wifi-qcom-ac', 'wifi-qcom', 'wireless'])
|
||||
p.add_argument('--keys', required=True)
|
||||
p.add_argument('--update-host', help='Redirect a verified 7.25.1 system updater to the project feed')
|
||||
p.add_argument('--output', required=True)
|
||||
p.add_argument('--option-root', help='Audited runtime payload directory')
|
||||
p.add_argument('--olsr-root', help='Built OLSR payload directory')
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Redirect the verified 7.25.1 ARM/ARM64 RouterOS updater host.
|
||||
|
||||
The two references are ELF32 absolute literal words. The second string has
|
||||
four trailing NUL bytes, enough for mikrotik.brunner.ninja without moving a
|
||||
section. The first reference is redirected to that second string.
|
||||
"""
|
||||
import argparse
|
||||
import hashlib
|
||||
from pathlib import Path
|
||||
import struct
|
||||
|
||||
|
||||
LAYOUTS = {
|
||||
"6fa7c71d0ff6014d3d65f3e696893bd87a4b3635b9a06d51de8619524aee9512":
|
||||
(0x37ce9, 0xfed4, 0x4200c, 0x359c4),
|
||||
"237ccca17a91d06a52b91d24b4cda60b2a5aca83a47fa1c38719b8b2bba47bea":
|
||||
(0x384bf, 0x118a8, 0x4286c, 0x35b1c),
|
||||
}
|
||||
OLD = b"upgrade.mikrotik.com\0"
|
||||
SLOT_SIZE = 24
|
||||
|
||||
|
||||
def patch(data: bytes, host: str) -> bytes:
|
||||
if host != "mikrotik.brunner.ninja":
|
||||
raise ValueError("Only the verified project hostname is supported")
|
||||
layout = LAYOUTS.get(hashlib.sha256(data).hexdigest())
|
||||
if layout is None:
|
||||
raise ValueError("Unsupported sys2 build; updater host was not changed")
|
||||
first, first_ref, second, second_ref = layout
|
||||
if (data[first:first + len(OLD)] != OLD or data[second:second + SLOT_SIZE] !=
|
||||
OLD + b"\0" * (SLOT_SIZE - len(OLD))):
|
||||
raise ValueError("Unexpected updater string layout")
|
||||
if (data[first_ref:first_ref + 4] != struct.pack("<I", first + 0x10000) or
|
||||
data[second_ref:second_ref + 4] != struct.pack("<I", second + 0x10000)):
|
||||
raise ValueError("Unexpected updater reference layout")
|
||||
result = bytearray(data)
|
||||
result[first:first + len(OLD)] = b"\0" * len(OLD)
|
||||
result[first_ref:first_ref + 4] = struct.pack("<I", second + 0x10000)
|
||||
replacement = host.encode("ascii") + b"\0"
|
||||
result[second:second + SLOT_SIZE] = replacement.ljust(SLOT_SIZE, b"\0")
|
||||
if OLD in result or result.count(replacement) != 1:
|
||||
raise ValueError("Updater host replacement did not verify")
|
||||
return bytes(result)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("source", type=Path)
|
||||
parser.add_argument("output", type=Path)
|
||||
parser.add_argument("--host", default="mikrotik.brunner.ninja")
|
||||
args = parser.parse_args()
|
||||
args.output.write_bytes(patch(args.source.read_bytes(), args.host))
|
||||
print(f"Patched updater host in {args.output}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+7
-3
@@ -17,6 +17,7 @@ from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey,
|
||||
from npk import NovaPackage, NpkPartID, NpkFileContainer
|
||||
from toyecc import ECPrivateKey, getcurvebyname
|
||||
import keypatch
|
||||
import patch_update_host
|
||||
|
||||
OFFICIAL_NPK_KEY = 'c293ced638a2a33c681fc8de98ee26c54eadc5390c2dfce197d35c83c416cf59'
|
||||
|
||||
@@ -84,16 +85,18 @@ def pack_squashfs(source, output):
|
||||
'-b', '256k', '-all-root', '-no-xattrs', '-no-progress', '-quiet'], check=True)
|
||||
|
||||
|
||||
def patch_tree(root, mapping):
|
||||
def patch_tree(root, mapping, update_host=None):
|
||||
hits = []
|
||||
for file in sorted(Path(root).rglob('*')):
|
||||
if file.is_symlink() or not file.is_file():
|
||||
continue
|
||||
data = file.read_bytes()
|
||||
if update_host and file.relative_to(root).as_posix() == 'nova/bin/sys2':
|
||||
data = patch_update_host.patch(data, update_host)
|
||||
updated = data
|
||||
for old, new in mapping.items():
|
||||
updated = keypatch.replace_key(old, new, updated, str(file.relative_to(root)))
|
||||
if updated != data:
|
||||
if updated != file.read_bytes():
|
||||
file.write_bytes(updated)
|
||||
hits.append(str(file.relative_to(root)))
|
||||
if not hits:
|
||||
@@ -142,7 +145,7 @@ def npk_build(args):
|
||||
tree = work / 'root'
|
||||
image.write_bytes(part.data)
|
||||
subprocess.run(['unsquashfs', '-no-progress', '-d', str(tree), str(image)], check=True, stdout=subprocess.DEVNULL)
|
||||
print('Patched system files:', ', '.join(patch_tree(tree, mapping)))
|
||||
print('Patched system files:', ', '.join(patch_tree(tree, mapping, args.update_host)))
|
||||
pack_squashfs(tree, work / 'output.sfs')
|
||||
part.data = (work / 'output.sfs').read_bytes()
|
||||
changed_system = True
|
||||
@@ -206,6 +209,7 @@ def main():
|
||||
if name == 'npk':
|
||||
p.add_argument('--arch', choices=['arm', 'arm64', 'x86', 'mipsbe', 'mmips', 'smips', 'ppc'], required=True)
|
||||
p.add_argument('--patch-system', action='store_true')
|
||||
p.add_argument('--update-host', help='Verified 7.25.1 ARM/ARM64 updater redirect')
|
||||
p.set_defaults(func=function)
|
||||
args = parser.parse_args()
|
||||
args.func(args)
|
||||
|
||||
Reference in new issue
Block a user