mirror of
https://github.com/home-assistant/core.git
synced 2025-07-29 18:28:14 +02:00
Merge feature branch with backup changes to dev (#132954)
* Reapply "Make WS command backup/generate send events" (#131530) This reverts commit9b8316df3f. * MVP implementation of Backup sync agents (#126122) * init sync agent * add syncing * root import * rename list to info and add sync state * Add base backup class * Revert unneded change * adjust tests * move to kitchen_sink * split * move * Adjustments * Adjustment * update * Tests * Test unknown agent * adjust * Adjust for different test environments * Change /info WS to contain a dictinary * reorder * Add websocket command to trigger sync from the supervisor * cleanup * Make mypy happier --------- Co-authored-by: Erik <erik@montnemery.com> * Make BackupSyncMetadata model a dataclass (#130555) Make backup BackupSyncMetadata model a dataclass * Rename backup sync agent to backup agent (#130575) * Rename sync agent module to agent * Rename BackupSyncAgent to BackupAgent * Fix test typo * Rename async_get_backup_sync_agents to async_get_backup_agents * Rename and clean up remaining sync things * Update kitchen sink * Apply suggestions from code review * Update test_manager.py --------- Co-authored-by: Erik Montnemery <erik@montnemery.com> * Add additional options to WS command backup/generate (#130530) * Add additional options to WS command backup/generate * Improve test * Improve test * Align parameter names in backup/agents/* WS commands (#130590) * Allow setting password for backups (#110630) * Allow setting password for backups * use is_hassio from helpers * move it * Fix getting psw * Fix restoring with psw * Address review comments * Improve docstring * Adjust kitchen sink * Adjust --------- Co-authored-by: Erik <erik@montnemery.com> * Export relevant names from backup integration (#130596) * Tweak backup agent interface (#130613) * Tweak backup agent interface * Adjust kitchen_sink * Test kitchen sink backup (#130609) * Test agents_list_backups * Test agents_info * Test agents_download * Export Backup from manager * Test agents_upload * Update tests after rebase * Use backup domain * Remove WS command backup/upload (#130588) * Remove WS command backup/upload * Disable failing kitchen_sink test * Make local backup a backup agent (#130623) * Make local backup a backup agent * Adjust * Adjust * Adjust * Adjust tests * Adjust * Adjust * Adjust docstring * Adjust * Protect members of CoreLocalBackupAgent * Remove redundant check for file * Make the backup.create service use the first local agent * Add BackupAgent.async_get_backup * Fix some TODOs * Add support for downloading backup from a remote agent * Fix restore * Fix test * Adjust kitchen_sink test * Remove unused method BackupManager.async_get_backup_path * Re-enable kitchen sink test * Remove BaseBackupManager.async_upload_backup * Support restore from remote agent * Fix review comments * Include backup agent error in response to WS command backup/info (#130884) * Adjust code related to WS command backup/info (#130890) * Include backup agent error in response to WS command backup/details (#130892) * Remove LOCAL_AGENT_ID constant from backup manager (#130895) * Add backup config storage (#130871) * Add base for backup config * Allow updating backup config * Test loading backup config * Add backup config update method * Add temporary check for BackupAgent.async_remove_backup (#130893) * Rename backup slug to backup_id (#130902) * Improve backup websocket API tests (#130912) * Improve backup websocket API tests * Add missing snapshot * Fix tests leaving files behind * Improve backup manager backup creation tests (#130916) * Remove class backup.backup.LocalBackup (#130919) * Add agent delete backup (#130921) * Add backup agent delete backup * Remove agents delete websocket command * Update docstring Co-authored-by: Erik Montnemery <erik@montnemery.com> --------- Co-authored-by: Erik Montnemery <erik@montnemery.com> * Disable core local backup agent in hassio (#130933) * Rename remove backup to delete backup (#130940) * Rename remove backup to delete backup * Revert "backup/delete" * Refactor BackupManager (#130947) * Refactor BackupManager * Adjust * Adjust backup creation * Copy in executor * Fix BackupManager.async_get_backup (#130975) * Fix typo in backup tests (#130978) * Adjust backup NewBackup class (#130976) * Remove class backup.BackupUploadMetadata (#130977) Remove class backup.BackupMetadata * Report backup size in bytes instead of MB (#131028) Co-authored-by: Robert Resch <robert@resch.dev> * Speed up CI for feature branch (#131030) * Speed up CI for feature branch * adjust * fix * fix * fix * fix * Rename remove to delete in backup websocket type (#131023) * Revert "Speed up CI for feature branch" (#131074) Revert "Speed up CI for feature branch (#131030)" This reverts commit791280506d. * Rename class BaseBackup to AgentBackup (#131083) * Rename class BaseBackup to AgentBackup * Update tests * Speed up CI for backup feature branch (#131079) * Add backup platform to the hassio integration (#130991) * Add backup platform to the hassio integration * Add hassio to after_dependencies of backup * Address review comments * Remove redundant hassio parametrization of tests * Add tests * Address review comments * Bump CI cache version * Revert "Bump CI cache version" This reverts commit2ab4d2b179. * Extend backup info class AgentBackup (#131110) * Extend backup info class AgentBackup * Update kitchen sink * Update kitchen sink test * Update kitchen sink test * Exclude cloud and hassio from core files (#131117) * Remove unnecessary **kwargs from backup API (#131124) * Fix backup tests (#131128) * Freeze backup dataclasses (#131122) * Protect CoreLocalBackupAgent.load_backups (#131126) * Use backup metadata v2 in core/container backups (#131125) * Extend backup creation API (#131121) * Extend backup creation API * Add tests * Fix merge * Fix merge * Return agent errors when deleting a backup (#131142) * Return agent errors when deleting a backup * Remove redundant calls to dict.keys() * Add enum type for backup folder (#131158) * Add method AgentBackup.from_dict (#131164) * Remove WS command backup/agents/list_backups (#131163) * Handle backup schedule (#131127) * Add backup schedule handling * Fix unrelated incorrect type annotation in test * Clarify delay save * Make the backup time compatible with the recorder nightly job * Update create backup parameters * Use typed dict for create backup parameters * Simplify schedule state * Group create backup parameters * Move parameter * Fix typo * Use Folder model * Handle deserialization of folders better * Fail on attempt to include addons or folders in core backup (#131204) * Fix AgentBackup test (#131201) * Add options to WS command backup/restore (#131194) * Add options to WS command backup/restore * Add tests * Fix test * Teach core backup to restore only database or only settings (#131225) * Exclude tmp_backups/*.tar from backups (#131243) * Add WS command backup/subscribe_events (#131250) * Clean up temporary directory after restoring backup (#131263) * Improve hassio backup agent list (#131268) * Include `last_automatic_backup` in reply to backup/info (#131293) Include last_automatic_backup in reply to backup/info * Handle backup delete after config (#131259) * Handle delete after copies * Handle delete after days * Add some test examples * Test config_delete_after_logic * Test config_delete_after_copies_logic * Test more delete after days * Add debug logs * Always delete the oldest backup first * Never remove the last backup * Clean up words Co-authored-by: Erik Montnemery <erik@montnemery.com> * Fix after cleaning words * Use utcnow * Remove duplicate guard * Simplify sorting * Delete backups even if there are agent errors on get backups --------- Co-authored-by: Erik Montnemery <erik@montnemery.com> * Rename backup delete after to backup retention (#131364) * Rename backup delete after to backup retention * Tweak * Remove length limit on `agent_ids` when configuring backup (#132057) Remove length limit on agent_ids when configuring backup * Rename backup retention_config to retention (#132068) * Modify backup agent API to be stream oriented (#132090) * Modify backup agent API to be stream oriented * Fix tests * Adjust after code review * Remove no longer needed pylint override * Improve test coverage * Change BackupAgent API to work with AsyncIterator objects * Don't close files in the event loop * Don't close files in the event loop * Fix backup manager create backup log (#132174) * Fix debug log level (#132186) * Add cloud backup agent (#129621) * Init cloud backup sync * Add more metadata * Fix typo * Adjust to base changes * Don't raise on list if more than one backup is available * Adjust to base branch * Fetch always and verify on download * Update homeassistant/components/cloud/backup.py Co-authored-by: Martin Hjelmare <marhje52@gmail.com> * Adjust to base branch changes * Not required anymore * Workaround * Fix blocking event loop * Fix * Add some tests * some tests * Add cloud backup delete functionality * Enable check * Fix ruff * Use fixture * Use iter_chunks instead * Remove read * Remove explicit export of read_backup * Align with BackupAgent API changes * Improve test coverage * Improve error handling * Adjust docstrings * Catch aiohttp.ClientError bubbling up from hass_nabucasa * Improve iteration --------- Co-authored-by: Erik <erik@montnemery.com> Co-authored-by: Robert Resch <robert@resch.dev> Co-authored-by: Martin Hjelmare <marhje52@gmail.com> Co-authored-by: Krisjanis Lejejs <krisjanis.lejejs@gmail.com> * Extract file receiver from `BackupManager.async_receive_backup` to util (#132271) * Extract file receiver from BackupManager.async_receive_backup to util * Apply suggestions from code review Co-authored-by: Martin Hjelmare <marhje52@gmail.com> --------- Co-authored-by: Martin Hjelmare <marhje52@gmail.com> * Make sure backup directory exists (#132269) * Make sure backup directory exists * Hand off directory creation to executor * Use mkdir's exist_ok feeature * Organize BackupManager instance attributes (#132277) * Don't store received backups in a TempDir (#132272) * Don't store received backups in a TempDir * Fix tests * Make sure backup directory exists * Address review comments * Fix tests * Rewrite backup manager state handling (#132375) * Rewrite backup manager state handling * Address review comments * Modify backup reader/writer API to be stream oriented (#132464) * Internalize backup tasks (#132482) * Internalize backup tasks * Update test after rebase * Handle backup error during automatic backup (#132511) * Improve backup manager state logging (#132549) * Fix backup manager state when restore completes (#132548) * Remove WS command backup/agents/download (#132664) * Add WS command backup/generate_with_stored_settings (#132671) * Add WS command backup/generate_with_stored_settings * Register the new command, add tests * Refactor local agent backup tests (#132683) * Refactor test_load_backups * Refactor test loading agents * Refactor test_delete_backup * Refactor test_upload * Clean up duplicate tests * Refactor backup manager receive tests (#132701) * Refactor backup manager receive tests * Clean up * Refactor pre and post platform tests (#132708) * Refactor backup pre platform test * Refactor backup post platform test * Bump aiohasupervisor to version 0.2.2b0 (#132704) * Bump aiohasupervisor to version 0.2.2b0 * Adjust tests * Publish event when manager is idle after creating backup (#132724) * Handle busy backup manager when uploading backup (#132736) * Adjust hassio backup agent to supervisor changes (#132732) * Adjust hassio backup agent to supervisor changes * Fix typo * Refactor test for create backup with wrong parameters (#132763) * Refactor test not loading bad backup platforms (#132769) * Improve receive backup coverage (#132758) * Refactor initiate backup test (#132829) * Rename Backup to ManagerBackup (#132841) * Refactor backup config (#132845) * Refactor backup config * Remove unnecessary condition * Adjust tests * Improve initiate backup test (#132858) * Store the time of automatic backup attempts (#132860) * Store the time of automatic backup attempts * Address review comments * Update test * Update cloud test * Save agent failures when creating backups (#132850) * Save agent failures when creating backups * Update tests * Store KnownBackups * Add test * Only clear known_backups on no error, add tests * Address review comments * Store known backups as a list * Update tests * Track all backups created with backup strategy settings (#132916) * Track all backups created with saved settings * Rename * Add explicit call to save the store * Don't register service backup.create in HassOS installations (#132932) * Revert changes to action service backup.create (#132938) * Fix logic for cleaning up temporary backup file (#132934) * Fix logic for cleaning up temporary backup file * Reduce scope of patch * Fix with_strategy_settings info not sent over websocket (#132939) * Fix with_strategy_settings info not sent over websocket * Fix kitchen sink tests * Fix cloud and hassio tests * Revert backup ci changes (#132955) Revert changes speeding up CI * Fix revert of CI changes (#132960) --------- Co-authored-by: Joakim Sørensen <joasoe@gmail.com> Co-authored-by: Martin Hjelmare <marhje52@gmail.com> Co-authored-by: Robert Resch <robert@resch.dev> Co-authored-by: Paul Bottein <paul.bottein@gmail.com> Co-authored-by: Krisjanis Lejejs <krisjanis.lejejs@gmail.com>
This commit is contained in:
Erik Montnemery
GitHub
@ -1,6 +1,10 @@
|
||||
"""Home Assistant module to handle restoring backups."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Iterable
|
||||
from dataclasses import dataclass
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
from pathlib import Path
|
||||
@ -14,7 +18,12 @@ import securetar
|
||||
from .const import __version__ as HA_VERSION
|
||||
|
||||
RESTORE_BACKUP_FILE = ".HA_RESTORE"
|
||||
KEEP_PATHS = ("backups",)
|
||||
KEEP_BACKUPS = ("backups",)
|
||||
KEEP_DATABASE = (
|
||||
"home-assistant_v2.db",
|
||||
"home-assistant_v2.db-wal",
|
||||
)
|
||||
|
||||
|
||||
_LOGGER = logging.getLogger(__name__)
|
||||
|
||||
@ -24,6 +33,21 @@ class RestoreBackupFileContent:
|
||||
"""Definition for restore backup file content."""
|
||||
|
||||
backup_file_path: Path
|
||||
password: str | None
|
||||
remove_after_restore: bool
|
||||
restore_database: bool
|
||||
restore_homeassistant: bool
|
||||
|
||||
|
||||
def password_to_key(password: str) -> bytes:
|
||||
"""Generate a AES Key from password.
|
||||
|
||||
Matches the implementation in supervisor.backups.utils.password_to_key.
|
||||
"""
|
||||
key: bytes = password.encode()
|
||||
for _ in range(100):
|
||||
key = hashlib.sha256(key).digest()
|
||||
return key[:16]
|
||||
|
||||
|
||||
def restore_backup_file_content(config_dir: Path) -> RestoreBackupFileContent | None:
|
||||
@ -32,20 +56,24 @@ def restore_backup_file_content(config_dir: Path) -> RestoreBackupFileContent |
|
||||
try:
|
||||
instruction_content = json.loads(instruction_path.read_text(encoding="utf-8"))
|
||||
return RestoreBackupFileContent(
|
||||
backup_file_path=Path(instruction_content["path"])
|
||||
backup_file_path=Path(instruction_content["path"]),
|
||||
password=instruction_content["password"],
|
||||
remove_after_restore=instruction_content["remove_after_restore"],
|
||||
restore_database=instruction_content["restore_database"],
|
||||
restore_homeassistant=instruction_content["restore_homeassistant"],
|
||||
)
|
||||
except (FileNotFoundError, json.JSONDecodeError):
|
||||
except (FileNotFoundError, KeyError, json.JSONDecodeError):
|
||||
return None
|
||||
|
||||
|
||||
def _clear_configuration_directory(config_dir: Path) -> None:
|
||||
"""Delete all files and directories in the config directory except for the backups directory."""
|
||||
keep_paths = [config_dir.joinpath(path) for path in KEEP_PATHS]
|
||||
config_contents = sorted(
|
||||
[entry for entry in config_dir.iterdir() if entry not in keep_paths]
|
||||
def _clear_configuration_directory(config_dir: Path, keep: Iterable[str]) -> None:
|
||||
"""Delete all files and directories in the config directory except entries in the keep list."""
|
||||
keep_paths = [config_dir.joinpath(path) for path in keep]
|
||||
entries_to_remove = sorted(
|
||||
entry for entry in config_dir.iterdir() if entry not in keep_paths
|
||||
)
|
||||
|
||||
for entry in config_contents:
|
||||
for entry in entries_to_remove:
|
||||
entrypath = config_dir.joinpath(entry)
|
||||
|
||||
if entrypath.is_file():
|
||||
@ -54,12 +82,15 @@ def _clear_configuration_directory(config_dir: Path) -> None:
|
||||
shutil.rmtree(entrypath)
|
||||
|
||||
|
||||
def _extract_backup(config_dir: Path, backup_file_path: Path) -> None:
|
||||
def _extract_backup(
|
||||
config_dir: Path,
|
||||
restore_content: RestoreBackupFileContent,
|
||||
) -> None:
|
||||
"""Extract the backup file to the config directory."""
|
||||
with (
|
||||
TemporaryDirectory() as tempdir,
|
||||
securetar.SecureTarFile(
|
||||
backup_file_path,
|
||||
restore_content.backup_file_path,
|
||||
gzip=False,
|
||||
mode="r",
|
||||
) as ostf,
|
||||
@ -88,22 +119,41 @@ def _extract_backup(config_dir: Path, backup_file_path: Path) -> None:
|
||||
f"homeassistant.tar{'.gz' if backup_meta["compressed"] else ''}",
|
||||
),
|
||||
gzip=backup_meta["compressed"],
|
||||
key=password_to_key(restore_content.password)
|
||||
if restore_content.password is not None
|
||||
else None,
|
||||
mode="r",
|
||||
) as istf:
|
||||
for member in istf.getmembers():
|
||||
if member.name == "data":
|
||||
continue
|
||||
member.name = member.name.replace("data/", "")
|
||||
_clear_configuration_directory(config_dir)
|
||||
istf.extractall(
|
||||
path=config_dir,
|
||||
members=[
|
||||
member
|
||||
for member in securetar.secure_path(istf)
|
||||
if member.name != "data"
|
||||
],
|
||||
path=Path(tempdir, "homeassistant"),
|
||||
members=securetar.secure_path(istf),
|
||||
filter="fully_trusted",
|
||||
)
|
||||
if restore_content.restore_homeassistant:
|
||||
keep = list(KEEP_BACKUPS)
|
||||
if not restore_content.restore_database:
|
||||
keep.extend(KEEP_DATABASE)
|
||||
_clear_configuration_directory(config_dir, keep)
|
||||
shutil.copytree(
|
||||
Path(tempdir, "homeassistant", "data"),
|
||||
config_dir,
|
||||
dirs_exist_ok=True,
|
||||
ignore=shutil.ignore_patterns(*(keep)),
|
||||
)
|
||||
elif restore_content.restore_database:
|
||||
for entry in KEEP_DATABASE:
|
||||
entrypath = config_dir / entry
|
||||
|
||||
if entrypath.is_file():
|
||||
entrypath.unlink()
|
||||
elif entrypath.is_dir():
|
||||
shutil.rmtree(entrypath)
|
||||
|
||||
for entry in KEEP_DATABASE:
|
||||
shutil.copy(
|
||||
Path(tempdir, "homeassistant", "data", entry),
|
||||
config_dir,
|
||||
)
|
||||
|
||||
|
||||
def restore_backup(config_dir_path: str) -> bool:
|
||||
@ -119,8 +169,13 @@ def restore_backup(config_dir_path: str) -> bool:
|
||||
backup_file_path = restore_content.backup_file_path
|
||||
_LOGGER.info("Restoring %s", backup_file_path)
|
||||
try:
|
||||
_extract_backup(config_dir, backup_file_path)
|
||||
_extract_backup(
|
||||
config_dir=config_dir,
|
||||
restore_content=restore_content,
|
||||
)
|
||||
except FileNotFoundError as err:
|
||||
raise ValueError(f"Backup file {backup_file_path} does not exist") from err
|
||||
if restore_content.remove_after_restore:
|
||||
backup_file_path.unlink(missing_ok=True)
|
||||
_LOGGER.info("Restore complete, restarting")
|
||||
return True
|
||||
|
||||
Reference in New Issue
Block a user