From f49fea71c90d5feb9f690a9b7ac61021285c9d40 Mon Sep 17 00:00:00 2001 From: DoTheEvo Date: Mon, 4 Sep 2023 12:29:42 +0200 Subject: [PATCH] update --- .../kopia_server_deploy_win/DEPLOY.cmd | 4 + .../kopia_server_deploy_win/Kopia.url | 6 + kopia_backup/readme.md | 354 ++++++++++++++---- 3 files changed, 294 insertions(+), 70 deletions(-) create mode 100644 kopia_backup/kopia_server_deploy_win/Kopia.url diff --git a/kopia_backup/kopia_server_deploy_win/DEPLOY.cmd b/kopia_backup/kopia_server_deploy_win/DEPLOY.cmd index e66a7ac..a32d801 100644 --- a/kopia_backup/kopia_server_deploy_win/DEPLOY.cmd +++ b/kopia_backup/kopia_server_deploy_win/DEPLOY.cmd @@ -41,10 +41,14 @@ if exist C:\Windows\System32\Tasks\kopia_server_backup_start ( echo - starting Kopia Server schtasks /run /tn kopia_server_backup_start +echo - copying link to Desktop +robocopy "%~dp0\" "%USERPROFILE%\Desktop" "Kopia.url" /NDL /NJH /NJS + echo. echo -------------------------------------------------------------- echo. echo DEPLOYMENT DONE echo KOPIA SERVER CAN NOW BE FIND AT WEB PAGE: localhost:51515 +echo A LINK SHOULD BE ON YOUR DESKTOP echo. pause diff --git a/kopia_backup/kopia_server_deploy_win/Kopia.url b/kopia_backup/kopia_server_deploy_win/Kopia.url new file mode 100644 index 0000000..fbd84cb --- /dev/null +++ b/kopia_backup/kopia_server_deploy_win/Kopia.url @@ -0,0 +1,6 @@ +[InternetShortcut] +URL=http://localhost:51515/ +IDList= +HotKey=0 +[{000214A0-0000-0000-C000-000000000046}] +Prop3=19,2 diff --git a/kopia_backup/readme.md b/kopia_backup/readme.md index bc19fb7..8931940 100644 --- a/kopia_backup/readme.md +++ b/kopia_backup/readme.md @@ -7,65 +7,93 @@ WORK IN PROGRESS
WORK IN PROGRESS
WORK IN PROGRESS
+#Kopia-on-a-linux-machine +* [Kopia on a linux machine](#Kopia-on-a-linux-machine) +* [Kopia on a Windows machine](#Kopia-on-a-windows-machine) +* [Kopia in Docker](#Kopia-in-Docker) # Purpose & Overview Backups. * [Official site](https://kopia.io/) +* [Official Forum](https://kopia.discourse.group/) * [Github](https://github.com/kopia/kopia) -Kopia is a new open source backup utility with basicly **all** modern features.
+Kopia is a new open source backup utility with basically **all** modern features.
Cross-platform, deduplication, encryption, compression, multithreaded speed, native cloud storage support, repository replication, snapshots mounting, GUI versions, server version,... -Written in golang. +Written in golang.
+Embedded webGUI for server mode is done in React. KopiaUI comes with electron. + +### Ways to use Kopia + +* **cli** - Command line.
+ You call the kopia binary passing some commands, it executes stuff, done.
+ Deployment requires extra work - scripts with configs, scheduling. +* **Kopia Server** - kopia binary runs in the server mode.
+ Runs in the background, with its web server answering at url: `localhost:51515`
+ Through the web GUI the management is easier than using cli. + As an addition to normal backup functions, the server mode can also act + as a centralized repository for other machines running kopia instances.
+ Deployment requires extra work similar to cli, but actual use is through web GUI. +* **KopiaUI** - GUI version.
+ KopiaUI comes packaged with electron to provide the feel of a standalone desktop app.
+ Good for simple deployment where average user just wants to backup stuff.
+ Benefits over cli or server is easier setup and management.
+ Drawback is that it runs under one user and only when that user is logged in. +* **Kopia in Docker** - Kopia Server running in docker
+ Can fulfill two needs: + * Backup docker-host stuff to a cloud or a mounted network storage. + Managed through webgui instead of cli. + * A centralized kopia repository where other machines on the network, + that also use kopia, backup their data. + +![repo_first](https://i.imgur.com/rbqhmzZ.png) # Some aspects of Kopia -There are 3 ways to go about running kopia - -* **cli** - Command line.
- You call the binary passing some commands, it executes stuff, done.
- Requires extra work - scripts with configs, scheduling. -* **KopiaUI** - GUI version of kopia.
- Easier managment, takes uppon itself scheduling.
- Drawback is that it runs under one user and only when logged in. -* **Kopia Server** - running kopia and its webserver in the background
- Managment through web browser at an url, can run as a docker container. - [Official Getting Started Guide](https://kopia.io/docs/getting-started/)
+[Kopia Build Architecture](https://github.com/kopia/kopia/blob/master/BUILD.md)
[Official Features](https://kopia.io/docs/features/) * Kopia is a single ~35MB binary file. * Backups are stored in a **repository** that needs to be created first, - and is always encrypted - requires password. - * Before any action, Kopia needs to connecto to a repo. -* **Snapshot**, appart from typical meaning, kopia also uses it to for - targets(paths) to be backed up. -* **Policy** is a term used to define behaviour of the backup/repo, + and is always encrypted. + * Before any action, Kopia needs to connect to a repo. +* **Snapshots**, apart from typical meaning, kopia also uses the term for + targets(paths) that are being backed up. +* **Policy** is a term used to define behavior of the backup/repo, like backups retention, what to ignore, logging, scheduling(server/UI), actions before and after backup,... * **Policies** are stored inside a repo and can apply at various levels and - can inherit from each other - - global policy, the default that comes predefined during repo creation - - per user policy and per machine policy - - snapshot level policy, only applying for that one path -* Maintence is automatic + can **inherit** from each other + - **Global** policy, the default that comes predefined during repo creation, + can be edited like any other. + - Per user policy, per machine policy. + - Snapshot level policy, only applying for that one path. +* **Maintenance** is automatic +* During snapshots Kopia uses local **cache**, location varies depending on the OS. + Default max size is 5GB, but it gets swept periodically every few minutes.
+ Useful commands are `kopia cache info` and `kopia cache clear` +* [Here's](https://kopia.discourse.group/t/trying-to-understand-retention-policies/164/4) + how **retention** works under the hood. * .. # Kopia on a linux machine -cli version of kopia will be used, with a script and systemd-timers for scheduling. +![list_snapshots_cli](https://i.imgur.com/5sG3osF.png) -### install kopia +cli version of kopia will be used to periodically backup to a mounted network storage.
+The backup script will be executed using systemd-timers for scheduling. -for arch linux, kopia is on AUR `yay kopia-bin` +### Install Kopia -### the initial steps +For arch linux, kopia is on AUR `yay kopia-bin` -General use of sudo so that kopia has access everywhere. +### The initial steps and general use commands * **repo creation** @@ -78,7 +106,7 @@ General use of sudo so that kopia has access everywhere. `sudo kopia policy get --global`
`sudo kopia policy list`
-`sudo kopia policy set --global --ignore-cache-dirs true --keep-annual 1 --keep-monthly 6 --keep-weekly 4 --keep-daily 14 --keep-hourly 0 --keep-latest 14`
+`sudo kopia policy set --global --keep-annual 2 --keep-monthly 6 --keep-weekly 4 --keep-daily 14 --keep-hourly 0 --keep-latest 3`
* **manual run** @@ -94,11 +122,17 @@ General use of sudo so that kopia has access everywhere. ### The backup script +In linux, passing multiple paths separated by space seems to work fine.
+So both `/home` and `/etc` are set to be backed up. + `/opt/kopia-backup-home-etc.sh` ```bash #!/bin/bash -#sudo kopia policy set --global --keep-annual 1 --keep-monthly 6 --keep-weekly 4 --keep-daily 14 --keep-hourly 0 --keep-latest 14 +# initialize repository +# sudo kopia repo create filesystem --path /mnt/mirror/KOPIA/docker_host_kopia +# adjust global policy +# sudo kopia policy set --global --keep-annual 2 --keep-monthly 6 --keep-weekly 4 --keep-daily 14 --keep-hourly 0 --keep-latest 3 REPOSITORY_PATH='/mnt/mirror/KOPIA/docker_host_kopia' BACKUP_THIS='/home /etc' @@ -108,6 +142,7 @@ kopia repository connect filesystem --path $REPOSITORY_PATH --password $KOPIA_PA kopia snapshot create $BACKUP_THIS kopia repository disconnect ``` + make the script executable
`sudo chmod +x /opt/kopia-backup-home-etc.sh` @@ -167,7 +202,9 @@ WantedBy=timers.target * `systemctl status kopia-home-etc.timer` * `journalctl -u kopia-home-etc.timer` - see history -# Mounting network storage using systemd + +
+

Mounting network storage using systemd

 * files are placed in `/etc/systemd/system` * the name of mount and automount files MUST correspond with the path
@@ -205,69 +242,246 @@ Where=/mnt/mirror WantedBy=multi-user.target ``` -# Remote backup +
-... some day ... +--- +--- -# Kopia in Windows +# Kopia on a Windows machine + +![windows_snapshot_history_gui](https://i.imgur.com/nqkng5k.png) ## Kopia Server on Windows -* download this repo, delete shit, keep `kopia_server_deploy` folder -* run `DEPLOY.cmd`, it will +* Download this repo, keep `kopia_server_deploy_win` folder, delete rest. +* Run `DEPLOY.cmd`, it will: * Removes powershell scripts restriction. * Creates folder `C:\Kopia` and kopies files there - * imports a task schedule that will start Kopia Server at boot + * Imports a task schedule that will start `C:\Kopia\kopia_server_start.cmd`
+ Can edit flags used for the server. + * Places kopia.url on the current user desktop * visit in browser `localhost:51515` * setup repo * setup what to backup and schedule -* edit the `kopia_backup_scipt.ps1`, set what to backup and where -* for the same repo location execute
- `sudo kopia repository create filesystem --path C:\Backup` -* run the script -## Kopia cli on Windows +Kopia should now run on boot and be easy to manage through web GUI.
+Be it creating backup jobs, mounting old snapshots, or just looking around +if all works as it should. + +It is also popular to use nssm to start up and manage kopia as a service.
+ +While Kopia runs now in server mode, the fact that we pass `--insecure` flag +means it cant serve as a repository for other kopia instances. +For that look at docker deployment section or in to making changes too. -While GUI version seems like a way to go.. IMO its not there yet. -The schedule seems to be dependant on user logging in... -and general weird feeling of poor GUI quality. +
+

Kopia cli on Windows

 -So here goes cli version. As after some hands-on experience with cli version -the GUI version might click in better too, later. +![windows_scoop_install_kopia](https://i.imgur.com/UPZFImh.png) -* download this repo, delete shit, keep `kopia-deploy` folder -* run `DEPLOY.cmd`, it will +*Written before I realiezed I could be using kopia server.*
+*Also, at the moment its the only way I know how to make actions work for VSS snapshots.* + +While KopiaUI version seems like the way to go I really dislike the idea +of not having trust in kopia if using it on servers that run with users +logged out. Since KopiaUI works only if user is logged in. + +* Download this repo, keep `kopia_cli_deploy_win` folder, delete rest. +* Run `DEPLOY.cmd`, it will: * Removes powershell scripts restriction. - * Install scoop, sudo, kopia. + * kopies kopia.exe in to `C:\Windows\System32` * Creates folder `C:\Kopia` and kopies there
`kopia_backup_scipt.ps1` and the VSS ps1 before and after files. * imports a task schedule -* edit the `kopia_backup_scipt.ps1`, set what to backup and where -* for the same repo location execute
- `sudo kopia repository create filesystem --path C:\Backup` -* run the script +* Open the `kopia_backup_scipt.ps1` and follow the instructions there.
+ Which is just to create repo before running the script.
+ `kopia repo create filesystem --path C:\kopia_repo --password aaa` +* edit the scheduled task to the prefered time, default is daily at 21:19 +* run scheduled task manually +* check if it worked + * `kopia repo status` + * `kopia snap list --all` -To do the above things manualy: +### VSS snapshots -* install kopia using scoop - * open terminal as admin - * `Set-ExecutionPolicy RemoteSigned` - * `iex "& {$(irm get.scoop.sh)} -RunAsAdmin"` - * `scoop install sudo --global` - * `sudo scoop install kopia --global` -* download this repo, extract to `c:\kopia` -* edit kopia-backup-home-etc.sh as see fit +To make use of VSS snapshots before running backup execute these two commands, +replacing `C:\Test` with the snapshot target path.
-* powershell as as administrator -* --enable-actions -* in tray, right click on the icon - `Launch At Startup`
- this creates registry entry - *HKCU\Software\Microsoft\Windows\CurrentVersion\Run\KopiaUI* -* +* `kopia policy set C:\Test --before-folder-action "powershell -WindowStyle Hidden C:\Kopia\win_vss_before.ps1"` +* `kopia policy set C:\Test --after-folder-action "powershell -WindowStyle Hidden C:\Kopia\win_vss_after.ps1"` -kopia policy set --before-folder-action "powershell -WindowStyle Hidden \before.ps1" -kopia policy set --after-folder-action "powershell -WindowStyle Hidden \after.ps1" +Note the use of `--enable-actions` in the backup script `kopia_backup_scipt.ps1`, +which is required for actions to work. + +### Kopia install using scoop, machine-wide + +Just something to have note of, if decided to switch to heavy scoop use. + +* open terminal as admin +* `Set-ExecutionPolicy Bypass` +* `iex "& {$(irm get.scoop.sh)} -RunAsAdmin"` +* `scoop install sudo --global` +* `sudo scoop install kopia --global` + +
+ +--- +--- + +# Kopia in Docker + +![kopia_docker_logs](https://i.imgur.com/w57KHvp.png) + +### Files and directory structure + +``` +/mnt/ +└── mirror/ + +/home/ +└── ~/ + └── docker/ + └── kopia/ + ├── kopia_config/ + ├── kopia_cache/ + ├── kopia_logs/ + ├── some_data/ + ├── kopia_repository/ + ├── kopia_tmp/ + ├── .env + └── docker-compose.yml +``` + +* `/mnt/mirror/...` - a mounted network storage share +* `kopia_config/` - repository.config and ui-preferences.json +* `kopia_cache/` - cache +* `kopia_logs/` - logs +* `some_data/` - some data to be backed up +* `kopia_repository/` - repository location +* `kopia_tmp/` - temp used for snapshots +* `.env` - a file containing environment variables for docker compose +* `docker-compose.yml` - a docker compose file, telling docker how to run the containers + +### docker-compose + +The data to be backed up are mounted in read only mode.
+To be able to mount snapshots, extra privilages are required and fuse access. + +``` +services: + + kopia: + image: kopia/kopia:latest + container_name: kopia + hostname: kopia + restart: unless-stopped + env_file: .env + privileged: true + cap_add: + - SYS_ADMIN + security_opt: + - apparmor:unconfined + devices: + - /dev/fuse:/dev/fuse:rwm + ports: + - "51515:51515" + command: + - server + - start + - --tls-generate-cert + - --disable-csrf-token-checks + - --address=0.0.0.0:51515 + - --server-username=$USERNAME + - --server-password=$KOPIA_PASSWORD + volumes: + # Mount local folders needed by kopia + - ./kopia_config:/app/config + - ./kopia_cache:/app/cache + - ./kopia_logs:/app/logs + # Mount local folders to snapshot + - ./some_data:/data:ro + # Mount repository location + - ./kopia_repository:/repository + # Mount path for browsing mounted snaphots + - ./kopia_tmp:/tmp:shared + +networks: + default: + name: $DOCKER_MY_NETWORK + external: true +``` +`.env` +```bash +DOCKER_MY_NETWORK=caddy_net +TZ=Europe/Bratislava +# KOPIA +USERNAME=admin +KOPIA_PASSWORD=aaa +``` + +**All containers must be on the same network**.
+Which is named in the `.env` file.
+If one does not exist yet: `docker network create caddy_net` + +### Reverse proxy + +Caddy is used, details +[here](https://github.com/DoTheEvo/selfhosted-apps-docker/tree/master/caddy_v2).
+ +To function as a repository server, Kopia must be setup with https, +which is achieved by using `--tls-generate-cert` flag and removal +of `--insecure` flag. + +So now Kopia sits behind Caddy, but caddy needs to be told the traffic is +https and to ignore that the certificate is not valid. + +`Caddyfile` +``` +kopia.{$MY_DOMAIN} { + reverse_proxy kopia:51515 { + transport http { + tls + tls_insecure_skip_verify + } + } +} +``` + +### First run + +![kopia_repo_setup_first_run](https://i.imgur.com/mnn66Hj.png) + +* visit kopia.example.com +* create new repository as `Local Directory or NAS`, set path to `/repository`, + set password + +Now this container can do backups of mounted stuff in to other mounted places +while managed through webgui.
+To also make it function as a repository server a user account needs to be added. +The users are stored in the repo. + +* exec in to the container
+ `docker container exec -it kopia /bin/bash` +* add user@machine and set the password
+ `kopia server user add user1@machine1` +* on another machine on network install kopia server or koppiaUI
+ on first run: + * Pick `Kopia Repository Server` + * Server address: `https://kopia.example.com:443` + * *Trusted server certificate fingerprint (SHA256)*
+ can be left empty, or if you put something there, it gives you error + where it tells you fingerprints of the server to pick from. + * In advanced option one can override user@machine with the one set + when exec-ed in to the docker container. + Or exec again there and add another one. + +### Troubleshooting + +* check kopia docker contaienr logs, I like using [ctop](https://github.com/bcicen/ctop) +* `nslookup kopia.example.com` check if you are gettin to you server from client +* Make sure you use port 443 in server address.