mirror of
https://github.com/DoTheEvo/selfhosted-apps-docker.git
synced 2026-07-03 14:06:40 +02:00
update
This commit is contained in:
DoTheEvo
@@ -41,10 +41,14 @@ if exist C:\Windows\System32\Tasks\kopia_server_backup_start (
|
||||
echo - starting Kopia Server
|
||||
schtasks /run /tn kopia_server_backup_start
|
||||
|
||||
echo - copying link to Desktop
|
||||
robocopy "%~dp0\" "%USERPROFILE%\Desktop" "Kopia.url" /NDL /NJH /NJS
|
||||
|
||||
echo.
|
||||
echo --------------------------------------------------------------
|
||||
echo.
|
||||
echo DEPLOYMENT DONE
|
||||
echo KOPIA SERVER CAN NOW BE FIND AT WEB PAGE: localhost:51515
|
||||
echo A LINK SHOULD BE ON YOUR DESKTOP
|
||||
echo.
|
||||
pause
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
[InternetShortcut]
|
||||
URL=http://localhost:51515/
|
||||
IDList=
|
||||
HotKey=0
|
||||
[{000214A0-0000-0000-C000-000000000046}]
|
||||
Prop3=19,2
|
||||
+284
-70
@@ -7,65 +7,93 @@
|
||||
WORK IN PROGRESS<br>
|
||||
WORK IN PROGRESS<br>
|
||||
WORK IN PROGRESS<br>
|
||||
#Kopia-on-a-linux-machine
|
||||
* [Kopia on a linux machine](#Kopia-on-a-linux-machine)
|
||||
* [Kopia on a Windows machine](#Kopia-on-a-windows-machine)
|
||||
* [Kopia in Docker](#Kopia-in-Docker)
|
||||
|
||||
# Purpose & Overview
|
||||
|
||||
Backups.
|
||||
|
||||
* [Official site](https://kopia.io/)
|
||||
* [Official Forum](https://kopia.discourse.group/)
|
||||
* [Github](https://github.com/kopia/kopia)
|
||||
|
||||
Kopia is a new open source backup utility with basicly **all** modern features.</br>
|
||||
Kopia is a new open source backup utility with basically **all** modern features.</br>
|
||||
Cross-platform, deduplication, encryption, compression, multithreaded speed,
|
||||
native cloud storage support, repository replication, snapshots mounting,
|
||||
GUI versions, server version,...
|
||||
|
||||
Written in golang.
|
||||
Written in golang.<br>
|
||||
Embedded webGUI for server mode is done in React. KopiaUI comes with electron.
|
||||
|
||||
### Ways to use Kopia
|
||||
|
||||
* **cli** - Command line.<br>
|
||||
You call the kopia binary passing some commands, it executes stuff, done.<br>
|
||||
Deployment requires extra work - scripts with configs, scheduling.
|
||||
* **Kopia Server** - kopia binary runs in the server mode.<br>
|
||||
Runs in the background, with its web server answering at url: `localhost:51515`<br>
|
||||
Through the web GUI the management is easier than using cli.
|
||||
As an addition to normal backup functions, the server mode can also act
|
||||
as a centralized repository for other machines running kopia instances.<br>
|
||||
Deployment requires extra work similar to cli, but actual use is through web GUI.
|
||||
* **KopiaUI** - GUI version.<br>
|
||||
KopiaUI comes packaged with electron to provide the feel of a standalone desktop app.<br>
|
||||
Good for simple deployment where average user just wants to backup stuff.<br>
|
||||
Benefits over cli or server is easier setup and management.<br>
|
||||
Drawback is that it runs under one user and only when that user is logged in.
|
||||
* **Kopia in Docker** - Kopia Server running in docker<br>
|
||||
Can fulfill two needs:
|
||||
* Backup docker-host stuff to a cloud or a mounted network storage.
|
||||
Managed through webgui instead of cli.
|
||||
* A centralized kopia repository where other machines on the network,
|
||||
that also use kopia, backup their data.
|
||||
|
||||
|
||||
|
||||
# Some aspects of Kopia
|
||||
|
||||
There are 3 ways to go about running kopia
|
||||
|
||||
* **cli** - Command line.<br>
|
||||
You call the binary passing some commands, it executes stuff, done.<br>
|
||||
Requires extra work - scripts with configs, scheduling.
|
||||
* **KopiaUI** - GUI version of kopia.<br>
|
||||
Easier managment, takes uppon itself scheduling.<br>
|
||||
Drawback is that it runs under one user and only when logged in.
|
||||
* **Kopia Server** - running kopia and its webserver in the background<br>
|
||||
Managment through web browser at an url, can run as a docker container.
|
||||
|
||||
[Official Getting Started Guide](https://kopia.io/docs/getting-started/)<br>
|
||||
[Kopia Build Architecture](https://github.com/kopia/kopia/blob/master/BUILD.md)<br>
|
||||
[Official Features](https://kopia.io/docs/features/)
|
||||
|
||||
* Kopia is a single ~35MB binary file.
|
||||
* Backups are stored in a **repository** that needs to be created first,
|
||||
and is always encrypted - requires password.
|
||||
* Before any action, Kopia needs to connecto to a repo.
|
||||
* **Snapshot**, appart from typical meaning, kopia also uses it to for
|
||||
targets(paths) to be backed up.
|
||||
* **Policy** is a term used to define behaviour of the backup/repo,
|
||||
and is always encrypted.
|
||||
* Before any action, Kopia needs to connect to a repo.
|
||||
* **Snapshots**, apart from typical meaning, kopia also uses the term for
|
||||
targets(paths) that are being backed up.
|
||||
* **Policy** is a term used to define behavior of the backup/repo,
|
||||
like backups retention, what to ignore, logging, scheduling(server/UI),
|
||||
actions before and after backup,...
|
||||
* **Policies** are stored inside a repo and can apply at various levels and
|
||||
can inherit from each other
|
||||
- global policy, the default that comes predefined during repo creation
|
||||
- per user policy and per machine policy
|
||||
- snapshot level policy, only applying for that one path
|
||||
* Maintence is automatic
|
||||
can **inherit** from each other
|
||||
- **Global** policy, the default that comes predefined during repo creation,
|
||||
can be edited like any other.
|
||||
- Per user policy, per machine policy.
|
||||
- Snapshot level policy, only applying for that one path.
|
||||
* **Maintenance** is automatic
|
||||
* During snapshots Kopia uses local **cache**, location varies depending on the OS.
|
||||
Default max size is 5GB, but it gets swept periodically every few minutes.<br>
|
||||
Useful commands are `kopia cache info` and `kopia cache clear`
|
||||
* [Here's](https://kopia.discourse.group/t/trying-to-understand-retention-policies/164/4)
|
||||
how **retention** works under the hood.
|
||||
* ..
|
||||
|
||||
# Kopia on a linux machine
|
||||
|
||||
cli version of kopia will be used, with a script and systemd-timers for scheduling.
|
||||
|
||||
|
||||
### install kopia
|
||||
cli version of kopia will be used to periodically backup to a mounted network storage.<br>
|
||||
The backup script will be executed using systemd-timers for scheduling.
|
||||
|
||||
for arch linux, kopia is on AUR `yay kopia-bin`
|
||||
### Install Kopia
|
||||
|
||||
### the initial steps
|
||||
For arch linux, kopia is on AUR `yay kopia-bin`
|
||||
|
||||
General use of sudo so that kopia has access everywhere.
|
||||
### The initial steps and general use commands
|
||||
|
||||
* **repo creation**
|
||||
|
||||
@@ -78,7 +106,7 @@ General use of sudo so that kopia has access everywhere.
|
||||
|
||||
`sudo kopia policy get --global`<br>
|
||||
`sudo kopia policy list`<br>
|
||||
`sudo kopia policy set --global --ignore-cache-dirs true --keep-annual 1 --keep-monthly 6 --keep-weekly 4 --keep-daily 14 --keep-hourly 0 --keep-latest 14`<br>
|
||||
`sudo kopia policy set --global --keep-annual 2 --keep-monthly 6 --keep-weekly 4 --keep-daily 14 --keep-hourly 0 --keep-latest 3`<br>
|
||||
|
||||
* **manual run**
|
||||
|
||||
@@ -94,11 +122,17 @@ General use of sudo so that kopia has access everywhere.
|
||||
|
||||
### The backup script
|
||||
|
||||
In linux, passing multiple paths separated by space seems to work fine.<br>
|
||||
So both `/home` and `/etc` are set to be backed up.
|
||||
|
||||
`/opt/kopia-backup-home-etc.sh`
|
||||
```bash
|
||||
#!/bin/bash
|
||||
|
||||
#sudo kopia policy set --global --keep-annual 1 --keep-monthly 6 --keep-weekly 4 --keep-daily 14 --keep-hourly 0 --keep-latest 14
|
||||
# initialize repository
|
||||
# sudo kopia repo create filesystem --path /mnt/mirror/KOPIA/docker_host_kopia
|
||||
# adjust global policy
|
||||
# sudo kopia policy set --global --keep-annual 2 --keep-monthly 6 --keep-weekly 4 --keep-daily 14 --keep-hourly 0 --keep-latest 3
|
||||
|
||||
REPOSITORY_PATH='/mnt/mirror/KOPIA/docker_host_kopia'
|
||||
BACKUP_THIS='/home /etc'
|
||||
@@ -108,6 +142,7 @@ kopia repository connect filesystem --path $REPOSITORY_PATH --password $KOPIA_PA
|
||||
kopia snapshot create $BACKUP_THIS
|
||||
kopia repository disconnect
|
||||
```
|
||||
|
||||
make the script executable<br>
|
||||
`sudo chmod +x /opt/kopia-backup-home-etc.sh`
|
||||
|
||||
@@ -167,7 +202,9 @@ WantedBy=timers.target
|
||||
* `systemctl status kopia-home-etc.timer`
|
||||
* `journalctl -u kopia-home-etc.timer` - see history
|
||||
|
||||
# Mounting network storage using systemd
|
||||
|
||||
<details>
|
||||
<summary><h3>Mounting network storage using systemd</h3></summary>
|
||||
|
||||
* files are placed in `/etc/systemd/system`
|
||||
* the name of mount and automount files MUST correspond with the path<br>
|
||||
@@ -205,69 +242,246 @@ Where=/mnt/mirror
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
# Remote backup
|
||||
</details>
|
||||
|
||||
... some day ...
|
||||
---
|
||||
---
|
||||
|
||||
# Kopia in Windows
|
||||
# Kopia on a Windows machine
|
||||
|
||||
|
||||
|
||||
## Kopia Server on Windows
|
||||
|
||||
* download this repo, delete shit, keep `kopia_server_deploy` folder
|
||||
* run `DEPLOY.cmd`, it will
|
||||
* Download this repo, keep `kopia_server_deploy_win` folder, delete rest.
|
||||
* Run `DEPLOY.cmd`, it will:
|
||||
* Removes powershell scripts restriction.
|
||||
* Creates folder `C:\Kopia` and kopies files there
|
||||
* imports a task schedule that will start Kopia Server at boot
|
||||
* Imports a task schedule that will start `C:\Kopia\kopia_server_start.cmd`<br>
|
||||
Can edit flags used for the server.
|
||||
* Places kopia.url on the current user desktop
|
||||
* visit in browser `localhost:51515`
|
||||
* setup repo
|
||||
* setup what to backup and schedule
|
||||
* edit the `kopia_backup_scipt.ps1`, set what to backup and where
|
||||
* for the same repo location execute<br>
|
||||
`sudo kopia repository create filesystem --path C:\Backup`
|
||||
* run the script
|
||||
|
||||
## Kopia cli on Windows
|
||||
Kopia should now run on boot and be easy to manage through web GUI.<br>
|
||||
Be it creating backup jobs, mounting old snapshots, or just looking around
|
||||
if all works as it should.
|
||||
|
||||
It is also popular to use nssm to start up and manage kopia as a service.<br>
|
||||
|
||||
While Kopia runs now in server mode, the fact that we pass `--insecure` flag
|
||||
means it cant serve as a repository for other kopia instances.
|
||||
For that look at docker deployment section or in to making changes too.
|
||||
|
||||
|
||||
While GUI version seems like a way to go.. IMO its not there yet.
|
||||
The schedule seems to be dependant on user logging in...
|
||||
and general weird feeling of poor GUI quality.
|
||||
<details>
|
||||
<summary><h2>Kopia cli on Windows</h2></summary>
|
||||
|
||||
So here goes cli version. As after some hands-on experience with cli version
|
||||
the GUI version might click in better too, later.
|
||||
|
||||
|
||||
* download this repo, delete shit, keep `kopia-deploy` folder
|
||||
* run `DEPLOY.cmd`, it will
|
||||
*Written before I realiezed I could be using kopia server.*<br>
|
||||
*Also, at the moment its the only way I know how to make actions work for VSS snapshots.*
|
||||
|
||||
While KopiaUI version seems like the way to go I really dislike the idea
|
||||
of not having trust in kopia if using it on servers that run with users
|
||||
logged out. Since KopiaUI works only if user is logged in.
|
||||
|
||||
* Download this repo, keep `kopia_cli_deploy_win` folder, delete rest.
|
||||
* Run `DEPLOY.cmd`, it will:
|
||||
* Removes powershell scripts restriction.
|
||||
* Install scoop, sudo, kopia.
|
||||
* kopies kopia.exe in to `C:\Windows\System32`
|
||||
* Creates folder `C:\Kopia` and kopies there<br>
|
||||
`kopia_backup_scipt.ps1` and the VSS ps1 before and after files.
|
||||
* imports a task schedule
|
||||
* edit the `kopia_backup_scipt.ps1`, set what to backup and where
|
||||
* for the same repo location execute<br>
|
||||
`sudo kopia repository create filesystem --path C:\Backup`
|
||||
* run the script
|
||||
* Open the `kopia_backup_scipt.ps1` and follow the instructions there.<br>
|
||||
Which is just to create repo before running the script.<br>
|
||||
`kopia repo create filesystem --path C:\kopia_repo --password aaa`
|
||||
* edit the scheduled task to the prefered time, default is daily at 21:19
|
||||
* run scheduled task manually
|
||||
* check if it worked
|
||||
* `kopia repo status`
|
||||
* `kopia snap list --all`
|
||||
|
||||
To do the above things manualy:
|
||||
### VSS snapshots
|
||||
|
||||
* install kopia using scoop
|
||||
* open terminal as admin
|
||||
* `Set-ExecutionPolicy RemoteSigned`
|
||||
* `iex "& {$(irm get.scoop.sh)} -RunAsAdmin"`
|
||||
* `scoop install sudo --global`
|
||||
* `sudo scoop install kopia --global`
|
||||
* download this repo, extract to `c:\kopia`
|
||||
* edit kopia-backup-home-etc.sh as see fit
|
||||
To make use of VSS snapshots before running backup execute these two commands,
|
||||
replacing `C:\Test` with the snapshot target path.<br>
|
||||
|
||||
* powershell as as administrator
|
||||
* --enable-actions
|
||||
* in tray, right click on the icon - `Launch At Startup`<br>
|
||||
this creates registry entry - *HKCU\Software\Microsoft\Windows\CurrentVersion\Run\KopiaUI*
|
||||
*
|
||||
* `kopia policy set C:\Test --before-folder-action "powershell -WindowStyle Hidden C:\Kopia\win_vss_before.ps1"`
|
||||
* `kopia policy set C:\Test --after-folder-action "powershell -WindowStyle Hidden C:\Kopia\win_vss_after.ps1"`
|
||||
|
||||
kopia policy set <target_dir> --before-folder-action "powershell -WindowStyle Hidden <path_to_script>\before.ps1"
|
||||
kopia policy set <target_dir> --after-folder-action "powershell -WindowStyle Hidden <path_to_script>\after.ps1"
|
||||
Note the use of `--enable-actions` in the backup script `kopia_backup_scipt.ps1`,
|
||||
which is required for actions to work.
|
||||
|
||||
### Kopia install using scoop, machine-wide
|
||||
|
||||
Just something to have note of, if decided to switch to heavy scoop use.
|
||||
|
||||
* open terminal as admin
|
||||
* `Set-ExecutionPolicy Bypass`
|
||||
* `iex "& {$(irm get.scoop.sh)} -RunAsAdmin"`
|
||||
* `scoop install sudo --global`
|
||||
* `sudo scoop install kopia --global`
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
---
|
||||
|
||||
# Kopia in Docker
|
||||
|
||||
|
||||
|
||||
### Files and directory structure
|
||||
|
||||
```
|
||||
/mnt/
|
||||
└── mirror/
|
||||
|
||||
/home/
|
||||
└── ~/
|
||||
└── docker/
|
||||
└── kopia/
|
||||
├── kopia_config/
|
||||
├── kopia_cache/
|
||||
├── kopia_logs/
|
||||
├── some_data/
|
||||
├── kopia_repository/
|
||||
├── kopia_tmp/
|
||||
├── .env
|
||||
└── docker-compose.yml
|
||||
```
|
||||
|
||||
* `/mnt/mirror/...` - a mounted network storage share
|
||||
* `kopia_config/` - repository.config and ui-preferences.json
|
||||
* `kopia_cache/` - cache
|
||||
* `kopia_logs/` - logs
|
||||
* `some_data/` - some data to be backed up
|
||||
* `kopia_repository/` - repository location
|
||||
* `kopia_tmp/` - temp used for snapshots
|
||||
* `.env` - a file containing environment variables for docker compose
|
||||
* `docker-compose.yml` - a docker compose file, telling docker how to run the containers
|
||||
|
||||
### docker-compose
|
||||
|
||||
The data to be backed up are mounted in read only mode.<br>
|
||||
To be able to mount snapshots, extra privilages are required and fuse access.
|
||||
|
||||
```
|
||||
services:
|
||||
|
||||
kopia:
|
||||
image: kopia/kopia:latest
|
||||
container_name: kopia
|
||||
hostname: kopia
|
||||
restart: unless-stopped
|
||||
env_file: .env
|
||||
privileged: true
|
||||
cap_add:
|
||||
- SYS_ADMIN
|
||||
security_opt:
|
||||
- apparmor:unconfined
|
||||
devices:
|
||||
- /dev/fuse:/dev/fuse:rwm
|
||||
ports:
|
||||
- "51515:51515"
|
||||
command:
|
||||
- server
|
||||
- start
|
||||
- --tls-generate-cert
|
||||
- --disable-csrf-token-checks
|
||||
- --address=0.0.0.0:51515
|
||||
- --server-username=$USERNAME
|
||||
- --server-password=$KOPIA_PASSWORD
|
||||
volumes:
|
||||
# Mount local folders needed by kopia
|
||||
- ./kopia_config:/app/config
|
||||
- ./kopia_cache:/app/cache
|
||||
- ./kopia_logs:/app/logs
|
||||
# Mount local folders to snapshot
|
||||
- ./some_data:/data:ro
|
||||
# Mount repository location
|
||||
- ./kopia_repository:/repository
|
||||
# Mount path for browsing mounted snaphots
|
||||
- ./kopia_tmp:/tmp:shared
|
||||
|
||||
networks:
|
||||
default:
|
||||
name: $DOCKER_MY_NETWORK
|
||||
external: true
|
||||
```
|
||||
|
||||
|
||||
`.env`
|
||||
```bash
|
||||
DOCKER_MY_NETWORK=caddy_net
|
||||
TZ=Europe/Bratislava
|
||||
|
||||
# KOPIA
|
||||
USERNAME=admin
|
||||
KOPIA_PASSWORD=aaa
|
||||
```
|
||||
|
||||
**All containers must be on the same network**.</br>
|
||||
Which is named in the `.env` file.</br>
|
||||
If one does not exist yet: `docker network create caddy_net`
|
||||
|
||||
### Reverse proxy
|
||||
|
||||
Caddy is used, details
|
||||
[here](https://github.com/DoTheEvo/selfhosted-apps-docker/tree/master/caddy_v2).</br>
|
||||
|
||||
To function as a repository server, Kopia must be setup with https,
|
||||
which is achieved by using `--tls-generate-cert` flag and removal
|
||||
of `--insecure` flag.
|
||||
|
||||
So now Kopia sits behind Caddy, but caddy needs to be told the traffic is
|
||||
https and to ignore that the certificate is not valid.
|
||||
|
||||
`Caddyfile`
|
||||
```
|
||||
kopia.{$MY_DOMAIN} {
|
||||
reverse_proxy kopia:51515 {
|
||||
transport http {
|
||||
tls
|
||||
tls_insecure_skip_verify
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### First run
|
||||
|
||||
|
||||
|
||||
* visit kopia.example.com
|
||||
* create new repository as `Local Directory or NAS`, set path to `/repository`,
|
||||
set password
|
||||
|
||||
Now this container can do backups of mounted stuff in to other mounted places
|
||||
while managed through webgui.<br>
|
||||
To also make it function as a repository server a user account needs to be added.
|
||||
The users are stored in the repo.
|
||||
|
||||
* exec in to the container<br>
|
||||
`docker container exec -it kopia /bin/bash`
|
||||
* add user@machine and set the password<br>
|
||||
`kopia server user add user1@machine1`
|
||||
* on another machine on network install kopia server or koppiaUI<br>
|
||||
on first run:
|
||||
* Pick `Kopia Repository Server`
|
||||
* Server address: `https://kopia.example.com:443`
|
||||
* *Trusted server certificate fingerprint (SHA256)*<br>
|
||||
can be left empty, or if you put something there, it gives you error
|
||||
where it tells you fingerprints of the server to pick from.
|
||||
* In advanced option one can override user@machine with the one set
|
||||
when exec-ed in to the docker container.
|
||||
Or exec again there and add another one.
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
* check kopia docker contaienr logs, I like using [ctop](https://github.com/bcicen/ctop)
|
||||
* `nslookup kopia.example.com` check if you are gettin to you server from client
|
||||
* Make sure you use port 443 in server address.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user