Initial commit

README.md

@@ -0,0 +1,57 @@
+# Backup helpers
+
+With the right tooling (I use `restic` for instance), taking backups is very easy, so is distributing them to multiple locations including cloud storage. It is so easy, that the main challenge for me keeping track of them. Questions such as _Where can I find the most recent backup of **X**_ or even _Am I taking backups according to the schedule or is anything missing_ are more and more difficult to answer. It is supported by the fact that some storage locations aren't always online.
+
+The goal of this project is to provide a collection of simple scripts to keep an offline list of that is backed up where, perform simple queries on the list and evaluate, whether it fulfills simple criteria (such as whether there is a recent enough backup of specified files). It is designed to keep the information in a simple format (actually JSON files scattered across a couple of directories) to be easily queryable and editable directly and so that it can easily be synchronized across multiple devices (Git, Syncthing, …). It deliberately doesn't provide any encryption; your backups should be protected by the actual backup software and the metadata gathered by these scripts are probably gonna be stored on your laptop or similar, so if it is breached, you have a much more urgent problem (like someone replacing you `gpg` binary with their shell script).
+
+## Importing backups
+
+The only currently implemented import mechanism is for `restic`. If you are brave enough however, you can write the files by hand. Every storage, that can hold some backups, is called a `repo` and has a configuration file in `repos/<repo-name>`. The configuration is needed only for the import, which then creates `backups/<repo-name>/<backup-id>` for each backup in the repository.
+
+For `restic` repos, the config looks like
+
+```json
+{
+	"name": "<repo-name>",
+	"type": "restic",
+	"restic-path": "<What one passes to `-r` to restic>"
+}
+```
+
+and you can import the backups with
+
+```sh
+./import.sh <repo-name>
+```
+
+**Make sure to always call the script with the root of this repository as your working directory**
+
+## Declaring backup targets
+
+The files/directories you want to back up are called `targets`. You configure each one of them by creating a file `targets/<target-name>`, where you list the source machine's hostname and the local path to the file/directory. Together it looks like
+
+```json
+{
+	"name": "<target-name>",
+	"hostname": "<hostname>",
+	"path": "<path>"
+}
+```
+
+The `hostname` and `path` values do not necessarily have to be real, it is just something we match on the backups in repos to recognize what's what.
+
+## Checking freshness
+
+When we have all the metadata downloaded, we can evaluate whether they match our expectations. This can be done offline without access to the repositories and should be resource non-intensive in general.
+
+We can define the rules in `check.sh` by appending them at the bottom of the file (between the markers in comments). If we then execute the checks with
+
+```sh
+./check.sh
+``
+
+it prints those that didn't pass. The return code is 0 on successful execution and all checks passing, 2 on successful execution but some checks non-passing and other exit codes signify a failure.
+
+**Again make sure to always call the script with the root of this repository as your working directory**
+
+The only currently implemented check is `isRecentInRepo <repo-name> <target-name> <maxAge>` where the first two arguments are the filenames of the corresponding configuration files and the last one is maximum age in whole seconds to still consider the backup to be fresh.

check.sh

@@ -0,0 +1,54 @@
+#!/usr/bin/env bash
+
+set -eu -o pipefail
+verbose='' # we test for (non)emptiness
+quiet='' # we test for (non)emptiness
+
+source lib.sh
+
+exitCode=0
+checkNotPassed() {
+	echo "WARN: $1"
+	if [ "$exitCode" -eq 0 ]; then
+		exitCode=2
+	fi
+}
+
+isRecentInRepo() {
+	local repoName="$1" targetName="$2" maxAge="$3"
+	checkPathComponent "$repoName"
+	checkPathComponent "$targetName"
+	local targetFile="targets/$targetName" backupsDir="backups/$repoName"
+	local hostname="$(< "$targetFile" jq --raw-output .hostname)" backedUpPath="$(< "$targetFile" jq --raw-output .path)"
+	local dates=""
+	while IFS='' read -r -d '' backupFile; do
+		verbose "Processing $backupFile"
+		local date
+		if date="$(< "$backupFile" jq --exit-status --raw-output --arg hostname "$hostname" --arg path "$backedUpPath" 'if $hostname == .hostname and any(.paths[]; . == $path) then .time | sub("\\.\\d+"; "") | strptime("%Y-%m-%dT%H:%M:%S%z") | mktime else false end')"; then
+			verbose "The snapshot is relevant and was created at $date"
+			dates="$dates"$'\n'"$date"
+		fi
+	done < <(find "$backupsDir" -maxdepth 1 -type f -print0)
+	local max="$(echo "$dates" | sort --numeric-sort --reverse | head -n 1)"
+	if [ -n "$max" ]; then
+		local age="$(("$now"-"$max"))"
+		verbose "Last backup was at $max, now it's $now, the age is $age"
+		if [ "$age" -gt "$maxAge" ]; then
+			checkNotPassed "$targetName on $repoName is too old"
+		else
+			info "$targetName on $repoName is fresh enough"
+		fi
+	fi
+}
+
+isRecent() {
+	abort "isRecent is not implemented yet"
+}
+
+now="$(date +%s)"
+
+##### End of definitions, user defined checks go below this line
+
+##### End of user defined checks, the rest of this file is a predefined trailer
+
+exit "$exitCode"

import.sh

@@ -0,0 +1,47 @@
+#!/usr/bin/env bash
+
+set -eu -o pipefail
+verbose='y' # we test for (non)emptiness
+quiet='' # we test for (non)emptiness
+
+source lib.sh
+
+if [ "$#" -ne 1 ]; then
+	abort "Usage: $0 repositoryName"
+fi
+
+info "BEWARE that this script can't garbage collect no longer existing snapshots yet"
+
+restic() { command restic --no-cache "$@"; }
+repoName="$1"
+checkPathComponent "$repoName"
+repoFile="repos/$repoName"
+if [ ! -f "$repoFile" ]; then
+	abort "Repository $repoName does not exist"
+fi
+repoType="$(< "$repoFile" jq --raw-output .type)"
+
+if [ "$repoType" = "restic" ]; then
+	repoPath="$(< "$repoFile" jq --raw-output '."restic-path"')"
+	info "Inspecting repository $repoName"
+	snapshots="$(restic -r "$repoPath" snapshots --json)"
+	len="$(echo "$snapshots" | jq '. | length')"
+	i=0
+	while [ "$i" -lt "$len" ]; do
+		snapshot="$(echo "$snapshots" | jq ".[$i]")"
+		id="$(echo "$snapshot" | jq --raw-output '.id')"
+		checkPathComponent "$id"
+		snapshotFile="backups/$repoName/$id"
+		if [ -f "$snapshotFile" ]; then
+			verbose "Snapshot $id is already known"
+		else
+			info "Found new snapshot $id"
+			mkdir -p "backups/$repoName"
+			echo "$snapshot" | jq --sort-keys --tab '. | pick(.hostname, .id, .paths, .tags, .time)' > "$snapshotFile"
+		fi
+		i=$(("$i"+1))
+	done
+	info "Processed $len snapshots"
+else
+	abort "Unsupported repository type $repoType"
+fi

lib.sh

@@ -0,0 +1,26 @@
+abort() {
+	echo "$1"
+	exit 1
+}
+
+info() {
+	if [ -z "$quiet" ]; then
+		echo "$1"
+	fi
+}
+
+verbose() {
+	if [ -n "$verbose" ]; then
+		echo "$1"
+	fi
+}
+
+checkPathComponent() {
+	local msg="Dangerous path component discovered: $1"
+	if [ "$1" = "." -o "$1" = ".." ]; then
+		abort "$msg"
+	fi
+	if echo "$1" | grep --quiet -F "/"; then
+		abort "$msg"
+	fi
+}