158 lines
6.0 KiB
Markdown
158 lines
6.0 KiB
Markdown
# rgl-photo-backup
|
|
|
|
Back up the photos the [Randomness Guide to London](https://london.randomness.org.uk/)
|
|
shows on its pages. Nearly all of them live on Flickr, which is a single
|
|
point of failure. This fetches the largest available version of each into a
|
|
directory, alongside a JSON sidecar of what the wiki and Flickr know about
|
|
it, then mirrors the directory into an [Immich](https://immich.app) album
|
|
with the source, photographer, licence and wiki pages carried into each
|
|
asset's description, tags, capture date and location.
|
|
|
|
## Requirements
|
|
|
|
Python 3 and nothing else.
|
|
|
|
## Setup
|
|
|
|
Two credentials, both reused from other tools if you already have them:
|
|
|
|
- `~/.config/flickrtouchr/auth.yml` with a `key:` line holding a Flickr API
|
|
key. Public photos need no OAuth, so the key alone is enough. Create one at
|
|
flickr.com/services/apps/create if you have none.
|
|
- `~/.config/immich/auth.yml` with `url:` (your Immich API base, ending in
|
|
`/api`) and `key:` (an API key from Immich's account settings).
|
|
|
|
Everything else can go in `~/.config/rgl-photo-backup/settings.yml`:
|
|
|
|
```yaml
|
|
directory: /srv/backup/rgl
|
|
pace: 1
|
|
quiet: true
|
|
```
|
|
|
|
Make the credential files `chmod 600`.
|
|
|
|
## Running
|
|
|
|
```
|
|
./rgl-photo-backup /srv/backup/rgl # fetch, then sync
|
|
./rgl-photo-backup --dry-run /srv/backup/rgl # report only
|
|
./rgl-photo-backup --fetch-only --limit 5 DIR # a trial
|
|
./rgl-photo-backup --sync-only DIR # disk to Immich only
|
|
./rgl-photo-backup --settings other.yml DIR # a different settings file
|
|
./rgl-photo-backup --debug DIR # log every request to stderr
|
|
./rgl-photo-backup -q DIR # same as --quiet
|
|
```
|
|
|
|
The first run downloads about 4800 photos, one Flickr info call and one
|
|
sizes call each, so at the default one-second pace it takes a few hours.
|
|
The pace applies to Flickr alone; Immich and the wiki index are not slowed.
|
|
Later runs skip everything already on disk, and in Immich only write an
|
|
album membership or a description that has actually changed.
|
|
|
|
Exit status is 0 for a clean run, 1 if any photo failed, 2 for a
|
|
configuration problem, and 130 if you interrupt it.
|
|
|
|
### The first run
|
|
|
|
Take it in stages, so a misconfiguration costs a few photos rather than
|
|
thousands:
|
|
|
|
1. `--fetch-only` on its own, to fill the backup directory.
|
|
2. `--sync-only --limit 20`, and look at the twenty assets in Immich.
|
|
3. the full run.
|
|
|
|
Immich queues its own metadata extraction after an upload, so on a first
|
|
run a description written straight afterwards can be overwritten by the
|
|
extraction finishing. This tool writes descriptions after everything else
|
|
for that reason, but if any of them come out wrong, run it again: the next
|
|
run compares against what Immich holds and puts them right.
|
|
|
|
Three things worth checking with curl against your own Immich version
|
|
before leaving it unattended, since the API has moved before:
|
|
|
|
- `POST /api/search/metadata` accepts `withExif`, `withArchived` and `size`
|
|
- `POST /api/assets` honours the `x-immich-checksum` header
|
|
- `PUT /api/tags` returns `value` as the full slash-separated path
|
|
|
|
## What lands in Immich
|
|
|
|
- An album named **RGL** holding every photo. Membership is read first, so
|
|
only photos not already in it are added.
|
|
- Tags `rgl`, `rgl/licence/<licence>` and `rgl/photographer/<name>`. Unlike
|
|
the album and the description, tag assignments are sent every run: Immich
|
|
answers an asset that already carries the tag with `duplicate`, and those
|
|
are reported as already tagged rather than treated as a failure.
|
|
- A description block, appended below anything already there and refreshed
|
|
in place on later runs:
|
|
|
|
```
|
|
Randomness Guide to London photo
|
|
Photo: <Flickr title>
|
|
Source: <Flickr photo page>
|
|
Photographer: <name>
|
|
Licence: CC BY-SA 2.0 <licence url>
|
|
Used on:
|
|
<page name> <page url>
|
|
```
|
|
|
|
- The capture date from Flickr and the page's coordinates, only where the
|
|
asset has none already.
|
|
|
|
A photo already in your library, for instance one of your own brought in by
|
|
another backup, is recognised by checksum: it joins the album and gets the
|
|
block appended rather than being uploaded twice.
|
|
|
|
## The backup directory
|
|
|
|
One file per photo, named by Flickr id (`3750465402.jpg`), beside a sidecar
|
|
(`3750465402.json`) holding the wiki fields, the Flickr info and the pages
|
|
that use it. The twenty-odd photos not on Flickr have no id to use, so they
|
|
are named `url-` and the first sixteen hex characters of the SHA1 of their
|
|
image URL (`url-0f4b1c2d3e4f5a6b.jpg`), with a matching sidecar. The
|
|
directory is a complete backup on its own. A `.part` file is an interrupted
|
|
download and is refetched next run.
|
|
|
|
## Running under cron
|
|
|
|
```cron
|
|
23 4 * * * /home/you/bin/rgl-photo-backup
|
|
```
|
|
|
|
With `quiet: true` a clean run prints nothing, so cron only mails when a
|
|
photo failed. A quiet run that failed ends with a single `N failed` line
|
|
after the individual failures. Failures list the photo and the wiki pages
|
|
using it, which is how you learn a Flickr photo has gone.
|
|
|
|
## Settings
|
|
|
|
A flag beats the environment, which beats the file, which beats the default.
|
|
|
|
| Key | Flag | Environment | Default |
|
|
|---|---|---|---|
|
|
| `directory` | positional | | required |
|
|
| `flickr_auth` | `--flickr-config FILE` | | `~/.config/flickrtouchr/auth.yml` |
|
|
| `immich_auth` | `--immich-config FILE` | | `~/.config/immich/auth.yml` |
|
|
| `index_url` | `--index-url URL` | | the RGL index endpoint |
|
|
| `fetch` | `--sync-only` sets false | | `true` |
|
|
| `sync` | `--fetch-only` sets false | | `true` |
|
|
| `dry_run` | `--dry-run` | | `false` |
|
|
| `limit` | `--limit N` | | unlimited |
|
|
| `pace` | `--pace SECONDS` | `RGL_PHOTO_BACKUP_PACE` | `1` |
|
|
| `quiet` | `--quiet`, `-q` | | `false` |
|
|
| `attempts` | | `RGL_PHOTO_BACKUP_ATTEMPTS` | `6` |
|
|
| `backoff` | | `RGL_PHOTO_BACKUP_BACKOFF` | `5` |
|
|
| `timeout` | | `RGL_PHOTO_BACKUP_TIMEOUT` | `60` |
|
|
| `user_agent` | | `RGL_PHOTO_BACKUP_USER_AGENT` | `rgl-photo-backup/1.0` |
|
|
| `debug` | `--debug` | `RGL_PHOTO_BACKUP_DEBUG` | `false` |
|
|
|
|
`--settings FILE` reads a different settings file; it is a flag only, since
|
|
it says where the rest of the settings come from.
|
|
|
|
## Tests
|
|
|
|
```
|
|
make test # offline unit tests
|
|
make lint
|
|
```
|