diff --git a/README.md b/README.md new file mode 100644 index 0000000..99e435e --- /dev/null +++ b/README.md @@ -0,0 +1,157 @@ +# rgl-photo-backup + +Back up the photos the [Randomness Guide to London](https://london.randomness.org.uk/) +shows on its pages. Nearly all of them live on Flickr, which is a single +point of failure. This fetches the largest available version of each into a +directory, alongside a JSON sidecar of what the wiki and Flickr know about +it, then mirrors the directory into an [Immich](https://immich.app) album +with the source, photographer, licence and wiki pages carried into each +asset's description, tags, capture date and location. + +## Requirements + +Python 3 and nothing else. + +## Setup + +Two credentials, both reused from other tools if you already have them: + +- `~/.config/flickrtouchr/auth.yml` with a `key:` line holding a Flickr API + key. Public photos need no OAuth, so the key alone is enough. Create one at + flickr.com/services/apps/create if you have none. +- `~/.config/immich/auth.yml` with `url:` (your Immich API base, ending in + `/api`) and `key:` (an API key from Immich's account settings). + +Everything else can go in `~/.config/rgl-photo-backup/settings.yml`: + +```yaml +directory: /srv/backup/rgl +pace: 1 +quiet: true +``` + +Make the credential files `chmod 600`. + +## Running + +``` +./rgl-photo-backup /srv/backup/rgl # fetch, then sync +./rgl-photo-backup --dry-run /srv/backup/rgl # report only +./rgl-photo-backup --fetch-only --limit 5 DIR # a trial +./rgl-photo-backup --sync-only DIR # disk to Immich only +./rgl-photo-backup --settings other.yml DIR # a different settings file +./rgl-photo-backup --debug DIR # log every request to stderr +./rgl-photo-backup -q DIR # same as --quiet +``` + +The first run downloads about 4800 photos, one Flickr info call and one +sizes call each, so at the default one-second pace it takes a few hours. +The pace applies to Flickr alone; Immich and the wiki index are not slowed. +Later runs skip everything already on disk, and in Immich only write an +album membership or a description that has actually changed. + +Exit status is 0 for a clean run, 1 if any photo failed, 2 for a +configuration problem, and 130 if you interrupt it. + +### The first run + +Take it in stages, so a misconfiguration costs a few photos rather than +thousands: + +1. `--fetch-only` on its own, to fill the backup directory. +2. `--sync-only --limit 20`, and look at the twenty assets in Immich. +3. the full run. + +Immich queues its own metadata extraction after an upload, so on a first +run a description written straight afterwards can be overwritten by the +extraction finishing. This tool writes descriptions after everything else +for that reason, but if any of them come out wrong, run it again: the next +run compares against what Immich holds and puts them right. + +Three things worth checking with curl against your own Immich version +before leaving it unattended, since the API has moved before: + +- `POST /api/search/metadata` accepts `withExif`, `withArchived` and `size` +- `POST /api/assets` honours the `x-immich-checksum` header +- `PUT /api/tags` returns `value` as the full slash-separated path + +## What lands in Immich + +- An album named **RGL** holding every photo. Membership is read first, so + only photos not already in it are added. +- Tags `rgl`, `rgl/licence/` and `rgl/photographer/`. Unlike + the album and the description, tag assignments are sent every run: Immich + answers an asset that already carries the tag with `duplicate`, and those + are reported as already tagged rather than treated as a failure. +- A description block, appended below anything already there and refreshed + in place on later runs: + + ``` + Randomness Guide to London photo + Photo: + Source: + Photographer: + Licence: CC BY-SA 2.0 + Used on: + + ``` + +- The capture date from Flickr and the page's coordinates, only where the + asset has none already. + +A photo already in your library, for instance one of your own brought in by +another backup, is recognised by checksum: it joins the album and gets the +block appended rather than being uploaded twice. + +## The backup directory + +One file per photo, named by Flickr id (`3750465402.jpg`), beside a sidecar +(`3750465402.json`) holding the wiki fields, the Flickr info and the pages +that use it. The twenty-odd photos not on Flickr have no id to use, so they +are named `url-` and the first sixteen hex characters of the SHA1 of their +image URL (`url-0f4b1c2d3e4f5a6b.jpg`), with a matching sidecar. The +directory is a complete backup on its own. A `.part` file is an interrupted +download and is refetched next run. + +## Running under cron + +```cron +23 4 * * * /home/you/bin/rgl-photo-backup +``` + +With `quiet: true` a clean run prints nothing, so cron only mails when a +photo failed. A quiet run that failed ends with a single `N failed` line +after the individual failures. Failures list the photo and the wiki pages +using it, which is how you learn a Flickr photo has gone. + +## Settings + +A flag beats the environment, which beats the file, which beats the default. + +| Key | Flag | Environment | Default | +|---|---|---|---| +| `directory` | positional | | required | +| `flickr_auth` | `--flickr-config FILE` | | `~/.config/flickrtouchr/auth.yml` | +| `immich_auth` | `--immich-config FILE` | | `~/.config/immich/auth.yml` | +| `index_url` | `--index-url URL` | | the RGL index endpoint | +| `fetch` | `--sync-only` sets false | | `true` | +| `sync` | `--fetch-only` sets false | | `true` | +| `dry_run` | `--dry-run` | | `false` | +| `limit` | `--limit N` | | unlimited | +| `pace` | `--pace SECONDS` | `RGL_PHOTO_BACKUP_PACE` | `1` | +| `quiet` | `--quiet`, `-q` | | `false` | +| `attempts` | | `RGL_PHOTO_BACKUP_ATTEMPTS` | `6` | +| `backoff` | | `RGL_PHOTO_BACKUP_BACKOFF` | `5` | +| `timeout` | | `RGL_PHOTO_BACKUP_TIMEOUT` | `60` | +| `user_agent` | | `RGL_PHOTO_BACKUP_USER_AGENT` | `rgl-photo-backup/1.0` | +| `debug` | `--debug` | `RGL_PHOTO_BACKUP_DEBUG` | `false` | + +`--settings FILE` reads a different settings file; it is a flag only, since +it says where the rest of the settings come from. + +## Tests + +``` +make test # offline unit tests +make lint +```