rgl-photo-backup/README.md

# rgl-photo-backup

Back up the photos the [Randomness Guide to London](https://london.randomness.org.uk/)
shows on its pages. Nearly all of them live on Flickr, which is a single
point of failure. This fetches the largest available version of each into a
directory, alongside a JSON sidecar of what the wiki and Flickr know about
it, then mirrors the directory into an [Immich](https://immich.app) album
with the source, photographer, licence and wiki pages carried into each
asset's description, tags, capture date and location.

## Requirements

Python 3 and nothing else.

## Setup

Two credentials, both reused from other tools if you already have them:

- `~/.config/flickrtouchr/auth.yml` with a `key:` line holding a Flickr API
  key. Public photos need no OAuth, so the key alone is enough. Create one at
  flickr.com/services/apps/create if you have none.
- `~/.config/immich/auth.yml` with `url:` (your Immich API base, ending in
  `/api`) and `key:` (an API key from Immich's account settings).

Everything else can go in `~/.config/rgl-photo-backup/settings.yml`:

```yaml
directory: /srv/backup/rgl
pace: 1
quiet: true
```

Make the credential files `chmod 600`.

## Running

```
./rgl-photo-backup /srv/backup/rgl            # fetch, then sync
./rgl-photo-backup --dry-run /srv/backup/rgl  # report only
./rgl-photo-backup --fetch-only --limit 5 DIR # a trial
./rgl-photo-backup --sync-only DIR            # disk to Immich only
./rgl-photo-backup --settings other.yml DIR   # a different settings file
./rgl-photo-backup --debug DIR                # log every request to stderr
./rgl-photo-backup -q DIR                     # same as --quiet
./rgl-photo-backup --report DIR               # which pages have no photo on disk, and why
```

The first run downloads about 4800 photos, one Flickr info call and one
sizes call each, so at the default one-second pace it takes a few hours.
The pace applies to Flickr alone; Immich and the wiki index are not slowed.
A 429 from Flickr doubles the pace for the rest of the run, up to ten seconds
between calls, and it relaxes again after twenty untroubled requests. The
photo that hit the limit is retried with backoff starting at fifteen seconds
and doubling, up to eight attempts.
Later runs skip everything already on disk, and in Immich only write an
album membership or a description that has actually changed.

Exit status is 0 for a clean run, 1 if any photo failed, 2 for a
configuration problem, and 130 if you interrupt it.

### The first run

Take it in stages, so a misconfiguration costs a few photos rather than
thousands:

1. `--fetch-only` on its own, to fill the backup directory.
2. `--sync-only --limit 20`, and look at the twenty assets in Immich.
3. the full run.

Immich queues its own metadata extraction after an upload, so on a first
run a description written straight afterwards can be overwritten by the
extraction finishing. This tool writes descriptions after everything else
for that reason, but if any of them come out wrong, run it again: the next
run compares against what Immich holds and puts them right.

Three things worth checking with curl against your own Immich version
before leaving it unattended, since the API has moved before:

- `POST /api/search/metadata` accepts `withExif`, `withArchived` and `size`
- `POST /api/assets` honours the `x-immich-checksum` header
- `PUT /api/tags` returns `value` as the full slash-separated path

## What lands in Immich

- An album named **RGL** holding every photo. Membership is read first, so
  only photos not already in it are added.
- Tags `rgl`, `rgl/licence/<licence>` and `rgl/photographer/<name>`. Unlike
  the album and the description, tag assignments are sent every run: Immich
  answers an asset that already carries the tag with `duplicate`, and those
  are reported as already tagged rather than treated as a failure.
- A description block, appended below anything already there and refreshed
  in place on later runs:

  ```
  Randomness Guide to London photo
  Photo: <Flickr title>
  Source: <Flickr photo page>
  Photographer: <name>
  Licence: CC BY-SA 2.0 <licence url>
  Used on:
    <page name> <page url>
  ```

- The capture date from Flickr and the page's coordinates, only where the
  asset has none already.

A photo already in your library, for instance one of your own brought in by
another backup, is recognised by checksum: it joins the album and gets the
block appended rather than being uploaded twice.

## The backup directory

One file per photo, named by Flickr id (`3750465402.jpg`), beside a sidecar
(`3750465402.json`) holding the wiki fields, the Flickr info and the pages
that use it. The twenty-odd photos not on Flickr have no id to use, so they
are named `url-` and the first sixteen hex characters of the SHA1 of their
image URL (`url-0f4b1c2d3e4f5a6b.jpg`), with a matching sidecar. The
directory is a complete backup on its own. A `.part` file is an interrupted
download and is refetched next run.

`.failures.json` records the last reason each photo failed, with the pages
that use it. A photo that later arrives drops out of it. `--report` reads the
wiki index and that file, and lists every page whose photo is not on disk
with the reason, or `not yet attempted`, without calling Flickr or Immich, so
it is the way to find photos that have gone from Flickr and need replacing
on the wiki.

## Running under cron

```cron
23 4 * * *  /home/you/bin/rgl-photo-backup
```

With `quiet: true` a clean run prints nothing, so cron only mails when a
photo failed. A quiet run that failed ends with a single `N failed` line
after the individual failures. Failures list the photo and the wiki pages
using it, which is how you learn a Flickr photo has gone.

## Settings

A flag beats the environment, which beats the file, which beats the default.

| Key | Flag | Environment | Default |
|---|---|---|---|
| `directory` | positional | | required |
| `flickr_auth` | `--flickr-config FILE` | | `~/.config/flickrtouchr/auth.yml` |
| `immich_auth` | `--immich-config FILE` | | `~/.config/immich/auth.yml` |
| `index_url` | `--index-url URL` | | the RGL index endpoint |
| `fetch` | `--sync-only` sets false | | `true` |
| `sync` | `--fetch-only` sets false | | `true` |
| `dry_run` | `--dry-run` | | `false` |
| `report` | `--report` | | `false` |
| `limit` | `--limit N` | | unlimited |
| `pace` | `--pace SECONDS` | `RGL_PHOTO_BACKUP_PACE` | `1` |
| `quiet` | `--quiet`, `-q` | | `false` |
| `attempts` | | `RGL_PHOTO_BACKUP_ATTEMPTS` | `8` |
| `backoff` | | `RGL_PHOTO_BACKUP_BACKOFF` | `15` |
| `timeout` | | `RGL_PHOTO_BACKUP_TIMEOUT` | `60` |
| `user_agent` | | `RGL_PHOTO_BACKUP_USER_AGENT` | `rgl-photo-backup/1.0` |
| `debug` | `--debug` | `RGL_PHOTO_BACKUP_DEBUG` | `false` |

`--settings FILE` reads a different settings file; it is a flag only, since
it says where the rest of the settings come from.

## Tests

```
make test    # offline unit tests
make lint
```