rgl-photo-backup

Back up the photos the Randomness Guide to London shows on its pages. Nearly all of them live on Flickr, which is a single point of failure. This fetches the largest available version of each into a directory, alongside a JSON sidecar of what the wiki and Flickr know about it, then mirrors the directory into an Immich album with the source, photographer, licence and wiki pages carried into each asset's description, tags, capture date and location.

Requirements

Python 3 and nothing else.

Setup

Two credentials, both reused from other tools if you already have them:

• ~/.config/flickrtouchr/auth.yml with a key: line holding a Flickr API key. Public photos need no OAuth, so the key alone is enough. Create one at flickr.com/services/apps/create if you have none.
• ~/.config/immich/auth.yml with url: (your Immich API base, ending in /api) and key: (an API key from Immich's account settings).

Everything else can go in ~/.config/rgl-photo-backup/settings.yml:

directory: /srv/backup/rgl
pace: 1
quiet: true

Make the credential files chmod 600.

Running

./rgl-photo-backup /srv/backup/rgl            # fetch, then sync
./rgl-photo-backup --dry-run /srv/backup/rgl  # report only
./rgl-photo-backup --fetch-only --limit 5 DIR # a trial
./rgl-photo-backup --sync-only DIR            # disk to Immich only
./rgl-photo-backup --settings other.yml DIR   # a different settings file
./rgl-photo-backup --debug DIR                # log every request to stderr
./rgl-photo-backup -q DIR                     # same as --quiet
./rgl-photo-backup --report DIR               # which pages have no photo on disk, and why

The first run downloads about 4800 photos, one Flickr info call and one sizes call each, so at the default one-second pace it takes a few hours. The pace applies to Flickr alone; Immich and the wiki index are not slowed. A 429 from Flickr doubles the pace for the rest of the run, up to ten seconds between calls, and it relaxes again after twenty untroubled requests. The photo that hit the limit is retried with backoff starting at fifteen seconds and doubling, up to eight attempts. Later runs skip everything already on disk, and in Immich only write an album membership or a description that has actually changed.

Exit status is 0 for a clean run, 1 if any photo failed, 2 for a configuration problem, and 130 if you interrupt it.

The first run

Take it in stages, so a misconfiguration costs a few photos rather than thousands:

1. --fetch-only on its own, to fill the backup directory.
2. --sync-only --limit 20, and look at the twenty assets in Immich.
3. the full run.

Immich queues its own metadata extraction after an upload, so on a first run a description written straight afterwards can be overwritten by the extraction finishing. This tool writes descriptions after everything else for that reason, but if any of them come out wrong, run it again: the next run compares against what Immich holds and puts them right.

Three things worth checking with curl against your own Immich version before leaving it unattended, since the API has moved before:

• POST /api/search/metadata accepts withExif, withArchived and size
• POST /api/assets honours the x-immich-checksum header
• PUT /api/tags returns value as the full slash-separated path

What lands in Immich

• An album named RGL holding every photo. Membership is read first, so only photos not already in it are added.

• Tags rgl, rgl/licence/<licence> and rgl/photographer/<name>. Unlike the album and the description, tag assignments are sent every run: Immich answers an asset that already carries the tag with duplicate, and those are reported as already tagged rather than treated as a failure.

• A description block, appended below anything already there and refreshed in place on later runs:

  Randomness Guide to London photo
  Photo: <Flickr title>
  Source: <Flickr photo page>
  Photographer: <name>
  Licence: CC BY-SA 2.0 <licence url>
  Used on:
    <page name> <page url>
  

• The capture date from Flickr and the page's coordinates, only where the asset has none already.

A photo already in your library, for instance one of your own brought in by another backup, is recognised by checksum: it joins the album and gets the block appended rather than being uploaded twice.

The backup directory

One file per photo, named by Flickr id (3750465402.jpg), beside a sidecar (3750465402.json) holding the wiki fields, the Flickr info and the pages that use it. The twenty-odd photos not on Flickr have no id to use, so they are named url- and the first sixteen hex characters of the SHA1 of their image URL (url-0f4b1c2d3e4f5a6b.jpg), with a matching sidecar. The directory is a complete backup on its own. A .part file is an interrupted download and is refetched next run.

.failures.json records the last reason each photo failed, with the pages that use it. A photo that later arrives drops out of it. --report reads the wiki index and that file, and lists every page whose photo is not on disk with the reason, or not yet attempted, without calling Flickr or Immich, so it is the way to find photos that have gone from Flickr and need replacing on the wiki.

Running under cron

23 4 * * *  /home/you/bin/rgl-photo-backup

With quiet: true a clean run prints nothing, so cron only mails when a photo failed. A quiet run that failed ends with a single N failed line after the individual failures. Failures list the photo and the wiki pages using it, which is how you learn a Flickr photo has gone.

Settings

A flag beats the environment, which beats the file, which beats the default.

Key	Flag	Environment	Default
directory	positional		required
flickr_auth	--flickr-config FILE		~/.config/flickrtouchr/auth.yml
immich_auth	--immich-config FILE		~/.config/immich/auth.yml
index_url	--index-url URL		the RGL index endpoint
fetch	--sync-only sets false		true
sync	--fetch-only sets false		true
dry_run	--dry-run		false
report	--report		false
limit	--limit N		unlimited
pace	--pace SECONDS	RGL_PHOTO_BACKUP_PACE	1
quiet	--quiet, -q		false
attempts		RGL_PHOTO_BACKUP_ATTEMPTS	8
backoff		RGL_PHOTO_BACKUP_BACKOFF	15
timeout		RGL_PHOTO_BACKUP_TIMEOUT	60
user_agent		RGL_PHOTO_BACKUP_USER_AGENT	rgl-photo-backup/1.0
debug	--debug	RGL_PHOTO_BACKUP_DEBUG	false

--settings FILE reads a different settings file; it is a flag only, since it says where the rest of the settings come from.

Tests

make test    # offline unit tests
make lint