Skip to content

fix(housekeeping): enforce backup retention (delete backups older than 14d)

Scaffold Digital Fleet requested to merge fix/backup-retention into risefor_main/master

Closes #216 (closed) (ticket originally filed in risefor-lobbying#54 (moved), moved to this repo by Catalyst — housekeeping lives in united4earth).

What was broken

The daily backup (django-q, 01:00) had no working retention path:

  1. historical/ was never created — the mv (subprocess, stderr→DEVNULL) failed silently forever, so cleanup_old_backups() listed a nonexistent directory → nothing moved, nothing deleted.
  2. Deletion was prompt/count-gated: >2 files past-due → auto-delete (prompt commented out), exactly 2 → nothing, 1 → delete. Stale backups lingered indefinitely.
  3. rm failures invisible (stderr=DEVNULL) while logger.success("deleted backup") fired unconditionally.
  4. Zips were written then source removed with no failure check between.

Net effect: housekeeping/database-backup/ grows unbounded → instances fill their disk.

The fix (housekeeping/backup_database.py)

  • enforce_retention() — final guaranteed step of every backup run: deletes any backup_*.json/backup_*.zip older than days_to_keep=14 in the main dir AND historical/. Unconditional, idempotent, no prompts (runs headless under qcluster). Logs each deletion with filename + age, or a clean "nothing older" line. Runs even if dump/zip failed earlier — disk must never fill silently.
  • historical/ created via os.makedirs(exist_ok=True) before moves.
  • Move via os.replace, zip verified before removing the source.
  • cleanup_old_backups() kept as deprecated alias for compatibility.
  • Bonus: apps.py fresh-install guard (apps.ready() queried django_q_schedule before migrations → migrate crashed on new DBs; found locally 2026-08-31).

Verified locally

Seeded historical/ with 15/20/25d zips + a 25d stale .json in main dir + a fresh zip:

SUCCESS Retention: deleted backup_2026-08-10.json (25d old)
SUCCESS Retention: deleted backup_2026-08-20.zip (15d old)
SUCCESS Retention: deleted backup_2026-08-15.zip (20d old)
SUCCESS Retention: deleted backup_2026-08-10.zip (25d old)
NOTICE  Retention: deleted 4 backup(s) older than 14d

Fresh zip kept ✓. Second run = no-op (idempotent) ✓.

Prod notes

  • The schedule is created by apps.py on startup (func unchanged) — merge + pull = the 01:00 run self-cleans.
  • First prod run may delete a large batch at once (weeks/months of accumulated zips) — that's the point; disk will breathe. Worth a glance at housekeeping/database-backup/ after the first nightly run.
  • One existing backup_* file can be kept deliberately by touching its mtime if you want a manual snapshot older than 14d.
Edited by Scaffold Digital Fleet

Merge request reports