fix(housekeeping): enforce backup retention (delete backups older than 14d)
Closes #216 (closed) (ticket originally filed in risefor-lobbying#54 (moved), moved to this repo by Catalyst — housekeeping lives in united4earth).
What was broken
The daily backup (django-q, 01:00) had no working retention path:
-
historical/was never created — themv(subprocess, stderr→DEVNULL) failed silently forever, socleanup_old_backups()listed a nonexistent directory → nothing moved, nothing deleted. - Deletion was prompt/count-gated:
>2files past-due → auto-delete (prompt commented out), exactly 2 → nothing, 1 → delete. Stale backups lingered indefinitely. -
rmfailures invisible (stderr=DEVNULL) whilelogger.success("deleted backup")fired unconditionally. - Zips were written then source removed with no failure check between.
Net effect: housekeeping/database-backup/ grows unbounded → instances fill their disk.
The fix (housekeeping/backup_database.py)
-
enforce_retention()— final guaranteed step of every backup run: deletes anybackup_*.json/backup_*.zipolder thandays_to_keep=14in the main dir ANDhistorical/. Unconditional, idempotent, no prompts (runs headless under qcluster). Logs each deletion with filename + age, or a clean "nothing older" line. Runs even if dump/zip failed earlier — disk must never fill silently. -
historical/created viaos.makedirs(exist_ok=True)before moves. - Move via
os.replace, zip verified before removing the source. -
cleanup_old_backups()kept as deprecated alias for compatibility. - Bonus:
apps.pyfresh-install guard (apps.ready() querieddjango_q_schedulebefore migrations →migratecrashed on new DBs; found locally 2026-08-31).
Verified locally
Seeded historical/ with 15/20/25d zips + a 25d stale .json in main dir + a fresh zip:
SUCCESS Retention: deleted backup_2026-08-10.json (25d old)
SUCCESS Retention: deleted backup_2026-08-20.zip (15d old)
SUCCESS Retention: deleted backup_2026-08-15.zip (20d old)
SUCCESS Retention: deleted backup_2026-08-10.zip (25d old)
NOTICE Retention: deleted 4 backup(s) older than 14d
Fresh zip kept ✓. Second run = no-op (idempotent) ✓.
Prod notes
- The schedule is created by
apps.pyon startup (func unchanged) — merge + pull = the 01:00 run self-cleans. - First prod run may delete a large batch at once (weeks/months of accumulated zips) — that's the point; disk will breathe. Worth a glance at
housekeeping/database-backup/after the first nightly run. - One existing
backup_*file can be kept deliberately bytouching its mtime if you want a manual snapshot older than 14d.