Skip to content

Housekeeping backups: no effective retention — instances fill their disk over time

Problem

The backup routine (scheduled daily at 01:00 via django-q in housekeeping) is supposed to keep only ~14 days of database backups, but the deletion path is broken in practice, so backups accumulate forever and instances run out of storage after a while.

Where the code actually lives

housekeeping/backup_database.py is part of the united4earth Django shell (risefor-front repo), not this repo — filing here per Catalyst's ask since risefor-lobbying is the backend home; the fix MR will reference this issue. (If you'd rather mirror the ticket in the front repo, say so.)

The holes (read of housekeeping/backup_database.py)

  1. historical/ dir is never created — the mv to historical/ silently fails (stderr=subprocess.DEVNULL), so cleanup_old_backups(HISTORICAL_DIR, 14) runs os.listdir on a nonexistent directory → raises FileNotFoundError inside the daily task (or silently kills the run depending on qcluster handling). Nothing ever gets moved, nothing ever gets deleted.
  2. >2 gating on deletion — when more than 2 backups are past due, it logs a warning and (after the interactive prompt was commented out) auto-deletes; but if exactly 2 are past due it does nothing ("else: no backups older") — up to 2 stale zips linger forever, repeatedly.
  3. rm via subprocess with stderr swallowed — failures are invisible; logger.success("deleted backup") logs even if rm failed.
  4. JSON files >2 days are zipped but zipping failure leaves both files (json + zip can duplicate) — plus no check that zip succeeded before os.remove.
  5. No size/spacing safety between backup zip rotation steps, and no verification that historical cleanup covers the fresh backup_YYYY-MM-DD.json if the zip step ever fails.

Net effect: on prod instances, housekeeping/database-backup/ grows unbounded → disk full.

Expected behavior

  • After every daily backup: enforce retention — delete backups (zip or json) older than the configured number of days (14, matching fc8fbed3 "backup: update to 14 days").
  • Retention must be enforced unconditionally and idempotently on every run (no prompt, no count-gating), with clear logs of what was deleted and why.
  • The historical/ directory must exist (created on demand) before any move.

Fix approach (MR incoming)

  • Create HISTORICAL_DIR with os.makedirs(..., exist_ok=True) before moving.
  • Replace prompt/>2-gated deletion with straightforward age-based deletion (no interaction — it runs under qcluster).
  • Use os.remove directly (check success), log every deletion with filename + age.
  • Keep zip + move behavior, add retention as the final guaranteed step, run it even if zip/move failed earlier.
  • --test-files mode extended to validate retention end-to-end locally.
Edited by Scaffold Digital Fleet