Skip to content

feat(representatives): self-contained data pipeline - fresh-by-default imports + senator contacts & photos

Scaffold Digital Fleet requested to merge feature/reps-data-pipeline into master

What this MR does

Makes the representative pipeline self-contained in risefor-lobbying and fresh-by-default. Root cause fixed: the AN master files the import read were git-tracked in law-tracker and silently frozen at 2024-07-30 for 2+ years (senators elected after that date, e.g. Maingault PA885050, were invisible).

New commands

  • update-representative-data - download AN dep/sen/min zip, extract per-uid splits, merge into risefor_lobbying/data/src/an/.../master/*.json (git-ignored, rebuilt every run - never stale again).
  • harvest-senat-contacts - scrape senat.fr senator profile Contact blocks: personal email (AN data has only 27/348; harvest took us to 344/365), twitter/facebook/instagram/linkedin (institutional accounts filtered), HATVP -> pageHatvp. Incremental, polite delay, --dry-run/--limit/--force. Phone on senat.fr pages is the switchboard - never imported.

Changed

  • import_elected_gouv: DEFAULT = download fresh data first (via update-representative-data), then import. Opt-outs: --no-download-data (warns 'may be stale'), --no-harvest-senat (skips the default senat.fr harvest that runs after import when --type senateur/all). Download failure aborts rather than silently importing stale data.
  • Source paths resolve own-package dir first; law_tracker/ legacy dir still works as fallback -> pipeline runs on instances WITHOUT the law-tracker plugin.
  • Data loading lazy (module-level file opens removed) -> manage.py boots safely even without data files.
  • 'Adresse officielle' now imported (senators official postal address; never overwrites a circo address).
  • Unknown department codes (977/978/986/988) log+skip instead of raising and killing the run.
  • get_representative_pictures v2: senators via senat.fr /senimg/{slug}_carre.jpg (AN hosts no senator photos); deputies via AN photo CDN direct pattern; MEDIA_ROOT instead of relative ./media; timeouts; honest counters (double-increment fixed); --dry-run/--limit.

Verified (isolated DB copies, update-path from full dev DB with 2,188 reps)

  • 348/348 fresh senator uids matched; senators: 344/365 email, 365/365 photo, 348 pageParlement, 49 pageHatvp (was 0)
  • deputies: 564 (1 genuine new deputy), 577/577 hemicycle seats, photos + emails dedup exactly (zero new dup rows)
  • idempotent re-runs verified (harvest 51->23 targets on 2nd run; pictures skip already-photographed)
  • chained flags verified live: default downloads+imports+harvests in ONE command; opt-outs warn and skip

Prod flow after merge

python manage.py import_elected_gouv            # all types, fresh data + senat harvest
python manage.py get_representative_pictures    # photos

No migrations, no model changes. reversion per save. Requires qcluster running (same as before - email post_save signal) and circo-commune-cp.csv present (unchanged requirement).

Closes the senator-data-freeze class of bugs: masters are generated artifacts now, never committed.

Merge request reports