feat(scraper): import club events from TartanConnect #93

Merged

jalenluorion merged 4 commits from feat/tartanconnect-import into main

2026-10-03 22:47:01 +00:00

jalenluorion commented

2026-10-03 21:56:46 +00:00

Member

Imports club events from TartanConnect, so Explore has non-class events. Closes #70.

python -m scraper.scripts.export_tartanconnect [--dry-run]: one CLUB org per host club, a TartanConnect category in each, one event per listing (RSVP link in source_url, tags in the description). Also a tartanconnect choice in scrape.yml.
The monitor parses the public feed with json.loads and pages through all of it (it fetched 300 by string splitting before), with retries on the feed's 502/503s.
fix(api): batch occurrence regeneration skipped events without a recurrence rule, so one-time events never reached the calendar or saved events. They now get exactly one ONETIME occurrence. SOC events all have rules, so they are unaffected.
Events this source created that are still in the future but no longer in the feed are deleted (cancelled or changed). Past events, hand-made events and other sources are never touched, and the deletion is skipped if more than a fifth of the source's future events would go, so a short feed cannot wipe saves.
regenerate_occurrences moved out of export_soc.py so both exports share it.

Tested on dev: 785 events from 60 clubs, one occurrence each, four reruns with identical ids and no duplicates. 270 API tests pass; the fixture is a trimmed copy of the real feed.

Not in this PR: a Register button (no component renders source_url yet; see #74).

Imports club events from TartanConnect, so Explore has non-class events. Closes #70. - `python -m scraper.scripts.export_tartanconnect [--dry-run]`: one CLUB org per host club, a TartanConnect category in each, one event per listing (RSVP link in source_url, tags in the description). Also a `tartanconnect` choice in scrape.yml. - The monitor parses the public feed with json.loads and pages through all of it (it fetched 300 by string splitting before), with retries on the feed's 502/503s. - fix(api): batch occurrence regeneration skipped events without a recurrence rule, so one-time events never reached the calendar or saved events. They now get exactly one ONETIME occurrence. SOC events all have rules, so they are unaffected. - Events this source created that are still in the future but no longer in the feed are deleted (cancelled or changed). Past events, hand-made events and other sources are never touched, and the deletion is skipped if more than a fifth of the source's future events would go, so a short feed cannot wipe saves. - `regenerate_occurrences` moved out of export_soc.py so both exports share it. Tested on dev: 785 events from 60 clubs, one occurrence each, four reruns with identical ids and no duplicates. 270 API tests pass; the fixture is a trimmed copy of the real feed. Not in this PR: a Register button (no component renders source_url yet; see #74).

jalenluorion added 4 commits

2026-10-03 21:56:46 +00:00

refactor(scraper): share occurrence regeneration between exports ecbc05e19a

Move regenerate_occurrences from export_soc.py to
scraper/persistence/api_occurrences.py so the TartanConnect export can
call it too. No behavior change.

fix(api): give one-time events their occurrence in batch regeneration c466ac77c3

regenerate_occurrences_by_events skipped every event without a recurrence
rule, so a scraper importing one-time events got no event_occurrences and
the events never reached the calendar or saved events. Replace such an
event's occurrences with one ONETIME row built from the event instead.

feat(scraper): import club events from TartanConnect a170a0169c

Rewrite the TartanConnect monitor on json.loads: skip separator rows,
page through range until the feed runs out, and keep the event ID, RSVP
URL, host club, categories and times (America/New_York; the few events
listed in another zone are converted).

Add scraper/scripts/export_tartanconnect.py, modeled on export_soc.py:
one CLUB org and one TartanConnect category per host club, one-time CLUB
events upserted on the existing identity, then occurrence regeneration.
Future events from earlier runs that the feed no longer lists (cancelled,
removed or changed) are deleted; past events and other sources' events
are left alone.

Refs #70

fix(scraper): skip stale-event deletion when the feed looks short

kennel/build build succeeded

Details

CI / check-1 (pull_request) Successful in 2m23s

Details

CI / build (pull_request) Successful in 2m18s

Details

CI / check (pull_request) Successful in 0s

Details

4c50a23e60

Deleting cascades to users' saves, so a feed that loads but is missing
events must not wipe them. Skip the deletion when more than a fifth of the
source's future events would go, and never delete rows without a run id.

jalenluorion merged commit 78f654051f into main

2026-10-03 22:47:01 +00:00

jalenluorion deleted branch feat/tartanconnect-import

2026-10-03 22:47:01 +00:00

jalenluorion referenced this pull request from a commit

2026-10-03 22:47:02 +00:00

feat(scraper): import club events from TartanConnect (#93)