Import club events from TartanConnect #70

New issue

Closed

opened 2026-10-03 19:44:43 +00:00 by jalenluorion · 0 comments

jalenluorion commented

2026-10-03 19:44:43 +00:00

Member

Goal
Most club events live on TartanConnect. Importing them puts club life on the same calendar as classes, which is the core of the vision.

Current state

Monitor: api/scraper/monitors/career_club/tartanconnect.py (TartanConnectScraper). It calls mobile_ws/v17/mobile_events_list, which needs no login (checked Oct 3: HTTP 200, 860 events listed).
Problems: it asks for limit=300 only, parses the JSON by string splitting instead of json.loads, and builds the RSVP link but throws it away. Many locations come back as "Private Location (sign in to display)". scrape() writes through the legacy update_database, and scrape_data_only() duplicates all of it.
Only an Excel exporter exists (api/scraper/exporters/tartanconnect_export_to_excel.py), with no Supabase export script.

Scope

Rewrite the parser on json.loads, skip separator rows (listingSeparator == "true"), and page through range until all events are fetched.
Keep eventId, the RSVP URL (source_url), host club, categories and times (America/New_York).
Add api/scraper/scripts/export_tartanconnect.py modeled on export_soc.py: --dry-run, agent run, upsert one CLUB org per host (upsert_orgs, conflict on name), one category per org, insert_events with event_type CLUB, then regenerate occurrences. Factor regenerate_occurrences out of export_soc.py instead of copying it.
Handle events that are cancelled or removed between runs.
Add tartanconnect to the scrape.yml source choices.
Tests with a saved JSON fixture in api/tests/scraper_monitors/.

Done when

Dry run shows sane counts, the dev load shows club events in Explore with working Register links, and a rerun creates no duplicates.

Notes
Club orgs match on name, so pilot clubs created by hand will merge with their TartanConnect org. Check spelling before the prod run.

**Goal** Most club events live on TartanConnect. Importing them puts club life on the same calendar as classes, which is the core of the vision. **Current state** - Monitor: `api/scraper/monitors/career_club/tartanconnect.py` (`TartanConnectScraper`). It calls `mobile_ws/v17/mobile_events_list`, which needs no login (checked Oct 3: HTTP 200, 860 events listed). - Problems: it asks for `limit=300` only, parses the JSON by string splitting instead of `json.loads`, and builds the RSVP link but throws it away. Many locations come back as "Private Location (sign in to display)". `scrape()` writes through the legacy `update_database`, and `scrape_data_only()` duplicates all of it. - Only an Excel exporter exists (`api/scraper/exporters/tartanconnect_export_to_excel.py`), with no Supabase export script. **Scope** - [ ] Rewrite the parser on `json.loads`, skip separator rows (`listingSeparator == "true"`), and page through `range` until all events are fetched. - [ ] Keep `eventId`, the RSVP URL (`source_url`), host club, categories and times (America/New_York). - [ ] Add `api/scraper/scripts/export_tartanconnect.py` modeled on `export_soc.py`: `--dry-run`, agent run, upsert one `CLUB` org per host (`upsert_orgs`, conflict on name), one category per org, `insert_events` with `event_type` CLUB, then regenerate occurrences. Factor `regenerate_occurrences` out of `export_soc.py` instead of copying it. - [ ] Handle events that are cancelled or removed between runs. - [ ] Add `tartanconnect` to the `scrape.yml` source choices. - [ ] Tests with a saved JSON fixture in `api/tests/scraper_monitors/`. **Done when** - [ ] Dry run shows sane counts, the dev load shows club events in Explore with working Register links, and a rerun creates no duplicates. **Notes** Club orgs match on name, so pilot clubs created by hand will merge with their TartanConnect org. Check spelling before the prod run.

jalenluorion added this to the Phase 2 milestone

2026-10-03 19:44:43 +00:00

jalenluorion added the

feature

data

labels

2026-10-03 19:44:43 +00:00

jalenluorion added this to the CMUCal project

2026-10-03 19:45:09 +00:00

jalenluorion referenced this issue from a pull request that will close it,

2026-10-03 21:56:46 +00:00

feat(scraper): import club events from TartanConnect #93