Run the scrapers automatically every week #84

New issue

Open

opened 2026-10-03 19:44:45 +00:00 by jalenluorion · 0 comments

jalenluorion commented

2026-10-03 19:44:45 +00:00

Member

Goal
Campus events change every day. If scrapers only run when someone remembers, the calendar goes stale and students stop trusting it.

Current state

Scrapers run by hand. .forgejo/workflows/scrape.yml is manual dispatch only (sources: soc, peer_tutoring, supplemental_instruction). Its header says to add a schedule: trigger once a run has been watched end to end under kennel.
It sets SECRETSPEC_PROFILE to dev or prod with the openbao-ci provider, but the OpenBao ci JWT role can read only secret/secretspec/cal/ci/ (see the keepalive.yml header). Neither profile can resolve SUPABASE_SERVICE_ROLE_KEY, so the workflow cannot run as written.
Imported iCal sources are never re-synced (sync_ical_source in api/app/services/ical.py has no callers).

Scope

Choose and document an approach:
- a kennel-managed cron job (check whether kennel supports scheduled jobs yet; scrape.yml says it does not), or
- a narrowly scoped OpenBao policy letting ci read only the prod keys the scrapers need (SUPABASE_URL, SUPABASE_SERVICE_ROLE_KEY, API_BASE_URL, SCRAPER_API_TOKEN). This needs a governance change.
Weekly runs for TartanConnect, SASC, SI and peer tutoring; SOC only during add/drop and registration.
Re-sync active calendar sources on the same schedule.
Failures show up somewhere people look (red run plus a message or alert), like the keepalive job.
Update the scrape.yml header (and its stale "roughly an hour" SOC estimate).

Done when

Two weekly runs in a row write to prod with no manual step, and a failed run is noticed within a day.

**Goal** Campus events change every day. If scrapers only run when someone remembers, the calendar goes stale and students stop trusting it. **Current state** - Scrapers run by hand. `.forgejo/workflows/scrape.yml` is manual dispatch only (sources: soc, peer_tutoring, supplemental_instruction). Its header says to add a `schedule:` trigger once a run has been watched end to end under kennel. - It sets `SECRETSPEC_PROFILE` to `dev` or `prod` with the `openbao-ci` provider, but the OpenBao `ci` JWT role can read only `secret/secretspec/cal/ci/` (see the `keepalive.yml` header). Neither profile can resolve `SUPABASE_SERVICE_ROLE_KEY`, so the workflow cannot run as written. - Imported iCal sources are never re-synced (`sync_ical_source` in `api/app/services/ical.py` has no callers). **Scope** - [ ] Choose and document an approach: - a kennel-managed cron job (check whether kennel supports scheduled jobs yet; `scrape.yml` says it does not), or - a narrowly scoped OpenBao policy letting `ci` read only the prod keys the scrapers need (`SUPABASE_URL`, `SUPABASE_SERVICE_ROLE_KEY`, `API_BASE_URL`, `SCRAPER_API_TOKEN`). This needs a governance change. - [ ] Weekly runs for TartanConnect, SASC, SI and peer tutoring; SOC only during add/drop and registration. - [ ] Re-sync active calendar sources on the same schedule. - [ ] Failures show up somewhere people look (red run plus a message or alert), like the keepalive job. - [ ] Update the `scrape.yml` header (and its stale "roughly an hour" SOC estimate). **Done when** - [ ] Two weekly runs in a row write to prod with no manual step, and a failed run is noticed within a day.

jalenluorion added this to the Phase 3 milestone

2026-10-03 19:44:45 +00:00

jalenluorion added the

feature

intermediate

data

labels

2026-10-03 19:44:45 +00:00

jalenluorion added this to the CMUCal project

2026-10-03 19:45:09 +00:00