Fix HAN, OSCM, and WUR scrapers broken by site redesigns - #1376
Open
ioan-alexandra wants to merge 3 commits into
Open
Fix HAN, OSCM, and WUR scrapers broken by site redesigns#1376ioan-alexandra wants to merge 3 commits into
ioan-alexandra wants to merge 3 commits into
Conversation
- WUR: old RSS feed is dead; parse activity data out of the new Next.js page's embedded React Server Components payload instead. - OSCM: site migrated events plugins, update the list selector to match the new events-table markup. - HAN: course detail pages lost their structured date/venue markup; extract the start date and venue from the "Data, tijden & locatie" accordion text instead. Also fixes a latent bug that double-prefixed the one listing entry that already had an absolute URL. Test cassettes rebuilt from live site snapshots.
process_wur discarded its url argument and hardcoded the activities page URL internally, so ingestion.yml's url field had no effect on what actually got scraped. Use the passed-in url so ingestion.yml stays the single source of truth if WUR moves the page again.
ioan-alexandra
force-pushed
the
fix/broken-scrapers
branch
from
September 8, 2026 12:27
f2fb4a4 to
cf3e1f4
Compare
The cassette recorded 25 candidate courses, of which 5 genuinely have no exact date published (only month+year) and are correctly skipped - 20 succeed, not 21 as the test asserted. Verified this was already wrong in the original commit, independent of the later rebase: running the same test against the pre-rebase commit in isolation reproduces the identical 20-vs-21 failure.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
All three of these scrapers had been silently returning 0 events/materials because the sites they scrape changed structure:
WUR: the RSS calendar feed (
.../Resources-1/RSS/Calendar.htm) has been retired entirely (404). Rewrote the ingestor to instead parse activity data out of the new Next.js page's embedded React Server Components JSON payload (https://www.wur.nl/en/news-insights/activities-at-wur).OSCM: the events page still works, but the site migrated to a different events plugin. Updated the list selector from
.eventname > a(no longer present) to.events-table a, matching the new table-based markup.HAN: the course listing page is unchanged, but course detail pages lost the structured CSS classes the ingestor relied on for date/venue (
.course-superhero__payoff__content,.course-details__sidebar__item). These now returnnil, causingundefined method 'css' for nilon every course. Fixed by extracting the start date and venue from the "Data, tijden & locatie" accordion text via regex instead of relying on specific CSS classes.Updated the corresponding cassettes.
Verified against the live sites via a local TeSS/Taxila stack: