MI events: failing on duplicate data in scrape

[braykuka]

1261

• added status value to dedupe_key

[jessemortenson]

Thanks! Can you share your thinking behind adding the status variable to the dedupe key? I'm imagining a scenario:

1. Yesterday: the scraper ingests Event A which has status tentative
2. Today: the scraper ingests the same Event A, but status has been changed on the source website to cancelled.
3. What should happen is that the data pipeline updates the existing Event A to change its status from tentative to cancelled, because it matches on the existing dedupe key

I'm worried that adding status to the dedupe key will instead cause a duplicate event to be created, so that there will be both an Event A - Tentative and an Event A - Cancelled.

[braykuka]

As you can see, there are same duplicates for 11/7 and 11/14. They are same event name and same event date & time.
For 11/7, the first meeting is tentative and the second meeting is cancelled.
For 11/14, the first meeting is cancelled and the second meeting is tentative.

Could you please let me know which one should be the duplicated event in these cases?
Thanks.

[jessemortenson]

Thank you for the screenshot, I can see how that is confusing!

Looking at the source website, it appears that each of those events has a unique URL:

• https://legislature.mi.gov/Committees/Meeting?meetingID=e74dc891fd6c4c3dadd31619b75139c0
• https://legislature.mi.gov/Committees/Meeting?meetingID=fd39b8de57ed45de87225401b059c139
• https://legislature.mi.gov/Committees/Meeting?meetingID=484957deeeea470d9576e39819b506a7
• https://legislature.mi.gov/Committees/Meeting?meetingID=51c4adda616f41cbae02bd6d5209dfa2

The content of the pages on those URLs does differ: the agenda seems to be different between the "cancelled" and the "new" events (even though the name, location, time are all the same). So it seems these are actually not duplicate events in the source website (my assumption was wrong).

Those URLs seem to imply each event has a unique meetingID. If that is true, perhaps the meetingID is the best dedupe key for MI events? When we have a "natural" unique identifier that seems to be consistent at a source site, that can be a good candidate for the dedupe key.

[braykuka]

@jessemortenson Thank you for the update. I've changed dedupe_key with meetingID. Please review it again.

[jessemortenson]

LGTM, merging in