IL: Bill scraper rewrite to new beta site #5121
Conversation
|
@jessemortenson I've rewritten bill and event scraper for new website. Please review it and let me know if you have issues. |
There was a problem hiding this comment.
Thanks! overall code looks good, but I do want to request a change to reduce the number of domain name string literals in here (beta.ilga.gov) so we have an easier transition later.
Events scraper fully reviewed and looks good.
I'm still running the bills scraper so will finish review of that later.
scrapers/il/events.py
Outdated
| event, name = self.scrape_page( | ||
| meeting.attrib["href"], chamber_names[chamber] | ||
| ) | ||
| meeting_url = "https://beta.ilga.gov" + meeting.attrib[ |
There was a problem hiding this comment.
Please use a BASE_URL constant (or similar variable) in place of the literal string https://beta.ilga.gov. I expect this URL will change when they promote the beta site, and I'd like to be able to change one variable per scraper at that time (rather than having to go through code and find all the instances of this domain name string).
The urls array constant at the top of this file is OK
scrapers/il/bills.py
Outdated
| params["num2"] = "10000" | ||
| params["DocTypeID"] = doc_type | ||
| html = self.get(self.LEGISLATION_URL, params=params).text | ||
| url = "https://beta.ilga.gov/Legislation/RegularSession/{}?SessionId={}".format( |
There was a problem hiding this comment.
Please use a BASE_URL constant (or similar variable) in place of the literal string https://beta.ilga.gov
scrapers/il/bills.py
Outdated
| def scrape_archive_bills(self, session): | ||
| session_abr = session[0:2] | ||
| url = f"https://www.ilga.gov/legislation/legisnet{session_abr}/{session_abr}gatoc.html" | ||
| url = f"https://beta.ilga.gov/documents/legislation/legisnet{session_abr}/{session_abr}gatoc.html" |
There was a problem hiding this comment.
Please use a BASE_URL constant (or similar variable) in place of the literal string https://beta.ilga.gov
scrapers/il/bills.py
Outdated
| ) | ||
| ) | ||
| # if it's html, extract the pdf link too while we're here. | ||
| pdf_url = f"https://beta.ilga.gov/documents/legislation/{session_number}/{doctype}/PDF/{version_id}.pdf" |
There was a problem hiding this comment.
Please use a BASE_URL constant (or similar variable) in place of the literal string https://beta.ilga.gov
|
@braykuka Did you notice anything interesting about the Beta site compared to the current site? I ran the scraper ( |
|
I fixed the issue for Base url and amount of different. |
There was a problem hiding this comment.
Bill data looks good! I am still checking one thing before I merge.
I do have one change request: there is a single Appointment "bill" that has an Abstract with an empty string:
jesse@greenbookwork:~/repo/openstates/braykuka/openstates-scrapers/scrapers/_data/il$ rg AM1030365
bill_91f98c90-b876-11ef-ad10-4c034f4dc15f.json
1:{"legislative_session": "103rd", "identifier": "AM1030365", "title": "APPOINT-RENYSHA BROWN", "from_organization": "~{\"classification\": \"upper\"}", "classification": ["appointment"], "subject": [], "abstracts": [{"note": "", "abstract": ""}], "other_titles": [], "other_identifiers": [], "actions": [{"description": "Received by the Senate Sen. Laura M. Murphy", "date": "2023-11-06", "organization_id": "~{\"classification\": \"lower\"}", "classification": [], "related_entities": []}, {"description": "Assigned to Executive Appointments", "date": "2023-11-06", "organization_id": "~{\"classification\": \"lower\"}", "classification": ["referral-committee"], "related_entities": []}, {"description": "Recommends Do Consent Executive Appointments; 005-000-000", "date": "2024-11-20", "organization_id": "~{\"classification\": \"lower\"}", "classification": [], "related_entities": []}, {"description": "Placed on Calendar Order of Executive Appointments November 21, 2024", "date": "2024-11-20", "organization_id": "~{\"classification\": \"lower\"}", "classification": [], "related_entities": []}], "sponsorships": [{"name": "Laura M. Murphy", "classification": "primary", "entity_type": "person", "primary": true, "person_id": "~{\"chamber\": \"upper\", \"name\": \"Laura M. Murphy\"}", "organization_id": null}], "related_bills": [], "versions": [{"note": "Introduced", "links": [{"url": "https://beta.ilga.gov/documents/legislation/103/AM/PDF/10300AM1030365.pdf", "media_type": "application/pdf"}, {"url": "https://beta.ilga.gov/Legislation/BillStatus/FullText?LegDocId=189154&DocName=10300AM1030365&DocNum=1030365&DocTypeID=AM&LegID=150918&GAID=17&SessionID=112&SpecSess=&Session=&Print=1", "media_type": "text/html"}], "date": "", "classification": ""}], "documents": [], "citations": [], "sources": [{"url": "https://beta.ilga.gov/Legislation/BillStatus?DocNum=1030365&GAID=17&DocTypeID=AM&LegId=150918&SessionID=112", "note": ""}], "extras": {}, "_id": "91f98c90-b876-11ef-ad10-4c034f4dc15f"}
jesse@greenbookwork:~/repo/openstates/braykuka/openstates-scrapers/scrapers/_data/il$ cat bill_91f98c90-b876-11ef-ad10-4c034f4dc15f.json | jq .abstracts
[
{
"note": "",
"abstract": ""
}
]
This fails validation, which throws an error at the end of the scrape. Can you please change to ensure that no Abstract is created that has an empty abstract string property? Basically: if there is no abstract/summary text for the bill, then we don't want to add an abstract at all for that bill.
|
@jessemortenson , I have fixed the abstract issue.
|
|
Thanks! I'll merge in |
1271