Initial build of PrimeGov scraper

[krammy19]

We need to design a web scraper for the PrimeGov agenda hosting platform. Examples of cities that use this software include San Mateo and Pleasanton

Input: a path endpoint or a batch .csv file of endpoints to scrape. This url should point to the specific page on a city website where agendas are listed out for public review.

Output: a .csv table with the following column data for each agenda listed on this page:

{Column} | {Description}
Index | Autoincrement index
City | City or agency name
Meeting Name | Title of government body
Date / Time | Date / time of meeting
Agenda | URL of agenda pdf
Meeting video | URL of meeting video (if available)
Published minutes | URL of minutes pdf (if available)

For reference and examples, please see this scraping walkthrough for Legistar.

As you build this scraper, keep in mind that we will need to eventually add additional features, including:

Accessing past agendas not included on the city’s main page, e.g. past years.

Filtering agenda scraping by Date range

Filtering agenda scraping by Meeting Name

Downloading all agendas from scraped urls into a specified directory

Scraping staff report urls from scraped agendas.

[skyheat47295]

@krammy19 I'd like to volunteer to tackle this one.
Thanks, Hamilton Dutcher

[krammy19]

Sure, that would be a huge help! Thanks @skyheat47295

[skyheat47295]

@krammy19 , @shengxio Hello, my first draft is ready for review. I can create a PR, or you can review the code here.
https://github.com/skyheat47295/PrimeGov_scraper

The code works, and brings down a .csv file with the required data, however;
There is an issue is regarding the javascript interpretation of the href link.
Shall we set up a time to discuss?

[shengxio]

@skyheat47295 , @krammy19 Hello and yes definitely.
I am normally free on Saturday, Monday all day and Thursday before 3pm.
Let me know which day works the best for you guys, and we can shuffle from there.

To be honest, I am still learning what you did in the repo so.
We may need 30 mins to an hour time for the meeting if you don't mind.

Thanks!

Roland Ding

[skyheat47295]

@shengxio @krammy19 I'm available during those times. If you want to just touch base, you can DM me on Slack, and we can schedule something for tomorrow or next week. I can walk you through what I did, and the thoughts I have.

Thanks!

H.

[shengxio]

@krammy19 Hey Mark, I wonder if this task has been completed?

Thanks!
R.