Skip to content

Commit

Permalink
Add install & run instructions
Browse files Browse the repository at this point in the history
  • Loading branch information
SimmonsRitchie committed Dec 13, 2023
1 parent 3066a5f commit cd52b9e
Showing 1 changed file with 40 additions and 8 deletions.
48 changes: 40 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,21 +3,53 @@
[![CI build status](https://github.com/City-Bureau/city-scrapers/workflows/CI/badge.svg)](https://github.com/City-Bureau/city-scrapers/actions?query=workflow%3ACI)
[![Cron build status](https://github.com/City-Bureau/city-scrapers/workflows/Cron/badge.svg)](https://github.com/City-Bureau/city-scrapers/actions?query=workflow%3ACron)

## Who are the City Bureau Documenters, and why do they want to scrape websites?
A collection of scrapers for public meetings in Illinois, with a particular focus on Chicago.

## Getting Started

### Install

View our [installation guide](https://cityscrapers.org/docs/development/) for detailed instructions.

### Run

1. Activate the virtual environment:

```bash
pipenv shell
```

2. Run a scraper:

```bash
scrapy crawl <spider_name>
```

3. For example, to trigger the `chi_citycouncil` spider to scrape Chicago City Council meeting information run:

```bash
scrapy crawl chi_citycouncil
```

Alternatively, you can watch this [three minute video](https://www.youtube.com/watch?v=4Z3zXQYJXZc) to see how to run a scraper.

## About this project

### Who are the City Bureau Documenters, and why do they want to scrape websites?

Public meetings are important spaces for democracy where any resident can participate and hold public figures accountable. [City Bureau's Documenters program](https://www.documenters.org/) pays community members an hourly wage to inform and engage their communities by attending and documenting public meetings.

How does the Documenters program know when meetings are happening? It isn’t easy! These events are spread across dozens of websites, rarely in useful data formats.

That’s why City Bureau is working together with a team of civic coders to develop and coordinate the City Scrapers, a community open source project to scrape and store these meetings in a central database.

## What are the City Scrapers?
### What are the City Scrapers?

The City Scrapers collect information about public meetings. Every day, the City Scrapers go out and fetch information about new meetings from the Chicago city council's website, the local school council's website, the Chicago police department's website, and many more. This happens automatically so that a person doesn't have to do it. The scrapers store all of the meeting information in a database for journalists at City Bureau to report on them.

Community members are also welcome to use this code to set up their own databases.

## What can I learn from working on the City Scrapers?
### What can I learn from working on the City Scrapers?

A lot about the City of Chicago! What is City Council talking about this week? What are the local school councils, and what community power do they have? What neighborhoods is the police department doing outreach in? Who governs our water?

Expand All @@ -29,33 +61,33 @@ From building a scraper, you'll gain experience with:
- a basic data file format (JSON), working with a schema and data validation
- problem-solving, finding patterns, designing robust code

## How can I set up City Scrapers for my area?
### How can I set up City Scrapers for my area?

This repo is focused on Chicago, but you can set up City Scrapers for your area by following the instructions in the [City-Bureau/city-scrapers-template](https://github.com/city-bureau/city-scrapers-template) repo.

## Community Mission
### Community Mission

The City Bureau Labs community welcomes contributions from everyone. We prioritize learning and leadership opportunities for under-represented individuals in tech and journalism.

We hope that working with us will fill experience gaps (like using git/GitHub, working with data, or having your ideas taken seriously), so that more under-represented people will become decision-makers in both our community and Chicago’s tech and media scenes at large.

## Ready to code with us?
### Ready to code with us?

1. [Fill out this form](https://airtable.com/shrRv027NLgToRFd6) to join our Slack channel and meet the community.
2. Read about [how we collaborate](https://github.com/City-Bureau/city-scrapers/blob/main/CONTRIBUTING.md) and review our [Code of Conduct](https://github.com/City-Bureau/city-scrapers/blob/main/CODE_OF_CONDUCT.md).
3. Check out our [documentation](https://cityscrapers.org/docs/development/), and get started with [Installation](https://cityscrapers.org/docs/development/#installation) and [Contributing a spider](https://cityscrapers.org/docs/development/#contribute).

We ask all new contributors to start by writing a spider and its documentation or fixing a bug in an existing one in order to gain familiarity with our code and culture. Reach out on Slack for support if you need it.

## Don't want to code?
### Don't want to code?

[Join our Slack channel](https://airtable.com/shrRv027NLgToRFd6) (chatroom) to discuss ideas and meet the community!

A. We have ongoing conversations about what sort of data we should collect and how it should be collected. Help us make these decisions by commenting on [issues with a non-coding label](https://github.com/City-Bureau/city-scrapers/issues?q=is%3Aissue+is%3Aopen+label%3Anon-coding).

B. Research sources for public meetings. Answer questions like: Are we scraping events from the right websites? Are there local agencies that we're missing? Should events be updated manually or by a scraper? [Triage events sources on these issues](https://github.com/City-Bureau/city-scrapers/issues?q=is%3Aissue+is%3Aopen+label%3A%22non-coding%3A+triage+events+source%22).

## Support this work
### Support this work

This project is organized and maintained by [City Bureau](http://www.citybureau.org/).

Expand Down

0 comments on commit cd52b9e

Please sign in to comment.