RFC: Action Categorization Overhaul

[jamesturk]

As part of the work we're doing we want to be able to improve action categorization, part of that will involve reworking how it works so that we can re-categorize actions without a new scrape (important for old sessions). I'd also like to make it easier for people to submit simple fixes to the action categorizers without editing a bunch of code in the scraper itself.

With those goals in mind, here's my proposal:

• pupa gains a new command, essentially pupa categorize-actions nc
  • by default this will run on all actions and print a report of how it categorized actions with enough detail for debugging
• this command relies on writing some new action categorization code and making a few code changes in each state, it will look for two things (in this order)
  • [state].actions.categorize_actions - a function that takes action details (text, chamber, session?) and returns a list (can be empty) of categories
  • [state].actions.categorizers - a list of tuples mapping regex to actions, used if no function is defined (probably sufficient in 90% of cases)

other things that'll need to be done:

• existing action categorization needs to continue to work for backwards compatibility (at least for now)
• presumably the categorizer should be called after a scrape automatically (categorize-actions is mainly for adding categories to historical data)
• we'll need to special case action categories on import to not be overwritten so that update detection doesn't break

Open questions:

• will these two approaches be sufficient? / are any states a lot more complex to categorize?

[showerst]

+1 to overhauling that, would be nice to simplify. A few thoughts:

1. I'd like at least the option to keep categorization in the scrape so that it emits bill objects w/ actions and categories like it does today. Happy to bury this behind a flag or something, but we emit our scraped bills to a queue at the time each bill is 'saved', so a second mass pass after that would require a bunch of extra architecture. It also kinda mucks with the idea that the scrape emits all the data you need (at least for a given data type being scraped).
2. I vaguely remember there being 1 or 2 states where you need to look at previous actions to properly categorize something, i'll check the code base and try to find out if that's still valid.
3. A few states (without looking, maybe OH, IN, KS? maybe CA?) that are api driven take other metadata to categorize (usually codes from a lookup table), so we'll need to accommodate that
4. This might be more of a pupa issue, but I remember that adding new action categorization types was a bit of a haul when I did it a few years ago. It would be nice if this were override-able at the scraper level, both for weird state specific actions we might want to capture, and to make writing scrapers of other levels (municipal, foreign) easier.

[jamesturk]

1. Can you remind me, where does your emit step happen?

@schneidy can you post the sheet you started here so we can collaborate on these issues

and as for 4. 100% agree- Making that customizable is something that's long overdue. I have a few ideas I'll try to write up on a pupa issue at some point

[schneidy]

Here is the sheet that has the initial notes on the levels of difficulty to categorize the actions for each state.

[showerst]

Can you remind me, where does your emit step happen?

In pupa's do_scrape in scrape/base.py we added a handler for custom outputters with their own save_object() function.

https://github.com/GovHawkDC/pupa/blob/25d6b397038bb61805aac1200ea2053f5b2981f1/pupa/scrape/base.py#L94

[jamesturk]

gotcha, so you're bypassing all of pupa.import right?

[showerst]

Yup, skipping that step entirely I believe, and running them all with --scrape

[jamesturk]

Just noting the fact that entity recognition (e.g. 'ADDED H.SMITH AS SPONSOR') is currently part of the categorizers, and we'd want to revisit that too.

[showerst]

Yeah, one vote here for splitting that out into separate logic.

Spit-balling here, does it make sense to do a multi-pass approach:

1. classify action
2. resolve people
3. resolve entities (thinking principally of committees here)

Then we could re-use the people and entity resolution code elsewhere for looking up voters and sponsors.

I'm imagining some built-ins that's are easy to override on a per-jurisdiction level, maybe in the init.py or something.

Not sure if this makes sense as one function that can take an action or a voter or a sponsor (maybe with an arg to differentiate?) or if the logic is going to be different enough that they should be different.

[showerst]

Side note on my idea of the entity resolvers living in the scrapers, that might make running in scrape mode a real mess, so maybe that should be left in core where it lives now?

It would be cool to make that easier to update though, so maybe there's another way.

[jamesturk]

thoughts on moving forward on this a bit (and other related things) in #20