Home

An Overview of dd-Data3.0 Digital Democracy Database Team

Group

Manager

• Alex Dekhtyar, Ph.D. (Master of Deep Learning), [email protected]

Team

• Nick Russo, [email protected]
• Andrew Rose, [email protected]
• Nathan Philliber

Who Do I Ask Questions to:

For emergencies (ex. rm -rf on /var directory), email your manager directly. For everything else contact Nick Russo, email [email protected] or slack. If he doesn’t respond, send another email with a parrot emoji. If he still doesn’t respond, contact your manager.

Style Guide

Follow PEP-8. Document code or else the Pythonista will come after you. Follow the project structure below. Do not write duplicate code. pls do not blindly use list comprehension. pls.

Project Structure

• Constants (Python Package)
  • Allowed items:
    • Constants
    • Queries
  • No code should be in this folder.
• Models (Python Package)
  • These are all the model representations of data structures
  • Person
  • Legislator
  • Committee etc.
  • Note: When adding new data to the project, make a model object. It is important to have these models so it is predictable what objects contain.
• OpenStatesParsers (Python Package)
  • This directory contains all parsing classes related to parsing open states data.
  • These classes contain shared code that can be used for all states.
    • Override necessary methods for state dependent irregularities.
• Utils (Python Package)
  • Contains generic code used for all states.
    • Insertion classes
      • Handle all logic for insertion a specific model into the database.
  • Generic Utils
    • Generic repetitive code
    • Contains logger
      • !Note! Please use this logger, this is a standardized logger.
        • Printing is not logging. Logging is meant for important messages. For example, a failed insertion or an exception.
        • Sure, print to debug. I do that too. No one debugs with a logger.
  • Generic MySQL
    • Standardized Generic MySQL Calls
  • Database Connection
    • Standardized way for connecting to MySQL Database
• (STATE)-Build
  • New scripts for importing data
    • This should be extremely minimal.
    • Most new code should be in the parser.
      • If the new datasource is OpenStates, please review the generic parser that is already written.
        • Nothing is worse than duplicate work/code.
    • No new code should be written for inserting data into the database. If you feel there is an instance to add to an insertion manager please contact the person stated above.
  • Custom implementations of parsers
    • These are not generic parsers for gathering and formatting data.

Adding a New State

Steps

1. Gather data sources
   • OpenStates is a good place to start.
2. Add the new state to State table.
3. Add session year for the new state to Session table.
4. Add the chambers for the new state to the House table (Senate, Assembly, House etc).
5. Minimum data and tables to be filled for Transcription Tool
   • Legislators
     • Person
     • Legislator
     • Term
     • PersonStateAffiliation
     • AlternateID (If applies)
   • Committees
     • Committee
     • CommitteeNames
     • ServesOn
     • author
   • Bills
     • Bills
     • BillVersion
     • BillVoteSummary
     • BillVoteDetail
     • Action
     • BillAnalysis (If applies)
     • Motion
   • Hearings
     • Hearing
     • HearingAgenda
     • CommitteeHearing
   • Districts
     • District
   • Lobbyists
     • Lobbyists
     • LobbyingFirm
     • LobbyingFirmState
     • LobbyistEmployer
     • LobbyistEmployment
     • LobbyistDirectEmployment
     • LobbyistContract
     • LobbyistContractWork
     • PersonStateAffiliation
     • Organization
6. Consult with Alex Dekhtyar about adding the remaining data into the following tables.
   • Behest (if applies)
   • Gifts
   • Contribution
   • LegislativeStaff
     • All tables related.