Skip to content

Latest commit

 

History

History
278 lines (180 loc) · 14 KB

README.md

File metadata and controls

278 lines (180 loc) · 14 KB
Raw

Aggie

Introduction

Aggie is a web application for using social media and other resources to track groups around real-time events such as elections or natural disasters.

Aggie can retrieve data from several sources:

  • Twitter (tweets matching a keyword search)
  • Telegram
  • Crowdtangle (Facebook, Instagram, and Reddit posts from publicly accessible groups and pages)
  • RSS (article titles and descriptions)
  • ELMO (answers to survey questions)

Items (called reports) from all sources are streamed into the application. Monitors can quickly triage incoming reports by marking them as relevant or irrelevant.

Relevant reports can be grouped into groups for further monitoring and follow-up.

Reports are fully searchable and filterable via a fast web interface.

Report queries can be saved and tracked over time via a series of visual analytics.

Aggie is built for scalability and can handle hundreds of incoming reports per second. The backend fetching and analytics systems feature a modular design well-suited to parallelism and multi-core architectures.

Users can be assigned to admin, manager, monitor, and viewer roles, each with appropriate permissions.

Aggie is built using React and Express.js, commonly used and popular web frameworks.

Contact [email protected] for more information on the Aggie project.

Sassafras Tech Collective offers managed instances of Aggie, along with development and support services.

Table of Contents

Using the Application

Extensive documentation about using the application can be found in ReadTheDocs page.

Source Installation

Software Requirements

  1. node.js (v18.20 LTS)
    1. Use Node Version Manager.
      • Node Version Manager (nvm) allows multiple versions of node.js to be used on your system and manages the versions within each project.
      • on windows, you can either use nvm-windows and nvs
      • After installing nvm:
        1. Navigate to the aggie project directory: cd aggie.
        2. Run nvm install to install the version specified in .nvmrc.
        3. then nvm use to switch to that version.
  2. Mongo DB (requires >= 4.2.0)
    1. Follow the installation instructions for your operating system.
    2. You can connect to the live database, ask a maintainer for a copy of the db access token. you will need mongoCompass installed.
    3. if you are running a copy of the dabase locally:
      1. Make sure MongoDB is running:
        • On Linux run sudo systemtl status mongod to see whether the mongod daemon started MongoDB successfully. If there are any errors, you can check out the logs in /var/log/mongodb to see them.
      2. Note: You do not need to create a user or database for aggie in Mongo DB. These will be generated during the installation process below.

Installation

  1. Clone the aggie repo.

    • you can use github-desktop, or clone using git.
    • In your terminal, navigate to your main projects folder (e.g. Documents).
    • Use this command: git clone https://github.com/TID-Lab/aggie.git.
    • cd aggie

  2. Copy backend/config/secrets.json.example to backend/config/secrets.json.

    • ask current developers for a copy of the secrets.json

  3. Copy .env.example to .env .

    • ask current developers for a copy of the .env
    • the DATABASE_URL key should be the current mongo database, ask developers for a copy of this key

  4. (optional, rarely needed) You might have issues with HTTPS. if so, copy your SSL certificate information to the config folder (two files named key.pem and cert.pem).

    • If you do not have the certificate you can create a new self-signed certificate with the following command: openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 365
    • This will allow you to start the server but it will generate unsafe warnings in the browser. You will need a real trusted certificate for production use.
    • Adding the -nodes flag will generate an unencrypted private key, allowing you to run tests without passphrase prompt

  5. Run npm install from the project directory.

    • This installs all dependencies.

Development

  1. Use npm run dev for development.
    • you can run frontend and backend in separate shells with npm run dev:frontend and npm run dev:backend
  2. Navigate to https://localhost:8000 in your browser.
    • This will show you the running site. Login with the user name and password, which you can obtain from the current devs

Production

  1. build react app with npm run build
  2. run npm start
    • remember to have your ENVIRONMENT=production in the .env file.

Pull Requests

When collaborating with multiple developers, we adopt a feature-branch workflow. If you are unfamiliar, read this article.

the develop branch is our main/staging branch. production will be built from this branch. try not to push directly to this branch.

when writing PRs, include high-level changes and notable/interesting engineering challenges. However, you don't need to be particularly granular.

PR's should be reviewed by another developer, ideally the developer lead or the developer with domain knowledge of the feature before merging.

PR merge conflicts

resolve conflicts with the main develop branch by merging the latest into the current branch. for example, if you are working on a PR example-feature-branch then:

  1. checkout develop
  2. git fetch --all
  3. git pull
  4. checkout example-feature-branch
  5. git merge develop
  6. resolve any merge conflicts then push to branch

Maintenance

TODO: create proper maintenance

1. To run migrations run npx migrate.

Project Configuration

You can adjust the settings in the config/secrets.json file to configure the application.

Tests

TODO: tests are broken at the moment. this is a work in progress

Set config.adminParty=true if you want to run tests.

Social Media and Feeds

2424 update:

changes to many social media APIs means we need new ways of obtaining data from these sites. crowdtangle is being sunset. work in progress.

Twitter

  1. Follow these instructions to generate tokens to use the Twitter API.
  2. Go to Settings > Configuration and edit the Twitter settings. Remember to toggle the switch on, once you have saved the settings.

CrowdTangle

  1. Create a dashboard on CrowdTangle and generate the dashboard token.
  2. Add your CT API token to config/secrets.json.
  3. Run npm run update-ct-lists to fetch data.
    • This will update config/crowdtangle_list.json.
    • This also happens automatically every night at midnight while Aggie is running.

Note: To have git ignore changes, run git update-index --skip-worktree config/crowdtangle_list.json

WhatsApp

The WhatsApp feature is documented in a conference paper. As WhatsApp does not currently offer an API, a Firefox extension in Linux is used to redirect notifications from web.whatsapp.com to Aggie server. Thus, you need a Linux computer accessing WhatsApp through Firefox for this to work. Follow these steps to have it working.

  1. Install Firefox in Linux using your distribution preferred method.
  2. Install GNotifier add-on in Firefox.
  3. Configure the add-on about:addons:
    • Set Notification Engine to Custom command
    • Set the custom command to curl --data-urlencode "keyword=<your own keyword>" --data-urlencode "from=%title" --data-urlencode "text=%text" http://<IP address|domain name>:2222/whatsapp
      • We suggest setting your keyword to a unique string of text with out spaces or symbols, e.g., the phone number of the WhatsApp account used for Aggie. This keyword must be the same one as the one specified in the Aggie application, when creating the WhatsApp Aggie source.
      • Replace IP address|domain with the address or domain where Aggie is installed (e.g., localhost for testing).
  4. Visit web.whatsapp.com, follow instructions, and enable browser notifications
  5. Notifications will not be sent to Aggie when browser focus is on the WhatsApp tab, so move away from that tab if not replying to anyone.

ELMO

  1. Log in to your ELMO instance with an account having coordinator or higher privileges on the mission you want to track.
  2. In your ELMO instance, mark one or more forms as public (via the Edit Form page). Note the Form ID in the URL bar (e.g. if URL ends in /m/mymission/forms/123, the ID is 123).
  3. Visit your profile page (click the icon bearing your username in the top-right corner) and copy your API key (click 'Regenerate' if necessary).
  4. Go to Settings > Configuration and edit the ELMO settings. Remember to toggle the switch on, once you have saved the settings.

Google Places

Aggie uses Google Places for guessing locations in the application. To make it work:

  1. You will need to get an API key from Google API console for Google Places API.
  2. Read about Google API usage limits and consider whitelisting your Aggie deployment to avoid surprises.
  3. Go to Settings > Configuration and edit the Google Places settings and add the key.

Emails

The current build does not have email support

Email service is required to create new users.

  1. fromEmail is the email address from which system emails come. Also used for the default admin user.
  2. email.from is the address from which application emails will come
  3. email.transport is the set of parameters that will be passed to NodeMailer. Valid transport method values are: 'SES', 'sendgrid' and 'SMTP'.
  4. If you are using SES for sending emails, make sure config.fromEmail has been authorized in your Amazon SES configuration.

Fetching

  1. Set fetching value to enable/disable fetching for all sources at global level.
  • This is also changed during runtime based on user choice.

Logging

Set various logging options in logger section.

  • console section is for console logging. For various options, see [winston](see https://github.com/winstonjs/winston#transports)
  • file section is for file logging. For various options, see [winston](see https://github.com/winstonjs/winston#transports)
  • SES section is for email notifications.
    • Set appropriate AWS key and secret values.
    • Set to and from email ids. Make sure from has been authorised in your Amazon SES configuration.
  • Slack section is for Slack messages.
    • Set the webhook URL to send logs to a specific Slack channel
  • DO NOT set level to debug. Recommended value is error.

Only the console and file transports are enabled by default. Transports can be disabled using the "disabled" field included in each section in the config/secrets.json file.

Remote access

See the first part of the Tableau docs in BI Connector setup.

Data visualization using Tableau

Setting up and viewing Tableau visualizations in Aggie requires installing Tableau's MongoDB BI Connector on the server that acts as a bridge between Tableau and MongoDB. To set up the BI Connector, follow these steps: BI Connector setup.

Architecture

Aggie consists of two largely separate frontend and backend apps. Some model code (in /shared) is shared between them.

Backend

The backend is a Node.js/Express app responsible for fetching and analyzing data and servicing API requests. There are three main modules, each of which runs in its own process:

  • API module
  • Fetching module
  • Analytics module

See README files in the lib subdirectories for more info on each module.

The model layer (in /models) is shared among all three modules.

Frontend

See detailed Frontend at FRONTEND.md

The frontend is a SPA react app that runs in the browser and interfaces with the API, via both pull (REST) and push (WebSockets) modalities. It source files contained in /src and /public. when built, files are served from /build

Building and Publishing Aggie's documentation

would be nice to have modern docs, but we dont at the moment

The documentation is in the `docs` directory. These are automatically built and pushed on each commit for the `master` and `develop` branches in Github:

To build the docs locally, do the following:

  1. Install Python and pip
  2. Install Sphinx with pip install -U Sphinx
  3. Install recommonmark: pip install recommonmark
  4. Install Read The Docs theme: pip install sphinx_rtd_theme
  5. From the docs directory in Aggie, run make html
  6. The compiled documentation has its root at docs/_build/html/index.html