This is a final project for ITMS448 at IIT:
An OSINT dashboard for gathering intelligence from multiple sources.
The project's name is Bowser, named after Mary Bowser.
We are focusing on domestic terrorism.
You can view our final presentation PDF here.
Clone or download this repository.
All commands take place in the repo's directory.
Install Python 3.6 or higher.
pip install pipenv installs a Python environment manager.
pipenv install --dev installs dependencies plus dev deps specified in /Pipfile.
pipenv shell opens a shell with all your Python deps installed.
Without this, the deps installed from pipenv install aren't referred to when you type python.
cd bowser/
python bowserScraper.py
Running this will generate a CSV file in bowser/out/.
Edit the end of bowser.py to make larger CSV files or gather from different boards.
cd client and run npm run-script build.
This generates JS and HTML that will be served under 0.0.0.0:1839/ as Flask static files, when Flask is running.
Alternatively, you can run npm start if you wish to run the web UI under Node.js.
cd bowser
python bowserHTTPAPI.py
This will start an HTTP API on http://0.0.0.0:1839/api.
Also, a web UI from Node.js static files will be served on http://0.0.0.0:1839/.
This web UI must be built using npm as described in the "Web UI" section.
Alternatively, http://bowser-webapi.herokuapp.com can be used if you need to access it over the internet and not locally.
The Heroku app may fail due to Cloudflare WAF blocking Heroku's IPs. Running it locally should prevent this.
http://bowser-web-app.freemyip.com:1839 can be used with no restrictions to browse a live version of our project. Try going to /api!
The DigitalOcean instance was shut down as I (Henry P) have graduated and do not want to pay for it. Please run it yourself if you want to use this project.
cd scrapy/
scrapy runspider testScrapyProject/testScrapyProject/spiders/aarchivedmoe.py
See /client for a more detailed README file.
Project was bootstraped with Template: https://www.creative-tim.com/product/material-dashboard-react
-
If using VS Code need Flow extension
Name: Flow Language Support
Id: flowtype.flow-for-vscode
Description: Flow support for VS Code
Version: 1.5.0
Publisher: flowtype VS Marketplace Link: https://marketplace.visualstudio.com/items?itemName=flowtype.flow-for-vscode -
Turn off VS Code default javascript/script checker
- Create a .vscode directory at root project level
- Create a .vscode/settings.json file
- copy json into the file
{
"extensions.enabled": [
/*whatever extensions wanted to be*/
],
"extensions.disabled": ["vscode.typescript-language-features"]
}Role: Project Manager
Role: Backend Developer
Role: Frontend Developer
Role: DevOps Engineer
Role: Cloud Developer, Backend Developer
Role: Data Analyst
Role: Assistant Project Manager, Data Analyst
Role: Data Analyst
4chan API:
https://github.com/4chan/4chan-API
4plebs API documentation:
https://archive.4plebs.org/_/articles/faq/
Other adjacent tools:
https://github.com/eksopl/asagi
https://github.com/FoolCode/FoolFuuka
https://github.com/Grayson112233/python-4chan-scraper/
A list of sources for keywords whose use may indicate terrorist activity.
http://cryptome.org/2014/01/nsa-codenames.htm
https://buggedplanet.info/index.php?title=Category:NSA_codewords
https://www.businessinsider.com/nsa-prism-keywords-for-domestic-spying-2013-6
https://www.theregister.co.uk/2001/05/31/what_are_those_words/
http://www.scribd.com/doc/82701103/Analyst-Desktop-Binder-REDACTED
https://github.com/mogade/badwords/blob/master/en.txt
https://epic.org/foia/epic-v-dhs-media-monitoring/Analyst-Desktop-Binder-REDACTED.pdf