modern-data-stack.md

MODERN DATA STACK

• Data Sources
  • APIs
  • Datasets
  • Databases
  • SaaS Apps
  • Event Streams
• Data Integration & ETL
• Data Storage & Processing
  • Data Lake/Lakehouse
  • Cloud Data Warehouse
• Data Transformation
• Data Governance
  • Data Labeling
  • Workflow Orchestration
  • Version Control for Data Science
  • Feature Store
• Free Cloud Services
• Applications
  • Web Application Development
    • Frontend Development
    • Backend Development
    • [Database]
    • [Deployment]
    • [DevOps]
  • Data Reporting
  • Data Visualization
  • Data Science
  • MLOps
    • End-to-End Machine Learning
    • AutoML
    • ML Frameworks
    • Deep Learning Frameworks
    • Model Visualization and Debugging
    • Model Optimization
    • Experiment Tracking
    • Model Packaging
    • Model Serving and Deployment
    • Workflow Orchestration
    • Model Monitoring and Management
    • Explainability
    • CI/CD
    • Infrastructure Management
    • Specialized Hardware
• Specialized Tools
  • NLP
  • Computer Vision
  • Reinforcement Learning Frameworks
  • DevOps
  • Cloud Platforms
  • Big Data Technologies
  • Time Series Analysis
  • Cloud-Native AI
  • Federate Learning Frameworks
  • Low-Code AI Platforms
  • AI Chatbot Frameworks
  • AI Ethics and Governance

Data Sources

APIs

• Public APIs
• Public APIs Lists

Datasets

• Public Datasets
• Public Data Repositories [1] [2]

Databases

• PostgreSQL
• MySQL
• MongoDB
• SQLite
• MS SQL Server

SaaS Apps

• Salesforce
• Hubspot
• Stripe

Event Streams

• Snowplow
• Segment

Data Integration & ETL

• Apache Spark
• Apache Kafka
• Apache Flink
• Airflow
• Airbyte
• Stitch

Data Storage & Processing

Data Lake/Lakehouse

• AWS S3
• Azure Data Lake
• Google Cloud Storage

Cloud Data Warehouse

• Snowflake
• Amazon Redshift
• Google BigQuery

Data Transformation

• dbt

Data Governance

Data Labeling

• Labelbox
• Amazon SageMaker Ground Truth
• SuperAnnotate

Workflow Orchestration

• Apache Airflow
• GitHub Actions
• Luigi

Version Control for Data Science

• DVC
• Pachyderm

Feature Store

• Tecton
• Feast

Free Cloud Services

• AWS
• Azure
• GCP

Applications

Web Application Development

Frontend Development

• [React]
• [RxJS]
• [WebGL]
• [D3.js]
• [three.js]
• [Plotly.js]
• [Sigma.js]
• [Leaflet]
• [Cesium]
• [Deck.gl]
• [Recharts]
• [Redux]

Backend Development

• [Flask]
• [Django]
• [Node.js]
• [Express.js]
• [GraphQL]
• [RESTful API]

Database

• [PostgreSQL]

Deployment

• [Heroku]
• [AWS]

DevOps

• [GitHub Actions]

Data Reporting

• Jupyter
• Streamlit
• Voila
• Mercury
• Shiny
• Metabase

Data Visualization

• Tableau
• Power BI

Data Science

• Data Science Tools

MLOps

End-to-End Machine Learning

• AWS SageMaker
• Azure Machine Learning
• Google TFX
• IBM Watson
• DataBricks
• DataRobot

AutoML

• H20.ai
• DataRobot
• Google AutoML

ML Frameworks

• TensorFlow
• PyTorch
• scikit-learn

Deep Learning Frameworks

• TensorFlow
• Keras
• PyTorch
• Fast.ai

Model Visualization and Debugging

• Neptune.ai
• TensorBoard
• Netron

Model Optimization

• Hyperopt
• Optuna
• Talos

Experiment Tracking

• MLflow
• Weights&Biases
• TensorBoard
• Neptune.ai

Model Packaging

• Flask
• FastAPI
• Streamlit
• Django
• Gradio
• Docker

Model Serving and Deployment

• AWS Sagemaker
• Google AI Platform
• TensorBoard Serving
• TorchServe
• Azure ML
• Kubeflow
• Seldon
• MLflow

Workflow Orchestration

• Kubeflow
• Kedro
• Airflow

Model Monitoring and Management

• Arize
• EvidentlyAI
• Prometheus
• Grafana
• SageMAker Model Monitor
• Kibana
• Onnx

Explainability

• SHAP
• LIME
• [What-If Tool]
• ELI5

CI/CD

• Jenkins
• Travis CI
• GitLab CI/CD

Infrastructure Management

• Terraform
• Ansible
• Docker

Specialized Hardware

• GPUs
• TPUs

Specialized Tools

NLP

• Langchain
• Hugging Face
• spaCy

Computer Vision

• OpenCV
• [Dlib](http://dlib.net
• ImageAI

Reinforcement Learning Frameworks

• OpenAI Gym
• PySC2
• Keras-RL

DevOps

• Docker
• Kubernetes
• Jenkins

Cloud Platforms

• AWS
• GCP
• Azure

Big Data Technologies

• Hadoop
• Apache Kafta
• Apache Cassandra

Time Series Analysis

• Prophet
• ARIMA
• Statsmodels

Cloud-Native AI

• AWS Lambda
• Google Cloud Functions
• Azure Functions

Federate Learning Frameworks

• TensorFlow Federated
• PySyft
• IBM Federated Learning

Low-Code AI Platforms

• IBM Watson Studio
• Microsoft Power Platform
• Google Cloud AutoML

AI Chatbot Frameworks

• Rasa
• Dialogflow
• Botpress

AI Ethics and Governance

• AI Fairness 360
• IBM Watson OpenScale
• Microsoft Azure Responsible AI