- Section 1: Software Architecture Mindmap
- Section 2: Software Architecture Reference
- Section 3: Computer Science Papers
- Section 4: Data Science (ML/NN)
- Section 5: Terminology and Comparisons
Software terminologies and concepts, software architecture overview
Summarized the keywords and solutions have faced in my learning and experience.
- Full Version
Software_Architecture_Mindmap.png
Three main pillars upon software architecture
-
Modern Application Development
-
Cloud Computing (AWS/Azure/GCP)
-
Data Science (ML/NN)
and
Numerous technologies and methodologies.
ⓒ 2022. (https://github.com/kimtth) all rights reserved.
This mindmap created by https://app.mindmapmaker.org/
- System Design 101: ByteByteGo
- Awesome Lists: 😎 Awesome lists about all kinds of interesting topics /
awesome.re
/ github topic - Awesome Software Architecture (simskij)
- Awesome Software Architecture: A curated list of awesome articles, videos, and other resources to learn and practice software architecture, patterns, and principles
- Software Architecture Books: A comprehensive list of books on Software Architecture
- System Design: Learn how to design systems at scale and prepare for system design interviews
- Microsoft .NET Application Architecture - Reference Apps
- Software Architecture Books
- System Design Fight Club
- System Design - Neo Kim
- Awesome System Design Resources
- InfoQ: News and Articles
- Dzone: RefCards and Trend Reports
- Thoughtworks: Technology Radar
- Microsoft Learn: Documentation and Code samples
- Trendshift: GitHub Trending repositories
- Design Gurus: Portal For Tech Interviews
- System Design Blueprint: The Ultimate Guide
-
Expand
🔹
Latency
is the response time of your application, usually expressed in milliseconds🔹
Throughput
is how many transactions per second or minute your application can handle🔹
Errors
is usually measured in a percent of🔹
Saturation
is the ability of your application to use the available CPU and Memory -
InfoQ minibooks: Architectures You’ve Always Wondered About .. 2021 / 2023 / 2024 / Cell-Based Architecture
- Web Browser Engineering: Building a basic but complete web browser from scratch
- Curated list of project-based tutorials
- Master programming by recreating your favorite technologies from scratch
- Build frontend applications at scale
- AWS to Azure services comparison
- Google Cloud to Azure services comparison
- Compare AWS and Azure services to Google Cloud
- Microsoft Azure Developer's Cheat Sheet: Every product, feature and service in the Azure family
- Azure Cloud Adoption Framework :CAF: organization-wide adoption guidance
- Azure Well-architected Framework :WAF: workload-focussed design and continuous improvement guidance
- Azure Architecture Center :AAC: architecture patterns and reference architectures
Expand
🔹 Abstractly speaking, a landing zone helps you plan for and design an Azure deployment, by conceptualizing a designated area for placement and integration of resources.
There are two types of landing zones:
1. `platform landing zone`: provides centralized enterprise-scale foundational services for workloads and applications.
2. `application landing zone`: provides services specific to an application or workload.
- Kaggle Solutions and Ideas: Collection of Kaggle Solutions and Ideas
- Best-of Machine Learning with Python: A ranked list of awesome machine learning Python libraries. Updated weekly.
- freeCodeCamp: Learn to code for free. youtube
- Ultimate Collection of 60 YouTube Courses for 21 Programming Languages
- Computer Science courses with video lectures
- Software Industry Statistics: Statista Industry Insight
- Gartner Top Strategic Technology Trends 2024
- MAD (ML/AI/Data) Landscape
- Hype Cycle for Emerging Technologies 2024
- Microsoft Digital Defense Report
- Substack Leaderboard: Newsletter
- Algorithm Visualizer: Interactive Online Platform that Visualizes Algorithms from Code
- Hello Algo
- OOP Design Patterns
- Dev Encyclopedia: Encyclopedia for developers / git
- Data Engineering Wiki
- Best Kubernetes Tools: Bluelight Consulting
- PostgreSQL Wiki: Don't Do This
- Power BI DAX Patterns
- Semantic Scholar > Semantic Reader
- AI by Hand
- Visualizing relationships between research: Litmaps: / Connected Papers
- Finding Papers: Ask R Discovery / scite_
- Visualizer for neural network: netron
- Excalidraw
- AWS Architecture Blog
- Azure Architecture Blog
- GCP Cloud Blog
- Netflix TechBlog
- Uber Blog
- The Cloudflare Blog
- Engineering at Meta
- LinkedIn Engineering
- Stripe Blog: Engineering
- Discord Blog: Engineering & Developers
- Slack Engineering
- 79 Engineering Blogs To Level Up Your System Design Skills
- How to Architect Software for a Greener Future
- Generative Search: Practical Advice for Retrieval Augmented Generation (RAG)
- What are the Greenest Programing Languages?
- Frugal Architect Handbook
-
General
- The Pragmatic Programmer by David Thomas and Andrew Hunt
- Modern Software Engineering by David Farley
- Code Complete by Steve McConnell
- Software Engineering at Google by Titus Winters, Tom Manshreck, and Hyrum Wright
-
Good Practices
- Clean Code by Uncle Bob Martin
- Head First Design Patterns by Eric Freeman
- Refactoring by Martin Fowler
- Design Patterns by Eric Gamma and Others
-
Data Structures and Algorithms
- Grokking Algorithms by Aditya Bhargava
- Introduction to Algorithms by Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, Clifford Stein
- Cracking the Coding Interview by Gayle Laakmann McDowell
-
Data
- Designing Data-Intensive Applications by Martin Kleppman
- Learning SQL by Alan Beaulieu
-
Testing
- Growing Object-Oriented Software, Guided by Tests by Steve Freeman
- Unit Testing Principles, Practices, and Patterns by Vladimir Khorikov
- The Art of Unit Testing by Roy Osherove
- TDD by Example by Kent Beck
-
Software Architecture
- Fundamentals Of Software Architecture by Mark Richards and Neil Ford
- Clean Architecture by Uncle Bob Martin
- Software Architecture: The Hard Parts by Neal Ford, Mark Richards, Pramod Sadalage, and Zhamak Dehghani
- Domain-Driven Design Quickly by Abel Avram and Floyd Marinescu
- A Philosophy of Software Design by John Ousterhout
- System Design Interview by Alex Xu
- Domain-Driven Design by Eric Evans
-
Distributed Systems
- Understanding Distributed Systems by Roberto Vitillo
- Designing Distributed Systems by Brendan Burns
-
DevOps
- DevOps Handbook by Gene Kim, Patrick Debois, John Willis, and Jez Humble
- Continuous Delivery by Jez Humble and David Farley
- Accelerate by Nicole Forsgren, Jez Humble, and Gene Kim
-
Machine Learning
- The Hundred-Page Machine Learning Book by Andriy Burkov
- Designing Machine Learning Systems by Chip Huyen
- On the Criteria To Be Used in Decomposing Systems into Modules (1972): D.L. Parnas
- An Axiomatic Basis for Computer Programming (1969): C.A.R. Hoare
- Time, Clocks, and the Ordering of Events in a Distributed System (1978): L. Lamport
- Out of the Tar Pit (2006): B. Moseley, P. Marks
- Dynamo: Amazon’s Highly Available Key-value Store (2007): G. DeCandia et al.
- MapReduce: Simplified Data Processing on Large Clusters (2004): J. Dean, S. Ghemawat
- A Note On Distributed Computing (1994): J. Waldo, G. Wyant, A. Wollrath, S. Kendall
- A Metrics Suite for Object-Oriented Design (1994): S.R. Chidamber
- A Relational Model of Data for Large Shared Data Banks (1969): E.F. Codd
- Why Functional Programming Matters (1990): J. Hughes
- Here's a reading list of 70+ Distributed Systems papers mostly from conferences in just last 2 years! 70+ Distributed Systems papers [Jan 2024]
- ref [May 2024]
- Dynamo: Amazon’s Highly Available Key Value Store
- Google File System: Insights into a highly scalable file system
- Scaling Memcached at Facebook: A look at the complexities of caching
- BigTable: The design principles behind a distributed storage system
- Borg: Large Scale Cluster Management at Google
- Cassandra: A look at the design and architecture of a distributed NoSQL database
- Attention Is All You Need: Into a new deep learning architecture known as the transformer
- Kafka: Internals of the distributed messaging platform
- FoundationDB: A look at how a distributed database works
- Amazon Aurora: How Amazon provides high availability and performance
- Spanner: Design and architecture of Google’s globally distributed database
- MapReduce: A detailed look at how MapReduce enables parallel processing of massive volumes of data
- Shard Manager: Understanding the generic shard management framework
- Dapper: Insights into Google’s distributed systems tracing infrastructure
- Flink: A detailed look at the unified architecture of stream and batch processing
- A Comprehensive Survey on Vector Databases
- Zanzibar: A look at the design, implementation, and deployment of a global system for managing access control lists at Google
- Monarch: Architecture of Google’s in-memory time series database
- Thrift: Explore the design choices behind Facebook’s code-generation tool
- Bitcoin: The ground-breaking introduction to the peer-to-peer electronic cash system
- WTF - Who to Follow Service at Twitter: Twitter’s (now X) user recommendation system
- MyRocks: LSM-Tree Database Storage Engine
- GoTo Considered Harmful
- Raft Consensus Algorithm: Learn about the more understandable consensus algorithm
- Time Clocks and Ordering of Events: The extremely important paper that explains the concept of time and event ordering in a distributed system
- Deep Learning - Ian Goodfellow, Yoshua Bengio, and Aaron Courville
- Mathematics for Machine Learning - Marc Peter Deisenroth, A. Aldo Faisal, Cheng Soon Ong
- An Introduction to Statistical Learning - Gareth James, Daniela Witten, Trevor Hastie, Robert Tibshirani, Jonathan Taylor
- The Elements of Statistical Learning - Jerome H. Friedman, Robert Tibshirani, and Trevor Hastie
- Probabilistic Machine Learning: An Introduction - Kevin Patrick Murphy
- Probabilistic Machine Learning: Advanced Topics - Kevin Patrick Murphy
- Understanding Machine Learning - Shai Shalev-Shwartz and Shai Ben-David
- Automated Machine Learning - Frank Hutter, Lars Kotthoff, Joaquin Vanschoren
- Applied Causal Inference - Uday Kamath, Kenneth Graham, Mitchell Naylor
- Reinforcement Learning: An Introduction - Richard S. Sutton and Andrew G. Barto
- The Hundred-Page Machine Learning Book - Andriy Burkov
- Machine Learning Engineering - Andriy Burkov
- Natural Language Processing with Python - Steven Bird, Ewan Klein, and Edward Loper
- Dive into Deep Learning - Aston Zhang, Zachary C. Lipton, Mu Li, Alexander J. Smola
- Machine Learning Yearning - Andrew NG
- Machine Learning for Humans - Vishal Maini, Samer Sabri
- Pattern Recognition and Machine Learning - Christopher M. Bishop
- Deep Learning on Graphs - Yao Ma and Jiliang Tang
- Approaching (Almost) Any Machine Learning Problem - Abhishek Thakur
- Feature Engineering and Selection - Max Kuhn and Kjell Johnson
- Hands-On Machine Learning with R - Bradley Boehmke & Brandon Greenwell
- Deep Learning Interviews - Shlomo Kashani and Amir Ivry
- Machine Learning Interpretability - Patrick Hall and Navdeep Gill
- Interpretable Machine Learning - Christoph Molnar
- Boosting: Foundations and Algorithms - Robert E. Schapire, Yoav Freund
- A Brief Introduction to Machine Learning for Engineers - Osvaldo Simeone
- Speech and Language Processing - Daniel Jurafsky & James Martin
- Computer Vision: Models, Learning, and Inference - Simon J.D. Prince
- Information Theory, Inference and Learning Algorithms - David J. C. MacKay
- Machine Learning For Dummies - Judith Hurwitz and Daniel Kirsch
- Algebra, Topology, Differential Calculus, and Optimization Theory for Computer Science and Machine Learning
- The Data Engineering Handbook
- Machine Learning for Beginners
- Machine Learning YouTube Videos
- Mathematics for Machine Learning
- Deep Learning Book
- Machine Learning ZoomCamp
- Machine Learning Tutorials
- Awesome Machine Learning
- CS 229 Machine Learning Cheatsheets
- Machine Learning Interview Guide
- Awesome Production Machine Learning
- 365 Data Science Flashcards
- ref > Virgilio | Python Data Science Handbook | Microsoft: 10 Weeks, 20 Lessons, Data Science | Data science Python notebooks | 📚 Papers & tech blog | Open Source Data Science Masters | Awesome Data Science | Data science interview questions and answers | free self-taught education in Data Science!
- data science project template
- labml.ai Deep Learning Paper Implementations: 60+ Implementations/tutorials of deep learning papers with side-by-side notes
- Deep Learning Models: A collection of various deep learning architectures, models, and tips
- fastai book: The fastai book, published as Jupyter Notebooks
-
Gartner's PACE Layered Application Strategy: A methodology for categorizing, selecting, managing and governing applications based on their characteristics and the speed of change they require1.
-
JIT vs AOT: JIT and AOT are two types of compilers that differ in when they convert a program from one language to another, either at run-time or build-time.
-
SSG: Static site generator list: A tool that generates a full static HTML website based on raw data and a set of templates.
-
Popular Enterprise Architecture Frameworks: TOGAF, Zachman, Federal Enterprise Architecture (FEA), Gartner Enterprise Architecture Framework, Business Architecture Guild’s BIZBOK, Department of Defense Architecture Framework (DoDAF), ArchiMate, and Sherwood Applied Business Security Architecture (SABSA).
-
Are Architecture Styles, Patterns, and Design Patterns Different?
Architecture Styles vs Patterns vs Design Patterns
𝟭. 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗮𝗹 𝘀𝘁𝘆𝗹𝗲𝘀 This is the highest level of abstraction, where architectural designs instruct us on structuring our code. The highest level of granularity describes the application's layers and high-level modules and how they relate to and interact with one another. Examples of architectural styles include: 🔹 Monolith 🔹 Layered 🔹 Event-driven 🔹 Self-contained Systems 🔹 Microservices 🔹 Space-Based 𝟮. 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗮𝗹 𝗽𝗮𝘁𝘁𝗲𝗿𝗻𝘀 These patterns represent a way to implement an architectural style, so we can do this regularly. Some examples are how to separate the user interface (UI) and data, how internal modules interact, and what layers we will use. Patterns answer these types of questions. They usually impact the code base and how to structure the code inside. Examples of architectural patterns include: 🔹 Model-View-Presenter (MVP): 1:1 Relationship between View and Presenter. e.g., Windows forms 🔹 Model-View-Controller (MVC): e.g., Smalltalk, ASP.Net MVC 🔹 Model–View–Viewmodel (MVVM): One to Many relationship between View and ViewModel. e.g., Silverlight, WPF, AngularJs: 🔹 Domain-Driven Design 𝟯. 𝗗𝗲𝘀𝗶𝗴𝗻 𝗽𝗮𝘁𝘁𝗲𝗿𝗻𝘀 These differ from architectural patterns in that they focus on a smaller code base area and have a smaller influence (focus on a local problem). These include limiting the creation of a class to only one object or notifying all dependent objects when the internal state of an object is changed. These patterns are described in the book "Design Patterns: Elements of Reusable Object-Oriented Software" by Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides from 1994. We have 𝘁𝗵𝗿𝗲𝗲 𝗴𝗿𝗼𝘂𝗽𝘀 𝗼𝗳 𝗗𝗲𝘀𝗶𝗴𝗻 𝗣𝗮𝘁𝘁𝗲𝗿𝗻𝘀: 🔹 𝗖𝗿𝗲𝗮𝘁𝗶𝗼𝗻𝗮𝗹: here we have Factory Method, Builder, Singleton, ... 🔹 𝗦𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗮𝗹: here we have an Adapter, Bridge, and Decorator, ... 🔹 𝗕𝗲𝗵𝗮𝘃𝗶𝗼𝗿𝗮𝗹: here we have Command, Iterator, State, Strategy, ...
-
Memory consistency model: A Primer on Memory Consistency and Cache Coherence
SC vs TSO vs Relaxed Memory Consistency
𝟭. Sequential Consistency (SC): Operations execute in order as per the program. 🔹 SC preserves order for two memory operations from the same thread for all four combinations of loads and stores (Load → Load, Load → Store, Store → Store, and Store → Load). 🔹 MIPS R10000 𝟮. Total Store Order (TSO): Reads can happen before preceding writes complete. 🔹 TSO preserves the first three orders (Load → Load, Load → Store, Store → Store) but not Store → Load order. 🔹 x86 CPU. 𝟯. Relaxed Memory Consistency: Allows more reordering of operations for performance. 🔹 ARM and RISC-V
-
API Gateway vs Load Balancer
Expand
🔹 API Gateway: Manages access to backend services, handles tasks like rate-limiting, authentication, logging, and security policies.
🔹 Load Balancer: Distributes network traffic across multiple servers for high availability and even load distribution.
-
Data engineering & Data Scientists Vocab 101 ref / 15 DS/ML Cheat Sheets
Expand
🔹 Data engineering Vocab 101🔹 75 Key Terms That Data Scientists Remember by Heart
🔹 A Comprehensive NumPy Cheat Sheet Of 40 Most Used Methods
🔹 15 Pandas ↔ Polars ↔ SQL ↔ PySpark Translations
🔹 11 Key Probability Distributions
🔹 6 Must-Know Types of Clustering Algorithms in Machine Learning
🔹 25 Most Important Mathematical Definitions in Data Science
🔹 10 Regression and Classification Loss Functions
-
Transfer Learning, Fine-tuning, Multitask Learning and Federated Learning ref
-
DevOps, Platform engineering and SRE (site reliability engineering) ref
SRE vs. DevOps vs. Platform Engineering
🔹DevOps, SRE, and Platform Engineering are practices that streamline software development and maintenance. They all involve automation and collaboration.
🔹DevOps covers the entire software development process promoting team collaboration.
🔹SRE focuses on system reliability, including application monitoring and emergency response.
🔹Platform Engineering manages the infrastructure and tools needed for software development and operations.
🔹DevOps is about the whole development process, SRE emphasizes reliability and scalability, and Platform Engineering is about infrastructure and tool management.
-
API Protocols (ref. ByteByteGo)
-
Web services and APIs (SOAP, RestAPI, GraphQL, gRPC and Kafka) ref
SOAP, RestAPI, GraphQL, gRPC and Kafka
🔹SOAP (Simple Object Access Protocol): XML-based protocol for web services, heayweight, favored for security and reliability.
🔹REST (Representational State Transfer): Uses HTTP methods, simple and easy to use, but can be resource-heavy.
🔹GraphQL: Allows flexible data queries, reduces data over-fetching.
🔹gRPC (Google Remote Procedure Call)**: High-performance RPC framework, ideal for connecting microservices. Built on top of HTTP/2 and uses Protocol Buffers for data exchange.
🔹Kafka: Distributed streaming platform, uses publish-subscribe model for message queueing. real-time consistency. "at-least-once" delivery.
-
Real-time communication and messaging (MQTT, AMQP and WebSocket) ref
MQTT vs AMQP vs WebSocket
🔹MQTT (Message Queuing Telemetry Transport): Lightweight messaging protocol, uses publish-subscribe model, ideal for IoT and M2M communication. Three levels of Quality of Service (QoS): “At most once” (QoS 0), “At least once” (QoS 1), and “Exactly once” (QoS 2).
🔹AMQP (Advanced Message Queuing Protocol): Open-standard application layer protocol, robust message delivery, routing, and security features. Two qualities of service: “At most once (delivered once or lost)” and “At least once (delivered one or more times.)”.
🔹WebSocket: Enables full-duplex communication channels over a single TCP connection
-
Reactive programming vs event-driven architecture ref
- Event-Driven: Handles user actions or system events. More general and can be used in any context where an event occurs
- Reactive: Data-driven approach. managing data streams and propagating changes, like in a spreadsheet model.
-
RABC vs ReABC: RBAC (Role-Based Access Control) is an authorization model that assigns permissions based on predefined roles. On the other hand, ReBAC (Relationship-Based Access Control) extends RBAC’s capabilities by considering relationships between entities.
-
Conway's law: Software engineering principle that states that the structure of a system reflects the structure of the organization that designs it.
-
Data Management in Distributed systems (Partitioning, Shuffling and Bucketing)
Partitioning vs Shuffling vs Bucketing
🔹Partitioning: The process of dividing a large dataset into smaller parts, known as partitions. This process splits Hive table's files into multiple files. For example,
../hive/warehouse/sales_table/product_id=P1
.🔹Shuffling: Shuffling is the process of redistributing data across different partitions. The overhead of operations can be ranked as follows:
orderby
>join
>groupby
.🔹Bucketing: This is the process of decomposing data into manageable parts based on a certain column, thereby improving query performance and storage efficiency. It is best used when there are very few repeating values in a column (for example 1. a primary key column). For instance, Bucket0:
../hive/warehouse/sales_table/product_id=P1/000000_0
, Bucket1:../hive/warehouse/sales_table/product_id=P1/000001_0
, and so on. -
SSO (Single Sign-On) is an authentication scheme that allows a user to log in with a single ID and password to any of several related, yet independent, software systems.
SSO workflow, Types of SSO, SSO Implementations
🔹SSO workflow: Identity Provider (IdP), Service Provider (SP), SSO Server
- IdP: Central Authentication server e.g., Google
- SP: Individual Applications rely on SSO e.g, Trello
- SSO Server: Bridge between IdP and SPs
🔹Types of SSO: SAML, OAuth (Open Authorization) 2.0, Open ID Connect (OIDC)
Protocol Purpose Token Format - OAuth 2.0 Open standard for Authorization Access Tokens Temporary access to 3rd party app OpenID Connect (OIDC) Open standard for Authentication JSON Web Token (JWT) Newer type of SSO based on OAuth 2.0, Straightforward protocol than SAML SAML Authentication, Authorization XML Most common, Use SAML Protocol to exchange authentication between SSO server and SP 🔹Some other Types of SSO: Kerberos, Smart card authentication
- Kerberos: Less suitable for internet-facing SSO due to the shared secret between KDC (Key Distributin Center) and all participants.
- Smart card authentication: Physical card
🔹SSO Implementations: Microsoft Entra ID (FKA Micorsoft Active Directory), Okta, Ping Identity, OneLogin, Auth0
-
Deployment Styles: Blue/Green, Canary, and A/B
Blue/Green, Canary, A/B
🔹Blue/Green Deployment: Two identical environments, "Blue" and "Green". Deploy new version in inactive environment, test, then switch users to it. For example, AWS supports blue/green deployment strategies including Elastic Beanstalk, OpsWorks, CloudFormation, CodeDeploy, and Amazon ECS.
🔹Canary Deployment: Roll out new version to a small group of users, monitor feedback, then do a full-scale release.
🔹A/B Testing: Compare two versions of a webpage or app to see which performs better. A typical example of A/B testing is website usability testing.
-
Flaky Test: A Flaky Test is a test that sometimes passes and sometimes fails, despite no changes in the code. Causes can include poorly written tests, async waits, test order dependency, and concurrency issues. They can slow down CI/CD pipelines and cause issues for end users. ref
-
Hadoop Ecosystem
Hadoop vs Azure, AWS, GCP
🔹1. HDFS (File Storage): Azure Data Lake Storage, Amazon S3, Google Cloud Storage
🔹2. YARN (Resource Management): No direct equivalent in Azure, AWS, GCP
🔹3. MapReduce (Data Processing): HDInsight, Amazon EMR, Google Cloud Dataproc
🔹4. Spark (Fast Data Processing): Databricks, Spark in HDInsight, Azure Synapse Analytics, Amazon EMR, Google Cloud Dataproc
🔹5. PIG, HIVE (Query Data): HDInsight, Azure Synapse Analytics, Amazon EMR, Google Cloud Dataproc
🔹6. HBase (NoSQL DB): Azure Cosmos DB, HBase on a virtual machine (VM), HBase in Azure HDInsight, Amazon DynamoDB, Google Cloud Bigtable
🔹7. Mahout, Spark MLLib (ML Libraries): Databricks, Amazon SageMaker, No direct equivalent in GCP
🔹8. Solar, Lucene (Search/Index): Azure Cognitive Search, Amazon CloudSearch, Google Cloud Search
🔹9. Zookeeper (Cluster Management): No direct equivalent in Azure, Amazon Managed Apache ZooKeeper, No direct equivalent in GCP
🔹10. Oozie (Job Scheduling): Azure Data Factory, AWS Step Functions, Google Cloud Composer
-
Software defined Networking(SDN) Northbound vs Southbound
Expand
graph TD A[Application layer - routing, load balancing, etc] -->|Northbound APIs| B[Control layer - SDN controller] B -->|Southbound APIs| C[Infrastructure layer - physical switches, data plane]
🔹The Controller is the SDN network's brain, directing traffic flows.
🔹 The Southbound Interface communicates the controller's decisions to the switches using protocols like OpenFlow.
🔹SDN Switches direct traffic based on the controller's instructions.
🔹Network Devices (servers, routers, etc.) send and receive data flows as directed by the SDN switches.
🔹The Northbound Interface uses APIs to exchange data between the controller and applications.
🔹SDN Applications use network data to perform tasks, communicating their needs to the controller.
graph LR A[Controller] -- API --> B[Southbound Interface] B -- OpenFlow --> C[SDN Switches] C -- Data Flow --> D[Network Devices] A -- API --> E[Northbound Interface] E -- Applications --> F[SDN Applications]
-
Cracking coding interviews
Expand
🔹src: ref
🔹Two Pointers: Navigating arrays with two indices. ref
🔹Intervals: Working with ranges of values. ref / ref / ref
🔹Dynamic Programming: Solving complex problems by breaking them down into simpler subproblems. ref / ref
🔹Tree Traversal: Visiting all nodes in a tree. ref / ref
🔹DFS-BFS: Depth-first and breadth-first search algorithms. ref / ref / ref / ref
🔹Binary Search: Finding an element in a sorted array. ref
🔹Array: A data structure holding elements. ref
🔹Sliding Window: A subset of data that moves. ref / ref / ref / ref / ref
🔹Backtracking: Trying out all possibilities to find a solution. ref / ref / ref
🔹Combination: Finding all possible arrangements of elements. ref
🔹Trie: A tree-like data structure for storing strings. ref
🔹Word Break: Dividing a string into words. ref
🔹Bit Manipulation: Performing operations on binary numbers. ref / ref
🔹Sum: Adding numbers together. ref
🔹Monotonic Stack: A stack keeping elements in an ordered manner. ref
🔹Big-O-Notation: ref
🔹src: ref
🔹Master Graph Algorithms for Coding Interviews
🔹20 Patterns to Master Dynamic Programming
-
Medallion architecture: A data design pattern for lakehouses. It enhances data quality across three layers: bronze (raw), silver (curated), and gold (presentation). This “multi-hop” architecture allows data to transition between layers as required. ref
-
Slowly changing dimensions (SCD): Slowly Changing Dimensions change over time, but at a slow pace and unpredictably. For example, a customer’s address in a retail business.
-
Star schema: The Star Schema is a data model for data warehouses. It has a central fact table for measurable data and surrounding dimension tables for descriptive data. ref
-
OLAP vs OLTP:
OLAP
: Used for complex data analysis and business reporting, such as financial analysis and sales forecasting.OLTP
: Used for real-time processing of online transactions, including everyday transactions like ATM withdrawals and in-store purchases. -
Test-Driven Development
Expand
🔹F.I.R.S.T: A testing principle where tests are Fast, Isolated, Repeatable, Self-validating, and Timely/Thorough.
🔹DAMP: Stands for "Descriptive And Meaningful Phrases" in testing.
🔹BDD: Behavior Driven Development uses "Given-When-Then" format.
🔹DRY: "Don’t Repeat Yourself" principle avoids redundancy.
🔹TDD: Test-Driven Development focuses on tests first.
🔹Exploratory Testing: Simultaneous learning, test design, and test execution. It is about exploring the application and finding defects that were not anticipated.
🔹Smoke Testing: A preliminary test to check the basic functionality of an application to ensure that the most crucial functions work.
🔹Alpha/Beta Testing: 🔹Alpha: Initial testing performed by internal staff / 🔹Beta: Testing performed by actual users
-
SLA, SLO, and SLI
Expand
🔹SLA (Service Level Agreement): A contract defining the expected level of service.
99.9% uptime
🔹SLO (Service Level Objective): A measure of service performance agreed upon in an SLA.
200ms response
🔹SLI (Service Level Indicator): A quantitative measure of a specific aspect of the level of service.
Query latency
-
Space-Based Architecture (SBA) vs Cell-based architecture
- Space-Based Architecture (SBA): Removing the database and instead using a shared memory (memory grids) model
- Cell-based architecture: multiple isolated workload instances (cells) for fault isolation and handling subsets of workload requests
-
Azure vs AWS vs GCP in Cloud Netwrok
Expand
🔹Cloud Network Structures:
- AWS: region → VPC → availability zone → subnet (public, private)
- Azure: region → VNet → availability zone → subnet
- GCP: global → VPC → subnet (region-specific)
🔹Traffic Between VNet or VPC:
- Set up a VNet or VPC gateway and configure the routing table.
- Azure: One VPN gateway per virtual network, supporting external and on-premise connectivity.
- Azure: Routing tables can be assigned at the subnet level.
- AWS: Routing tables are associated at the VPC level, determining traffic between subnets.
- GCP: Routing tables are associated with the VPC, not the subnet. Routes are determined by subnet CIDR ranges.
🔹Regional Traffic:
- Use peering for traffic between VNets or VPCs in the same region.
- Peering provides lower latency, higher bandwidth, and reduced costs compared to gateways
🔹Hybrid Connectivity:
- Required for on-premise networks to connect to cloud networks.
- Services used: Azure ExpressRoute, AWS Direct Connect, GCP Cloud Interconnect.
🔹Connectivity Scenario
Connectivity Scenario AWS Azure GCP VPC (VNet) <-> VPC (VNet) VPC Peering, AWS Transit Gateway VNet Peering, Azure Virtual WAN, VNet Gateway VPC Network Peering, Cloud VPN On-Premise <-> VPC (VNet) Virtual Private Gateway, AWS Direct Connect, AWS Transit Gateway VPN Gateway, Azure ExpressRoute, Azure Virtual WAN Cloud VPN, Cloud Interconnect VNet (VPC) <-> Internet Internet Gateway (public subnet), NAT Gateway (private subnet) Internet Gateway, NAT Gateway Cloud Router, Cloud NAT Subnet Traffic Control Network Access Control Lists (ACLs) Network Security Groups (NSGs) 1. Shared VPC (XPN) subnets and set permissions at project or subnet level 2. Firewall Rules (VM level). - In Azure, the equivalent of AWS Transit Gateway is Azure Virtual WAN
- Azure and GCP don’t have a direct concept of public and private subnets like AWS.
- An Azure Virtual Network (VNet) Gateway can serve as a VPN gateway or an ExpressRoute gateway. Site-to-Site, Point-to-Site, and VNet-to-VNet connections all use a VPN gateway.
- Point-to-site: Individual device to network.
- Site-to-site: Network to network.
- Private IP vs Public IP:
192.168.1.4
(not routable on the internet) vs34.207.152.137
🔹Resource hierarchy
Level AWS Azure GCP 1 Organization Management Group Organization 2 Organizational Unit (OU) Subscription Folder 3 Account Resource Group Project 4 Resources Resources Resources -
Security Words 101
Expand
-
- MIM/PAM: Microsoft Identity Manager / Privileged Access Management
- PAW: Privileged Account Workstations
- AADIS: Azure Active Directory Implementation Services
- DIAD: Design and Implementation for Azure Active Directory
- LAPS/SLAM: Local Administrator Password Solution / Security Lifecycle Automation & Management
- IAM: Identity and Access Management
- SSO: Single Sign-On
- MFA: Multi-Factor Authentication
-
- ATA: Advanced Threat Analytics
- PADS: Persistent Advisory Detection Service
- IR&R: Incident Response & Recovery
- ATP: Advanced Threat Protection
- OMS: Operations Management Suite
- ETD: Enterprise Threat Detection
- SIEM: Security Information and Event Management
- EDR: Endpoint Detection and Response
- XDR: Extended Detection and Response
- SCEP: System Center Endpoint Protection (Microsoft Defender for Endpoint)
-
- AIP: Azure Information Protection (=AD RMS+On-premise files)
- AD RMS: Active Directory Rights Management Services
- WIP: Windows Information Protection
- DLP: Data Loss Prevention
- IRM: Information Rights Management
-
- SDL: Security Development Lifecycle
- MSRA: Microsoft Security Risk Assessment
- DIF: Dynamic Identity Framework
- OAWSS: Offline Assessment for Windows Server Security
- OAADS: Offline Assessment for Active Directory Security
- SAST: Static Application Security Testing
- DAST: Dynamic Application Security Testing
-
- ESAE: Enhanced Security Administrative Environment
- SCCM: System Center Configuration Manager
- EMS: Enterprise Mobility Suite
- SCOM/ACS: System Center Operations Manager / Audit Collection Services
- GRC: Governance, Risk, and Compliance
- SOC: Security Operations Center
-
-
Cloud Design Patterns. ref
Expand
- Cache-Aside: Cache frequently used data for performance.
- CQRS: Separate reads/writes for scalability.
- Event Sourcing: Record full data change history.
- Materialized View: Precompute query results for speed.
- Sharding: Partition data to scale storage.
- Strangler Fig: Gradually migrate legacy systems.
- Anti-Corruption Layer: Isolate new systems from old ones.
- Bulkhead: Prevent failure spread across components.
- Sidecar: Add functionality without changing the core.
- BFF: Tailor backend for different clients.
- Queue-Based Load Leveling: Buffer requests for smooth load handling.
- Publisher-Subscriber: Broadcast messages to multiple consumers.
- Competing Consumers: Process messages in parallel for scalability.
- Message Broker: Route messages via intermediary.
- Pipes and Filters: Sequentially process data through components.
- Valet Key: Provide secure temporary access.
- Gatekeeper: Filter requests to protect backends.
- Federated Identity: Use third-party login credentials.
- Secret Store: Secure sensitive data.
- Validation: Ensure data input is sanitized.
- Retry: Retry failed operations automatically.
- Circuit Breaker: Stop repeated failing operations.
- Throttling: Limit resource usage via request control.
- Health Endpoint Monitoring: Expose health checks for monitoring.
-
Principles & Concepts: YAGNI, KISS, DRY, CAP Theorem, PACELC, ACID, BASE
Expand
- YAGNI (You Aren't Gonna Need It): Don’t add features until necessary.
- KISS (Keep It Simple, Stupid): Keep designs simple.
- DRY (Don't Repeat Yourself): Avoid code duplication.
- CAP Theorem (Consistency, Availability, Partition Tolerance): Choose between consistency, availability, and partition tolerance.
- PACELC (Partition Tolerance, Availability, Consistency, Else Latency/Consistency): Trade-offs exist in availability/consistency and latency/consistency.
- ACID (Atomicity, Consistency, Isolation, Durability): Properties ensuring reliable database transactions.
- BASE (Basically Available, Soft State, Eventually Consistent): Prioritizes availability and eventual consistency in distributed systems.
-
Windows UI Development Frameworks: WinUI3, Windows Form, WPF, UWP, Win32, .NET MAUI, Uno
Expand
- For new Windows apps, use the Windows App SDK and WinUI instead of UWP, which is no longer actively developed.
- .NET MAUI is backed by Microsoft, whereas the Uno Platform is supported by nventive.
graph TD A[WinUI 3 <br/>'Windows UI Library 3 <br/>Supports only Windows'] -->|Builds on| B[UWP <br/>'Universal Windows Platform <br/>Supports only Windows'] A -->|Supports| E[.NET MAUI <br/>'Multi-platform App UI <br/>Cross-platform support: not support Linux'] B -->|Shares components with| E C[WPF <br/>'Windows Presentation Foundation <br/>Supports only Windows'] -->|Integrates with| E D[Windows Forms <br/>'Older framework <br/>Supports only Windows'] -->|Legacy framework| C F[Uno Platform <br/>'Cross-platform apps <br/>Cross-platform support incl. WebAssembly'] -->|Uses| A F -->|Leverages| B G[Win32 MFC <br/>'Microsoft Foundation Classes <br/>Supports only Windows'] -->|Legacy framework| C H[PWA <br/>'Progressive Web Apps <br/>Cross-platform support'] -->|Builds on| B I[React Native for Windows <br/>'Cross-platform mobile framework <br/>Cross-platform support'] -->|Supports| A J[Blazor Hybrid <br/>'Web UI with native capabilities <br/>Cross-platform support'] -->|Integrates with| E %% Set styles for the backgrounds style A fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px; %% WinUI 3 as Windows-only style B fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px; %% UWP as Windows-only style E fill:#e0f7fa,stroke:#4d94ff,stroke-width:2px; %% Cross-platform style F fill:#e0f7fa,stroke:#4d94ff,stroke-width:2px; %% Cross-platform style C fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px; %% WPF as Windows-only style D fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px; %% Windows Forms as Windows-only style G fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px; %% Win32 MFC as Windows-only %% Set styles for web app frameworks style H fill:#c8e6c9,stroke:#388e3c,stroke-width:2px; %% PWA as Cross-platform style I fill:#c8e6c9,stroke:#388e3c,stroke-width:2px; %% React Native for Windows style J fill:#c8e6c9,stroke:#388e3c,stroke-width:2px; %% Blazor Hybrid %% Add color legend K[Legend: <br/> <span style='color:#4d94ff'>Blue Background:</span> Cross-platform support <br/> <span style='color:#6a1b9a'>Purple Background:</span> Supports only Windows <br/> <span style='color:#388e3c'>Green Background:</span> Web app frameworks] %% Set styles for the legend background style K fill:#fff9c4,stroke:#f57f17,stroke-width:2px; %% Comments click A "https://docs.microsoft.com/en-us/windows/apps/winui/winui3/" "WinUI 3: Modern UI framework for Windows apps." click B "https://docs.microsoft.com/en-us/windows/uwp/" "UWP: Build universal apps for all Windows devices." click C "https://docs.microsoft.com/en-us/dotnet/desktop/wpf/" "WPF: Rich desktop applications with advanced graphics." click D "https://docs.microsoft.com/en-us/dotnet/desktop/winforms/" "Windows Forms: Simplified desktop app development." click E "https://docs.microsoft.com/en-us/dotnet/maui/" ".NET MAUI: Cross-platform UI framework for mobile and desktop." click F "https://platform.uno/" "Uno Platform: Build cross-platform apps using WinUI." click G "https://learn.microsoft.com/en-us/cpp/mfc/overview-of-mfc?view=msvc-160" "Win32 MFC: C++ framework for Windows desktop applications." click H "https://learn.microsoft.com/en-us/microsoft-edge/progressive-web-apps-chromium/" "PWA: Build web apps that behave like native apps." click I "https://microsoft.github.io/react-native-windows/" "React Native for Windows: Build cross-platform apps using React Native." click J "https://docs.microsoft.com/en-us/aspnet/core/client-side/blazor/hybrid?view=aspnetcore-7.0" "Blazor Hybrid: Build web UIs with native capabilities."
-
Mixin = Interface + actual implementation. It is a class that provides reusable methods/behavior to other classes without requiring full inheritance.