-
+
1 Why Mamba and S
1 Why Mamba and S
-
+
Figure 1.1: Spectrum of Efficiency vs Effectiveness of State Representation in Different Model Architecture Families [1]
@@ -287,7 +301,7 @@ 1 Why Mamba and S
-
+
Figure 1.2: Discrete - Continuous Spectrum of Data Sources and Examples [4]
@@ -299,7 +313,7 @@ 1 Why Mamba and S
-
+
Figure 1.3: Long Range Arena: Benchmark Spanning Text Images, Symbolic Reasoning (1K-16K token length) [6]
@@ -314,7 +328,7 @@ 1 Why Mamba and S
-
+
@@ -323,7 +337,7 @@ 1 Why Mamba and S
-
+
@@ -348,7 +362,7 @@
-
+
Head View: Visualising attention head activations between different layers. Connecting lines are weighted based on the attention score between respective words.
@@ -356,7 +370,7 @@
-
+
Neuron View: Visualising query, key and value embeddings when computing attention between each token and other tokens within the sequence. Positive values are colored as blue and negative values as orange.
@@ -376,7 +390,7 @@
-
+
OpenAI’s GPT-4-128K Long Context Performance
@@ -384,7 +398,7 @@
-
+
Anthropic’s Claude 2.1 Long Context Performance
@@ -400,7 +414,7 @@
-
+
Figure 1.7: Lost in the Middle: Performance Degrades When Information Access is in the Middle of Document [10]
@@ -416,7 +430,7 @@
-
+
Figure 1.8: Comparison of scaled dot-product attention with and without KV caching [12]
@@ -465,7 +479,7 @@
-
+
Figure 1.9: Unrolling Recurrent Neural Network Architecture Over Time
@@ -550,7 +564,7 @@ 2 What are Struct
-
+
Figure 2.1: The Three Representations of Linear State Space Layers in S4: (Left) State space models allow us to model continuous-time systems .(Center) The discretised recurrent format can be used for fast autoregressive inference. Recent theory on continuous-time memorisation of the hidden state transition matrix \(\mathbf{\bar{A}}\) enables us to capture LRDs mathematically and empirically. (Right) Unrolling the RNN into a global convolutional representation allows for efficient training by computing the layer depthwise in parallel [15].
@@ -582,14 +596,14 @@
-
+
-
+
@@ -623,7 +637,7 @@
-
+
Zero Order Hold Sampling Function
@@ -635,7 +649,7 @@
-
+
Discrete SSM Diagram [1]
@@ -657,7 +671,7 @@
-
+
Figure 2.4: Visualising 1D Convolution with 1x3 Kernel [16]
@@ -677,7 +691,7 @@
-
+
Signal in Time and Frequency Domain [17]
@@ -689,7 +703,7 @@
-
+
Legendre Polynomials [18]
@@ -711,7 +725,7 @@
-
+
@@ -720,7 +734,7 @@
-
+
@@ -729,7 +743,7 @@
-
+
@@ -752,7 +766,7 @@
-
+
Diagonal Plus Low-rank Approximation
@@ -764,7 +778,7 @@
-
+
S4D Recurrent and Convolutional View: Colors denote independent 1D SSMs; purple denotes trainable parameters [20]
@@ -787,7 +801,7 @@
-
+
Visualising S4 vs S4D Results
@@ -799,7 +813,7 @@
-
+
S4 vs S4D Long Range Arena Results
@@ -822,7 +836,7 @@ 3 How does Mamba
-
+
Figure 3.1: Differences between S4 and Mamba (S6) [3]
@@ -841,7 +855,7 @@
-
+
Selective Copying: This requires time-varying models that can selectively remember or ignore inputs depending on their content.
@@ -853,7 +867,7 @@
-
+
Induction Heads: This is an associative recall task which requires retrieving an answer based on context, a key ability of LLMs.
@@ -881,7 +895,7 @@
-
+
Selective Copying Results: Accuracy for combinations of architectures
@@ -893,7 +907,7 @@
-
+
Induction Heads Extrapolation: Mamba has ability to maintain high induction test accuracy for sequence length up to 1 million tokens
@@ -920,7 +934,7 @@
-
+
(Left): Average Memory Bandwidth for A100 (Right): Selective SSM Architecture Simplified: The select state layer is kept and computed in SRAM. [1]
@@ -934,7 +948,7 @@
-
+
State Selection with Hardware-Aware State Expansion: The selection mechanism ensures the expanded matrix states only materialise in SRAM to reduce data transfer and computation between SRAM<>HBM. [3]
@@ -960,7 +974,7 @@
-
+
Visualisation of Linear Scan
@@ -972,7 +986,7 @@
-
+
Visualisation of Blelloch Algorithm (Work-Efficient Parallel Prefix Scan)
@@ -997,7 +1011,7 @@
-
+
(Left): FlashAttention: The \(\mathbf{(QK)V}\) matrix of size \(N^2\) is computed in SRAM using tiling before being written to HBM. (Right): Speedup of Attention on GPT-2
@@ -1022,7 +1036,7 @@
-
+
Neural Network Computation Graph Source
@@ -1036,7 +1050,7 @@
-
+
Recomputing of Activations on Backward Pass: Blue = forward, Red = backward Source
@@ -1050,7 +1064,7 @@
-
+
Saving GPU Memory with Re-computation [24]
@@ -1079,7 +1093,7 @@
-
+
From H3 to the Mamba Block [25]
@@ -1091,7 +1105,7 @@
-
+
Mamba Block Decoder Architecture [1]
@@ -1118,7 +1132,7 @@
-
+
Comparison of Mamba variants with different popular 7B LLMs on Piqa, Winogrande, Lambada, and Hellaswag Source
@@ -1130,7 +1144,7 @@
-
+
Evaluation Comparison of Mamba variants with several similar-sized LLMs [3]
@@ -1159,7 +1173,7 @@ 4 Conclusion and
-
+
Timeline of SSM based Models [27]
@@ -1171,7 +1185,7 @@ 4 Conclusion and
-
+
SSM Model Landscape Over Various Domains [28]
@@ -1200,7 +1214,7 @@
-
+
Vision Mamba Scan Techniques
@@ -1212,7 +1226,7 @@
-
+
Vision Mamba Model Landscape
@@ -1843,7 +1857,7 @@ 5 References
-
-
+
+
1 Why Mamba and S
-
+
Figure 1.2: Discrete - Continuous Spectrum of Data Sources and Examples [4]
@@ -299,7 +313,7 @@ 1 Why Mamba and S
-
+
Figure 1.3: Long Range Arena: Benchmark Spanning Text Images, Symbolic Reasoning (1K-16K token length) [6]
@@ -314,7 +328,7 @@ 1 Why Mamba and S
-
+
@@ -323,7 +337,7 @@ 1 Why Mamba and S
-
+
@@ -348,7 +362,7 @@
-
+
Head View: Visualising attention head activations between different layers. Connecting lines are weighted based on the attention score between respective words.
@@ -356,7 +370,7 @@
-
+
Neuron View: Visualising query, key and value embeddings when computing attention between each token and other tokens within the sequence. Positive values are colored as blue and negative values as orange.
@@ -376,7 +390,7 @@
-
+
OpenAI’s GPT-4-128K Long Context Performance
@@ -384,7 +398,7 @@
-
+
Anthropic’s Claude 2.1 Long Context Performance
@@ -400,7 +414,7 @@
-
+
Figure 1.7: Lost in the Middle: Performance Degrades When Information Access is in the Middle of Document [10]
@@ -416,7 +430,7 @@
-
+
Figure 1.8: Comparison of scaled dot-product attention with and without KV caching [12]
@@ -465,7 +479,7 @@
-
+
Figure 1.9: Unrolling Recurrent Neural Network Architecture Over Time
@@ -550,7 +564,7 @@ 2 What are Struct
-
+
Figure 2.1: The Three Representations of Linear State Space Layers in S4: (Left) State space models allow us to model continuous-time systems .(Center) The discretised recurrent format can be used for fast autoregressive inference. Recent theory on continuous-time memorisation of the hidden state transition matrix \(\mathbf{\bar{A}}\) enables us to capture LRDs mathematically and empirically. (Right) Unrolling the RNN into a global convolutional representation allows for efficient training by computing the layer depthwise in parallel [15].
@@ -582,14 +596,14 @@
-
+
-
+
@@ -623,7 +637,7 @@
-
+
Zero Order Hold Sampling Function
@@ -635,7 +649,7 @@
-
+
Discrete SSM Diagram [1]
@@ -657,7 +671,7 @@
-
+
Figure 2.4: Visualising 1D Convolution with 1x3 Kernel [16]
@@ -677,7 +691,7 @@
-
+
Signal in Time and Frequency Domain [17]
@@ -689,7 +703,7 @@
-
+
Legendre Polynomials [18]
@@ -711,7 +725,7 @@
-
+
@@ -720,7 +734,7 @@
-
+
@@ -729,7 +743,7 @@
-
+
-
+
+
1 Why Mamba and S
-
+
Figure 1.3: Long Range Arena: Benchmark Spanning Text Images, Symbolic Reasoning (1K-16K token length) [6]
@@ -314,7 +328,7 @@ 1 Why Mamba and S
-
+
@@ -323,7 +337,7 @@ 1 Why Mamba and S
-
+
@@ -348,7 +362,7 @@
-
+
Head View: Visualising attention head activations between different layers. Connecting lines are weighted based on the attention score between respective words.
@@ -356,7 +370,7 @@
-
+
Neuron View: Visualising query, key and value embeddings when computing attention between each token and other tokens within the sequence. Positive values are colored as blue and negative values as orange.
@@ -376,7 +390,7 @@
-
+
OpenAI’s GPT-4-128K Long Context Performance
@@ -384,7 +398,7 @@
-
+
Anthropic’s Claude 2.1 Long Context Performance
@@ -400,7 +414,7 @@
-
+
Figure 1.7: Lost in the Middle: Performance Degrades When Information Access is in the Middle of Document [10]
@@ -416,7 +430,7 @@
-
+
Figure 1.8: Comparison of scaled dot-product attention with and without KV caching [12]
@@ -465,7 +479,7 @@
-
+
Figure 1.9: Unrolling Recurrent Neural Network Architecture Over Time
@@ -550,7 +564,7 @@ 2 What are Struct
-
+
Figure 2.1: The Three Representations of Linear State Space Layers in S4: (Left) State space models allow us to model continuous-time systems .(Center) The discretised recurrent format can be used for fast autoregressive inference. Recent theory on continuous-time memorisation of the hidden state transition matrix \(\mathbf{\bar{A}}\) enables us to capture LRDs mathematically and empirically. (Right) Unrolling the RNN into a global convolutional representation allows for efficient training by computing the layer depthwise in parallel [15].
@@ -582,14 +596,14 @@
-
+
-
+
@@ -623,7 +637,7 @@
-
+
Zero Order Hold Sampling Function
@@ -635,7 +649,7 @@
-
+
Discrete SSM Diagram [1]
@@ -657,7 +671,7 @@
-
+
Figure 2.4: Visualising 1D Convolution with 1x3 Kernel [16]
@@ -677,7 +691,7 @@
-
+
Signal in Time and Frequency Domain [17]
@@ -689,7 +703,7 @@
-
+
Legendre Polynomials [18]
@@ -711,7 +725,7 @@
-
+
@@ -720,7 +734,7 @@
-
+
-
+
+
1 Why Mamba and S
-
+
@@ -323,7 +337,7 @@ 1 Why Mamba and S
-
+
@@ -348,7 +362,7 @@
-
+
Head View: Visualising attention head activations between different layers. Connecting lines are weighted based on the attention score between respective words.
@@ -356,7 +370,7 @@
-
+
Neuron View: Visualising query, key and value embeddings when computing attention between each token and other tokens within the sequence. Positive values are colored as blue and negative values as orange.
@@ -376,7 +390,7 @@
-
+
OpenAI’s GPT-4-128K Long Context Performance
@@ -384,7 +398,7 @@
-
+
Anthropic’s Claude 2.1 Long Context Performance
@@ -400,7 +414,7 @@
-
+
Figure 1.7: Lost in the Middle: Performance Degrades When Information Access is in the Middle of Document [10]
@@ -416,7 +430,7 @@
-
+
Figure 1.8: Comparison of scaled dot-product attention with and without KV caching [12]
@@ -465,7 +479,7 @@
-
+
Figure 1.9: Unrolling Recurrent Neural Network Architecture Over Time
@@ -550,7 +564,7 @@ 2 What are Struct
-
+
Figure 2.1: The Three Representations of Linear State Space Layers in S4: (Left) State space models allow us to model continuous-time systems .(Center) The discretised recurrent format can be used for fast autoregressive inference. Recent theory on continuous-time memorisation of the hidden state transition matrix \(\mathbf{\bar{A}}\) enables us to capture LRDs mathematically and empirically. (Right) Unrolling the RNN into a global convolutional representation allows for efficient training by computing the layer depthwise in parallel [15].
@@ -582,14 +596,14 @@
-
+
-
+
@@ -623,7 +637,7 @@
-
+
Zero Order Hold Sampling Function
@@ -635,7 +649,7 @@
-
+
Discrete SSM Diagram [1]
@@ -657,7 +671,7 @@
-
+
Figure 2.4: Visualising 1D Convolution with 1x3 Kernel [16]
@@ -677,7 +691,7 @@
-
+
Signal in Time and Frequency Domain [17]
@@ -689,7 +703,7 @@
-
+
Legendre Polynomials [18]
@@ -711,7 +725,7 @@
-
+
-
+
Head View: Visualising attention head activations between different layers. Connecting lines are weighted based on the attention score between respective words.
@@ -356,7 +370,7 @@
-
+
Neuron View: Visualising query, key and value embeddings when computing attention between each token and other tokens within the sequence. Positive values are colored as blue and negative values as orange.
@@ -376,7 +390,7 @@
-
+
OpenAI’s GPT-4-128K Long Context Performance
@@ -384,7 +398,7 @@
-
+
Anthropic’s Claude 2.1 Long Context Performance
@@ -400,7 +414,7 @@
-
+
Figure 1.7: Lost in the Middle: Performance Degrades When Information Access is in the Middle of Document [10]
@@ -416,7 +430,7 @@
-
+
Figure 1.8: Comparison of scaled dot-product attention with and without KV caching [12]
@@ -465,7 +479,7 @@
-
+
Figure 1.9: Unrolling Recurrent Neural Network Architecture Over Time
@@ -550,7 +564,7 @@ 2 What are Struct
-
+
Figure 2.1: The Three Representations of Linear State Space Layers in S4: (Left) State space models allow us to model continuous-time systems .(Center) The discretised recurrent format can be used for fast autoregressive inference. Recent theory on continuous-time memorisation of the hidden state transition matrix \(\mathbf{\bar{A}}\) enables us to capture LRDs mathematically and empirically. (Right) Unrolling the RNN into a global convolutional representation allows for efficient training by computing the layer depthwise in parallel [15].
@@ -582,14 +596,14 @@
-
+
-
+
@@ -623,7 +637,7 @@
-
+
Zero Order Hold Sampling Function
@@ -635,7 +649,7 @@
-
+
Discrete SSM Diagram [1]
@@ -657,7 +671,7 @@
-
+
Figure 2.4: Visualising 1D Convolution with 1x3 Kernel [16]
@@ -677,7 +691,7 @@
-
+
Signal in Time and Frequency Domain [17]
@@ -689,7 +703,7 @@
-
+
Legendre Polynomials [18]
@@ -711,7 +725,7 @@
-
+
-
+
OpenAI’s GPT-4-128K Long Context Performance
@@ -384,7 +398,7 @@
-
+
Anthropic’s Claude 2.1 Long Context Performance
@@ -400,7 +414,7 @@
-
+
Figure 1.7: Lost in the Middle: Performance Degrades When Information Access is in the Middle of Document [10]
@@ -416,7 +430,7 @@
-
+
Figure 1.8: Comparison of scaled dot-product attention with and without KV caching [12]
@@ -465,7 +479,7 @@
-
+
Figure 1.9: Unrolling Recurrent Neural Network Architecture Over Time
@@ -550,7 +564,7 @@ 2 What are Struct
-
+
Figure 2.1: The Three Representations of Linear State Space Layers in S4: (Left) State space models allow us to model continuous-time systems .(Center) The discretised recurrent format can be used for fast autoregressive inference. Recent theory on continuous-time memorisation of the hidden state transition matrix \(\mathbf{\bar{A}}\) enables us to capture LRDs mathematically and empirically. (Right) Unrolling the RNN into a global convolutional representation allows for efficient training by computing the layer depthwise in parallel [15].
@@ -582,14 +596,14 @@
-
+
-
+
@@ -623,7 +637,7 @@
-
+
Zero Order Hold Sampling Function
@@ -635,7 +649,7 @@
-
+
Discrete SSM Diagram [1]
@@ -657,7 +671,7 @@
-
+
Figure 2.4: Visualising 1D Convolution with 1x3 Kernel [16]
@@ -677,7 +691,7 @@
-
+
Signal in Time and Frequency Domain [17]
@@ -689,7 +703,7 @@
-
+
Legendre Polynomials [18]
@@ -711,7 +725,7 @@
-
+
-
+
Figure 1.7: Lost in the Middle: Performance Degrades When Information Access is in the Middle of Document [10]
@@ -416,7 +430,7 @@
-
+
Figure 1.8: Comparison of scaled dot-product attention with and without KV caching [12]
@@ -465,7 +479,7 @@
-
+
Figure 1.9: Unrolling Recurrent Neural Network Architecture Over Time
@@ -550,7 +564,7 @@ 2 What are Struct
-
+
Figure 2.1: The Three Representations of Linear State Space Layers in S4: (Left) State space models allow us to model continuous-time systems .(Center) The discretised recurrent format can be used for fast autoregressive inference. Recent theory on continuous-time memorisation of the hidden state transition matrix \(\mathbf{\bar{A}}\) enables us to capture LRDs mathematically and empirically. (Right) Unrolling the RNN into a global convolutional representation allows for efficient training by computing the layer depthwise in parallel [15].
@@ -582,14 +596,14 @@
-
+
-
+
@@ -623,7 +637,7 @@
-
+
Zero Order Hold Sampling Function
@@ -635,7 +649,7 @@
-
+
Discrete SSM Diagram [1]
@@ -657,7 +671,7 @@
-
+
Figure 2.4: Visualising 1D Convolution with 1x3 Kernel [16]
@@ -677,7 +691,7 @@
-
+
Signal in Time and Frequency Domain [17]
@@ -689,7 +703,7 @@
-
+
Legendre Polynomials [18]
@@ -711,7 +725,7 @@
-
+
-
+
+
-
+
Figure 1.8: Comparison of scaled dot-product attention with and without KV caching [12]
@@ -465,7 +479,7 @@
-
+
Figure 1.9: Unrolling Recurrent Neural Network Architecture Over Time
@@ -550,7 +564,7 @@ 2 What are Struct
-
+
Figure 2.1: The Three Representations of Linear State Space Layers in S4: (Left) State space models allow us to model continuous-time systems .(Center) The discretised recurrent format can be used for fast autoregressive inference. Recent theory on continuous-time memorisation of the hidden state transition matrix \(\mathbf{\bar{A}}\) enables us to capture LRDs mathematically and empirically. (Right) Unrolling the RNN into a global convolutional representation allows for efficient training by computing the layer depthwise in parallel [15].
@@ -582,14 +596,14 @@
-
+
-
+
@@ -623,7 +637,7 @@
-
+
Zero Order Hold Sampling Function
@@ -635,7 +649,7 @@
-
+
Discrete SSM Diagram [1]
@@ -657,7 +671,7 @@
-
+
Figure 2.4: Visualising 1D Convolution with 1x3 Kernel [16]
@@ -677,7 +691,7 @@
-
+
Signal in Time and Frequency Domain [17]
@@ -689,7 +703,7 @@
-
+
Legendre Polynomials [18]
@@ -711,7 +725,7 @@
-
+
-
+
+
-
+
Figure 1.9: Unrolling Recurrent Neural Network Architecture Over Time
@@ -550,7 +564,7 @@ 2 What are Struct
-
+
Figure 2.1: The Three Representations of Linear State Space Layers in S4: (Left) State space models allow us to model continuous-time systems .(Center) The discretised recurrent format can be used for fast autoregressive inference. Recent theory on continuous-time memorisation of the hidden state transition matrix \(\mathbf{\bar{A}}\) enables us to capture LRDs mathematically and empirically. (Right) Unrolling the RNN into a global convolutional representation allows for efficient training by computing the layer depthwise in parallel [15].
@@ -582,14 +596,14 @@
-
+
-
+
@@ -623,7 +637,7 @@
-
+
Zero Order Hold Sampling Function
@@ -635,7 +649,7 @@
-
+
Discrete SSM Diagram [1]
@@ -657,7 +671,7 @@
-
+
Figure 2.4: Visualising 1D Convolution with 1x3 Kernel [16]
@@ -677,7 +691,7 @@
-
+
Signal in Time and Frequency Domain [17]
@@ -689,7 +703,7 @@
-
+
Legendre Polynomials [18]
@@ -711,7 +725,7 @@
-
+
-
+
+
2 What are Struct
-
+
Figure 2.1: The Three Representations of Linear State Space Layers in S4: (Left) State space models allow us to model continuous-time systems .(Center) The discretised recurrent format can be used for fast autoregressive inference. Recent theory on continuous-time memorisation of the hidden state transition matrix \(\mathbf{\bar{A}}\) enables us to capture LRDs mathematically and empirically. (Right) Unrolling the RNN into a global convolutional representation allows for efficient training by computing the layer depthwise in parallel [15].
@@ -582,14 +596,14 @@
-
+
-
+
@@ -623,7 +637,7 @@
-
+
Zero Order Hold Sampling Function
@@ -635,7 +649,7 @@
-
+
Discrete SSM Diagram [1]
@@ -657,7 +671,7 @@
-
+
Figure 2.4: Visualising 1D Convolution with 1x3 Kernel [16]
@@ -677,7 +691,7 @@
-
+
Signal in Time and Frequency Domain [17]
@@ -689,7 +703,7 @@
-
+
Legendre Polynomials [18]
@@ -711,7 +725,7 @@
-
+
-
+
+
-
+
-
+
Discrete SSM Diagram [1]
@@ -657,7 +671,7 @@
-
+
Figure 2.4: Visualising 1D Convolution with 1x3 Kernel [16]
@@ -677,7 +691,7 @@
-
+
Signal in Time and Frequency Domain [17]
@@ -689,7 +703,7 @@
-
+
Legendre Polynomials [18]
@@ -711,7 +725,7 @@
-
+
-
+
+
-
+
Signal in Time and Frequency Domain [17]
@@ -689,7 +703,7 @@
-
+
Legendre Polynomials [18]
@@ -711,7 +725,7 @@
-
+
-
+
-
+
S4D Recurrent and Convolutional View: Colors denote independent 1D SSMs; purple denotes trainable parameters [20]
@@ -787,7 +801,7 @@
-
+
Visualising S4 vs S4D Results
@@ -799,7 +813,7 @@
-
+
S4 vs S4D Long Range Arena Results
@@ -822,7 +836,7 @@ 3 How does Mamba
-
+
Figure 3.1: Differences between S4 and Mamba (S6) [3]
@@ -841,7 +855,7 @@
-
+
Selective Copying: This requires time-varying models that can selectively remember or ignore inputs depending on their content.
@@ -853,7 +867,7 @@
-
+
Induction Heads: This is an associative recall task which requires retrieving an answer based on context, a key ability of LLMs.
@@ -881,7 +895,7 @@
-
+
Selective Copying Results: Accuracy for combinations of architectures
@@ -893,7 +907,7 @@
-
+
Induction Heads Extrapolation: Mamba has ability to maintain high induction test accuracy for sequence length up to 1 million tokens
@@ -920,7 +934,7 @@
-
+
(Left): Average Memory Bandwidth for A100 (Right): Selective SSM Architecture Simplified: The select state layer is kept and computed in SRAM. [1]
@@ -934,7 +948,7 @@
-
+
State Selection with Hardware-Aware State Expansion: The selection mechanism ensures the expanded matrix states only materialise in SRAM to reduce data transfer and computation between SRAM<>HBM. [3]
@@ -960,7 +974,7 @@
-
+
Visualisation of Linear Scan
@@ -972,7 +986,7 @@
-
+
Visualisation of Blelloch Algorithm (Work-Efficient Parallel Prefix Scan)
@@ -997,7 +1011,7 @@
-
+
(Left): FlashAttention: The \(\mathbf{(QK)V}\) matrix of size \(N^2\) is computed in SRAM using tiling before being written to HBM. (Right): Speedup of Attention on GPT-2
@@ -1022,7 +1036,7 @@
-
+
Neural Network Computation Graph Source
@@ -1036,7 +1050,7 @@
-
+
Recomputing of Activations on Backward Pass: Blue = forward, Red = backward Source
@@ -1050,7 +1064,7 @@
-
+
Saving GPU Memory with Re-computation [24]
@@ -1079,7 +1093,7 @@
-
+
From H3 to the Mamba Block [25]
@@ -1091,7 +1105,7 @@
-
+
Mamba Block Decoder Architecture [1]
@@ -1118,7 +1132,7 @@
-
+
Comparison of Mamba variants with different popular 7B LLMs on Piqa, Winogrande, Lambada, and Hellaswag Source
@@ -1130,7 +1144,7 @@
-
+
Evaluation Comparison of Mamba variants with several similar-sized LLMs [3]
@@ -1159,7 +1173,7 @@ 4 Conclusion and
-
+
Timeline of SSM based Models [27]
@@ -1171,7 +1185,7 @@ 4 Conclusion and
-
+
SSM Model Landscape Over Various Domains [28]
@@ -1200,7 +1214,7 @@
-
+
Vision Mamba Scan Techniques
@@ -1212,7 +1226,7 @@
-
+
Vision Mamba Model Landscape
@@ -1843,7 +1857,7 @@ 5 References
-
-
+
S4 vs S4D Long Range Arena Results
@@ -822,7 +836,7 @@ 3 How does Mamba
-
+
Figure 3.1: Differences between S4 and Mamba (S6) [3]
@@ -841,7 +855,7 @@
-
+
Selective Copying: This requires time-varying models that can selectively remember or ignore inputs depending on their content.
@@ -853,7 +867,7 @@
-
+
Induction Heads: This is an associative recall task which requires retrieving an answer based on context, a key ability of LLMs.
@@ -881,7 +895,7 @@
-
+
Selective Copying Results: Accuracy for combinations of architectures
@@ -893,7 +907,7 @@
-
+
Induction Heads Extrapolation: Mamba has ability to maintain high induction test accuracy for sequence length up to 1 million tokens
@@ -920,7 +934,7 @@
-
+
(Left): Average Memory Bandwidth for A100 (Right): Selective SSM Architecture Simplified: The select state layer is kept and computed in SRAM. [1]
@@ -934,7 +948,7 @@
-
+
State Selection with Hardware-Aware State Expansion: The selection mechanism ensures the expanded matrix states only materialise in SRAM to reduce data transfer and computation between SRAM<>HBM. [3]
@@ -960,7 +974,7 @@
-
+
Visualisation of Linear Scan
@@ -972,7 +986,7 @@
-
+
Visualisation of Blelloch Algorithm (Work-Efficient Parallel Prefix Scan)
@@ -997,7 +1011,7 @@
-
+
(Left): FlashAttention: The \(\mathbf{(QK)V}\) matrix of size \(N^2\) is computed in SRAM using tiling before being written to HBM. (Right): Speedup of Attention on GPT-2
@@ -1022,7 +1036,7 @@
-
+
Neural Network Computation Graph Source
@@ -1036,7 +1050,7 @@
-
+
Recomputing of Activations on Backward Pass: Blue = forward, Red = backward Source
@@ -1050,7 +1064,7 @@
-
+
Saving GPU Memory with Re-computation [24]
@@ -1079,7 +1093,7 @@
-
+
From H3 to the Mamba Block [25]
@@ -1091,7 +1105,7 @@
-
+
Mamba Block Decoder Architecture [1]
@@ -1118,7 +1132,7 @@
-
+
Comparison of Mamba variants with different popular 7B LLMs on Piqa, Winogrande, Lambada, and Hellaswag Source
@@ -1130,7 +1144,7 @@
-
+
Evaluation Comparison of Mamba variants with several similar-sized LLMs [3]
@@ -1159,7 +1173,7 @@ 4 Conclusion and
-
+
Timeline of SSM based Models [27]
@@ -1171,7 +1185,7 @@ 4 Conclusion and
-
+
SSM Model Landscape Over Various Domains [28]
@@ -1200,7 +1214,7 @@
-
+
Vision Mamba Scan Techniques
@@ -1212,7 +1226,7 @@
-
+
Vision Mamba Model Landscape
@@ -1843,7 +1857,7 @@ 5 References
-
-
+
+
-
+
Selective Copying: This requires time-varying models that can selectively remember or ignore inputs depending on their content.
@@ -853,7 +867,7 @@
-
+
Induction Heads: This is an associative recall task which requires retrieving an answer based on context, a key ability of LLMs.
@@ -881,7 +895,7 @@
-
+
Selective Copying Results: Accuracy for combinations of architectures
@@ -893,7 +907,7 @@
-
+
Induction Heads Extrapolation: Mamba has ability to maintain high induction test accuracy for sequence length up to 1 million tokens
@@ -920,7 +934,7 @@
-
+
(Left): Average Memory Bandwidth for A100 (Right): Selective SSM Architecture Simplified: The select state layer is kept and computed in SRAM. [1]
@@ -934,7 +948,7 @@
-
+
State Selection with Hardware-Aware State Expansion: The selection mechanism ensures the expanded matrix states only materialise in SRAM to reduce data transfer and computation between SRAM<>HBM. [3]
@@ -960,7 +974,7 @@
-
+
Visualisation of Linear Scan
@@ -972,7 +986,7 @@
-
+
Visualisation of Blelloch Algorithm (Work-Efficient Parallel Prefix Scan)
@@ -997,7 +1011,7 @@
-
+
(Left): FlashAttention: The \(\mathbf{(QK)V}\) matrix of size \(N^2\) is computed in SRAM using tiling before being written to HBM. (Right): Speedup of Attention on GPT-2
@@ -1022,7 +1036,7 @@
-
+
Neural Network Computation Graph Source
@@ -1036,7 +1050,7 @@
-
+
Recomputing of Activations on Backward Pass: Blue = forward, Red = backward Source
@@ -1050,7 +1064,7 @@
-
+
Saving GPU Memory with Re-computation [24]
@@ -1079,7 +1093,7 @@
-
+
From H3 to the Mamba Block [25]
@@ -1091,7 +1105,7 @@
-
+
Mamba Block Decoder Architecture [1]
@@ -1118,7 +1132,7 @@
-
+
Comparison of Mamba variants with different popular 7B LLMs on Piqa, Winogrande, Lambada, and Hellaswag Source
@@ -1130,7 +1144,7 @@
-
+
Evaluation Comparison of Mamba variants with several similar-sized LLMs [3]
@@ -1159,7 +1173,7 @@ 4 Conclusion and
-
+
Timeline of SSM based Models [27]
@@ -1171,7 +1185,7 @@ 4 Conclusion and
-
+
SSM Model Landscape Over Various Domains [28]
@@ -1200,7 +1214,7 @@
-
+
Vision Mamba Scan Techniques
@@ -1212,7 +1226,7 @@
-
+
Vision Mamba Model Landscape
@@ -1843,7 +1857,7 @@ 5 References
-
+
Induction Heads: This is an associative recall task which requires retrieving an answer based on context, a key ability of LLMs.
@@ -881,7 +895,7 @@
-
+
Selective Copying Results: Accuracy for combinations of architectures
@@ -893,7 +907,7 @@
-
+
Induction Heads Extrapolation: Mamba has ability to maintain high induction test accuracy for sequence length up to 1 million tokens
@@ -920,7 +934,7 @@
-
+
(Left): Average Memory Bandwidth for A100 (Right): Selective SSM Architecture Simplified: The select state layer is kept and computed in SRAM. [1]
@@ -934,7 +948,7 @@
-
+
State Selection with Hardware-Aware State Expansion: The selection mechanism ensures the expanded matrix states only materialise in SRAM to reduce data transfer and computation between SRAM<>HBM. [3]
@@ -960,7 +974,7 @@
-
+
Visualisation of Linear Scan
@@ -972,7 +986,7 @@
-
+
Visualisation of Blelloch Algorithm (Work-Efficient Parallel Prefix Scan)
@@ -997,7 +1011,7 @@
-
+
(Left): FlashAttention: The \(\mathbf{(QK)V}\) matrix of size \(N^2\) is computed in SRAM using tiling before being written to HBM. (Right): Speedup of Attention on GPT-2
@@ -1022,7 +1036,7 @@
-
+
Neural Network Computation Graph Source
@@ -1036,7 +1050,7 @@
-
+
Recomputing of Activations on Backward Pass: Blue = forward, Red = backward Source
@@ -1050,7 +1064,7 @@
-
+
Saving GPU Memory with Re-computation [24]
@@ -1079,7 +1093,7 @@
-
+
From H3 to the Mamba Block [25]
@@ -1091,7 +1105,7 @@
-
+
Mamba Block Decoder Architecture [1]
@@ -1118,7 +1132,7 @@
-
+
Comparison of Mamba variants with different popular 7B LLMs on Piqa, Winogrande, Lambada, and Hellaswag Source
@@ -1130,7 +1144,7 @@
-
+
Evaluation Comparison of Mamba variants with several similar-sized LLMs [3]
@@ -1159,7 +1173,7 @@ 4 Conclusion and
-
+
Timeline of SSM based Models [27]
@@ -1171,7 +1185,7 @@ 4 Conclusion and
-
+
SSM Model Landscape Over Various Domains [28]
@@ -1200,7 +1214,7 @@
-
+
Vision Mamba Scan Techniques
@@ -1212,7 +1226,7 @@
-
+
Vision Mamba Model Landscape
@@ -1843,7 +1857,7 @@ 5 References
-
+
Selective Copying Results: Accuracy for combinations of architectures
@@ -893,7 +907,7 @@
-
+
Induction Heads Extrapolation: Mamba has ability to maintain high induction test accuracy for sequence length up to 1 million tokens
@@ -920,7 +934,7 @@
-
+
(Left): Average Memory Bandwidth for A100 (Right): Selective SSM Architecture Simplified: The select state layer is kept and computed in SRAM. [1]
@@ -934,7 +948,7 @@
-
+
State Selection with Hardware-Aware State Expansion: The selection mechanism ensures the expanded matrix states only materialise in SRAM to reduce data transfer and computation between SRAM<>HBM. [3]
@@ -960,7 +974,7 @@
-
+
Visualisation of Linear Scan
@@ -972,7 +986,7 @@
-
+
Visualisation of Blelloch Algorithm (Work-Efficient Parallel Prefix Scan)
@@ -997,7 +1011,7 @@
-
+
(Left): FlashAttention: The \(\mathbf{(QK)V}\) matrix of size \(N^2\) is computed in SRAM using tiling before being written to HBM. (Right): Speedup of Attention on GPT-2
@@ -1022,7 +1036,7 @@
-
+
Neural Network Computation Graph Source
@@ -1036,7 +1050,7 @@
-
+
Recomputing of Activations on Backward Pass: Blue = forward, Red = backward Source
@@ -1050,7 +1064,7 @@
-
+
Saving GPU Memory with Re-computation [24]
@@ -1079,7 +1093,7 @@
-
+
From H3 to the Mamba Block [25]
@@ -1091,7 +1105,7 @@
-
+
Mamba Block Decoder Architecture [1]
@@ -1118,7 +1132,7 @@
-
+
Comparison of Mamba variants with different popular 7B LLMs on Piqa, Winogrande, Lambada, and Hellaswag Source
@@ -1130,7 +1144,7 @@
-
+
Evaluation Comparison of Mamba variants with several similar-sized LLMs [3]
@@ -1159,7 +1173,7 @@ 4 Conclusion and
-
+
Timeline of SSM based Models [27]
@@ -1171,7 +1185,7 @@ 4 Conclusion and
-
+
SSM Model Landscape Over Various Domains [28]
@@ -1200,7 +1214,7 @@
-
+
Vision Mamba Scan Techniques
@@ -1212,7 +1226,7 @@
-
+
Vision Mamba Model Landscape
@@ -1843,7 +1857,7 @@ 5 References
-
+
Induction Heads Extrapolation: Mamba has ability to maintain high induction test accuracy for sequence length up to 1 million tokens
@@ -920,7 +934,7 @@
-
+
(Left): Average Memory Bandwidth for A100 (Right): Selective SSM Architecture Simplified: The select state layer is kept and computed in SRAM. [1]
@@ -934,7 +948,7 @@
-
+
State Selection with Hardware-Aware State Expansion: The selection mechanism ensures the expanded matrix states only materialise in SRAM to reduce data transfer and computation between SRAM<>HBM. [3]
@@ -960,7 +974,7 @@
-
+
Visualisation of Linear Scan
@@ -972,7 +986,7 @@
-
+
Visualisation of Blelloch Algorithm (Work-Efficient Parallel Prefix Scan)
@@ -997,7 +1011,7 @@
-
+
(Left): FlashAttention: The \(\mathbf{(QK)V}\) matrix of size \(N^2\) is computed in SRAM using tiling before being written to HBM. (Right): Speedup of Attention on GPT-2
@@ -1022,7 +1036,7 @@
-
+
Neural Network Computation Graph Source
@@ -1036,7 +1050,7 @@
-
+
Recomputing of Activations on Backward Pass: Blue = forward, Red = backward Source
@@ -1050,7 +1064,7 @@
-
+
Saving GPU Memory with Re-computation [24]
@@ -1079,7 +1093,7 @@
-
+
From H3 to the Mamba Block [25]
@@ -1091,7 +1105,7 @@
-
+
Mamba Block Decoder Architecture [1]
@@ -1118,7 +1132,7 @@
-
+
Comparison of Mamba variants with different popular 7B LLMs on Piqa, Winogrande, Lambada, and Hellaswag Source
@@ -1130,7 +1144,7 @@
-
+
Evaluation Comparison of Mamba variants with several similar-sized LLMs [3]
@@ -1159,7 +1173,7 @@ 4 Conclusion and
-
+
Timeline of SSM based Models [27]
@@ -1171,7 +1185,7 @@ 4 Conclusion and
-
+
SSM Model Landscape Over Various Domains [28]
@@ -1200,7 +1214,7 @@
-
+
Vision Mamba Scan Techniques
@@ -1212,7 +1226,7 @@
-
+
Vision Mamba Model Landscape
@@ -1843,7 +1857,7 @@ 5 References
-
+
(Left): Average Memory Bandwidth for A100 (Right): Selective SSM Architecture Simplified: The select state layer is kept and computed in SRAM. [1]
@@ -934,7 +948,7 @@
-
+
State Selection with Hardware-Aware State Expansion: The selection mechanism ensures the expanded matrix states only materialise in SRAM to reduce data transfer and computation between SRAM<>HBM. [3]
@@ -960,7 +974,7 @@
-
+
Visualisation of Linear Scan
@@ -972,7 +986,7 @@
-
+
Visualisation of Blelloch Algorithm (Work-Efficient Parallel Prefix Scan)
@@ -997,7 +1011,7 @@
-
+
(Left): FlashAttention: The \(\mathbf{(QK)V}\) matrix of size \(N^2\) is computed in SRAM using tiling before being written to HBM. (Right): Speedup of Attention on GPT-2
@@ -1022,7 +1036,7 @@
-
+
Neural Network Computation Graph Source
@@ -1036,7 +1050,7 @@
-
+
Recomputing of Activations on Backward Pass: Blue = forward, Red = backward Source
@@ -1050,7 +1064,7 @@
-
+
Saving GPU Memory with Re-computation [24]
@@ -1079,7 +1093,7 @@
-
+
From H3 to the Mamba Block [25]
@@ -1091,7 +1105,7 @@
-
+
Mamba Block Decoder Architecture [1]
@@ -1118,7 +1132,7 @@
-
+
Comparison of Mamba variants with different popular 7B LLMs on Piqa, Winogrande, Lambada, and Hellaswag Source
@@ -1130,7 +1144,7 @@
-
+
Evaluation Comparison of Mamba variants with several similar-sized LLMs [3]
@@ -1159,7 +1173,7 @@ 4 Conclusion and
-
+
Timeline of SSM based Models [27]
@@ -1171,7 +1185,7 @@ 4 Conclusion and
-
+
SSM Model Landscape Over Various Domains [28]
@@ -1200,7 +1214,7 @@
-
+
Vision Mamba Scan Techniques
@@ -1212,7 +1226,7 @@
-
+
Vision Mamba Model Landscape
@@ -1843,7 +1857,7 @@ 5 References
-
+
Visualisation of Linear Scan
@@ -972,7 +986,7 @@
-
+
Visualisation of Blelloch Algorithm (Work-Efficient Parallel Prefix Scan)
@@ -997,7 +1011,7 @@
-
+
(Left): FlashAttention: The \(\mathbf{(QK)V}\) matrix of size \(N^2\) is computed in SRAM using tiling before being written to HBM. (Right): Speedup of Attention on GPT-2
@@ -1022,7 +1036,7 @@
-
+
Neural Network Computation Graph Source
@@ -1036,7 +1050,7 @@
-
+
Recomputing of Activations on Backward Pass: Blue = forward, Red = backward Source
@@ -1050,7 +1064,7 @@
-
+
Saving GPU Memory with Re-computation [24]
@@ -1079,7 +1093,7 @@
-
+
From H3 to the Mamba Block [25]
@@ -1091,7 +1105,7 @@
-
+
Mamba Block Decoder Architecture [1]
@@ -1118,7 +1132,7 @@
-
+
Comparison of Mamba variants with different popular 7B LLMs on Piqa, Winogrande, Lambada, and Hellaswag Source
@@ -1130,7 +1144,7 @@
-
+
Evaluation Comparison of Mamba variants with several similar-sized LLMs [3]
@@ -1159,7 +1173,7 @@ 4 Conclusion and
-
+
Timeline of SSM based Models [27]
@@ -1171,7 +1185,7 @@ 4 Conclusion and
-
+
SSM Model Landscape Over Various Domains [28]
@@ -1200,7 +1214,7 @@
-
+
Vision Mamba Scan Techniques
@@ -1212,7 +1226,7 @@
-
+
Vision Mamba Model Landscape
@@ -1843,7 +1857,7 @@ 5 References
-
+
Visualisation of Blelloch Algorithm (Work-Efficient Parallel Prefix Scan)
@@ -997,7 +1011,7 @@
-
+
(Left): FlashAttention: The \(\mathbf{(QK)V}\) matrix of size \(N^2\) is computed in SRAM using tiling before being written to HBM. (Right): Speedup of Attention on GPT-2
@@ -1022,7 +1036,7 @@
-
+
Neural Network Computation Graph Source
@@ -1036,7 +1050,7 @@
-
+
Recomputing of Activations on Backward Pass: Blue = forward, Red = backward Source
@@ -1050,7 +1064,7 @@
-
+
Saving GPU Memory with Re-computation [24]
@@ -1079,7 +1093,7 @@
-
+
From H3 to the Mamba Block [25]
@@ -1091,7 +1105,7 @@
-
+
Mamba Block Decoder Architecture [1]
@@ -1118,7 +1132,7 @@
-
+
Comparison of Mamba variants with different popular 7B LLMs on Piqa, Winogrande, Lambada, and Hellaswag Source
@@ -1130,7 +1144,7 @@
-
+
Evaluation Comparison of Mamba variants with several similar-sized LLMs [3]
@@ -1159,7 +1173,7 @@ 4 Conclusion and
-
+
Timeline of SSM based Models [27]
@@ -1171,7 +1185,7 @@ 4 Conclusion and
-
+
SSM Model Landscape Over Various Domains [28]
@@ -1200,7 +1214,7 @@
-
+
Vision Mamba Scan Techniques
@@ -1212,7 +1226,7 @@
-
+
Vision Mamba Model Landscape
@@ -1843,7 +1857,7 @@ 5 References
-
+
(Left): FlashAttention: The \(\mathbf{(QK)V}\) matrix of size \(N^2\) is computed in SRAM using tiling before being written to HBM. (Right): Speedup of Attention on GPT-2
@@ -1022,7 +1036,7 @@
-
+
Neural Network Computation Graph Source
@@ -1036,7 +1050,7 @@
-
+
Recomputing of Activations on Backward Pass: Blue = forward, Red = backward Source
@@ -1050,7 +1064,7 @@
-
+
Saving GPU Memory with Re-computation [24]
@@ -1079,7 +1093,7 @@
-
+
From H3 to the Mamba Block [25]
@@ -1091,7 +1105,7 @@
-
+
Mamba Block Decoder Architecture [1]
@@ -1118,7 +1132,7 @@
-
+
Comparison of Mamba variants with different popular 7B LLMs on Piqa, Winogrande, Lambada, and Hellaswag Source
@@ -1130,7 +1144,7 @@
-
+
Evaluation Comparison of Mamba variants with several similar-sized LLMs [3]
@@ -1159,7 +1173,7 @@ 4 Conclusion and
-
+
Timeline of SSM based Models [27]
@@ -1171,7 +1185,7 @@ 4 Conclusion and
-
+
SSM Model Landscape Over Various Domains [28]
@@ -1200,7 +1214,7 @@
-
+
Vision Mamba Scan Techniques
@@ -1212,7 +1226,7 @@
-
+
Vision Mamba Model Landscape
@@ -1843,7 +1857,7 @@ 5 References
-
+
Recomputing of Activations on Backward Pass: Blue = forward, Red = backward Source
@@ -1050,7 +1064,7 @@
-
+
Saving GPU Memory with Re-computation [24]
@@ -1079,7 +1093,7 @@
-
+
From H3 to the Mamba Block [25]
@@ -1091,7 +1105,7 @@
-
+
Mamba Block Decoder Architecture [1]
@@ -1118,7 +1132,7 @@
-
+
Comparison of Mamba variants with different popular 7B LLMs on Piqa, Winogrande, Lambada, and Hellaswag Source
@@ -1130,7 +1144,7 @@
-
+
Evaluation Comparison of Mamba variants with several similar-sized LLMs [3]
@@ -1159,7 +1173,7 @@ 4 Conclusion and
-
+
Timeline of SSM based Models [27]
@@ -1171,7 +1185,7 @@ 4 Conclusion and
-
+
SSM Model Landscape Over Various Domains [28]
@@ -1200,7 +1214,7 @@
-
+
Vision Mamba Scan Techniques
@@ -1212,7 +1226,7 @@
-
+
Vision Mamba Model Landscape
@@ -1843,7 +1857,7 @@ 5 References
-
+
From H3 to the Mamba Block [25]
@@ -1091,7 +1105,7 @@
-
+
Mamba Block Decoder Architecture [1]
@@ -1118,7 +1132,7 @@
-
+
Comparison of Mamba variants with different popular 7B LLMs on Piqa, Winogrande, Lambada, and Hellaswag Source
@@ -1130,7 +1144,7 @@
-
+
Evaluation Comparison of Mamba variants with several similar-sized LLMs [3]
@@ -1159,7 +1173,7 @@ 4 Conclusion and
-
+
Timeline of SSM based Models [27]
@@ -1171,7 +1185,7 @@ 4 Conclusion and
-
+
SSM Model Landscape Over Various Domains [28]
@@ -1200,7 +1214,7 @@
-
+
Vision Mamba Scan Techniques
@@ -1212,7 +1226,7 @@
-
+
Vision Mamba Model Landscape
@@ -1843,7 +1857,7 @@ 5 References
-
+
Comparison of Mamba variants with different popular 7B LLMs on Piqa, Winogrande, Lambada, and Hellaswag Source
@@ -1130,7 +1144,7 @@
-
+
Evaluation Comparison of Mamba variants with several similar-sized LLMs [3]
@@ -1159,7 +1173,7 @@ 4 Conclusion and
-
+
Timeline of SSM based Models [27]
@@ -1171,7 +1185,7 @@ 4 Conclusion and
-
+
SSM Model Landscape Over Various Domains [28]
@@ -1200,7 +1214,7 @@
-
+
Vision Mamba Scan Techniques
@@ -1212,7 +1226,7 @@
-
+
Vision Mamba Model Landscape
@@ -1843,7 +1857,7 @@ 5 References
4 Conclusion and
-
+
Timeline of SSM based Models [27]
@@ -1171,7 +1185,7 @@ 4 Conclusion and
-
+
SSM Model Landscape Over Various Domains [28]
@@ -1200,7 +1214,7 @@
-
+
Vision Mamba Scan Techniques
@@ -1212,7 +1226,7 @@
-
+
Vision Mamba Model Landscape
@@ -1843,7 +1857,7 @@ 5 References
4 Conclusion and
-
+
SSM Model Landscape Over Various Domains [28]
@@ -1200,7 +1214,7 @@
-
+
Vision Mamba Scan Techniques
@@ -1212,7 +1226,7 @@
-
+
Vision Mamba Model Landscape
@@ -1843,7 +1857,7 @@ 5 References
-
+
Vision Mamba Scan Techniques
@@ -1212,7 +1226,7 @@
-
+
Vision Mamba Model Landscape
@@ -1843,7 +1857,7 @@ 5 References