Skip to content

Commit

Permalink
fix T
Browse files Browse the repository at this point in the history
  • Loading branch information
iliadmitriev committed Dec 1, 2024
1 parent f87df8c commit 7a9f788
Showing 1 changed file with 3 additions and 3 deletions.
6 changes: 3 additions & 3 deletions ML/RNN/rnn.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ $$c^{\langle t\rangle }=\Gamma_u*\tilde{c}^{\langle t\rangle }+(1-\Gamma_u)*c^{\
- `value` (значение) - векторы, образующие контекст в входной последовательности $h_i$
- `query` (запрос) - векторы которые нужны в выходной последовательности $h_{t-1}'$

$$a(h_{i}, h_{t-1}') = (W_{k}h_{i})^T(W_{q}h_{t-1}')/\sqrt{d}$$
$$a(h_{i}, h_{t-1}') = (W_{k}h_{i})^\top(W_{q}h_{t-1}')/\sqrt{d}$$

$$\alpha_{ti}=SoftMax(a(h_{i}, h_{t-1}'))$$

Expand Down Expand Up @@ -100,8 +100,8 @@ $\alpha_{ij} = SoftMax_{i} a( W^j_k h_i, W^j_q q )$,

декоррелируются

$$(\alpha_{s}^{T} \alpha_{j} \to 0)$$
$$(\mathbf{\alpha_{s}}^\top \alpha_{j} \to 0)$$

и разреживаются

$$(\alpha_{k}^{T} \alpha_{j} \to 1)$$
$$(\mathbf{\alpha_{k}}^\top \alpha_{j} \to 1)$$

0 comments on commit 7a9f788

Please sign in to comment.