Skip to content

Commit

Permalink
improve docs
Browse files Browse the repository at this point in the history
  • Loading branch information
CarloLucibello committed Jan 5, 2023
1 parent 16b9fe1 commit 820d45e
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion src/attention.jl
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@ See also [`dot_product_attention_scores`](@ref) if you only need the attention s
- `value`: Value array of size `(v_dim, kv_len, batch_size...)`.
- `bias`: Either `nothing` or an array broadcastable to size `(kv_len, q_len, nheads, batch_size)`.
It will be added to the attention scores before applying the softmax. Default `nothing`.
- `fdrop`: A dropout function or layer to apply on the attention scores. Default `identity` (no dropout).
- `fdrop`: A dropout function or layer to be applied on the attention scores right after the softmax.
Default `identity` (no dropout).
- `mask`: Either `nothing` or a boolean array broadcastable to size `(kv_len, q_len, nheads, batch_size)`.
The mask is applied to the attention scores before the softmax.
Can also be set to `mask=:causal` to apply a causal mask. Default `nothing`.
Expand Down

0 comments on commit 820d45e

Please sign in to comment.