selfAttentionLayer

Self-attention layer

Since R2023a

Description

A self-attention layer computes single-head or multihead self-attention of its input.

The layer:

1. Computes the queries, keys, and values from the input

2. Computes the scaled dot-product attention across heads using the queries, keys, and values

3. Merges the results from the heads

4. Performs a linear transformation on the merged result

Creation

Syntax

layer = selfAttentionLayer(numHeads,numQueryChannels)

layer = selfAttentionLayer(numHeads,numQueryChannels,Name=Value)

Description

layer = selfAttentionLayer(numHeads,numQueryChannels) creates a self-attention layer with the specified number of heads and query channels.

layer = selfAttentionLayer(numHeads,numQueryChannels,Name=Value) sets writable properties using one or more name-value arguments. For example, selfAttentionLayer(3,12,DropoutProbability=0.1) creates a self attention layer with 3 heads, 12 query channels, and sets the DropoutProbability property to 0.1.

Input Arguments

numHeads — Number of attention heads

Read-only: positive integer

This property is read-only.

Number of attention heads, specified as a positive integer.

Each head performs a separate linear transformation of the input and computes attention weights independently. The layer uses these attention weights to compute a weighted sum of the input representations, generating a context vector. Increasing the number of heads lets the model capture different types of dependencies and attend to different parts of the input simultaneously. Reducing the number of heads can lower the computational cost of the layer.

The value of numHeads must divide numQueryChannels evenly.

This arguments sets the NumHeads property.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

numQueryChannels — Number of channels for the queries

positive integer

Number of channels for the queries, specified as a positive integer.

The value of numQueryChannels must be divisible by numHeads.

This arguments sets the NumQueryChannels property (since R2026a).

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

Properties

Self-Attention

NumHeads — Number of attention heads

positive integer

This property is read-only after object creation. To set this property, use the corresponding positional input argument when you create the SelfAttentionLayer object.

Number of attention heads, specified as a positive integer.

Each head performs a separate linear transformation of the input and computes attention weights independently. The layer uses these attention weights to compute a weighted sum of the input representations, generating a context vector. Increasing the number of heads lets the model capture different types of dependencies and attend to different parts of the input simultaneously. Reducing the number of heads can lower the computational cost of the layer.

Data Types: double

NumQueryChannels — Number of channels for queries

positive integer

Since R2026a

This property is read-only after object creation. To set this property, use the corresponding positional input argument when you create the SelfAttentionLayer object.

Number of channels for the queries, specified as a positive integer.

Data Types: double

NumQueryGroups — Number of query groups

positive integer

Since R2026a

This property is read-only after object creation. To set this property, use the corresponding name-value argument when you create the SelfAttentionLayer object.

Number of query groups (equivalent to the number of key-value heads), specified as a positive integer. The value of NumQueryGroups must divide NumQueryChannels.

The value of NumQueryGroups specifies the type of attention operation:

• For multihead attention, set NumQueryGroups to numHeads.

• For multiquery attention (MQA), set NumQueryGroups to 1.

• For grouped-query attention (GQA), set NumQueryGroups to a positive integer between 1 and numHeads.

When the number of query groups is greater than 1, the operation creates groups of query channels-per-head, and applies the attention operation within each group.

For example, for six heads with three query groups, the operation splits the query channels into the heads (h₁, …, h₆) and then creates the groups of heads g₁=(h₁,h₂), g₂=(h₃,h₄), and g₃=(h₅,h₆). The operation also splits the key and value channels into the heads g₁, g₂, and g₃.

When the number of query groups matches the number of heads, the groups have one head each and is equivalent to multihead attention. When the number of query groups is 1, then all the heads are in the same group and is equivalent to multiquery attention.

The default value is the NumHeads property value. To programmatically set the number of query groups to the number of heads when you create the layer, you can set the NumQueryGroups name-value argument to "num-heads".

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | char | string

NumKeyChannels — Number of channels for keys

Read-only: positive integer

This property is read-only.

Number of channels for the keys, specified as a positive integer.

The layer uses NumQueryChannels*NumQueryGroups/NumHeads as the number of channels for the keys (since R2026a).

Before R2026a: The layer uses numQueryChannels as the number of the channels for the keys. This is equivalent when the number of query groups matches the number of heads.

Data Types: double

NumValueChannels — Number of channels for values

"auto" (default) | positive integer

This property is read-only after object creation. To set this property, use the corresponding name-value argument when you create the SelfAttentionLayer object.

Number of channels for the values, specified as one of these values:

• "auto" — Use NumKeyChannels.

• Positive integer — Use the specified number of channels. This value must be divisible by NumHeads.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | char | string

OutputSize — Number of channels of layer output

"auto" (default) | positive integer

This property is read-only after object creation. To set this property, use the corresponding name-value argument when you create the SelfAttentionLayer object.

Number of channels of the layer output, specified as one of these values:

• "auto" — Use the number of channels in the layer input.

• Positive integer — Use the specified number of channels.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | char | string

HasPaddingMaskInput — Flag indicating whether layer has mask input

0 (false) (default) | 1 (true)

This property is read-only after object creation. To set this property, use the corresponding name-value argument when you create the SelfAttentionLayer object.

Flag indicating whether the layer has an input that represents the padding mask, specified as 0 (false) or 1 (true).

If the HasPaddingMaskInput property is 0 (false), then the layer has one input with the name "in", which corresponds to the input data. In this case, the layer treats all elements as data.

If the HasPaddingMaskInput property is 1 (true), then the layer has two inputs with the names "in" and "mask", which correspond to the input data and the mask, respectively. In this case, the padding mask is an array of ones and zeros. The layer uses and ignores elements of the input when the corresponding element in the mask is one or zero, respectively.

The dimension labels of the padding mask must match the dimension labels of the input data, ignoring any "C" (channel) and "U" (unspecified) dimensions. (since R2026a).

Before R2026a: The format of the padding mask must match that of the input data.

The size of the "S" (spatial) or "T" (time) dimension of the padding mask must match the sum of the size of the corresponding dimension in the input and the size of the second dimension of the KeyState and ValueState properties. The size of the "B" (batch) dimension of the padding mask must match the size of the corresponding dimension in the input.

The padding mask can have any number of channels. The software uses the values in the first channel only to indicate padding values.

AttentionMask — Mask preventing attention to elements in key-value pairs

"none" (default) | "causal"

This property is read-only after object creation. To set this property, use the corresponding name-value argument when you create the SelfAttentionLayer object.

Mask preventing attention to elements in key-value pairs, specified as one of these values:

• "none" — Do not prevent attention to elements based on their positions. If HasPaddingMaskInput is 1 (true), then the layer prevents attention to padding elements only.

• "causal" — Prevent elements in position M from attending to elements in position N, where N is greater than M. Use this option for autoregressive models.

DropoutProbability — Dropout probability for attention scores

0 (default) | scalar in the range [0, 1)

Probability of dropping out attention scores, specified as a scalar in the range [0, 1).

During training, the software randomly sets values in the attention scores to zero using the specified probability. These dropouts can encourage the model to learn more robust and generalizable representations by preventing it from relying too heavily on specific dependencies.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

HasScoresOutput — Flag indicating whether layer has scores output

0 (false) (default) | 1 (true)

This property is read-only after object creation. To set this property, use the corresponding name-value argument when you create the SelfAttentionLayer object.

Flag indicating whether the layer has an output that represents the scores (also known as the attention weights), specified as 0 (false) or 1 (true).

If the HasScoresOutput property is 0 (false), then the layer has one output with the name "out", which corresponds to the output data.

If the HasScoresOutput property is 1 (true), then the layer has two inputs with the names "out" and "scores", which correspond to the output data and the attention scores, respectively.

InputSize — Number of input channels

Read-only: "auto" (default) | positive integer

This property is read-only.

Number of input channels, specified as one of these values:

• "auto" — Automatically determine the number of input channels when you initialize the network

• Positive integer — Configure the layer for the specified number of input channels. InputSize and the number of channels in the layer input data must match.

Data Types: double | char

Parameters and Initialization

WeightsInitializer — Function to initialize weights

"glorot" (default) | "he" | "narrow-normal" | "zeros" | "ones" | function handle

Function to initialize the query, key, value, and output weights, specified as one of these values:

• "glorot" — Initialize the weights with the Glorot initializer (also known as Xavier initializer) [2]. The Glorot initializer independently samples from a uniform distribution with zero mean and a variance of 2/(numIn + numOut). The values of numIn and numOut depend on the weight matrix:

  Weight	numIn	numOut
  Query	InputSize	NumQueryChannels
  Key	InputSize	NumKeyChannels
  Value	InputSize	NumValueChannels
  Output	NumValueChannels	OutputSize

• "he" — Initialize the weights with the He initializer [3]. The He initializer samples from a normal distribution with zero mean and a variance of 2/numIn. The values of numIn and numOut depend on the weight matrix:

  Weight	numIn	numOut
  Query	InputSize	NumQueryChannels
  Key	InputSize	NumKeyChannels
  Value	InputSize	NumValueChannels
  Output	NumValueChannels	OutputSize

• "narrow-normal" — Initialize the weights by independently sampling from a normal distribution with zero mean and a standard deviation of 0.01.

• "zeros" — Initialize the weights with zeros.

• "ones" — Initialize the weights with ones.

• Function handle — Initialize the weights with a custom function. If you specify a function handle, then the function syntax must be of the form weights = func(sz), where sz is the size of the weights. For an example, see Specify Custom Weight Initialization Function.

The layer only initializes the weights when the corresponding weights property is empty.

The SelfAttentionLayer object stores this property as a character vector or a function handle.

Data Types: char | string | function_handle

BiasInitializer — Function to initialize biases

"zeros" (default) | "narrow-normal" | "ones" | function handle

Function to initialize the query, key, value, and output biases, specified as one of these values:

• "zeros" — Initialize the biases with zeros.

• "ones" — Initialize the biases with ones.

• "narrow-normal" — Initialize the biases by independently sampling from a normal distribution with zero mean and a standard deviation of 0.01.

• Function handle — Initialize the biases with a custom function. If you specify a function handle, then the function must have the form bias = func(sz), where sz is the size of the biases.

The layer only initializes the biases when the corresponding bias property is empty.

The SelfAttentionLayer object stores this property as a character vector or a function handle.

Data Types: char | string | function_handle

QueryWeights — Query weights

[] (default) | matrix

Query weights, specified as a NumQueryChannels-by-numInputChannels matrix or [], where numInputChannels is the number of channels in the layer input.

Data Types: single | double

KeyWeights — Key weights

[] (default) | matrix

Key weights, specified as a NumKeyChannels-by-numInputChannels matrix or [], where numInputChannels is the number of channels in the layer input.

Data Types: single | double

ValueWeights — Value weights

[] (default) | matrix

Value weights, specified as a NumValueChannels-by-numInputChannels matrix or [], where numInputChannels is the number of channels in the layer input.

Data Types: single | double

OutputWeights — Output weights

[] (default) | matrix

Output weights, specified as an OutputSize-by-NumValueChannels matrix or [].

Data Types: single | double

QueryBias — Query biases

[] (default) | vector

Query biases, specified as a NumQueryChannels-by-1 vector or [].

Data Types: single | double

KeyBias — Key biases

[] (default) | vector

Key biases, specified as a NumKeyChannels-by-1 vector or [].

Data Types: single | double

ValueBias — Value biases

[] (default) | vector

Value biases, specified as a NumValueChannels-by-1 vector or [].

Data Types: single | double

OutputBias — Output biases

[] (default) | vector

Output biases, specified as an OutputSize-by-1 vector or [].

Data Types: single | double

State

KeyState — Key state

[] (default) | numeric array

Since R2026a

Key state, specified as a NumKeyChannels-by-x-by-b numeric array or [], where b is the size of the "B" (batch) dimension of the input and x can be any value.

When you update the state of a network containing a SelfAttentionLayer object with the AttentionMask property set to "causal", the KeyState property contains keys used in previous forward passes of the network. If AttentionMask is "none", then the KeyState property must be empty.

After you set this property manually, calls to the resetState function set the key state to this value.

Data Types: single | double

ValueState — Value state

[] (default) | numeric array

Since R2026a

Value state, specified as a NumValueChannels-by-x-by-b numeric array or [], where b is the size of the "B" (batch) dimension of the input and x can be any value.

When you update the state of a network containing a SelfAttentionLayer with the AttentionMask property set to "causal", this property contains values used in previous forward passes of the network. If AttentionMask is "none", then the ValueState property must be empty.

After you set this property manually, calls to the resetState function set the value state to this value.

Data Types: single | double

Learning Rate and Regularization

WeightLearnRateFactor — Learning rate factor for weights

1 (default) | nonnegative scalar

Learning rate factor for the query, key, value, and output weights, specified as a nonnegative scalar.

The software multiplies this factor by the global learning rate to determine the learning rate for the weights in this layer. For example, if WeightLearnRateFactor is 2, then the learning rate for the weights in this layer is twice the current global learning rate. The software determines the global learning rate based on the settings you specify using the trainingOptions function.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

BiasLearnRateFactor — Learning rate factor for biases

1 (default) | nonnegative scalar

Learning rate factor for the query, key, value, and output biases, specified as a nonnegative scalar.

The software multiplies this factor by the global learning rate to determine the learning rate for the biases in this layer. For example, if BiasLearnRateFactor is 2, then the learning rate for the biases in the layer is twice the current global learning rate. The software determines the global learning rate based on the settings you specify using the trainingOptions function.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

WeightL2Factor — L₂ regularization factor for weights

1 (default) | nonnegative scalar

L₂ regularization factor for the query, key, value, and output weights, specified as a nonnegative scalar.

The software multiplies this factor by the global L₂ regularization factor to determine the L₂ regularization for the weights in this layer. For example, if WeightL2Factor is 2, then the L₂ regularization for the weights in this layer is twice the global L₂ regularization factor. You can specify the global L₂ regularization factor using the trainingOptions function.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

BiasL2Factor — L₂ regularization factor for biases

0 (default) | nonnegative scalar

L₂ regularization factor for the query, key, value, and output biases, specified as a nonnegative scalar.

The software multiplies this factor by the global L₂ regularization factor to determine the L₂ regularization for the biases in this layer. For example, if BiasL2Factor is 2, then the L₂ regularization for the biases in this layer is twice the global L₂ regularization factor. The software determines the global L₂ regularization factor based on the settings you specify using the trainingOptions function.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

Layer

Name — Layer name

'' (default) | character vector

Layer name, specified as a character vector. For Layer array input, the trainnet and dlnetwork functions automatically assign names to unnamed layers.

Data Types: char

NumInputs — Number of inputs

Read-only: 1 | 2

This property is read-only.

Number of inputs to the layer, returned as 1 or 2.

If the HasPaddingMaskInput property is 0 (false), then the layer has one input with the name "in", which corresponds to the input data. In this case, the layer treats all elements as data.

If the HasPaddingMaskInput property is 1 (true), then the layer has two inputs with the names "in" and "mask", which correspond to the input data and the mask, respectively. In this case, the padding mask is an array of ones and zeros. The layer uses and ignores elements of the input when the corresponding element in the mask is one or zero, respectively.

Data Types: double

InputNames — Input names

Read-only: "in" | ["in" "mask"]

This property is read-only.

Input names of the layer, returned as a cell array of character vectors.

If the HasPaddingMaskInput property is 0 (false), then the layer has one input with the name "in", which corresponds to the input data. In this case, the layer treats all elements as data.

If the HasPaddingMaskInput property is 1 (true), then the layer has two inputs with the names "in" and "mask", which correspond to the input data and the mask, respectively. In this case, the padding mask is an array of ones and zeros. The layer uses and ignores elements of the input when the corresponding element in the mask is one or zero, respectively.

The SelfAttentionLayer object stores this property as a cell array of character vectors.

NumOutputs — Number of outputs

Read-only: 1 (default) | 2

This property is read-only.

Number of outputs of the layer.

If the HasScoresOutput property is 0 (false), then the layer has one output with the name "out", which corresponds to the output data.

If the HasScoresOutput property is 1 (true), then the layer has two inputs with the names "out" and "scores", which correspond to the output data and the attention scores, respectively.

Data Types: double

OutputNames — Output names

Read-only: "out" (default) | ["out" "scores"]

This property is read-only.

Output names of the layer.

If the HasScoresOutput property is 0 (false), then the layer has one output with the name "out", which corresponds to the output data.

If the HasScoresOutput property is 1 (true), then the layer has two inputs with the names "out" and "scores", which correspond to the output data and the attention scores, respectively.

The SelfAttentionLayer object stores this property as a cell array of character vectors.

Examples

Create Self-Attention Layer

Create a self-attention layer with eight heads and 256 key and query channels.

layer = selfAttentionLayer(8,256)

layer = 
  SelfAttentionLayer with properties:

                   Name: ''
          AttentionMask: 'none'
    HasPaddingMaskInput: 0
        HasScoresOutput: 0

   Hyperparameters
              InputSize: 'auto'
               NumHeads: 8
         NumQueryGroups: 8
       NumQueryChannels: 256
         NumKeyChannels: 256
       NumValueChannels: 'auto'
             OutputSize: 'auto'
     DropoutProbability: 0

   Learnable Parameters
           QueryWeights: []
             KeyWeights: []
           ValueWeights: []
          OutputWeights: []
              QueryBias: []
                KeyBias: []
              ValueBias: []
             OutputBias: []

   State Parameters
               KeyState: []
             ValueState: []

  Show all properties

Include a self-attention layer in a layer array.

layers = [
    sequenceInputLayer(12)
    selfAttentionLayer(4,12)
    layerNormalizationLayer
    fullyConnectedLayer(9)
    softmaxLayer];

Algorithms

Dot-Product Attention

The attention operation focuses on parts of the input using weighted multiplication operations.

The single-head dot-product attention operation is given by

$$\text{attention}(Q,K,V)=\text{dropout}\left(\text{softmax}\left(\text{mask}\left(\lambda Q{K}^{\top },M\right)\right),p\right)V,$$

where:

• Q denotes the queries.

• K denotes the keys.

• V denotes the values.

• $$\lambda$$ denotes the scaling factor.

• M is a mask array of ones and zeros.

• p is the dropout probability.

The mask operation includes or excludes the values of the matrix multiplication by setting values of the input to $$-\infty$$ for zero-valued mask elements. The mask is the union of the padding and attention masks. The softmax function normalizes the value of the input data across the channel dimension such that it sums to one. The dropout operation sets elements to zero with probability p.

Multihead Attention

The multihead attention operation applies the attention operation across multiple heads. Each head uses its own learnable query, key, and value projection matrices.

The multihead attention operation for the queries Q, keys K, and values V is given by

$$\text{multiheadAttention}(Q,K,V)=\text{concatenate}({\text{head}}_1,\dots ,{\text{head}}_h)W^O,$$

where

• h is the number of heads.

• W^O is a learnable projection matrix for the output.

For the multihead attention operation, each learnable projection matrix for the queries, keys, and values are composed of concatenated matrices W_i, where i indexes over the heads.

The head operation is given by

$${\text{head}}_i=\text{attention}\left(Q{W}_i^Q,K{W}_i^K,V{W}_i^V\right),$$

where:

• i indexes over the heads.

• W^Q is a learnable projection matrix for the queries.

• W^K is a learnable projection matrix for the keys.

• W^V is a learnable projection matrix for the values.

Multiquery Attention

The multiquery attention (MQA) operation applies the attention operation across multiple heads. Each head uses its own learnable query projection matrix. The operation uses the same learnable key and value projection matrices across all heads.

The multiquery attention operation for the queries Q, keys K, and values V is given by

$$\text{multiqueryAttention}(Q,K,V)=\text{concatenate}({\text{head}}_1,\dots ,{\text{head}}_h)W^O,$$

where

• h is the number of heads.

• W^O is a learnable projection matrix for the output.

For the multiquery attention operation, only the learnable projection matrix for the queries is composed of concatenated matrices W_i, where i indexes over the heads.

For multiquery attention, the head operation is given by

$${\text{head}}_i=\text{attention}\left(Q{W}_i^Q,K{W}^K,V{W}^V\right),$$

where:

• i indexes over the heads.

• W^Q is a learnable projection matrix for the queries.

• W^K is a learnable projection matrix for the keys.

• W^V is a learnable projection matrix for the values.

Grouped-Query Attention

The grouped query attention (GQA) operation applies the attention operation across several heads. The operation partitions the heads into groups that use the same learnable query projection matrix. The operation uses the same learnable key and value projection matrices for each group of query heads.

The grouped-query attention operation for the queries Q, keys K, and values V is given by

$$\text{groupedQueryAttention}(Q,K,V)=\text{concatenate}({\text{head}}_1,\dots ,{\text{head}}_h)W^O,$$

where

• h is the number of heads.

• W^O is a learnable projection matrix for the output.

For the grouped query attention operation:

• The learnable projection matrix for the queries is composed of concatenated matrices W_i, where i indexes over the heads.

• The learnable projection matrices for the keys and values are composed of concatenated matrices W_j, where j indexes over the groups.

When the number of query groups is greater than 1, the operation creates groups of query channels-per-head, and applies the attention operation within each group.

For example, for six heads with three query groups, the operation splits the query channels into the heads (h₁, …, h₆) and then creates the groups of heads g₁=(h₁,h₂), g₂=(h₃,h₄), and g₃=(h₅,h₆). The operation also splits the key and value channels into the heads g₁, g₂, and g₃.

When the number of query groups matches the number of heads, the groups have one head each and is equivalent to multihead attention. When the number of query groups is 1, then all the heads are in the same group and is equivalent to multiquery attention.

The head operation is given by

$${\text{head}}_i=\text{attention}\left(Q{W}_i^Q,K{W}_{g(i)}^K,V{W}_{g(i)}^V\right),$$

where:

• i indexes over the heads.

• g(i) is the group number of head i.

• W^Q is a learnable projection matrix for the queries.

• W^K is a learnable projection matrix for the keys.

• W^V is a learnable projection matrix for the values.

Self-Attention

The self-attention operation is equivalent to setting the queries, keys, and values to the input data:

• selfAttention(X) = attention(X,X,X)

• multiheadSelfAttention(X) = multiheadAttention(X,X,X)

• multiquerySelfAttention(X) = multiqueryAttention(X,X,X)

• groupedQuerySelfAttention(X) = groupedQueryAttention(X,X,X)

In each of these cases, the operation still uses separate learnable parameters for the keys, queries, and values. For example, to calculate multiheadSelfAttention(X), the operation uses the head operation given by

$${\text{head}}_i=\text{attention}\left(X{W}_i^Q,X{W}_i^K,X{W}_i^V\right),$$

where:

• i indexes over the heads.

• X is the input data.

• W^Q is a learnable projection matrix for the queries.

• W^K is a learnable projection matrix for the keys.

• W^V is a learnable projection matrix for the values.

Layer Input and Output Formats

Most layers in a layer array or layer graph pass data to subsequent layers as formatted dlarray objects. The format of a dlarray object is a string of characters in which each character describes the corresponding dimension of the data. The format consists of one or more of these characters:

• "S" — Spatial

• "C" — Channel

• "B" — Batch

• "T" — Time

• "U" — Unspecified

For example, you can describe 2-D image data that is represented as a 4-D array, where the first two dimensions correspond to the spatial dimensions of the images, the third dimension corresponds to the channels of the images, and the fourth dimension corresponds to the batch dimension, as having the format "SSCB" (spatial, spatial, channel, batch).

You can interact with these dlarray objects in automatic differentiation workflows, such as those for:

• developing a custom layer

• using a functionLayer object

• using the forward and predict functions with dlnetwork objects

This table shows the supported input formats of SelfAttentionLayer objects and the corresponding output format. If the software passes the output of the layer to a custom layer that does not inherit from the nnet.layer.Formattable class, or to a FunctionLayer object with the Formattable property set to 0 (false), then the layer receives an unformatted dlarray object with dimensions ordered according to the formats in this table. The formats listed here are only a subset of the formats that the layer supports. The layer might support additional formats, such as formats with additional "S" (spatial) or "U" (unspecified) dimensions.

Input Format	Output Format
"CB" (channel, batch)	"CB" (channel, batch)
"SCB" (spatial, channel, batch)	"SCB" (spatial, channel, batch)
"CBT" (channel, batch, time)	"CBT" (channel, batch, time)
"SC" (spatial, channel)	"SC" (spatial, channel)
"CT" (channel, time)	"CT" (channel, time)
"SB" (spatial, batch)	"SCB" (spatial, channel, batch)
"BT" (batch, time)	"CBT" (channel, batch, time)

References

[1] Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. "Attention is all you need." In Advances in Neural Information Processing Systems, Vol. 30. Curran Associates, Inc., 2017. https://papers.nips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.

[2] Glorot, Xavier, and Yoshua Bengio. "Understanding the Difficulty of Training Deep Feedforward Neural Networks." In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, 249–356. Sardinia, Italy: AISTATS, 2010. https://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf

[3] He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. "Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification." In 2015 IEEE International Conference on Computer Vision (ICCV), 1026–34. Santiago, Chile: IEEE, 2015. https://doi.org/10.1109/ICCV.2015.123

Extended Capabilities

C/C++ Code Generation

Generate C and C++ code using MATLAB® Coder™.

Usage notes and limitations:

• Code generation is not supported when HasScoresOutput is set to true.

• Code generation does not support passing dlarray objects with "U" (unspecified) dimensions to this layer.

• For code generation, you must pass a dlarray object with a channel (C) dimension as the input to this layer. For example, code generation supports data format such as "SSC" or "SSCBT".

GPU Code Generation

Generate CUDA® code for NVIDIA® GPUs using GPU Coder™.

Refer to the usage notes and limitations in the C/C++ Code Generation section. The same usage notes and limitations apply to GPU code generation.

Version History

Introduced in R2023a

R2026a: Apply multiquery and grouped-query attention

The value of NumQueryGroups specifies the type of attention operation:

• For multihead attention, set NumQueryGroups to numHeads.

• For multiquery attention (MQA), set NumQueryGroups to 1.

• For grouped-query attention (GQA), set NumQueryGroups to a positive integer between 1 and numHeads.

Using multiquery attention and grouped-query attention in conjunction with the KeyState and ValueState properties can reduce memory and computation time for large inputs.

R2026a: Cache previous keys and values

You can now cache previous keys and values using the new KeyState and ValueState properties to accelerate making autoregressive predictions with transformer networks. Caching previous keys and values allows you to pass only one previous prediction as input, instead of an entire sequence. Previously, forecasting a future value required using all previous time steps as input.

To cache the keys and values and use them to make autoregressive predictions, set the state of a transformer network after each prediction. For example, this code uses a transformer network to forecast future values of a sequence using one previous time step as input.

% Reset the network state.
net = resetState(net);

% Get the first time step in the sequence.
sequence = initialValue;

% Predict values for the next 100 time steps.
for idx = 1:100
    % Make predicitions by passing the previous value as input.
    previousStep = sequence(end);
    [sequence(end+1),state] = predict(net,previousStep);

    % Update the network state.
    net.State = state;
end

R2026a: Padding mask does not require channel dimension

If the HasPaddingMaskInput property value is 1 (true), then the input data for the corresponding padding mask input does not require a channel dimension. For masks that do not specify a channel dimension, the operation assumes a singleton channel dimension.