sinusoidalPositionEncodingLayer

Sinusoidal position encoding layer

Since R2023b

Description

A sinusoidal position encoding layer maps position indices to vectors using sinusoidal operations. Use this layer in transformer neural networks to provide information about the position of the data in a sequence or image.

Creation

Syntax

layer = sinusoidalPositionEncodingLayer(outputSize)

layer = sinusoidalPositionEncodingLayer(outputSize,Name=Value)

Description

layer = sinusoidalPositionEncodingLayer(outputSize) creates a sinusoidal position encoding layer and sets the OutputSize property.

layer = sinusoidalPositionEncodingLayer(outputSize,Name=Value) creates a sinusoidal position encoding layer and sets the Positions and Name properties using one or more name-value arguments.

Properties

Sinusoidal Position Encoding

OutputSize — Number of channels in layer output

even positive integer

This property is read-only.

Number of channels in the layer output, specified as an even positive integer.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

Positions — Positions in input

"auto" (default) | "temporal-indicies" | "spatial-indicies" | "data-values"

This property is read-only.

Positions in the input, specified as one of these values:

• "auto" — For sequence or spatial-temporal input, use the temporal indices as positions, which is equivalent to using "temporal-indices". For one-dimensional image input, use the spatial indices as positions, which is equivalent to using "spatial-indices". For other input, use the input values as positions, which is equivalent to using "data-values".

• "temporal-indices" — Use the temporal indices of the input as positions.

• "spatial-indices" — Use the spatial indices of the input as positions.

• "data-values" — Use the values in the input as positions.

Layer

Name — Layer name

"" (default) | character vector | string scalar

Layer name, specified as a character vector or a string scalar. For Layer array input, the trainnet and dlnetwork functions automatically assign names to layers with the name "".

The SinusoidalPositionEncodingLayer object stores this property as a character vector.

Data Types: char | string

NumInputs — Number of inputs

1 (default)

This property is read-only.

Number of inputs to the layer, returned as 1. This layer accepts a single input only.

Data Types: double

InputNames — Input names

{'in'} (default)

This property is read-only.

Input names, returned as {'in'}. This layer accepts a single input only.

Data Types: cell

NumOutputs — Number of outputs

1 (default)

This property is read-only.

Number of outputs from the layer, returned as 1. This layer has a single output only.

Data Types: double

OutputNames — Output names

{'out'} (default)

This property is read-only.

Output names, returned as {'out'}. This layer has a single output only.

Data Types: cell

Examples

Create Sinusoidal Position Encoding Layer

Create a sinusoidal position encoding layer with an output size of 300.

layer = sinusoidalPositionEncodingLayer(300)

layer = 
  SinusoidalPositionEncodingLayer with properties:

          Name: ''
    OutputSize: 300
     Positions: 'auto'

   Learnable Parameters
    No properties.

   State Parameters
    No properties.

Use properties method to see a list of all properties.

Create a neural network containing a sinusoidal position encoding layer.

net = dlnetwork;

numChannels = 1;

embeddingOutputSize = 64;
numWords = 128;
maxPosition = 128;

numHeads = 4;
numKeyChannels = 4*embeddingOutputSize;

layers = [ 
    sequenceInputLayer(numChannels,Name="input")
    wordEmbeddingLayer(embeddingOutputSize,numWords,Name="word-emb")
    sinusoidalPositionEncodingLayer(embeddingOutputSize,Name="pos-enc");
    additionLayer(2,Name="add")
    selfAttentionLayer(numHeads,numKeyChannels,AttentionMask="causal")
    fullyConnectedLayer(numWords)
    softmaxLayer];

net = addLayers(net,layers);

net = connectLayers(net,"word-emb","add/in2");

View the neural network architecture.

plot(net)
axis off
box off

Algorithms

Sinusoidal Position Encoding Layer

A sinusoidal position encoding layer maps position indices to vectors using sinusoidal operations. The layer encodes position information of data for transformer neural networks.

The output of the layer has the same number of dimensions as the input. In the output, each vector in position p over the channel dimension is given by:

$$Y_p=\left[\begin{array}{c}\sin \left({\omega }_1·p\right)\\ \cos \left({\omega }_1·p\right)\\ \sin \left({\omega }_2·p\right)\\ \cos \left({\omega }_2·p\right)\\ ⋮\\ \sin \left({\omega }_{d/2}·p\right)\\ \cos \left({\omega }_{d/2}·p\right)\end{array}\right],$$

where p is the position, d is the encoding output size given by OutputSize and $${\omega }_k$$ is the wavelength given by:

$${\omega }_k=\frac{1}{{10000}^{2k/d}},$$

for $$k=1,\dots ,d/2$$.

When Positions is "auto", the layout of the output depends on the type of data:

• For sequence data X represented by a numChannels-by-numObservations-by-numTimeSteps array, where numChannels, numObservations, and numTimeSteps are the numbers of channels, observations, and time steps of the input, respectively, the output is an OutputSize-by-numObservations-by-numTimeSteps array Y, where each vector in Y(:,:,t) over the channel dimension is $$Y_t$$.

• For 1-D image data X represented by a height-by-numChannels-by-numObservations array, where height, numChannels, and numObservations are the height, number of channels, and number of observations of the input images, respectively, the output is a height-by-OutputSize-by-numObservations array Y, where each vector in Y(i,:,:) over the channel dimension is $$Y_i$$.

• For 2-D image sequence data X represented by a height-by-width-by-numChannels-by-numObservations-by-numTimeSteps array, where height and width are the height and width of the input image sequences, respectively, and numChannels, numObservations, and numTimeSteps are the numbers of channels, observations, and time steps of the input image sequences, respectively, the output is a height-by-width-by-OutputSize-by-numObservations-by-numTimeSteps array Y, where each vector in Y(:,:,:,:,t) over the channel dimension is $$Y_t$$.

Layer Input and Output Formats

Layers in a layer array or layer graph pass data to subsequent layers as formatted dlarray objects. The format of a dlarray object is a string of characters, in which each character describes the corresponding dimension of the data. The formats consist of one or more of these characters:

• "S" — Spatial

• "C" — Channel

• "B" — Batch

• "T" — Time

• "U" — Unspecified

For example, 2-D image data that is represented as a 4-D array, where the first two dimensions correspond to the spatial dimensions of the images, the third dimension corresponds to the channels of the images, and the fourth dimension corresponds to the batch dimension, can be described as having the format "SSCB" (spatial, spatial, channel, batch).

You can interact with these dlarray objects in automatic differentiation workflows, such as those for developing a custom layer, using a functionLayer object, or using the forward and predict functions with dlnetwork objects.

This table shows the supported input formats of SinusoidalPositionEncodingLayer objects and the corresponding output format. If the software passes the output of the layer to a custom layer that does not inherit from the nnet.layer.Formattable class, or a FunctionLayer object with the Formattable property set to 0 (false), then the layer receives an unformatted dlarray object with dimensions ordered according to the formats in this table. The formats listed here are only a subset. The layer may support additional formats such as formats with additional "S" (spatial) or "U" (unspecified) dimensions.

Input Format	Positions	Output Format
"CB" (channel, batch)	"auto" "data-values"	"CB" (channel, batch)
"SCB" (spatial, channel, batch)	"auto" "spatial-indices" "data-values"	"SCB" (spatial, channel, batch)
"SSCB" (spatial, spatial, channel, batch)	"data-values"	"SSCB" (spatial, spatial, channel, batch)
"SSSCB" (spatial, spatial, spatial, channel, batch)	"data-values"	"SSSCB" (spatial, spatial, spatial, channel, batch)
"CBT" (channel, batch, time)	"auto" "temporal-indices" "data-values"	"CBT" (channel, batch, time)
"SCBT" (spatial, channel, batch, time)	"auto" "temporal-indices" "spatial-indices" "data-values"	"SCBT" (spatial, channel, batch, time)
"SSCBT" (spatial, spatial, channel, batch, time)	"auto" "temporal-indices" "data-values"	"SSCBT" (spatial, spatial, channel, batch, time)
"SSSCBT" (spatial, spatial, spatial, channel, batch, time)	"auto" "temporal-indices" "data-values"	"SSSCBT" (spatial, spatial, spatial, channel, batch, time)
"SC" (spatial, channel)	"auto" "spatial-indices" "data-values"	"SC" (spatial, channel)
"SSC" (spatial, spatial, channel)	"data-values"	"SSC" (spatial, spatial, channel)
"SSSC" (spatial, spatial, spatial, channel)	"data-values"	"SSSC" (spatial, spatial, spatial, channel)
"SB" (spatial, batch)	"auto" "spatial-indices" "data-values"	"SCB" (spatial, channel, batch)
"SSB" (spatial, spatial, batch)	"data-values"	"SSCB" (spatial, spatial, channel, batch)
"SSSB" (spatial, spatial, spatial, batch)	"data-values"	"SSSCB" (spatial, spatial, spatial, channel, batch)
"SS" (spatial, spatial)	"data-values"	"SSC" (spatial, spatial, channel)
"SSS" (spatial, spatial, spatial)	"data-values"	"SSSC" (spatial, spatial, spatial, channel)
"SU" (spatial, unspecified)	"auto" "spatial-indices" "data-values"	"SCU" (spatial, channel, unspecified)
"BU" (batch, unspecified)	"auto" "data-values"	"CBU" (channel, batch, unspecified)
"UU" (unspecified, unspecified)	"auto" "data-values"	"CUU" (channel, unspecified, unspecified)
"UUU" (unspecified, unspecified, unspecified)	"auto" "data-values"	"CUUU" (channel, unspecified, unspecified, unspecified)
"UUUU" (unspecified, unspecified, unspecified, unspecified)	"auto" "data-values"	"CUUUU" (channel, unspecified, unspecified, unspecified, unspecified)
"UUUUU" (unspecified, unspecified, unspecified, unspecified, unspecified)	"auto" "data-values"	"CUUUUU" (channel, unspecified, unspecified, unspecified, unspecified, unspecified)

In dlnetwork objects, SinusoidalPositionEncodingLayer objects also support these input and output format combinations.

Input Format	Positions	Output Format
"CT" (channel, time)	"auto" "temporal-indices" "data-values"	"CT" (channel, time)
"SCT" (spatial, channel, time)	"auto" "temporal-indices" "spatial-indices" "data-values"	"SCT" (spatial, channel, time)
"SSCT" (spatial, spatial, channel, time)	"auto" "temporal-indices" "data-values"	"SSCT" (spatial, spatial, channel, time)
"SSSCT" (spatial, spatial, spatial, channel, time)	"auto" "temporal-indices" "data-values"	"SSSCT" (spatial, spatial, spatial, channel, time)
"BT" (batch, time)	"auto" "temporal-indices" "data-values"	"CBT" (channel, batch, time)
"SBT" (spatial, batch, time)	"auto" "temporal-indices" "spatial-indices" "data-values"	"SCBT" (spatial, channel, batch, time)
"SSBT" (spatial, spatial, batch, time)	"auto" "temporal-indices" "data-values"	"SSCBT" (spatial, spatial, channel, batch, time)
"SSSBT" (spatial, spatial, spatial, batch, time)	"auto" "temporal-indices" "data-values"	"SSSCBT" (spatial, spatial, spatial, channel, batch, time)
"ST" (spatial, time)	"auto" "temporal-indices" "spatial-indices" "data-values"	"SCT" (spatial, channel, time)
"SST" (spatial, spatial, time)	"auto" "temporal-indices" "data-values"	"SSCT" (spatial, spatial, channel, time)
"SSST" (spatial, spatial, spatial, time)	"auto" "temporal-indices" "data-values"	"SSSCT" (spatial, spatial, spatial, channel, time)
"TU" (time, unspecified)	"auto" "temporal-indices" "data-values"	"CTU" (channel, time, unspecified)

References

[1] Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. "Attention is all you need." In Advances in Neural Information Processing Systems, Vol. 30. Curran Associates, Inc., 2017. https://papers.nips.cc/paper/7181-attention-is-all-you-need.

Version History

Introduced in R2023b