crossentropy

Cross-entropy loss for classification tasks

Since R2019b

Syntax

loss = crossentropy(Y,targets)

loss = crossentropy(Y,targets,weights)

loss = crossentropy(___,'DataFormat',FMT)

loss = crossentropy(___,Name,Value)

Description

The cross-entropy operation computes the cross-entropy loss between network predictions and target values for single-label and multi-label classification tasks.

The crossentropy function computes the cross-entropy loss between predictions and targets represented as dlarray data. Using dlarray objects makes working with high dimensional data easier by allowing you to label the dimensions. For example, you can label which dimensions correspond to spatial, time, channel, and batch dimensions using the "S", "T", "C", and "B" labels, respectively. For unspecified and other dimensions, use the "U" label. For dlarray object functions that operate over particular dimensions, you can specify the dimension labels by formatting the dlarray object directly, or by using the DataFormat option.

Note

To calculate the cross-entropy loss within a layerGraph object or Layer array for use with the trainNetwork function, use classificationLayer.

loss = crossentropy(Y,targets) returns the categorical cross-entropy loss between the formatted dlarray object Y containing the predictions and the target values targets for single-label classification tasks. The output loss is an unformatted dlarray scalar.

For unformatted input data, use the 'DataFormat' option.

loss = crossentropy(Y,targets,weights) applies weights to the calculated loss values. Use this syntax to weight the contributions of classes, observations, regions, or individual elements of the input to the calculated loss values.

loss = crossentropy(___,'DataFormat',FMT) also specifies the dimension format FMT when Y is not a formatted dlarray.

loss = crossentropy(___,Name,Value) specifies options using one or more name-value pair arguments in addition to the input arguments in previous syntaxes. For example, 'TargetCategories','independent' computes the cross-entropy loss for a multi-label classification task.

Examples

Cross-Entropy Loss for Single-Label Classification

Create an array of prediction scores for 12 observations over 10 classes.

numClasses = 10;
numObservations = 12;

Y = rand(numClasses,numObservations);
dlY = dlarray(Y,'CB');
dlY = softmax(dlY);

View the size and format of the prediction scores.

size(dlY)

ans = 1×2

    10    12

dims(dlY)

ans = 
'CB'

Create an array of targets encoded as one-hot vectors.

labels = randi(numClasses,[1 numObservations]);
targets = onehotencode(labels,1,'ClassNames',1:numClasses);

View the size of the targets.

size(targets)

ans = 1×2

    10    12

Compute the cross-entropy loss between the predictions and the targets.

loss = crossentropy(dlY,targets)

loss = 
  1x1 dlarray

    2.3343

Cross-Entropy Loss for Multi-Label Classification

Create an array of prediction scores for 12 observations over 10 classes.

numClasses = 10;
numObservations = 12;
Y = rand(numClasses,numObservations);
dlY = dlarray(Y,'CB');

View the size and format of the prediction scores.

size(dlY)

ans = 1×2

    10    12

dims(dlY)

ans = 
'CB'

Create a random array of targets encoded as a numeric array of zeros and ones. Each observation can have multiple classes.

targets = rand(numClasses,numObservations) > 0.75;
targets = single(targets);

View the size of the targets.

size(targets)

ans = 1×2

    10    12

Compute the cross-entropy loss between the predictions and the targets. To specify cross-entropy loss for multi-label classification, set the 'TargetCategories' option to 'independent'.

loss = crossentropy(dlY,targets,'TargetCategories','independent')

loss = 
  1x1 single dlarray

    9.8853

Weighted Cross-Entropy Loss

Create an array of prediction scores for 12 observations over 10 classes.

numClasses = 10;
numObservations = 12;

Y = rand(numClasses,numObservations);
dlY = dlarray(Y,'CB');
dlY = softmax(dlY);

View the size and format of the prediction scores.

size(dlY)

ans = 1×2

    10    12

dims(dlY)

ans = 
'CB'

Create an array of targets encoded as one-hot vectors.

labels = randi(numClasses,[1 numObservations]);
targets = onehotencode(labels,1,'ClassNames',1:numClasses);

View the size of the targets.

size(targets)

ans = 1×2

    10    12

Compute the weighted cross-entropy loss between the predictions and the targets using a vector class weights. Specify a weights format of 'UC' (unspecified, channel) using the 'WeightsFormat' option.

weights = rand(1,numClasses);
loss = crossentropy(dlY,targets,weights,'WeightsFormat','UC')

loss = 
  1x1 dlarray

    1.1261

Input Arguments

Y — Predictions

dlarray | numeric array

Predictions, specified as a formatted dlarray, an unformatted dlarray, or a numeric array. When Y is not a formatted dlarray, you must specify the dimension format using the DataFormat option.

If Y is a numeric array, targets must be a dlarray.

targets — Target classification labels

dlarray | numeric array

Target classification labels, specified as a formatted or unformatted dlarray or a numeric array.

Specify the targets as an array containing one-hot encoded labels with the same size and format as Y. For example, if Y is a numObservations-by-numClasses array, then targets(n,i) = 1 if observation n belongs to class i targets(n,i) = 0 otherwise.

If targets is a formatted dlarray, then its format must be the same as the format of Y, or the same as DataFormat if Y is unformatted.

If targets is an unformatted dlarray or a numeric array, then the function applies the format of Y or the value of DataFormat to targets.

Tip

Formatted dlarray objects automatically permute the dimensions of the underlying data to have order "S" (spatial), "C" (channel), "B" (batch), "T" (time), then "U" (unspecified). To ensure that the dimensions of Y and targets are consistent, when Y is a formatted dlarray, also specify targets as a formatted dlarray.

weights — Weights

dlarray | numeric array

Weights, specified as a dlarray or a numeric array.

To specify class weights, specify a vector with a 'C' (channel) dimension with size matching the 'C' (channel) dimension of the Y. Specify the 'C' (channel) dimension of the class weights by using a formatted dlarray object or by using the 'WeightsFormat' option.

To specify observation weights, specify a vector with a 'B' (batch) dimension with size matching the 'B' (batch) dimension of the Y. Specify the 'B' (batch) dimension of the class weights by using a formatted dlarray object or by using the 'WeightsFormat' option.

To specify weights for each element of the input independently, specify the weights as an array of the same size as Y. In this case, if weights is not a formatted dlarray object, then the function uses the same format as Y. Alternatively, specify the weights format using the 'WeightsFormat' option.

Name-Value Arguments

Specify optional pairs of arguments as Name1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Before R2021a, use commas to separate each name and value, and enclose Name in quotes.

Example: 'TargetCategories','independent','DataFormat','CB' evaluates the cross-entropy loss for multi-label classification tasks and specifies the dimension order of the input data as 'CB'

TargetCategories — Type of classification task

'exclusive' (default) | 'independent'

Type of classification task, specified as the comma-separated pair consisting of 'TargetCategories' and one of the following:

• 'exclusive' — Single-label classification. Each observation in the predictions Y is exclusively assigned to one category. The function computes the loss between the target value for the single category specified by targets and the corresponding prediction in Y, averaged over the number of observations.

• 'independent'— Multi-label classification. Each observation in the predictions Y can be assigned to one or more independent categories. The function computes the sum of the loss between each category specified by targets and the predictions in Y for those categories, averaged over the number of observations. Cross-entropy loss for this type of classification task is also known as binary cross-entropy loss.

Mask — Mask indicating which elements to include for loss computation

dlarray | logical array | numeric array

Mask indicating which elements to include for loss computation, specified as a dlarray object, a logical array, or a numeric array with the same size as Y.

The function includes and excludes elements of the input data for loss computation when the corresponding value in the mask is 1 and 0, respectively.

If Mask is a formatted dlarray object, then its format must match that of Y. If Mask is not a formatted dlarray object, then the function uses the same format as Y.

If you specify the DataFormat option, then the function also uses the specified format for the mask.

The size of each dimension of Mask must match the size of the corresponding dimension in Y. The default value is a logical array of ones.

Tip

Formatted dlarray objects automatically permute the dimensions of the underlying data to have this order: "S" (spatial), "C" (channel), "B" (batch), "T" (time), and "U" (unspecified). For example, dlarray objects automatically permute the dimensions of data with format "TSCSBS" to have format "SSSCBT".

To ensure that the dimensions of Y and the mask are consistent, when Y is a formatted dlarray, also specify the mask as a formatted dlarray.

Reduction — Mode for reducing array of loss values

"sum" (default) | "none"

Mode for reducing the array of loss values, specified as one of the following:

• "sum" — Sum all of the elements in the array of loss values. In this case, the output loss is scalar.

• "none" — Do not reduce the array of loss values. In this case, the output loss is an unformatted dlarray object with the same size as Y.

NormalizationFactor — Divisor for normalizing reduced loss

"batch-size" (default) | "all-elements" | "mask-included" | "none"

Divisor for normalizing the reduced loss when Reduction is "sum", specified as one of the following:

• "batch-size" — Normalize the loss by dividing it by the number of observations in Y.

• "all-elements" — Normalize the loss by dividing it by the number of elements of Y.

• "mask-included" — Normalize the loss by dividing the loss values by the number of included elements specified by the mask for each observation independently. To use this option, you must specify a mask using the Mask option.

• "none" — Do not normalize the loss.

DataFormat — Dimension order of unformatted data

character vector | string scalar

Dimension order of unformatted input data, specified as a character vector or string scalar FMT that provides a label for each dimension of the data.

When you specify the format of a dlarray object, each character provides a label for each dimension of the data and must be one of these options:

• "S" — Spatial

• "C" — Channel

• "B" — Batch (for example, samples and observations)

• "T" — Time (for example, time steps of sequences)

• "U" — Unspecified

You can specify multiple dimensions labeled "S" or "U". You can use the labels "C", "B", and "T" at most once.

You must specify DataFormat when the input data is not a formatted dlarray.

Data Types: char | string

WeightsFormat — Dimension order of weights

character vector | string scalar

Dimension order of the weights, specified as a character vector or string scalar that provides a label for each dimension of the weights.

When you specify the format of a dlarray object, each character provides a label for each dimension of the data and must be one of these options:

• "S" — Spatial

• "C" — Channel

• "B" — Batch (for example, samples and observations)

• "T" — Time (for example, time steps of sequences)

• "U" — Unspecified

You can specify multiple dimensions labeled "S" or "U". You can use the labels "C", "B", and "T" at most once.

You must specify WeightsFormat when weights is a numeric vector and Y has two or more nonsingleton dimensions.

If weights is not a vector, or both weights and Y are vectors, then default value of WeightsFormat is the same as the format of Y.

Data Types: char | string

Output Arguments

loss — Cross-entropy loss

dlarray

Cross-entropy loss, returned as an unformatted dlarray. The output loss is an unformatted dlarray with the same underlying data type as the input Y.

The size of loss depends on the 'Reduction' option.

Algorithms

Cross-Entropy Loss

For each element Y_j of the input, the crossentropy function computes the corresponding cross-entropy element-wise loss values using the formula

$${\text{loss}}_j=-\left(T_j\text{ln}{Y}_j+(1-T_j)\text{ln}(1-Y_j)\right),$$

where T_j is the corresponding target value to Y_j.

To reduce the loss values to a scalar, the function then reduces the element-wise loss using the formula

$$\text{loss}=\frac{1}{N}{\displaystyle \sum _j{m}_j{w}_j{\text{loss}}_j,}$$

where N is the normalization factor, m_j is the mask value for element j, and w_j is the weight value for element j.

If you do not opt to reduce the loss, then the function applies the mask and the weights to the loss values directly:

$${\text{loss}}_j^{*}=m_j{w}_j{\text{loss}}_j$$

This table shows the loss formulations for different tasks.

Task	Description	Loss
Single-label classification	Cross-entropy loss for mutually exclusive classes. This is useful when observations must have a single label only.	$$\text{loss}=-\frac{1}{N}{\displaystyle \sum _{n=1}^N{\displaystyle \sum }_{i=1}^K}{T}_{ni}\text{ln}{Y}_{ni},$$ where N and K are the numbers of observations, and classes, respectively.
Multi-label classification	Cross-entropy loss for independent classes. This is useful when observations can have multiple labels.	$$\text{loss}=-\frac{1}{N}{\displaystyle \sum _{n=1}^N{\displaystyle \sum _{i=1}^K\left(T_{ni}\ln (Y_{ni})+(1-T_{ni})\ln (1-Y_{ni})\right)}},$$ where N and K are the numbers of observations and classes, respectively.
Single-label classification with weighted classes	Cross-entropy loss with class weights. This is useful for datasets with imbalanced classes.	$$\text{loss}=-\frac{1}{N}{\displaystyle \sum _{n=1}^N{\displaystyle \sum }_{i=1}^K}{w}_i{T}_{ni}\text{ln}{Y}_{ni},$$ where N and K are the numbers of observations and classes, respectively, and wi denotes the weight for class i.
Sequence-to-sequence classification	Cross-entropy loss with masked time-steps. This is useful for ignoring loss values that correspond to padded data.	$$\text{loss}=-\frac{1}{N}{\displaystyle \sum _{n=1}^N{\displaystyle \sum }_{t=1}^S{m}_{nt}{\displaystyle \sum }_{i=1}^K}{T}_{nti}\text{ln}{Y}_{nti},$$ where N, S, and K are the numbers of observations, time steps, and classes, mnt denotes the mask value for time step t of observation n.

Extended Capabilities

GPU Arrays

Accelerate code by running on a graphics processing unit (GPU) using Parallel Computing Toolbox™.

Usage notes and limitations:

• When at least one of the following input arguments is a gpuArray or a dlarray with underlying data of type gpuArray, this function runs on the GPU:

  • Y

  • targets

  • weights

  • 'Mask'

For more information, see Run MATLAB Functions on a GPU (Parallel Computing Toolbox).

Version History

Introduced in R2019b