Is a Bi-GRU available - bidirectional Gated Recurrent Unit (GRU) - or a way to implement a Bi-GRU?

28 views (last 30 days)
The following artificial recurrent neural network (RNN) architectures are available:
Whereas, I would like to know if an Bi-GRU exists or can be defined?
Thank you for your help.

Accepted Answer

Amanjit Dulai
Amanjit Dulai on 21 Oct 2021
A bi-LSTM layer works by applying two LSTM layers on the data; one in the forward direction and one in the reverse direction. You can apply an LSTM function in the reverse direction by flipping the data. The results from these two LSTM layers is then concatenated together to form the output of the bi-LSTM layer. So if we want to implement a bi-GRU layer, we can do this by using a custom flip layer together with GRU layers. A custom flip layer can be implemented as follows:
classdef FlipLayer < nnet.layer.Layer
methods
function layer = FlipLayer(name)
layer.Name = name;
end
function Y = predict(~, X)
Y = flip(X,3);
end
end
end
We can implement a bi-GRU layer with OutputMode="sequence" by arranging layers in the way shown below:
We can implement a bi-GRU layer with with OutputMode="last" by arranging layers in the way shown below:
Below is a short example showing how to use the custom flip layer mentioned above to implement a network with two bi-GRU layers (one with OutputMode="sequence" and oine with OutputMode="last"):
[XTrain, YTrain] = japaneseVowelsTrainData;
lg = layerGraph();
lg = addLayers(lg, [
sequenceInputLayer(12, "Name", "input")
gruLayer( 100, 'OutputMode', 'sequence', ...
"Name", "gru1")
concatenationLayer(1, 2, "Name", "cat1")
gruLayer( 100, 'OutputMode', 'last', ...
"Name", "gru3")
concatenationLayer(1, 2, "Name", "cat2")
fullyConnectedLayer(9)
softmaxLayer
classificationLayer()] );
lg = addLayers( lg, [
FlipLayer("flip1")
gruLayer( 100, 'OutputMode', 'sequence', ...
"Name", "gru2" )
FlipLayer("flip2")] );
lg = addLayers(lg, [
FlipLayer("flip3")
gruLayer( 100, 'OutputMode', 'last', ...
"Name", "gru4" )] );
lg = connectLayers(lg, "input", "flip1");
lg = connectLayers(lg, "flip2", "cat1/in2");
lg = connectLayers(lg, "cat1", "flip3");
lg = connectLayers(lg, "gru4", "cat2/in2");
options = trainingOptions('adam', 'Plots', 'training-progress');
net = trainNetwork(XTrain, YTrain, lg, options);
[XTest, YTest] = japaneseVowelsTestData;
YPred = classify(net, XTest);
accuracy = sum(YTest == YPred)/numel(YTest)
  2 Comments
Ronny Guendel
Ronny Guendel on 22 Oct 2021
I found the issue:
The names for the layers must be not empty such as,
fullyConnectedLayer(9, 'Name', 'fc')
softmaxLayer('Name', 'softmax')
classificationLayer('Name', 'classoutput')
The full working code:
[XTrain, YTrain] = japaneseVowelsTrainData;
lg = layerGraph();
lg = addLayers(lg, [
sequenceInputLayer(12, "Name", "input")
gruLayer( 100, 'OutputMode', 'sequence', ...
"Name", "gru1")
concatenationLayer(1, 2, "Name", "cat1")
gruLayer( 100, 'OutputMode', 'last', ...
"Name", "gru3")
concatenationLayer(1, 2, "Name", "cat2")
fullyConnectedLayer(9, 'Name', 'fc')
softmaxLayer('Name', 'softmax')
classificationLayer('Name', 'classoutput')] );
lg = addLayers( lg, [
FlipLayer("flip1")
gruLayer( 100, 'OutputMode', 'sequence', ...
"Name", "gru2" )
FlipLayer("flip2")] );
lg = addLayers(lg, [
FlipLayer("flip3")
gruLayer( 100, 'OutputMode', 'last', ...
"Name", "gru4" )] );
lg = connectLayers(lg, "input", "flip1");
lg = connectLayers(lg, "flip2", "cat1/in2");
lg = connectLayers(lg, "cat1", "flip3");
lg = connectLayers(lg, "gru4", "cat2/in2");
options = trainingOptions('adam', 'Plots', 'training-progress');
net = trainNetwork(XTrain, YTrain, lg, options);
[XTest, YTest] = japaneseVowelsTestData;
YPred = classify(net, XTest);
accuracy = sum(YTest == YPred)/numel(YTest)

Sign in to comment.

More Answers (1)

Ronny Guendel
Ronny Guendel on 22 Oct 2021
The full working code for me:
[XTrain, YTrain] = japaneseVowelsTrainData;
lg = layerGraph();
lg = addLayers(lg, [
sequenceInputLayer(12, "Name", "input")
gruLayer( 100, 'OutputMode', 'sequence', ...
"Name", "gru1")
concatenationLayer(1, 2, "Name", "cat1")
gruLayer( 100, 'OutputMode', 'last', ...
"Name", "gru3")
concatenationLayer(1, 2, "Name", "cat2")
fullyConnectedLayer(9, 'Name', 'fc')
softmaxLayer('Name', 'softmax')
classificationLayer('Name', 'classoutput')] );
lg = addLayers( lg, [
FlipLayer("flip1")
gruLayer( 100, 'OutputMode', 'sequence', ...
"Name", "gru2" )
FlipLayer("flip2")] );
lg = addLayers(lg, [
FlipLayer("flip3")
gruLayer( 100, 'OutputMode', 'last', ...
"Name", "gru4" )] );
lg = connectLayers(lg, "input", "flip1");
lg = connectLayers(lg, "flip2", "cat1/in2");
lg = connectLayers(lg, "cat1", "flip3");
lg = connectLayers(lg, "gru4", "cat2/in2");
options = trainingOptions('adam', 'Plots', 'training-progress');
net = trainNetwork(XTrain, YTrain, lg, options);
[XTest, YTest] = japaneseVowelsTestData;
YPred = classify(net, XTest);
accuracy = sum(YTest == YPred)/numel(YTest)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!