Why is dlgradient giving different answers?

When I use the dlgradient function to compute the gradient of the expression (Parameters.fc2.Weights * tanh(Parameters.fc1.Weights * y(:,1) + Parameters.fc1.Bias) + Parameters.fc2.Bias) with respect to Parameters.fc2.Bias, it yields varying results instead of a consistent value of 1. According to theoretical calculations, it should be 1, but for different values of y(:,i), I observe discrepancies. What might be the issue?

Parameters = struct;
stateSize = 1;
hiddenSize = 20;

Parameters.fc1 = struct;
sz_fc1 = [hiddenSize stateSize];
Parameters.fc1.Weights = initializeGlorot(sz_fc1, hiddenSize, stateSize);
Parameters.fc1.Bias = initializeZeros([hiddenSize 1]);

Parameters.fc2 = struct;
sz_fc2 = [stateSize hiddenSize];
Parameters.fc2.Weights = initializeGlorot(sz_fc2, stateSize, hiddenSize);
Parameters.fc2.Bias = initializeZeros([stateSize 1]);

y(:,1) = 1;
y(:,2) = 0.976;
gradient1.fc2.Bias = dlgradient(Parameters.fc2.Weights * (tanh(Parameters.fc1.Weights * y(:,1) + Parameters.fc1.Bias)) + Parameters.fc2.Bias, Parameters.fc2.Bias)
gradient2.fc2.Bias = dlgradient(Parameters.fc2.Weights * (tanh(Parameters.fc1.Weights * y(:,2) + Parameters.fc1.Bias)) + Parameters.fc2.Bias, Parameters.fc2.Bias)

[Matt J]

Attach Parameters and y in a .mat file so we can test your code.

Answer 1

[Angelo Yeo]

You can try to incorporate dlfeval when using dlgradient. You can get the results of 1's as expected.

Parameters = struct;
stateSize = 1;
hiddenSize = 20;

Parameters.fc1 = struct;
sz_fc1 = [hiddenSize stateSize];
Parameters.fc1.Weights = initializeGlorot(sz_fc1, hiddenSize, stateSize);
Parameters.fc1.Bias = initializeZeros([hiddenSize 1]);

Parameters.fc2 = struct;
sz_fc2 = [stateSize hiddenSize];
Parameters.fc2.Weights = initializeGlorot(sz_fc2, stateSize, hiddenSize);
Parameters.fc2.Bias = initializeZeros([stateSize 1]);

y(:,1) = 1;
y(:,2) = 0.976;

[res1, res2] = dlfeval(@gradFun, Parameters, y)

res1 = 
  1×1 single dlarray

     1

res2 = 
  1×1 single dlarray

     1

function [res1, res2] = gradFun(Parameters, y)


res1 = dlgradient(Parameters.fc2.Weights * (tanh(Parameters.fc1.Weights * y(:,1) + Parameters.fc1.Bias)) + Parameters.fc2.Bias, Parameters.fc2.Bias);
res2 = dlgradient(Parameters.fc2.Weights * (tanh(Parameters.fc1.Weights * y(:,2) + Parameters.fc1.Bias)) + Parameters.fc2.Bias, Parameters.fc2.Bias);

end

function weights = initializeGlorot(sz,numOut,numIn)

Z = 2*rand(sz,'single') - 1;
bound = sqrt(6 / (numIn + numOut));

weights = bound * Z;
weights = dlarray(weights);

end

function parameter = initializeZeros(sz)

parameter = zeros(sz,'single');
parameter = dlarray(parameter);

end