setNormalizer

Set normalizer in function approximator object

Since R2024a

Syntax

newFcnAppx = setNormalizer(fcnAppx,normalizers)

newFcnAppx = setNormalizer(fcnAppx,normalizers,indices)

Description

newFcnAppx = setNormalizer(fcnAppx,normalizers) returns a new function approximator object, newFcnAppx, with the same configuration as the original object, fcnAppx, but with the normalizers specified in normalizers. The function assigns each normalizer specified in normalizers to an input channel, in the order specified in the observationInfo and actionInfo properties of fcnAppx.

newFcnAppx = setNormalizer(fcnAppx,normalizers,indices) assigns the kth normalizer specified in normalizers to the input channel whose number is specified in the kth position of indices. normalizers and indices must have the same dimension.

Examples

Assign Normalizers to Actor and Critic Inputs

This example shows how to assign normalizer objects to the actor and critics of a DDPG agent.

Create DDPG Agent and Extract Actor and Critic

Create specification objects to define observation and action channels. For this example, the agent has two observation channels. The first channel has a vector with three elements and the second channel has a vector with four elements.

The action channel carries a two-dimensional vector.

obsInfo = [rlNumericSpec([3,1]) rlNumericSpec([4,1])];
actInfo = rlNumericSpec([2,1]);

Create a default DDPG agent.

agent = rlDDPGAgent(obsInfo, actInfo);

Extract the approximator objects.

actor = getActor(agent);
critic = getCritic(agent);

Create Normalizer Objects

Create one normalizer object for each input channel. DDPG agents use a Q-value function critic, which requires both the observation and the action as inputs. The Mean, StandardDeviation, Min, and Max properties apply to each element of the channel.

Create the normalizer for the first observation channel.

obs1Nrz = rlNormalizer(obsInfo(1).Dimension, ...
    Normalization="zscore", Mean=2, StandardDeviation=3)

obs1Nrz = 
  rlNormalizer with properties:

        Normalization: "zscore"
                 Mean: 2
    StandardDeviation: 3

Create the normalizer for the second observation channel.

obs2Nrz = rlNormalizer(obsInfo(2).Dimension, ...
    Normalization="zerocenter", Mean=4)

obs2Nrz = 
  rlNormalizer with properties:

    Normalization: "zerocenter"
             Mean: 4

Create the normalizer for the action input channel of the Q-value function critic.

actInNrz = rlNormalizer(actInfo.Dimension, ...
    Normalization="rescale-symmetric", Min=-2, Max=2)

actInNrz = 
  rlNormalizer with properties:

    Normalization: "rescale-symmetric"
              Min: -2
              Max: 2

To check how the normalizer works on an input, use normalize.

normalize(obs2Nrz,6)

ans = 2

Assign Normalizer Objects to Actor and Critic

To assign new normalizers to the actor and critic, use setNormalizer.

actor = setNormalizer(actor, [obs1Nrz, obs2Nrz]);
critic = setNormalizer(critic, [obs1Nrz, obs2Nrz, actInNrz]);

You can also assign normalizers to selected channels only. For example, to assign normalizers only to the first observation channel (in the order specified by obsInfo) and the action channel, use an indices vector.

critic = setNormalizer(critic, [obs1Nrz actInNrz], [1 3]);

Display the normalization properties of the actor and critic.

actor.Normalization

ans = 1×2 string
    "zscore"    "zerocenter"

critic.Normalization

ans = 1×3 string
    "zscore"    "zerocenter"    "rescale-symmetric"

Assign Actor and Critic to Agent

To assign the new actor and critic to the agent, use setActor and setCritic.

setCritic(agent, critic);
setActor(agent, actor);

To check that agent works, use getAction.

a = getAction(agent, { ...
    rand(obsInfo(1).Dimension) ...
    rand(obsInfo(2).Dimension)});
a{1}

ans = 2×1

   -0.1039
    0.5166

Create DQN Agent and Extract Critic

Define a mixed observation space and a discrete action channel.

obsInfo = [rlNumericSpec([3,1]), rlFiniteSetSpec([5,3,4,2])];
actInfo = rlFiniteSetSpec([-1,0,1]);

Create a default DQN agent.

agent = rlDQNAgent(obsInfo, actInfo);

Extract the agent critic.

critic = getCritic(agent);

Extract Normalizer Objects from Actor or Critic

Use getNormalizer to extract an array of normalizer objects from the agent critic.

crtNrz = getNormalizer(critic)

crtNrz = 
  1×2 rlNormalizer array with properties:

    Dimension
    Normalization
    Mean
    StandardDeviation
    Min
    Max

Modify Normalizer Objects Using Dot Notation

Use dot notation to access and change the property of the second normalizer object (associated with the second observation channel).

crtNrz(2).Normalization="rescale-zero-one";
crtNrz(2).Min=-5;
crtNrz(2).Max=15;

Assign the modified normalizer to the critic and assign the critic to the agent.

critic = setNormalizer(critic, crtNrz);
setCritic(agent, critic);

To check that the agent works, use getAction.

a = getAction(agent, { ...
    rand(obsInfo(1).Dimension) ...
    rand(obsInfo(2).Dimension)});
a{1}

ans = 0

Input Arguments

fcnAppx — Function approximator

function approximator object

Function approximator, specified as one of the following:

• rlValueFunction object — Value function critic

• rlQValueFunction object — Q-value function critic

• rlVectorQValueFunction object — Multi-output Q-value function critic with a discrete action space

• rlContinuousDeterministicActor object — Deterministic policy actor with a continuous action space

• rlDiscreteCategoricalActor — Stochastic policy actor with a discrete action space

• rlContinuousGaussianActor object — Stochastic policy actor with a continuous action space

• rlContinuousDeterministicTransitionFunction object — Continuous deterministic transition function for a model-based agent

• rlContinuousGaussianTransitionFunction object — Continuous Gaussian transition function for a model-based agent

• rlContinuousDeterministicRewardFunction object — Continuous deterministic reward function for a model-based agent

• rlContinuousGaussianRewardFunction object — Continuous Gaussian reward function for a model-based agent.

• rlIsDoneFunction object — Is-done function for a model-based agent

To create an actor or critic function object, use one of the following methods.

• Create a function object directly.

• Obtain the existing critic from an agent using getCritic.

• Obtain the existing actor from an agent using getActor.

Note

For agents with more than one critic, such as TD3 and SAC agents, you must call getModel for each critic representation individually. You cannot call getModel for the array returned by getCritic.

critics = getCritic(myTD3Agent);
criticNet1 = getModel(critics(1));
criticNet2 = getModel(critics(2));

normalizers — Normalizer objects

rlNormalizer object | array of rlNormalizer objects

Normalizer objects, specified as an rlNormalizer object or an array of rlNormalizer objects.

indices — Channel indices

vector of unique positive integers

Channel indices, specified as a vector of unique positive integers. setNormalizer assigns the kth normalizer specified in normalizers to the input channel whose number is specified in the kth position of indices. normalizers and indices must have the same dimension.

For example, suppose you have a Q-value function critic with two observation channels and one action channel. Then newFcnAppx = setNormalizer(fcnAppx,[nobsB nact],[2 3]) assigns the normalizer nobsB to the second observation channel specified in fcnAppx.observationInfo, and the normalizer nact to the action input channel (which has an index of 3 because the action channel is always considered after all the observation channels).

Example: [3 1 2]

Output Arguments

newFcnAppx — New function approximator

function approximator object

New function approximator, returned as an object of the same type as fcnAppx. Except for its new normalizer object, newFcnAppx is the same as fcnAppx.

Version History

Introduced in R2024a