Clear Filters
Clear Filters

Custom DQN enviroment & Loss function.

8 views (last 30 days)
Answered: Aditya on 17 Apr 2024
Hi all. I am working on my thesis project which involves the use of the DQN applied to the dismantling of networks (Undirected Graph). The first problem is to create an environment in which the actions are the removal of individual node.
The second problem concerns the loss function used by the DQN. I would like to know if there is a way to modify the loss function by adding a penalty. I am attaching the function that I would like to use by the Routine DQN of matlab function:
I also add that the Q (s, a) values used will be calculated upstream through another algorithm, so I would need to use those values.

Answers (1)

Aditya on 17 Apr 2024
Creating a custom environment for a DQN (Deep Q-Network) that involves the dismantling of networks (graphs) and modifying the loss function to include a penalty can be achieved in MATLAB. Let's break down the process into steps to address both of your concerns.
1. Creating a Custom Environment for Network Dismantling
For your thesis project, you'll need to define an environment that represents the undirected graph and actions that correspond to the removal of individual nodes. MATLAB's Reinforcement Learning Toolbox allows you to create custom environments by defining the necessary components such as observations, actions, and the reward mechanism.
2. Modifying the Loss Function in DQN
To modify the loss function used by the DQN algorithm in MATLAB, especially to add a penalty, you might have to customize the training loop or the part of the code where the loss is computed. The DQN's loss function is typically the mean squared error (MSE) between the predicted Q-values and the target Q-values. To add a penalty, you would adjust the computation of the target Q-values or directly modify the loss calculation.
If you have specific Q-values calculated upstream and want to use them along with a penalty in the loss function, you could manually compute the loss and perform the gradient update steps. Here's a conceptual sketch of how you might implement this:
  1. Compute Q-Values: Use your algorithm to compute the Q-values for the current state-action pairs.
  2. Compute Target Q-Values: For the next state, use your algorithm to compute the Q-values and then apply your custom penalty to these values.
  3. Compute Loss: Calculate the loss using the modified target Q-values and the Q-values from the current state-action pairs. If you're adding a penalty, it could be a function of the action taken or the resulting state.
  4. Update the Network: Use the computed loss to perform a gradient descent step on the DQN's neural network parameters.
Since modifying the core DQN algorithm in MATLAB's Reinforcement Learning Toolbox might require extensive customization, consider implementing the critical parts of the DQN (such as the computation of Q-values, the loss function, and the update step) manually if the toolbox does not offer the flexibility you need for your specific modifications.


Find more on Sequence and Numeric Feature Data Workflows in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!