resources.md

General Knowledge

General

RL debugging

A DQN hyperparameters

Initiation

1. Initiation introduction

2. Introduction of Xavier and Kaiming

3. Math explaination

Notes

According to ref(1):

And,

Then Xavier recommended a 0-mean normal distribution for W initiation.

The ref(1) proved that:

• Suppose activation function = tanh and input of tanh in scope of active area, i.e., tanh(x) ≈ x.
• Xavier initiation match hold² in above figure.
• Var(a) remains in scope of Var(x). a is output of a certain layer
• Var(a) holds without vanish or exploding through deep layers.

Ref(2) demonstrated a simple case to show that a two form of Xavier initiation gives similar effect:

• in form of uniform distribution bounded between

• in form of normal distribution like:

For layers with Relu as activation function, Kaiming initiation is recommended by ref(2). Kaiming is exactly the second form mentioned above. Ref(2) presented an example to show where sqrt(2) came from. Ref(2) also presented another example to show that Xavier failed in Relu case.

Ref(3) explains math behind these algorithms (Xavier and Kaiming):

• Xavier treats activation function with linear approximation. It is not the case for Relu.
• Kaiming takes Relu as example, as P(Relu(x)) = 1/2, there is sqrt(2) in Kaiming initiation.
• Backward prove is the same with forward process, except that the input is from layer^{(l + 1)} instead of layer^l-1
• Take both forward and backward in consideration, Xavier takes mean of input layer of each direction. Kaiming argues taking one direction in consideration is good enough.

TODO

• Ref(1) declares Xavier is a normal distribution, Ref(2) declares it is a uniform one. To check the paper

• Replay buffer stores experiences (s, s', a, r, d), where d is accompanied with s' (note that we can not save an experience for the first step). (DLR-RM/stable-baselines3#105)

• However it may help (and is part of many of the optimized parameters in rl-zoo), as it decreases the overall step size. Especially towards end of the training the updates seem to oscilate around good points, and by decreasing learning rate you reach higher returns

Optimizer

Adam

eps ref

Update pytorch lr of optimizer

Continuous Space

• DDPG
• TD3

TODO

• ACER

• SAC for discrete action space

• Soft Actor-Critic for continuous and discrete actions

Math

• Quantile regression

• Wasserstein metric

CPP

Detect anomaly in C++