Query regarding 'advantages' in a2c

[akileshbadrinaaraayanan]

The actor net takes state as input and outputs a policy containing the probability of each action. In train_model(), the ground truth for training actor net is 'advantages' which is not a probability distribution over possible actions. So, how does the categorical cross-entropy computation between the predicted output of actor net and 'advantages' work?

Thanks,
Akilesh

[dnddnjs]

Advantage means r + gamma * V(s') - V(s)
and loss function of policy network will be log(action_prob) * advantages which is form of cross entropy.

action_prob will be [p1, p2] and advantages will be [0, advantages] if actual action of agent is 2.
then cross entropy calculation just becomes [log(action_prob) * advantages]

[akileshbadrinaaraayanan]

Thanks a lot!

[akileshbadrinaaraayanan]

Hi,

Categorical cross-entropy in Keras with TF backend expects a probability distribution. However, advantages is not a probability distribution in this case. I have observed that it works fairly good but could you please explain how exactly it works. Categorical cross-entropy is generally defined between two probability distributions p(x) and q(x)

Thanks,
Akilesh

[fredcallaway]

#54