TU Darmstadt / ULB / TUprints

Compatible natural gradient policy search

Pajarinen, Joni ; Thai, Hong Linh ; Akrour, Riad ; Peters, Jan ; Neumann, Gerhard (2022):
Compatible natural gradient policy search. (Publisher's Version)
In: Machine Learning, 108 (8-9), pp. 1443-1466. Springer, ISSN 0885-6125, e-ISSN 1573-0565,
DOI: 10.26083/tuprints-00020531,

[img] Text
Copyright Information: CC BY 4.0 International - Creative Commons, Attribution.

Download (7MB)
Item Type: Article
Origin: Secondary publication service
Status: Publisher's Version
Title: Compatible natural gradient policy search
Language: English

Trust-region methods have yielded state-of-the-art results in policy search. A common approach is to use KL-divergence to bound the region of trust resulting in a natural gradient policy update. We show that the natural gradient and trust region optimization are equivalent if we use the natural parameterization of a standard exponential policy distribution in combination with compatible value function approximation. Moreover, we show that standard natural gradient updates may reduce the entropy of the policy according to a wrong schedule leading to premature convergence. To control entropy reduction we introduce a new policy search method called compatible policy search (COPOS) which bounds entropy loss. The experimental results show that COPOS yields state-of-the-art results in challenging continuous control tasks and in discrete partially observable tasks.

Journal or Publication Title: Machine Learning
Volume of the journal: 108
Issue Number: 8-9
Publisher: Springer
Classification DDC: 000 Allgemeines, Informatik, Informationswissenschaft > 004 Informatik
600 Technik, Medizin, angewandte Wissenschaften > 600 Technik
Divisions: 20 Department of Computer Science > Intelligent Autonomous Systems
TU-Projects: EC/H2020|640554|SKILLS4ROBOTS
Date Deposited: 10 Feb 2022 13:10
Last Modified: 10 Feb 2022 13:11
DOI: 10.26083/tuprints-00020531
Corresponding Links:
URN: urn:nbn:de:tuda-tuprints-205319
URI: https://tuprints.ulb.tu-darmstadt.de/id/eprint/20531
Actions (login required)
View Item View Item