TU Darmstadt / ULB / TUprints

Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learning

Cui, Kai ; Koeppl, Heinz (2022):
Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learning. (Publisher's Version)
In: Proceedings of Machine Learning Research, 130, In: Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, pp. 1909-1917,
Darmstadt, PMLR, 24th International Conference on Artificial Intelligence and Statistics (AISTATS) 2021, Virtual, 13.-15.04.2021, DOI: 10.26083/tuprints-00021511,
[Conference or Workshop Item]

[img] Text
cui21a.pdf
Available under: CC BY 4.0 International - Creative Commons, Attribution.

Download (750kB)
[img] Text
cui21a-supp.pdf
Available under: CC BY 4.0 International - Creative Commons, Attribution.

Download (1MB)
Item Type: Conference or Workshop Item
Origin: Secondary publication service
Status: Publisher's Version
Title: Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learning
Language: English
Abstract:

The recent mean field game (MFG) formalism facilitates otherwise intractable computation of approximate Nash equilibria in many-agent settings. In this paper, we consider discrete-time finite MFGs subject to finite-horizon objectives. We show that all discrete-time finite MFGs with non-constant fixed point operators fail to be contractive as typically assumed in existing MFG literature, barring convergence via fixed point iteration. Instead, we incorporate entropy-regularization and Boltzmann policies into the fixed point iteration. As a result, we obtain provable convergence to approximate fixed points where existing methods fail, and reach the original goal of approximate Nash equilibria. All proposed methods are evaluated with respect to their exploitability, on both instructive examples with tractable exact solutions and high-dimensional problems where exact methods become intractable. In high-dimensional scenarios, we apply established deep reinforcement learning methods and empirically combine fictitious play with our approximations.

Book Title: Proceedings of The 24th International Conference on Artificial Intelligence and Statistics
Series: Proceedings of Machine Learning Research
Series Volume: 130
Place of Publication: Darmstadt
Publisher: PMLR
Classification DDC: 000 Allgemeines, Informatik, Informationswissenschaft > 004 Informatik
500 Naturwissenschaften und Mathematik > 510 Mathematik
600 Technik, Medizin, angewandte Wissenschaften > 620 Ingenieurwissenschaften
Divisions: 18 Department of Electrical Engineering and Information Technology > Institute for Telecommunications > Bioinspired Communication Systems
18 Department of Electrical Engineering and Information Technology > Self-Organizing Systems Lab
TU-Projects: HMWK|III L6-519/03/05.001-(0016)|emergenCity TP Bock
Event Title: 24th International Conference on Artificial Intelligence and Statistics (AISTATS) 2021
Event Location: Virtual
Event Dates: 13.-15.04.2021
Date Deposited: 20 Jul 2022 13:34
Last Modified: 20 Jul 2022 13:35
DOI: 10.26083/tuprints-00021511
Corresponding Links:
URN: urn:nbn:de:tuda-tuprints-215111
URI: https://tuprints.ulb.tu-darmstadt.de/id/eprint/21511
PPN:
Export:
Actions (login required)
View Item View Item