Does an Intermediate Price Facilitate Algorithmic Collusion?

Abstract

Extensive simulation studies of reinforcement learning algorithms in pricing environments suggest that it may be possible for them to learn to collude when including a memory of one period. Theoretical analyses of this phenomenon have focused on the setting with two prices, while the simulations have been carried out with up to fifteen prices. Here, we address this gap by analyzing the probability of observing collusion by Decentralized Q-learning in a pricing duopoly with three possible prices and a memory of one period. Using the logit demand model for the payoffs leads to a payoff structure that is a three-action generalization of the iterated PD. Our central question is whether the addition of an intermediate action (or price) between “defection” and “cooperation” increases the likelihood of observing cooperation/collusion.

The Decentralized Q-learning algorithm is provably convergent in this environment, and the basins of attraction of its equilibria can be calculated explicitly using Individual Best-Response Graphs. We find that even though the additional action increases the number of collusive equilibria, the likelihood of collusion does not increase. In addition, we find common structure in the possible strategy equilibria, identify conditions for their existence, and refine the notion of collusive strategy equilibria. Finally, we show that introducing fluctuations to the learning process drastically increases the likelihood of collusion, but not to the levels observed when introducing fluctuations in the environment with two prices.

Designing effective antitrust policy and detection tools against the threat of algorithmic collusion requires understanding the mechanisms by which algorithms learn to collude. To this end, our results provide new insight into the drivers of collusion among reinforcement learners.