Session: 02-02: AI for Energy Sustainability II

Paper Number: 156635

156635 - The Economic Dispatch of Power-to-Gas Systems With Deep Reinforcement Learning: Tackling the Challenge of Delayed Rewards With Long-Term Energy Storage 

Abstract: 

Electricity price spikes pose a challenge to energy markets as their occurrence, duration, and magnitude is hard to predict. Simultaneously, they offer substantial revenue opportunities for power producers. The inherent intermittency of renewable energy sources such as wind and solar prevents the operators of renewables from capitalizing on lucrative price spikes. In fact, the spikes are often caused by low renewable production.

Energy storage solutions are increasingly recognized for their potential to alleviate this issue. By storing energy during periods of low demand and prices - or when renewable energy generation would otherwise be curtailed - storages can conserve energy for release during peak demand at higher prices, thereby augmenting the profitability of renewable energy operations. However, a profitable plant operation can only be assured with a smart dispatch policy for the involved components. Given the uncertainty of renewable energy generation and electricity prices, deciding on the right moment to store and release energy is a challenging multi-period stochastic optimization problem. Optimal plant operation must balance the cost of operating the storage system and short-term revenue with long-term revenues.

In this study, we apply deep reinforcement learning (DRL) to learn dispatch policies for hybrid power plants involving wind turbines, battery energy storage (BES) systems, and power-to-gas (P2G) systems with gas turbines. DRL has been applied to a variety of energy management problems in recent years due to its ability to learn dispatch policies from historical data, effectively capturing the uncertainty embedded in the data.

We construct a challenging simulation environment in which wind power can be sold to the utility grid or used to charge the BES and P2G system. For the P2G system, we model electrolyzers followed by methanation to produce synthetic natural gas (SNG) that is stored in an underground storage. During price spikes, discharging the BES or burning the SNG in the gas turbine can yield higher profits. Notably, BESs and P2G systems have different limitations. Utility-scale BESs have limited capacities, enabling discharge for several minutes up to a few hours at most, while P2G systems suffer from low round-trip efficiencies.

We conduct a case study in Alberta, Canada, where we empirically show that DRL can learn complex policies and concurrently dispatch the BES, P2G system, and gas turbine to maximize the wind farm’s profits. Compared to conventional optimization methods, our best-performing DRL algorithm, deep Q-networks (DQN), increases profits by 32% by efficiently capitalizing on the frequent price spikes in the province’s wholesale grid. To allow reproducibility of our results and facilitate future research, we make our data and codebase publicly available on GitHub.

Presenting Author: Manuel Sage McGill University

Presenting Author Biography: Manuel Sage is a Ph.D. candidate at McGill University. His research interests include machine learning and reinforcement learning, and their applications to energy systems in particular. He is researching the applicability of deep reinforcement learning algorithms to control problems in hybrid energy systems.