Session: 01-02: AI for Energy Sustainability II

Paper Number: 130538

130538 - Enhancing Battery Storage Energy Arbitrage With Deep Reinforcement Learning and Time-Series Forecasting 

Abstract: 

The growing incorporation of intermittent renewable energy into electricity grids comes with an increasing demand for flexible energy storages. Due to technological improvements and cost reductions, battery energy storages (BESs) are a suitable candidate for this task. Among the different roles that BESs can play, energy arbitrage is one of the most profitable sources of revenue for battery operators. Energy arbitrage describes buying and storing power during low electricity prices and discharging and selling power back to the grid when prices are high.

Forecasting the revenues generated through energy arbitrage is crucial for the planning of new BES projects but is also difficult due to the inherent uncertainty of electricity prices. Deep reinforcement learning (DRL) emerged in recent years as a promising tool and is able to cope with uncertainty by training on large quantities of historical data. However, without access to future electricity prices, DRL agents can only react to the currently observed prices and not learn to plan battery dispatch. Therefore, in this study, we combine DRL with time-series forecasting methods from deep learning to enhance the performance on energy arbitrage. Both tools interact in the following way: 1) DRL controls the battery through sequential decision making, seeking to maximize revenues; 2) at every time step, the trained forecaster provides predictions on future prices to the DRL agent and thereby allows better informed decisions.

Our study centers around the possible performance gains on energy arbitrage through the combination of these two machine learning models. Besides training the models on difficult real-world data, we consider the cost of battery degradation as well as losses during charging and discharging to obtain realistic results. We conduct a case study using price data from Alberta, Canada that is characterized by irregular price spikes. In the first set of experiments, we investigate the upper bounds of obtainable revenues by training DRL agents with perfect price forecasts of different horizons. While knowledge of electricity prices in three hours yielded best results for single forecasts, grouping forecasts for multiple horizons further improved performance. We then tune and train state-of-the-art deep learning models on price forecasting, consisting of convolutional layers, recurrent layers, and attention modules. In addition to historical price data, the models have access to historical demand and various climate variables recorded in hourly intervals. Due to the nature of the data, with time and magnitude of price spikes being hard to predict, the forecasting errors on the test set remain high.

Our results show that energy arbitrage with DRL-enabled battery control still significantly benefits from these imperfect predictions, but only if predictors for several horizons are combined. For example, access to perfect price forecasts 3 hours in the future resulted in a revenue increase of 42.6%. Using predicted prices for 3 hours ahead instead, the performance improved by only 10%. Grouping multiple forecasts of electricity prices for the next 12-hour window, perfect forecasts yielded a 57.5% growth in revenues compared to a 48.7% growth with trained predictors. Thus, DRL agents benefit especially from combined trained predictors – even more compared to having access to single perfect forecasts. We hypothesize that multiple predictors, despite their imperfections, convey useful information regarding the future development of electricity prices through a “majority vote” principle, enabling the DRL agent to learn more profitable BES control policies.

Presenting Author: Manuel Sage McGill University

Presenting Author Biography: Manuel Sage is a PhD Candidate in the Additive Design and Manufacturing Laboratory (ADML) at the Department of Mechanical Engineering in McGill University, in Montreal, Canada. His research interests include machine learning and reinforcement learning in general, and their applications to energy systems in particular. He is researching the applicability of deep reinforcement learning algorithms to control problems in hybrid energy systems.

Authors:

Manuel Sage McGill University
Joshua Campbell McGill University
Yaoyao Fiona Zhao McGill University