Session: 01-01: AI for Energy Sustainability I

Paper Number: 122475

122475 - Improving Control of Energy Systems With Reinforcement Learning: Application to a Reversible Pump Turbine 

Abstract: 

The substantial increase of volatile renewable energy sources in the electricity mix is challenging the stability of the grid. Pumped hydro storage systems are well suited to account for the volatility and maintain the required grid frequency through their fast reaction times and reactive power support. With the transition from fossil-fueled power plants, which are currently still providing a large part of grid stability services, to a fully renewable energy supply, pumped hydro storage power systems will need to provide even more flexibility in the future. At the moment, operating mode changes have fixed timings with big safety margins. To increase flexibility, faster reaction times for the switches between operating modes become necessary, which can only be achieved with precise and automated process control. Traditional control strategies might not be flexible enough to deal with this level of complexity.

Reinforcement learning (RL), a type of machine learning (ML), regularly outperforms traditional process control because it learns an optimal control strategy through direct interaction with its environment and can hence adjust to changing conditions. The so-called RL agent maps each possible state of the environment with the best possible action, yielding the optimal policy for the control of a process.

Despite the great potential of RL, its application for process control at an industrial scale is still very rare. Especially for energy supply and storage units, the implementation barrier for ML technologies is high due to safety concerns of operators. Reliability and safety guarantees are needed to ensure the safe operation of RL in energy systems in general, and pumped hydro power systems in particular. In our work, we identify necessary requirements to apply RL to energy systems, using a reversible pump turbine, as used in pumped storage power plants. Therefore, we evaluate engineering measures so the RL agent does not bring the pump turbine into unwanted operating conditions. Further, we propose the use of off-policy RL algorithms to increase the reliability of RL. In this way, the performance of the policy can be evaluated without the execution of exploratory actions on the real machine unit and thus increases robustness of the learning algorithm’s behavior.

For a proof-of-concept, we use the simulation model of a lab-scale reversible pump turbine. We compare the results for the control of the pump start-up with RL for training with and without the implementation of the defined safety and reliability measures, and evaluate the performance of a state-of-the-art off-policy deep RL algorithm for controlling the start-up as a pump within the simulation model.

Future research will then focus on the transfer of the optimal target policy of the trained RL agent to the real model machine unit located at the laboratory of the Institute of Energy Systems and Thermodynamics, TU Wien, to control the complete pump start-up, and the evaluation of the algorithm’s reliability. Therefore, we set up a digital twin platform that connects the real machine unit with its virtual counterpart and enables training and deploying of the RL policy.

Presenting Author: Carlotta Tubeuf TU Wien

Presenting Author Biography: Carlotta Tubeuf received the B.S. and M.S. degrees in mechanical engineering – management from TU Wien in 2018 and 2021, respectively.

Since 2021 she is pursuing her Ph.D. in mechanical engineering at the research unit of industrial energy systems at the institute of energy systems and thermodynamics at TU Wien. Her research mainly focuses on applying machine learning concepts via digital twin technologies for renewable energy systems.

Authors:

Carlotta Tubeuf TU Wien
Jakob Aus Der Schmitten TU Wien
René Hofmann TU Wien
Clemens Heitzinger TU Wien
Felix Birkelbach TU Wien