Session: 01-05: AI for Energy Sustainability V

Paper Number: 130508

130508 - Evaluating Reduced Order Models for Training Reinforcement Learning Agents for Building HVAC Control 

Abstract: 

Electrification combined with a highly renewable energy supply is considered the main route for decarbonising buildings. To best integrate into this future energy system, building heating, ventilation and air conditioning (HVAC) systems must be energy efficient, flexible, and resilient to electrical grid disturbances. These grid-integrated buildings must be responsive to weather predictions, occupancy patterns, and grid economic signals to optimise energy consumption and meet building operational demands. This is a complex task, and the rule-based controllers which govern HVAC systems today are neither optimal nor flexible enough to operate in this manner. 

Reinforcement learning (RL) has shown promise in this area due to its ability to learn optimal control strategies and to adapt and improve autonomously over time. However, the cost of training RL algorithms in terms of time, computation and data is high and serves as a significant barrier to practical implementation. Previous simulations have required several days to train RL agents on detailed building models and  in some cases, the use of high-performance computing clusters to repeat the training for hyperparameter tuning. In real-world applications, it has been found that RL could deliver energy savings through HVAC control; however, it is highly dependent on a detailed building physics model requiring modelling expertise and historical data for calibration.

The research aims to reduce the costs of training RL algorithms by simplifying the building models used in training and then transferring this learning to a more complex building model for online testing. Our hypothesis is that simplified building energy models can be used to train a Deep Q-Learning algorithm, and this learning can be transferred to a detailed building model and operate within allowed temperature tolerances. The novelty in this work is in the learning transfer from RL networks trained on reduced-order building physics models to the control of more detailed system models.

To test our hypothesis, a detailed building physics model of a commercial building in New York State was made to represent a real-world environment. The commercial building was modelled using building physics from EnergyPlus, and a split system heat pump was modelled using the Modelica Buildings library. A Deep Q-learning algorithm was then trained in four simplified building environments and then tested in our detailed modelling environment. The states were the indoor temperature, outdoor temperature, and occupancy state. The goal of the DQN was to maintain the building’s indoor temperature between 20°C - 22°C during occupied hours and 15°C - 17°C elsewise by modulating the input power to the heat pump compressor. The four simplified building physics models evaluated were resistor-capacitor (RC) models consisting of (1) a four-element model with separate RC chains for the external walls, roof, floor, and internal walls, (2) a three-element model which removes the roof RC chain, (3) a two-element model which also removes the floor RC chain, and (4) a one-element model which only considers the external walls. A separate DQN was trained on each of these models using two weeks of historical temperature data during the heating season. The four RL networks were then used to control the detailed building simulation for one week.

 

The DQN trained on the highest-order RC model was successful in learning a suitable control strategy and, when transferred to the detailed building model, was able to generalise sufficiently to provide temperature control when tested using previously unseen weather data. The network was able to control the indoor temperature with behaviour which followed the required occupancy pattern but within larger temperature bounds. The lower-order RC models did not learn adequate control strategies due to poor representation of the thermal mass, leading to consistently low or high building temperatures. Overall, this work validates the hypothesis that a significant amount of knowledge can be transferred from a simplified model to the operation of a building energy system however, further work is needed to determine which parameters are essential to capture to maximize the transfer learning.

Presenting Author: Matthew Kerr Columbia University

Presenting Author Biography: Matthew is a PhD student in the Building Energy Research Lab at Columbia University. He has a master's degree in Mechanical Engineering with Aeronautics from the University of Glasgow and several years of industry experience in manufacturing and project management. His current research focuses on advanced control strategies for HVAC systems, especially concerning the practical implementation of reinforcement learning for optimal control. His interests are in building energy management and how buildings can play an important role in balancing the energy grid by acting as agents of flexible demand.

Authors:

Matthew Kerr Columbia University
Bianca Howard Columbia University