Heating, Ventilation and Air Conditioning (HVAC) systems in buildings are a major source of global operational CO2 emissions, primarily due to their high energy demands. Traditional controllers have shown effectiveness in managing building energy use. However, they either struggle to handle complex environments or cannot incorporate learning from experience into their decision-making processes, leading to increased computational requirements. The potential solution to these drawbacks is reinforcement learning (RL), which can overcome them with its versatile and learning-based characteristics. In this context, this study presents a thorough literature review, focusing on studies published since 2019 that applied RL for HVAC system control. It bridges theoretical concepts and literature findings to identify suitable algorithms for each problem and to find gaps. It was found that RL deployment in real buildings is limited (23% of studies), with common training methods revealing fundamental technical problems that prevent their safe use: lack of diversification in exogenous state components (e.g., occupancy schedule, electricity price, and weather) that the agent receives in each episode during training in a way that reflects the diversity or unexpected change in real life. This necessitates repetitive, extensive retraining before real deployment, which is computationally expensive. Future research should focus on applying RL to real buildings by solving the previous problem. The meta-RL emerges as an up-and-coming solution for the generalization capabilities because it trains an agent on a wide range of tasks, making the agent more adaptive and reducing the computational cost. Further research should explore this direction.