Uncrewed Aerial Vehicle-User Equipment (UAV-UE) is integral to millimeter wave (mmWave)-based wireless cellular systems. UAV-UE at high altitudes encounter limited connectivity with terrestrial base stations. Tethered Aerial Base Stations (TABS) are viable alternatives to terrestrial base stations. Optimal placement of a TABS in a three-dimensional environment is necessary and critical to serve multiple moving UAV-UE units with reliable connectivity. In this work, we propose a contextual multiarmed bandit framework to learn the optimal TABS locations. We consider multiple UAV-UE units moving at high altitudes in an uplink mmWave setting. Under this framework, the TABS acts as a learning agent leveraging position information about served UAV-UE units to provide connectivity with minimum Signal to Noise Ratio (SNR) threshold requirements. We first compare the Upper Confidence Bound (UCB) and Thompson Sampling (TS)-based learning strategies against the traditional naive-based approach. Our simulation results show that the TS-based approach learns optimal locations with a 31% and 51% average regret-reduction ratio (ARR) over UCB and naive-based approaches, respectively. Also, the TS-based learning strategy for TABS reliably achieves the required SNR for UAV-UE units under multiple contexts, compared to a static TABS location.