High-altitude uncrewed aerial vehicles (UAVs) with millimeter wave communication are well-suited for fifthgeneration (5G) and beyond applications. UAVs may cause significant interference to ground user equipment (GUE). We consider using a moving tethered aerial base station (TBS) as an alternative to a terrestrial base station. We consider, the UAV and GUE locations as contexts to perform joint TBS location, UAV and GUE power allocation optimization in a three-dimensional environment. We propose a contextual multiarmed bandit framework using a novel counterfactual Thompson sampling (CTS) algorithm. We compare its performance against a joint optimization using vanilla Thompson sampling (TS) and single optimization TS (SOTS) approaches. Our results show that the CTS approach converges faster. We conclude that the CTSbased approach achieves better interference mitigation for both aerial and ground users.