No, you are not getting voltage sag.
Your vesc is heating up due to hard use (hill climbing) and reaching the thermal protection limit. (The vesc has a temperature sensor onboard, as well as connections for external ones in the motors.)
Yes, your motors will get hot when climbing hills.
No, it has nothing to do with your batteries.
Decreasing your motor max will reduce the torque your motors have under heavy load at low-to-moderate speed (hill climbing and accelerating from a standstill).
If you want to climb hills with greater ease and you don’t have the resources to upgrade to 10S or 12S (Don’t go 12s with vescs based on stock 4.12 hardware, that ends in streetface), then your best bet is to lower your gearing: Smaller motor pulley, and/or bigger wheel pulley. This will reduce your top speed and increase the torque you have for hills.