Going for a dual setup would help significantly. Reducing gearing ratio might help as well. Cheapest option would be to add heat sinks to the Mosfets of the Vesc http://www.ebay.com/itm/30PCS-Aluminum-Heatsink-Cooler-Adhesive-Kit-for-Cooling-Raspberry-Pi-New-/131760090486?hash=item1ead836d76:g:nhoAAOSwZ8ZW8fmY.
Maybe also cut a hole in the enlosure to stick the heat sinks out of.