That is very strange behavior. 
Try this @bullrider12 measure between 5V and H(does not matter which one) and rotate the motor by hand slowly to a point where you get 0V on your meter. Now without moving the motor connect the meter instead of 5V to GND and you should be getting those 5V. Because you are using different “side” (5V being one side and GND the other) as a reference so where you get 5V against 5V you should get 0V against GND and vice versa.
@yaca the exact values may be different because of the saturation curve of the hall sensors used. But this is my theory with your case: When you get 5,13 against 5V (as I explained in previous part of this reply) it is normal that you get low value against GND. the VESC was supplying 5,16V (5.13+0.03) at that time it seems like, which is correct. The same goes for 3.62+1,54=5.16 It is OK, you are just referencing the voltage against a different part of the supply. If your VESC works/recognizes the sensors you are good.