If P gain is too high, the quadcopter is reacting strong to any control input or disturbance. So, it is more likely to oscillate even if no control input is given.
Try to increase thrust so that it would take off, loosen your grasp only a little bit, but provide a larger area by holding with your fingers beneath each other to decrease any influence. Write down this critical value - the P gain should always be adjusted below.
See next:
https://www.youtube.com/watch?v=R4CsujIUk7A
ATTENTION:
It is recommended to test it with a second person who gives commands on the remote control.
You may pay attention that you are influencing the quadcopter while holding - it is more likely to oscillate if you hold it slightly due to the turning point you create. On the other hand you are preventing the quadcopter's reaction for control the tighter you are holding it.