I want to rotate vertex point (blue point) to the new position around the second position with an angle between first and second position in order to make mesh looks better.
You’ve got a flipped sign in the equation. The standard rotation matrix is:
cos -sin
sin cos
You flipped the bottom right to a negative. But, jakovd’s advice is good – often easier, and a just a tiny bit CPU slower, to say Quaternion R = Quaternion.LookRotation(wantP-aroundP); and then use it: p2=R*p2;, which “applies” the rotation to a point.
@sakamotomiwa could you explain why are you trying to rotate these vertices through the script? Maybe you are facing the problem that would be much easier to solve using some 3D modeling tool…
I have no clue what is it that you are showing us with your pictures, mate