move object over another object normals

I have a mesh object and a cylinder object in my scene.

What I'm trying to do is detect the (mesh) collision of the object when the mouse is over it. When the mouse is over this collision mesh, I then want to instantate the cylinder so that its' y up is aligned with the mesh collision normal where the mouse is touching it and its' pivot transforms to the mesh collision surface.

The end result should look like a pen hovering on a curved piece of paper.

This should continue to update as long as the mouse is over the mesh object. When the mouse is not hovering over the object then the instance should stop.

Any pointers on a method to accomplish this?

The key here is building an orientation for the object that will align it with the surface normal at the specified point.

There's an infinite number of such orientations, so the trick is to narrow that down to one. There are several different ways this can be done, the simplest of which is probably the 'fixed reference vector' method, e.g. (not compiled or tested):

Vector3 up = hit.normal;
Vector3 forward = Vector3.up.Cross(up);
targetObject.transform.rotation = Quaternion.LookRotation(forward, up);

The catch is that this may fail if the normal and the reference vector (Vector3.up in this case) are parallel or nearly parallel. There's another method that doesn't have that problem, but that can cause discontinuities in the orientation (which may be noticeable if the object isn't completely radially symmetrical). Basically, you just have to decide which method is most appropriate for the context and then code accordingly (and handle special cases separately if needed).