The physics engine used by Unity is PhysX. The source code is available online, but that’s going to be a rude awakening to glean the math out of that.
More generally, the subject is quite in depth. The sphere is the simplest collider of all, as it merely uses the radius of the sphere to detect if another sphere is within range. Simply subtract the distance between the centers and check to see if that distance is within the range of the two radii.
It gets more complicated with the object being tested isn’t a sphere. The phrase “edge case” literally applies to rectangular objects, and more general mesh shapes work on the various planes the mesh defines. The whole thing is further complicated when continuous mode is enabled, as there is an imaginary capsule or cylinder created on the fly that encloses the previous and current location which is subsequently used (instead of just the sphere) for intersection calculations.
The study is considered to exhaust the undergraduate level, and extends all the way to the graduate level of mathematical study. I say this because it would appear self explanatory that if you’re asking then you don’t yet have the math background, and if you have the math background you wouldn’t need to ask the question. The study is that of math itself, and while you can get the gist of it through a read through to satisfy curiosity, the fact is that since it can’t just stop at spheres (even though you want to limit the subject to spheres), this is an all-or-nothing study to get the whole picture of collision testing.
I don’t recall the title, but there is a book on the subject of creating physics engines that was an extremely good tour (published around 2008 to maybe 2010). The ‘standard’ books on this subject come from David Eberly (a PhD and acknowledged authority on the subject), but the author I’m thinking of I think had the first name Ian, but I don’t recall. You can find some web based tours by searching for collision testing.
Some portion of Unity code is exposed on git, so you can search for it yourself there, which classes you’re interested in looking at.
The maths you’re looking for might be pre-compiled on the physics dlls, so might not be possible to expose, but you might find some interesting stuff here: