I wrote something like a point collider into Virtual RC Racing (2004 version) for its collision detection/response system. It was a pretty common approach at one time back in the days of much more scarce CPU resources (I had a 333Mhz laptop at the time), and because it was simple to do. I believe Microsoft Flight Simulator used something like it too, and I had several friends writing their own race car simulators that were doing it as well. While I don’t remember all the details about how mine worked (probably wrote it in 2001), the basic idea was this:
The mesh had a handful of “collision points” assigned to it that were strategically placed around the car meshes in places that usually worked reasonably well. The track and track object triangles were sitting in a quadtree. Each collision point would check which quadtree node it was in, then do a quick dot product with the normal of each triangle in the quadtree node to see which side of each triangle the point was on. If it was on the backside of a triangle, the point was projected to each triangle plane. That point was then checked to see if it was inside the triangle.
If it was inside a triangle, you’d generate a collision impulse somehow. I used the depth of penetration of the point like a spring which produced a force along the triangle normal direction. That’s enough to keep a car that’s flipped over from falling through the track (most of the time, anyway). There was also a velocity based force in there too. You compute the velocity of the point, dot that with the triangle normal and out pops the velocity in the triangle normal direction which is used to compute an impulse for higher speed collisions.
The “spring depth” approach is called a “penalty method” while the velocity approach is called an “impulse method.” Contrary to being impossible, this was very common years ago due to its simplicity and speed.
It comes with its own problems and isn’t the best (which is why years later I spent over a year rewriting the collision detection/response system to a much more robust box system that used the edges and so on too), but it was very fast.
Just last week I wrote something that was inspired by it for my boat game (Design it, Drive it: Speedboats). I use it to place objects like the seats and driver’s feet firmly on the floor of a procedurally generated mesh created by the user inside the game. It’s all just points. Here’s one of the stages of its operation rendered:
The green lines are just visual markers that extend from the center of some of the possible hit triangles to a vector where the seat is going to be pushed along (up in this case) to get the seat to pop up out of the floor. It’s not exactly the same thing, but pretty similar. I’m doing the raycasts myself on the mesh triangles directly. There are no Unity colliders or Unity RayCast() calls here.
Anyway, for all practical purposes that really is just a simple raycasting system which would be easy to add into Unity oneself. One of many downsides is that you can only collide with things from one direction. You could also do a raycast from the last point’s position to the current point’s position or something similar I suppose (that doesn’t work anymore once the object has pretty much stopped though), but yes, point colliders are possible and is what just about everybody used to do back in the day. They are quite limited compared to geometrical colliders like Unity and most other systems probably use now, but they do have their uses. The main benefit is that they’re very fast and much easier to write. The downside is they don’t always work very well. The 2004 version of VRC was notorious for cars that bounced around or occasionally were shot into outer space. 
