Limiting player movement (Partially done)

I am currently using this code to restrict player movement along the horizontal axis, to ensure that the player stays within the camera view. Right now it works for the horizontal axis, but not the vertical axis, for some reason.

In other words, topB and downB values do not seem to be read correctly.

void Update () {
        
        float distance = (transform.position.z - cam.transform.position.z);
        float leftB = cam.ViewportToWorldPoint(new Vector3(0, 0, distance)).x;
        float rightB = cam.ViewportToWorldPoint(new Vector3(1, 0, distance)).x;

        float topB = cam.ViewportToWorldPoint(new Vector3(0, 1, distance)).z;
        float downB = cam.ViewportToWorldPoint(new Vector3(0, 0, distance)).z;

        Vector3 pos = transform.position;
        pos.x = Mathf.Clamp(transform.position.x, leftB, rightB);
        pos.z = Mathf.Clamp(transform.position.z, downB, topB); //Not working well: 
        // topB is roughly the center of the screen while downB is well beyond the 
        // bottom of the  screen.

        transform.position = pos;
    }

Video describing problem: http://youtu.be/wfGoLZ2GV84

Why can’t I make the camera just follow the player?

I have two players on screen and a camera which increases the ortographicSize as they get farther away from each other. At a certain point it stops stretching and needs to start restricting player movement instead.

Notes

The game is top down at an angle. I have been looking for help on camera.ViewportToWorldPoint, but I think I am not getting the dist (distance) right, since the camera is rotated at an angle, not straight down on the players (the cubes in the pictures below).
Screenshot

If I understood your logic correctly you need the distance from the camera to a plane, parallel to the camera’s projection plane, wich contains the player. One way to get this value is projecting a vector from the camera to the player into the camera’s forward direction. This is accomplished by:

float distance = Vector3.Dot(cam.transform.forward, transform.position - cam.transform.position);