2d Sprite collision not working!!!

I am creating a simple puzzle game where the goal is to make the circular sprite reach its goal . I have used a simple drag and drop movement script . The dark grey sprites used immovable objects and all other colored objects can be moved. All the rectangular shaped sprites can be moved in order to make way for the circle to reach its goal.
Problem I have is none of the sprites colliding with each other I have tried adding colliders2d and also rigidbody2d but it stills passes through other objects. If I add only rigidbody2d without freezing the position and rotation , objects push each other out of the boundary. Even boundaries are not colliding!
Anyone got any ideas , on how to add collision between objects???!!!

Help would be much appreciated!!!

@logicandchaos
Colored Blocks

Borders and Immovable Grey blocks