How to make a heavy bag (for punching ) heavier ?

Im making a game in VR in which I punch a boxing heavy bag with gloves . I have a Configurable joint on it is attached (at the anchor) a heavy bag . In the heavy bag object I have a rigidbody with a maximum amount of mass but its still not heavy enough as It gets punched it should barely move . I dont wont to add drag to the heavy bag as it makes it very unrealistic.

I tried adding a script to the gloves which check the moment the gloves collide with the bag and tried to downplay the force that is generated by the punching, but that doesn`t work either . The script is BagPunchScript (only the collider code):

 void OnCollisionEnter(Collision other) {
       
  
    Rigidbody otherR = other.gameObject.GetComponent<Rigidbody>();
   //check the impact angle

    Vector3 avgPoint = Vector3.zero;
    foreach (ContactPoint p in other.contacts)
    {
        avgPoint += p.point;

    }
    avgPoint /= other.contacts.Length;

    Vector3 dir = (avgPoint - transform.position).normalized;
    otherR.AddForceAtPosition(dir * 0.01f * rBody.velocity.magnitude, avgPoint); //downplay the force of the hit 

    if (other == null)

         return;
     
    
    

}

Any suggestions ?

Meishin has it right. I remember seeing in a training video that the physics engine calculates how much two masses can affect each other based on their relative mass. So, to make the gloves make the bag move less you lower their mass. Or if you are just applying force to the bag, then reduce the amount of force being used.