Cannot detect touches on moving objects

Cannot get touch/poke(SpatialPointerKind.Touch) inputs on moving objects persistently.

I have some objects moving around with Transform.Translate method. Touch interactions with my hand are not recognized most of the time. If I aggressively poke the object the input registers sometimes. If the object is static, poke is registered just fine.

I use a simple setup like following code to test the pokes.

void Update()
{
   foreach (var touch in Touch.activeTouches)
   {
      var spatialPointerState = EnhancedSpatialPointerSupport.GetPointerState(touch);
      if(spatialPointerState.Kind == SpatialPointerKind.Touch)
      {
      Debug.Log("poked " + spatialPointerState.targetObject)
      }
   }
}