It seems that the texture is shrinking and unity starts pulling texture from further away than my designated area in the atlas. The area around the required texture in the atlas is white/whitish.
I have tackled the exact same issue in Unity a few months ago. You definitely don’t want to set the filter mode to “point” as its pixels will not get upsampled/downsampled with the frustum yielding a “visual” quality loss.
To avoid seeing the white lines, in the “Texture Type” select “Advanced” from the drop-down list. It will unlock extra parameters for your texture. Find the “Generate Mip-maps” checkbox, and uncheck it. The white lines disappear with no significant quality loss.