You can use GetAxisRaw and trivially recreate the smoothing of GetAxis on a single axis basis with Mathf.Lerp, or on a dual-axis basis with Vector2.Lerp().
You just need one storage variable and the notion of how snappy you want the smoothing.
float axis; // the "accumulator"
const float Snappiness = 3.0f; // adjust to suit
and then in Update():
// raw input:
float x = Input.GetAxisRaw( "Horizontal");
// smooth it
axis = Mathf.Lerp( axis, x, Snappiness * Time.deltaTime);
// TODO: now use axis for your input
If you need 2-axis smoothing (blended for vector magnitude), replace float above with Vector2, assemble a new Vector2 from the combined raw axis inputs, and use Vector2.Lerp instead.
As for your actual drift problem, print the values out, see what they are. If they drift, you can search high and low through your hardware drivers and device logic and settings and everything looking for a ghost, or else you can trivially fix it by zeroing them out when they are below a certain magnitude, such as 0.1f or whatever.