How to apply a normal map pixel to a vector3

Unity Engine
1

I have a need for the exact direction a wall is facing. I figured out all of the components, the wall normal is easy to get either from the mesh or even raycastHit.normal. I know how to move through the uvs, so getting the pixel of the bump map is also easy. I’m not sure wtf do to to combine them though. I understand that r is the x angle, g is y, and b is z, but how to actually apply it to the vector is eluding me.

//Get the world position and normal
if (normals == null || normals.Length == 0) normals = mesh.normals;
Vector3 faceNormal = transform.TransformDirection((normals[vert0] + normals[vert1] + normals[vert2]) / 3);

//Get the world position
Vector3 p0 = getPositionOfMeshVertex(vert0, mesh);
Vector3 p1 = getPositionOfMeshVertex(vert1, mesh);
Vector3 p2 = getPositionOfMeshVertex(vert2, mesh);
Vector3 position = DecalLayer.barycentricToPoint3D(baryCenter, p0, p1, p2);

//Get the bumpmap addition to the normal
if (uvBump == null || uvBump.Length == 0){
    uvBump = DecalLayer.getUV(DecalLayer.TextureCoordinateSet.TextureCoord1, mesh);
    bumpMap = (Texture2D)GetComponent<Renderer>().material.GetTexture("Bumpmap");
    if (bumpMap == null) Debug.Log("Didn't find bumpmap");
}
Vector2 v0Bump = uvBump[vert0] * (float)((int)bumpMap.width);
Vector2 v1Bump = uvBump[vert1] * (float)((int)bumpMap.width);
Vector2 v2Bump = uvBump[vert2] * (float)((int)bumpMap.width);
Vector2 bumpPosition = DecalLayer.barycentricToPoint2D(baryCenter, v0Bump, v1Bump, v2Bump);

//Apply rgb of pixel to face normal
Color bumpNormal = bumpMap.GetPixel((int)bumpPosition.x, (int)bumpPosition.y);
//Uhhh... Couldn't find this algorithm on Google.  How do I apply this to faceNormal?

//Draw line for debugging
Debug.DrawLine(position, position + faceNormal * 3, Color.green);
2

The exact mapping is explained very well on WP
There could be differences in how exactly it works in Unity, namely some value could be flipped (I think Unreal and Unity differ in this way).