RigidBody2D keeps getting stuck on Tilemap Collider

I have an enemy which has a RigidBody2D and a PolygonCollider2D, and he walks over a Tilemap with a TilemapCollider. When it walks over the map it keeps getting stuck on something. When I then push it with my player it works again, but then it might get stuck elsewhere. I did try it with a BoxCollider2D (as my player has no issue and easily walks over the path) but that gives the same issue.

The RigidBody2D has Freeze rotation X checked but is otherwise default settings. The Collider is set to match the sprite better. The TilemapCollider is fully default settings.

Below you will find my code:

    private void Update()
    {
        if (triggers.Length == 0)
            return;

        Vector2 currentTrigger = triggers[triggerIndex];
        Vector2 direction = (currentTrigger - (Vector2)transform.position).normalized;

        rb.velocity = new Vector2(direction.x * WALK_SPEED, rb.velocity.y);

        if (Vector2.Distance(transform.position, currentTrigger) < 0.1f)
        {
            triggerIndex = (triggerIndex + 1) % triggers.Length;
        }
    }

Edit:
I should add, I looked into the solution given here, but it requires the Tilemap to have a RigidBody which breaks a lot of my other code

Edit 2:
I have the same problem with the player jumping. Sometimes when the player jumps it bumps it’s head on a collider that doesn’t appear to be there.